Sep 10 20:55:56.366016 ip-10-0-139-165 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 10 20:55:56.366028 ip-10-0-139-165 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 10 20:55:56.366037 ip-10-0-139-165 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 10 20:55:56.366243 ip-10-0-139-165 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 10 20:56:06.432309 ip-10-0-139-165 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 10 20:56:06.432324 ip-10-0-139-165 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 6dd3e53ef64a48c68ee8da651cd25bd1 --
Sep 10 20:58:02.222579 ip-10-0-139-165 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 20:58:02.639442 ip-10-0-139-165 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 20:58:02.639442 ip-10-0-139-165 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 20:58:02.639442 ip-10-0-139-165 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 20:58:02.639442 ip-10-0-139-165 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 20:58:02.639442 ip-10-0-139-165 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 20:58:02.642012 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.641923    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 20:58:02.649016 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649000    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:58:02.649016 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649015    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:58:02.649085 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649019    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:58:02.649085 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649022    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:58:02.649085 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649025    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:58:02.649085 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649030    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:58:02.649085 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649033    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:58:02.649085 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649036    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:58:02.649085 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649039    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:58:02.649085 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649043    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:58:02.649085 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649047    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:58:02.649085 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649050    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:58:02.649085 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649060    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:58:02.649085 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649064    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:58:02.649085 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649067    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:58:02.649085 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649070    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:58:02.649085 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649073    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:58:02.649085 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649076    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:58:02.649085 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649078    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:58:02.649085 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649081    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:58:02.649085 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649084    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:58:02.649557 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649086    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:58:02.649557 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649090    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:58:02.649557 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649093    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:58:02.649557 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649095    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:58:02.649557 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649098    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:58:02.649557 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649101    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:58:02.649557 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649103    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:58:02.649557 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649106    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:58:02.649557 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649108    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:58:02.649557 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649110    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:58:02.649557 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649113    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:58:02.649557 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649116    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:58:02.649557 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649118    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:58:02.649557 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649121    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:58:02.649557 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649124    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:58:02.649557 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649126    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:58:02.649557 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649129    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:58:02.649557 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649131    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:58:02.649557 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649134    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:58:02.649557 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649136    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:58:02.650125 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649139    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:58:02.650125 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649141    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:58:02.650125 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649143    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:58:02.650125 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649146    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:58:02.650125 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649149    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:58:02.650125 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649151    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:58:02.650125 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649153    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:58:02.650125 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649156    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:58:02.650125 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649158    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:58:02.650125 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649161    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:58:02.650125 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649163    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:58:02.650125 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649168    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:58:02.650125 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649171    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:58:02.650125 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649175    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:58:02.650125 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649177    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:58:02.650125 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649180    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:58:02.650125 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649183    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:58:02.650125 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649186    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:58:02.650125 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649189    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:58:02.650629 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649191    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:58:02.650629 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649194    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:58:02.650629 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649196    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:58:02.650629 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649198    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:58:02.650629 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649201    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:58:02.650629 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649204    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:58:02.650629 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649207    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:58:02.650629 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649210    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:58:02.650629 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649212    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:58:02.650629 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649215    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:58:02.650629 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649218    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:58:02.650629 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649221    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:58:02.650629 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649223    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:58:02.650629 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649226    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:58:02.650629 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649228    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:58:02.650629 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649231    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:58:02.650629 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649234    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:58:02.650629 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649236    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:58:02.650629 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649238    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:58:02.650629 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649241    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:58:02.651392 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649244    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:58:02.651392 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649246    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:58:02.651392 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649249    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:58:02.651392 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649251    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:58:02.651392 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649254    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:58:02.651392 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649256    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:58:02.651392 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649263    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:58:02.651392 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649266    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:58:02.651392 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649269    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:58:02.651392 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649271    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:58:02.651392 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649273    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:58:02.651392 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649276    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:58:02.651392 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649279    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:58:02.651392 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649281    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:58:02.651392 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649285    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:58:02.651392 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649287    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:58:02.651392 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649290    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:58:02.651392 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649304    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:58:02.651392 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.649307    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:58:02.651392 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650011    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:58:02.651897 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650019    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:58:02.651897 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650022    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:58:02.651897 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650024    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:58:02.651897 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650027    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:58:02.651897 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650030    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:58:02.651897 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650033    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:58:02.651897 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650035    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:58:02.651897 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650038    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:58:02.651897 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650041    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:58:02.651897 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650043    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:58:02.651897 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650046    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:58:02.651897 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650048    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:58:02.651897 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650051    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:58:02.651897 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650053    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:58:02.651897 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650056    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:58:02.651897 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650059    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:58:02.651897 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650061    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:58:02.651897 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650064    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:58:02.651897 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650067    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:58:02.651897 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650070    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:58:02.652412 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650073    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:58:02.652412 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650076    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:58:02.652412 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650079    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:58:02.652412 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650081    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:58:02.652412 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650084    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:58:02.652412 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650086    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:58:02.652412 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650089    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:58:02.652412 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650091    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:58:02.652412 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650094    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:58:02.652412 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650096    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:58:02.652412 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650099    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:58:02.652412 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650101    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:58:02.652412 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650104    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:58:02.652412 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650107    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:58:02.652412 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650109    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:58:02.652412 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650112    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:58:02.652412 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650114    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:58:02.652412 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650117    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:58:02.652412 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650119    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:58:02.652412 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650122    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:58:02.652910 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650124    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:58:02.652910 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650127    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:58:02.652910 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650129    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:58:02.652910 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650133    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:58:02.652910 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650137    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:58:02.652910 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650140    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:58:02.652910 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650143    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:58:02.652910 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650145    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:58:02.652910 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650148    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:58:02.652910 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650150    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:58:02.652910 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650153    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:58:02.652910 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650155    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:58:02.652910 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650158    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:58:02.652910 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650160    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:58:02.652910 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650163    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:58:02.652910 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650166    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:58:02.652910 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650169    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:58:02.652910 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650171    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:58:02.652910 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650174    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:58:02.653396 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650176    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:58:02.653396 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650179    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:58:02.653396 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650181    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:58:02.653396 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650184    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:58:02.653396 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650187    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:58:02.653396 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650189    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:58:02.653396 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650192    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:58:02.653396 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650195    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:58:02.653396 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650197    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:58:02.653396 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650200    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:58:02.653396 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650202    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:58:02.653396 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650206    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:58:02.653396 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650210    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:58:02.653396 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650213    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:58:02.653396 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650216    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:58:02.653396 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650218    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:58:02.653396 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650221    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:58:02.653396 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650223    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:58:02.653396 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650226    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:58:02.653396 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650230    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:58:02.653913 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650232    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:58:02.653913 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650235    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:58:02.653913 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650237    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:58:02.653913 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650240    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:58:02.653913 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650243    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:58:02.653913 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650246    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:58:02.653913 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650249    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:58:02.653913 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650251    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:58:02.653913 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650254    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:58:02.653913 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650257    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:58:02.653913 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650261    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:58:02.653913 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650264    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:58:02.653913 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650267    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:58:02.653913 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650270    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:58:02.653913 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650273    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:58:02.653913 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650275    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:58:02.653913 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650278    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:58:02.653913 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650280    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:58:02.653913 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.650283    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:58:02.653913 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651751    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:58:02.654425 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651763    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:58:02.654425 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651766    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:58:02.654425 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651769    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:58:02.654425 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651773    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:58:02.654425 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651776    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:58:02.654425 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651782    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:58:02.654425 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651785    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:58:02.654425 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651788    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:58:02.654425 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651792    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:58:02.654425 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651794    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:58:02.654425 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651797    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:58:02.654425 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651800    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:58:02.654425 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651803    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:58:02.654425 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651805    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:58:02.654425 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651808    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:58:02.654425 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651812    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:58:02.654425 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651815    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:58:02.654425 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651817    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:58:02.654425 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651822    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:58:02.654425 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651826    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:58:02.654933 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651829    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:58:02.654933 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651831    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:58:02.654933 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651834    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:58:02.654933 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651837    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:58:02.654933 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651845    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:58:02.654933 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651848    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:58:02.654933 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651851    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:58:02.654933 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651853    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:58:02.654933 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651856    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:58:02.654933 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651859    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:58:02.654933 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651864    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:58:02.654933 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651866    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:58:02.654933 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651875    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:58:02.654933 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651878    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:58:02.654933 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651880    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:58:02.654933 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651883    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:58:02.654933 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651887    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:58:02.654933 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651891    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:58:02.654933 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651895    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:58:02.654933 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651897    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:58:02.655426 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651900    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:58:02.655426 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651903    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:58:02.655426 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651911    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:58:02.655426 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651914    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:58:02.655426 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651917    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:58:02.655426 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651920    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:58:02.655426 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651923    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:58:02.655426 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651928    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:58:02.655426 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651932    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:58:02.655426 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651934    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:58:02.655426 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651937    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:58:02.655426 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651940    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:58:02.655426 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651944    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:58:02.655426 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651946    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:58:02.655426 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651952    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:58:02.655426 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651955    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:58:02.655426 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651958    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:58:02.655426 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651961    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:58:02.655426 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651963    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:58:02.655426 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651966    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:58:02.655906 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651969    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:58:02.655906 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651972    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:58:02.655906 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651987    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:58:02.655906 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651991    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:58:02.655906 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.651995    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:58:02.655906 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.652006    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:58:02.655906 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.652009    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:58:02.655906 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.652012    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:58:02.655906 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.652015    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:58:02.655906 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.652017    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:58:02.655906 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.652020    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:58:02.655906 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.652022    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:58:02.655906 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.652025    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:58:02.655906 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.652028    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:58:02.655906 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.652032    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:58:02.655906 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.652036    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:58:02.655906 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.652039    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:58:02.655906 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.652042    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:58:02.655906 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.652047    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:58:02.656399 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.652051    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:58:02.656399 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.652054    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:58:02.656399 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.652057    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:58:02.656399 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.652059    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:58:02.656399 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.652062    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:58:02.656399 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.652064    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:58:02.656399 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.652067    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:58:02.656399 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.652070    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:58:02.656399 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.652073    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:58:02.656399 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.652075    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:58:02.656399 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.652078    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:58:02.656399 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.652081    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:58:02.656399 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.652086    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:58:02.656399 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.652088    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:58:02.656399 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.652091    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:58:02.656399 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.652094    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:58:02.656399 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.652096    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:58:02.656399 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.652099    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:58:02.656399 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.652101    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:58:02.656399 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652354    2581 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 20:58:02.656871 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652396    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 20:58:02.656871 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652403    2581 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 20:58:02.656871 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652408    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 20:58:02.656871 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652412    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 20:58:02.656871 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652415    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 20:58:02.656871 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652420    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 20:58:02.656871 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652424    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 20:58:02.656871 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652427    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 20:58:02.656871 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652430    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 20:58:02.656871 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652434    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 20:58:02.656871 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652437    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 20:58:02.656871 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652441    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 20:58:02.656871 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652446    2581 flags.go:64] FLAG: --cgroup-root=""
Sep 10 20:58:02.656871 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652449    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 20:58:02.656871 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652452    2581 flags.go:64] FLAG: --client-ca-file=""
Sep 10 20:58:02.656871 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652455    2581 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 20:58:02.656871 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652458    2581 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 20:58:02.656871 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652464    2581 flags.go:64] FLAG: --cluster-domain=""
Sep 10 20:58:02.656871 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652467    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 20:58:02.656871 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652470    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 20:58:02.656871 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652473    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 20:58:02.656871 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652476    2581 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 20:58:02.656871 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652480    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 20:58:02.656871 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652483    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 20:58:02.657471 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652486    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 20:58:02.657471 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652489    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 20:58:02.657471 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652492    2581 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 20:58:02.657471 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652495    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 20:58:02.657471 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652499    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 20:58:02.657471 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652503    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 20:58:02.657471 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652506    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 20:58:02.657471 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652510    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 20:58:02.657471 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652513    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 20:58:02.657471 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652516    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 20:58:02.657471 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652519    2581 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 20:58:02.657471 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652522    2581 flags.go:64] FLAG: --enable-server="true"
Sep 10 20:58:02.657471 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652525    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 20:58:02.657471 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652529    2581 flags.go:64] FLAG: --event-burst="100"
Sep 10 20:58:02.657471 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652532    2581 flags.go:64] FLAG: --event-qps="50"
Sep 10 20:58:02.657471 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652535    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 20:58:02.657471 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652538    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 20:58:02.657471 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652541    2581 flags.go:64] FLAG: --eviction-hard=""
Sep 10 20:58:02.657471 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652545    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 20:58:02.657471 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652548    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 20:58:02.657471 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652554    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 20:58:02.657471 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652557    2581 flags.go:64] FLAG: --eviction-soft=""
Sep 10 20:58:02.657471 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652560    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 20:58:02.657471 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652563    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 20:58:02.657471 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652566    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 20:58:02.658099 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652569    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 20:58:02.658099 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652572    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 20:58:02.658099 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652575    2581 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 20:58:02.658099 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652577    2581 flags.go:64] FLAG: --feature-gates=""
Sep 10 20:58:02.658099 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652581    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 20:58:02.658099 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652584    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 20:58:02.658099 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652587    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 20:58:02.658099 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652590    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 20:58:02.658099 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652593    2581 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 20:58:02.658099 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652596    2581 flags.go:64] FLAG: --help="false"
Sep 10 20:58:02.658099 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652599    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-139-165.ec2.internal"
Sep 10 20:58:02.658099 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652603    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 20:58:02.658099 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652606    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 20:58:02.658099 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652609    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 20:58:02.658099 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652612    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 20:58:02.658099 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652615    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 20:58:02.658099 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652618    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 20:58:02.658099 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652621    2581 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 20:58:02.658099 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652624    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 20:58:02.658099 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652627    2581 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 20:58:02.658099 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652629    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 20:58:02.658099 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652633    2581 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 20:58:02.658099 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652635    2581 flags.go:64] FLAG: --kube-reserved=""
Sep 10 20:58:02.658099 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652638    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 20:58:02.658715 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652641    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 20:58:02.658715 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652644    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 20:58:02.658715 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652647    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 20:58:02.658715 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652652    2581 flags.go:64] FLAG: --lock-file=""
Sep 10 20:58:02.658715 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652655    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 20:58:02.658715 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652658    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 20:58:02.658715 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652661    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 20:58:02.658715 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652666    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 20:58:02.658715 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652669    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 20:58:02.658715 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652681    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 20:58:02.658715 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652684    2581 flags.go:64] FLAG: --logging-format="text"
Sep 10 20:58:02.658715 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652687    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 20:58:02.658715 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652690    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 20:58:02.658715 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652693    2581 flags.go:64] FLAG: --manifest-url=""
Sep 10 20:58:02.658715 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652696    2581 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 20:58:02.658715 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652701    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 20:58:02.658715 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652704    2581 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 20:58:02.658715 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652708    2581 flags.go:64] FLAG: --max-pods="110"
Sep 10 20:58:02.658715 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652711    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 20:58:02.658715 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652713    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 20:58:02.658715 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652716    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 20:58:02.658715 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652719    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 20:58:02.658715 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652722    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 20:58:02.658715 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652725    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 20:58:02.658715 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652728    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 20:58:02.659334 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652737    2581 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 20:58:02.659334 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652740    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 20:58:02.659334 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652743    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 20:58:02.659334 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652746    2581 flags.go:64] FLAG: --pod-cidr=""
Sep 10 20:58:02.659334 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652748    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 10 20:58:02.659334 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652754    2581 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 20:58:02.659334 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652757    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 20:58:02.659334 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652760    2581 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 20:58:02.659334 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652763    2581 flags.go:64] FLAG: --port="10250"
Sep 10 20:58:02.659334 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652766    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 20:58:02.659334 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652770    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-049dc777ebe8b4d8f"
Sep 10 20:58:02.659334 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652775    2581 flags.go:64] FLAG: --qos-reserved=""
Sep 10 20:58:02.659334 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652778    2581 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 20:58:02.659334 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652781    2581 flags.go:64] FLAG: --register-node="true"
Sep 10 20:58:02.659334 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652784    2581 flags.go:64] FLAG: --register-with-taints=""
Sep 10 20:58:02.659334 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652787    2581 flags.go:64] FLAG: --registry-burst="10"
Sep 10 20:58:02.659334 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652790    2581 flags.go:64] FLAG: --registry-qps="5"
Sep 10 20:58:02.659334 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652799    2581 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 20:58:02.659334 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652802    2581 flags.go:64] FLAG: --reserved-memory=""
Sep 10 20:58:02.659334 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652806    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 20:58:02.659334 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652809    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 20:58:02.659334 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652812    2581 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 20:58:02.659334 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652814    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 20:58:02.659334 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652817    2581 flags.go:64] FLAG: --runonce="false"
Sep 10 20:58:02.659334 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652820    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 20:58:02.659969 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652823    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 20:58:02.659969 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652826    2581 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 20:58:02.659969 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652829    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 20:58:02.659969 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652832    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 20:58:02.659969 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652835    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 20:58:02.659969 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652838    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 20:58:02.659969 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652841    2581 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 20:58:02.659969 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652844    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 20:58:02.659969 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652847    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 20:58:02.659969 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652849    2581 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 20:58:02.659969 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652852    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 20:58:02.659969 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652855    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 20:58:02.659969 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652858    2581 flags.go:64] FLAG: --system-cgroups=""
Sep 10 20:58:02.659969 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652861    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 20:58:02.659969 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652867    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 20:58:02.659969 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652870    2581 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 20:58:02.659969 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652872    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 20:58:02.659969 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652880    2581 flags.go:64] FLAG: --tls-min-version=""
Sep 10 20:58:02.659969 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652884    2581 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 20:58:02.659969 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652887    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 20:58:02.659969 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652890    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 20:58:02.659969 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652893    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 20:58:02.659969 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652896    2581 flags.go:64] FLAG: --v="2"
Sep 10 20:58:02.659969 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652900    2581 flags.go:64] FLAG: --version="false"
Sep 10 20:58:02.659969 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652904    2581 flags.go:64] FLAG: --vmodule=""
Sep 10 20:58:02.660587 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652914    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 20:58:02.660587 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.652918    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 20:58:02.660587 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653063    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:58:02.660587 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653066    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:58:02.660587 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653069    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:58:02.660587 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653072    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:58:02.660587 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653075    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:58:02.660587 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653078    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:58:02.660587 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653080    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:58:02.660587 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653083    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:58:02.660587 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653085    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:58:02.660587 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653088    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:58:02.660587 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653090    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:58:02.660587 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653093    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:58:02.660587 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653095    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:58:02.660587 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653098    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:58:02.660587 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653100    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:58:02.660587 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653103    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:58:02.660587 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653105    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:58:02.660587 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653108    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:58:02.661072 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653111    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:58:02.661072 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653114    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:58:02.661072 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653116    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:58:02.661072 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653119    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:58:02.661072 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653123    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:58:02.661072 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653127    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:58:02.661072 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653129    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:58:02.661072 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653132    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:58:02.661072 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653135    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:58:02.661072 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653137    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:58:02.661072 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653140    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:58:02.661072 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653143    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:58:02.661072 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653145    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:58:02.661072 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653153    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:58:02.661072 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653156    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:58:02.661072 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653158    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:58:02.661072 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653161    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:58:02.661072 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653164    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:58:02.661072 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653166    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:58:02.661072 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653169    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:58:02.661578 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653171    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:58:02.661578 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653174    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:58:02.661578 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653176    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:58:02.661578 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653179    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:58:02.661578 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653181    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:58:02.661578 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653184    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:58:02.661578 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653186    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:58:02.661578 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653189    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:58:02.661578 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653192    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:58:02.661578 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653194    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:58:02.661578 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653197    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:58:02.661578 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653199    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:58:02.661578 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653202    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:58:02.661578 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653204    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:58:02.661578 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653207    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:58:02.661578 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653210    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:58:02.661578 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653212    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:58:02.661578 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653216    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:58:02.661578 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653219    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:58:02.662064 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653222    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:58:02.662064 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653224    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:58:02.662064 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653227    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:58:02.662064 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653229    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:58:02.662064 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653232    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:58:02.662064 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653234    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:58:02.662064 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653237    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:58:02.662064 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653245    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:58:02.662064 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653248    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:58:02.662064 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653250    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:58:02.662064 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653254    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:58:02.662064 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653258    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:58:02.662064 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653260    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:58:02.662064 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653263    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:58:02.662064 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653266    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:58:02.662064 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653269    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:58:02.662064 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653273    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:58:02.662064 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653276    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:58:02.662064 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653279    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:58:02.662536 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653282    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:58:02.662536 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653285    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:58:02.662536 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653287    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:58:02.662536 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653290    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:58:02.662536 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653306    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:58:02.662536 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653310    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:58:02.662536 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653313    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:58:02.662536 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653316    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:58:02.662536 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653319    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:58:02.662536 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653322    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:58:02.662536 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653326    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:58:02.662536 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653330    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:58:02.662536 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653333    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:58:02.662536 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653336    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:58:02.662536 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653338    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:58:02.662536 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653341    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:58:02.662536 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653344    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:58:02.662536 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653346    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:58:02.662536 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653349    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:58:02.662536 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653351    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:58:02.663023 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653355    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:58:02.663023 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653364    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:58:02.663023 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.653366    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:58:02.663023 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.653907    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 20:58:02.663023 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.660767    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 20:58:02.663023 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.660782    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 20:58:02.663023 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.661494    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 20:58:02.663023 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.661502    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 20:58:02.663023 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661562    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:58:02.663023 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661567    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:58:02.663023 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661570    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:58:02.663023 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661574    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:58:02.663023 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661577    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:58:02.663023 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661580    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:58:02.663023 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661583    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:58:02.663450 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661588    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:58:02.663450 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661591    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:58:02.663450 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661593    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:58:02.663450 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661597    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:58:02.663450 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661599    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:58:02.663450 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661602    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:58:02.663450 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661605    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:58:02.663450 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661608    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:58:02.663450 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661611    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:58:02.663450 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661613    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:58:02.663450 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661616    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:58:02.663450 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661619    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:58:02.663450 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661621    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:58:02.663450 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661624    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:58:02.663450 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661628    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:58:02.663450 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661630    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:58:02.663450 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661633    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:58:02.663450 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661635    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:58:02.663450 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661638    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:58:02.663908 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661641    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:58:02.663908 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661645    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:58:02.663908 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661649    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:58:02.663908 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661653    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:58:02.663908 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661656    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:58:02.663908 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661659    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:58:02.663908 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661662    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:58:02.663908 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661665    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:58:02.663908 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661668    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:58:02.663908 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661671    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:58:02.663908 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661674    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:58:02.663908 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661677    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:58:02.663908 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661680    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:58:02.663908 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661682    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:58:02.663908 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661685    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:58:02.663908 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661687    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:58:02.663908 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661691    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:58:02.663908 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661695    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:58:02.663908 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661697    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:58:02.663908 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661700    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:58:02.664406 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661702    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:58:02.664406 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661705    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:58:02.664406 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661708    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:58:02.664406 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661710    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:58:02.664406 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661713    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:58:02.664406 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661716    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:58:02.664406 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661718    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:58:02.664406 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661722    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:58:02.664406 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661725    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:58:02.664406 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661727    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:58:02.664406 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661730    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:58:02.664406 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661733    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:58:02.664406 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661735    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:58:02.664406 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661738    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:58:02.664406 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661741    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:58:02.664406 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661743    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:58:02.664406 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661747    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:58:02.664406 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661750    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:58:02.664406 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661752    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:58:02.664867 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661755    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:58:02.664867 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661758    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:58:02.664867 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661761    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:58:02.664867 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661763    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:58:02.664867 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661766    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:58:02.664867 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661769    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:58:02.664867 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661771    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:58:02.664867 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661774    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:58:02.664867 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661776    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:58:02.664867 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661779    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:58:02.664867 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661781    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:58:02.664867 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661784    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:58:02.664867 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661787    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:58:02.664867 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661790    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:58:02.664867 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661792    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:58:02.664867 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661794    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:58:02.664867 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661797    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:58:02.664867 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661799    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:58:02.664867 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661802    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:58:02.664867 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661805    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:58:02.665380 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661808    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:58:02.665380 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661810    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:58:02.665380 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661813    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:58:02.665380 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661815    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:58:02.665380 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661818    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:58:02.665380 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661821    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:58:02.665380 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661825    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:58:02.665380 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661828    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:58:02.665380 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661831    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:58:02.665380 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661833    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:58:02.665380 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661844    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:58:02.665380 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661847    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:58:02.665380 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661850    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:58:02.665380 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661852    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:58:02.665380 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.661857    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 20:58:02.665754 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661972    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:58:02.665754 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661978    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:58:02.665754 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661981    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:58:02.665754 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661984    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:58:02.665754 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661987    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:58:02.665754 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661990    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:58:02.665754 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661993    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:58:02.665754 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661995    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:58:02.665754 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.661998    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:58:02.665754 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662002    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:58:02.665754 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662005    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:58:02.665754 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662007    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:58:02.665754 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662010    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:58:02.665754 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662012    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:58:02.665754 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662015    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:58:02.665754 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662018    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:58:02.665754 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662020    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:58:02.665754 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662023    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:58:02.665754 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662026    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:58:02.666219 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662028    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:58:02.666219 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662031    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:58:02.666219 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662034    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:58:02.666219 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662036    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:58:02.666219 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662040    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:58:02.666219 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662042    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:58:02.666219 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662045    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:58:02.666219 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662047    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:58:02.666219 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662050    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:58:02.666219 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662053    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:58:02.666219 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662056    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:58:02.666219 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662058    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:58:02.666219 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662061    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:58:02.666219 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662063    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:58:02.666219 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662066    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:58:02.666219 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662069    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:58:02.666219 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662071    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:58:02.666219 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662074    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:58:02.666219 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662077    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:58:02.666219 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662079    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:58:02.666737 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662082    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:58:02.666737 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662085    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:58:02.666737 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662087    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:58:02.666737 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662090    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:58:02.666737 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662092    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:58:02.666737 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662095    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:58:02.666737 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662097    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:58:02.666737 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662100    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:58:02.666737 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662103    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:58:02.666737 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662107    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:58:02.666737 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662110    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:58:02.666737 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662113    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:58:02.666737 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662116    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:58:02.666737 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662118    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:58:02.666737 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662121    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:58:02.666737 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662123    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:58:02.666737 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662126    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:58:02.666737 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662130    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:58:02.666737 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662133    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:58:02.667196 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662136    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:58:02.667196 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662140    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:58:02.667196 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662142    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:58:02.667196 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662145    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:58:02.667196 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662148    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:58:02.667196 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662151    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:58:02.667196 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662154    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:58:02.667196 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662157    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:58:02.667196 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662159    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:58:02.667196 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662162    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:58:02.667196 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662164    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:58:02.667196 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662167    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:58:02.667196 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662170    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:58:02.667196 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662172    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:58:02.667196 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662175    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:58:02.667196 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662177    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:58:02.667196 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662180    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:58:02.667196 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662182    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:58:02.667196 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662184    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:58:02.667686 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662187    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:58:02.667686 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662189    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:58:02.667686 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662192    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:58:02.667686 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662194    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:58:02.667686 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662197    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:58:02.667686 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662200    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:58:02.667686 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662202    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:58:02.667686 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662205    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:58:02.667686 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662207    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:58:02.667686 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662210    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:58:02.667686 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662212    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:58:02.667686 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662215    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:58:02.667686 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662218    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:58:02.667686 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662220    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:58:02.667686 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662223    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:58:02.667686 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662225    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:58:02.667686 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662228    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:58:02.667686 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662233    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:58:02.667686 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662236    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:58:02.667686 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662238    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:58:02.668161 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662241    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:58:02.668161 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:02.662243    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:58:02.668161 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.662248    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 20:58:02.668161 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.662367    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 20:58:02.668161 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.664696    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 20:58:02.668161 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.665685    2581 server.go:1019] "Starting client certificate rotation"
Sep 10 20:58:02.668161 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.665779    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 20:58:02.668161 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.665812    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 20:58:02.668706 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.668693    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 20:58:02.671728 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.671703    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 20:58:02.682427 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.682410    2581 log.go:25] "Validated CRI v1 runtime API"
Sep 10 20:58:02.687138 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.687116    2581 log.go:25] "Validated CRI v1 image API"
Sep 10 20:58:02.687984 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.687969    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 20:58:02.694434 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.694406    2581 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7de9c2ea-211b-43fd-a17e-99bd7865dc85:/dev/nvme0n1p4 ccd009ab-3380-4363-be0b-8452b732edc3:/dev/nvme0n1p3]
Sep 10 20:58:02.694512 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.694428    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 20:58:02.698746 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.698626    2581 manager.go:216] Machine: {Timestamp:2026-09-10 20:58:02.698109972 +0000 UTC m=+0.355018734 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3078041 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ecaef08702fd95a109878985b7d42 SystemUUID:ec2ecaef-0870-2fd9-5a10-9878985b7d42 BootID:6dd3e53e-f64a-48c6-8ee8-da651cd25bd1 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:0e:28:f0:45:5f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:0e:28:f0:45:5f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:42:5b:4a:74:5e:51 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 20:58:02.698746 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.698731    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 20:58:02.698940 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.698867    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 20:58:02.701247 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.701224    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 20:58:02.701444 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.701251    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-165.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 20:58:02.702039 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.702027    2581 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 20:58:02.702039 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.702039    2581 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 20:58:02.702125 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.702047    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 20:58:02.703717 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.703692    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 20:58:02.704267 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.704253    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 20:58:02.705046 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.705036    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 20:58:02.708983 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.708971    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 20:58:02.709099 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.709089    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 20:58:02.709138 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.709116    2581 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 20:58:02.709138 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.709125    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 20:58:02.709138 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.709134    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 20:58:02.709216 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.709142    2581 kubelet.go:398] "Adding apiserver pod source"
Sep 10 20:58:02.709216 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.709152    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 20:58:02.709527 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.709502    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 20:58:02.709571 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.709536    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 20:58:02.712180 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.712163    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 20:58:02.714805 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.714792    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 20:58:02.714862 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.714822    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 20:58:02.716352 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.716340    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 20:58:02.716403 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.716358    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 20:58:02.716403 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.716364    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 20:58:02.716403 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.716369    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 20:58:02.716403 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.716377    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 20:58:02.716403 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.716382    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 20:58:02.716403 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.716395    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 20:58:02.716403 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.716400    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 20:58:02.716577 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.716413    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 20:58:02.716577 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.716418    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 20:58:02.716577 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.716426    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 20:58:02.716577 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.716431    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 20:58:02.717137 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.717128    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 20:58:02.717166 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.717137    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 20:58:02.721217 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:02.721200    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 20:58:02.721574 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.721563    2581 server.go:1268] "Started kubelet"
Sep 10 20:58:02.721615 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:02.721573    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-165.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 20:58:02.721660 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.721619    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 20:58:02.722673 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.722652    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-165.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 20:58:02.722984 ip-10-0-139-165 systemd[1]: Started Kubernetes Kubelet.
Sep 10 20:58:02.723198 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.722993    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 20:58:02.723198 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.723070    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 20:58:02.726045 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.725720    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 20:58:02.726045 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.725941    2581 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 20:58:02.727213 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:02.727190    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 20:58:02.730706 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.730684    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 20:58:02.731052 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.730998    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 20:58:02.731371 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:02.731350    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-165.ec2.internal\" not found"
Sep 10 20:58:02.731569 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.731556    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 20:58:02.731659 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.731557    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 20:58:02.731659 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.731661    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 20:58:02.731807 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.731770    2581 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 20:58:02.731807 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.731778    2581 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 20:58:02.731956 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.731930    2581 factory.go:55] Registering systemd factory
Sep 10 20:58:02.732016 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.731974    2581 factory.go:223] Registration of the systemd container factory successfully
Sep 10 20:58:02.732016 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:02.731023    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-165.ec2.internal.18d41137a88bb2f8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-165.ec2.internal,UID:ip-10-0-139-165.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-165.ec2.internal,},FirstTimestamp:2026-09-10 20:58:02.721530616 +0000 UTC m=+0.378439381,LastTimestamp:2026-09-10 20:58:02.721530616 +0000 UTC m=+0.378439381,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-165.ec2.internal,}"
Sep 10 20:58:02.732237 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.732216    2581 factory.go:153] Registering CRI-O factory
Sep 10 20:58:02.732237 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.732237    2581 factory.go:223] Registration of the crio container factory successfully
Sep 10 20:58:02.732399 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.732289    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 20:58:02.732399 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.732327    2581 factory.go:103] Registering Raw factory
Sep 10 20:58:02.732399 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.732342    2581 manager.go:1196] Started watching for new ooms in manager
Sep 10 20:58:02.732771 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.732753    2581 manager.go:349] Starting recovery of all containers
Sep 10 20:58:02.742386 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.742367    2581 manager.go:354] Recovery completed
Sep 10 20:58:02.746476 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.746462    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:58:02.748447 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:02.748416    2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-165.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 10 20:58:02.748547 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.748458    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-165.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:58:02.748547 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.748487    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-165.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:58:02.748547 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.748503    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-165.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:58:02.748665 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:02.748580    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 20:58:02.749185 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.749159    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 20:58:02.749185 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.749174    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 20:58:02.749336 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.749192    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 20:58:02.749336 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.749192    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:58:02.750764 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.750748    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-165.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:58:02.750839 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.750773    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-165.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:58:02.750839 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.750788    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-165.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:58:02.751804 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.751786    2581 policy_none.go:49] "None policy: Start"
Sep 10 20:58:02.751879 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.751809    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 20:58:02.751879 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.751822    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 20:58:02.753785 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.753770    2581 policy_none.go:47] "Start"
Sep 10 20:58:02.761701 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:02.761629    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-165.ec2.internal.18d41137aa26d443  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-165.ec2.internal,UID:ip-10-0-139-165.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-165.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-165.ec2.internal,},FirstTimestamp:2026-09-10 20:58:02.748474435 +0000 UTC m=+0.405383207,LastTimestamp:2026-09-10 20:58:02.748474435 +0000 UTC m=+0.405383207,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-165.ec2.internal,}"
Sep 10 20:58:02.765698 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.765677    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zlmxp"
Sep 10 20:58:02.767085 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:02.767007    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-165.ec2.internal.18d41137aa271f0a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-165.ec2.internal,UID:ip-10-0-139-165.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-165.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-165.ec2.internal,},FirstTimestamp:2026-09-10 20:58:02.748493578 +0000 UTC m=+0.405402341,LastTimestamp:2026-09-10 20:58:02.748493578 +0000 UTC m=+0.405402341,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-165.ec2.internal,}"
Sep 10 20:58:02.771344 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:02.771265    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-165.ec2.internal.18d41137aa276b08  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-165.ec2.internal,UID:ip-10-0-139-165.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-139-165.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-139-165.ec2.internal,},FirstTimestamp:2026-09-10 20:58:02.748513032 +0000 UTC m=+0.405421794,LastTimestamp:2026-09-10 20:58:02.748513032 +0000 UTC m=+0.405421794,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-165.ec2.internal,}"
Sep 10 20:58:02.776580 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.776563    2581 manager.go:337] "Starting Device Plugin manager"
Sep 10 20:58:02.776669 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:02.776606    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 20:58:02.776669 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.776619    2581 server.go:87] "Starting device plugin registration server"
Sep 10 20:58:02.776847 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.776832    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 20:58:02.776900 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.776848    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 20:58:02.777012 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.776998    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 20:58:02.777321 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.777078    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 20:58:02.777321 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.777089    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 20:58:02.777662 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:02.777606    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 20:58:02.777662 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:02.777659    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-165.ec2.internal\" not found"
Sep 10 20:58:02.779307 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.779189    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zlmxp"
Sep 10 20:58:02.780832 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:02.780761    2581 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-139-165.ec2.internal.18d41137aa26d443\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-165.ec2.internal.18d41137aa26d443  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-165.ec2.internal,UID:ip-10-0-139-165.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-165.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-165.ec2.internal,},FirstTimestamp:2026-09-10 20:58:02.748474435 +0000 UTC m=+0.405383207,LastTimestamp:2026-09-10 20:58:02.750762789 +0000 UTC m=+0.407671553,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-165.ec2.internal,}"
Sep 10 20:58:02.860553 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.860522    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 20:58:02.861676 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.861657    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 20:58:02.861748 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.861715    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 20:58:02.861748 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.861741    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 20:58:02.861865 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:02.861833    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 20:58:02.864664 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.864648    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 20:58:02.877120 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.877107    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:58:02.878011 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.877992    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-165.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:58:02.878087 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.878026    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-165.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:58:02.878087 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.878040    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-165.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:58:02.878087 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.878069    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-165.ec2.internal"
Sep 10 20:58:02.889711 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.889668    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-165.ec2.internal"
Sep 10 20:58:02.889711 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:02.889688    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-165.ec2.internal\": node \"ip-10-0-139-165.ec2.internal\" not found"
Sep 10 20:58:02.910830 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:02.910811    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-165.ec2.internal\" not found"
Sep 10 20:58:02.962051 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.962024    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-165.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-165.ec2.internal"]
Sep 10 20:58:02.962151 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.962094    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:58:02.963070 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.963055    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-165.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:58:02.963146 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.963084    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-165.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:58:02.963146 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.963098    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-165.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:58:02.965472 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.965459    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:58:02.965600 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.965588    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-165.ec2.internal"
Sep 10 20:58:02.965636 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.965615    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:58:02.966247 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.966231    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-165.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:58:02.966333 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.966252    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-165.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:58:02.966333 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.966262    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-165.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:58:02.966333 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.966281    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-165.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:58:02.966333 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.966318    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-165.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:58:02.966333 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.966329    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-165.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:58:02.968522 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.968509    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-165.ec2.internal"
Sep 10 20:58:02.968582 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.968531    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:58:02.969281 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.969260    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-165.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:58:02.969383 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.969292    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-165.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:58:02.969383 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:02.969321    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-165.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:58:02.993903 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:02.993886    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-165.ec2.internal\" not found" node="ip-10-0-139-165.ec2.internal"
Sep 10 20:58:02.998239 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:02.998218    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-165.ec2.internal\" not found" node="ip-10-0-139-165.ec2.internal"
Sep 10 20:58:03.011906 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:03.011890    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-165.ec2.internal\" not found"
Sep 10 20:58:03.033000 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:03.032982    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65f1e16d97785601ead433dc27092e06-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-165.ec2.internal\" (UID: \"65f1e16d97785601ead433dc27092e06\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-165.ec2.internal"
Sep 10 20:58:03.033051 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:03.033008    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d43f2c9998219eadf34d68ce7c7d50dc-config\") pod \"kube-apiserver-proxy-ip-10-0-139-165.ec2.internal\" (UID: \"d43f2c9998219eadf34d68ce7c7d50dc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-165.ec2.internal"
Sep 10 20:58:03.033051 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:03.033033    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/65f1e16d97785601ead433dc27092e06-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-165.ec2.internal\" (UID: \"65f1e16d97785601ead433dc27092e06\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-165.ec2.internal"
Sep 10 20:58:03.112149 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:03.112123    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-165.ec2.internal\" not found"
Sep 10 20:58:03.133174 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:03.133153    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/65f1e16d97785601ead433dc27092e06-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-165.ec2.internal\" (UID: \"65f1e16d97785601ead433dc27092e06\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-165.ec2.internal"
Sep 10 20:58:03.133236 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:03.133179    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65f1e16d97785601ead433dc27092e06-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-165.ec2.internal\" (UID: \"65f1e16d97785601ead433dc27092e06\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-165.ec2.internal"
Sep 10 20:58:03.133236 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:03.133205    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d43f2c9998219eadf34d68ce7c7d50dc-config\") pod \"kube-apiserver-proxy-ip-10-0-139-165.ec2.internal\" (UID: \"d43f2c9998219eadf34d68ce7c7d50dc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-165.ec2.internal"
Sep 10 20:58:03.133322 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:03.133263    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65f1e16d97785601ead433dc27092e06-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-165.ec2.internal\" (UID: \"65f1e16d97785601ead433dc27092e06\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-165.ec2.internal"
Sep 10 20:58:03.133366 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:03.133311    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d43f2c9998219eadf34d68ce7c7d50dc-config\") pod \"kube-apiserver-proxy-ip-10-0-139-165.ec2.internal\" (UID: \"d43f2c9998219eadf34d68ce7c7d50dc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-165.ec2.internal"
Sep 10 20:58:03.133366 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:03.133263    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/65f1e16d97785601ead433dc27092e06-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-165.ec2.internal\" (UID: \"65f1e16d97785601ead433dc27092e06\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-165.ec2.internal"
Sep 10 20:58:03.212923 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:03.212834    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-165.ec2.internal\" not found"
Sep 10 20:58:03.301682 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:03.301647    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-165.ec2.internal"
Sep 10 20:58:03.301682 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:03.301672    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-165.ec2.internal"
Sep 10 20:58:03.313160 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:03.313142    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-165.ec2.internal\" not found"
Sep 10 20:58:03.414182 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:03.414141    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-165.ec2.internal\" not found"
Sep 10 20:58:03.516661 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:03.516602    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-165.ec2.internal\" not found"
Sep 10 20:58:03.600805 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:03.600777    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 20:58:03.617138 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:03.617103    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-165.ec2.internal\" not found"
Sep 10 20:58:03.665548 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:03.665522    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 20:58:03.666166 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:03.665642    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 20:58:03.666166 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:03.665674    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 20:58:03.717073 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:03.717054    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 20:58:03.717171 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:03.717154    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-165.ec2.internal\" not found"
Sep 10 20:58:03.784638 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:03.784554    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 20:53:03 +0000 UTC" deadline="2028-05-20 11:31:57.67025625 +0000 UTC"
Sep 10 20:58:03.784638 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:03.784597    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14822h33m53.885663027s"
Sep 10 20:58:03.818003 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:03.817976    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-165.ec2.internal\" not found"
Sep 10 20:58:03.898919 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:03.898898    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:58:03.899843 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:03.899826    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-165.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:58:03.899913 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:03.899858    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-165.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:58:03.899913 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:03.899869    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-165.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:58:03.918522 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:03.918499    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-165.ec2.internal\" not found"
Sep 10 20:58:04.019728 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:04.019688    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-165.ec2.internal\" not found"
Sep 10 20:58:04.120502 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:04.120433    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-165.ec2.internal\" not found"
Sep 10 20:58:04.155668 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:04.155618    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65f1e16d97785601ead433dc27092e06.slice/crio-8c9e3d8510cb75573d34cb85eeca5e26dba194ec47fa2346b696bb1c795ef9da WatchSource:0}: Error finding container 8c9e3d8510cb75573d34cb85eeca5e26dba194ec47fa2346b696bb1c795ef9da: Status 404 returned error can't find the container with id 8c9e3d8510cb75573d34cb85eeca5e26dba194ec47fa2346b696bb1c795ef9da
Sep 10 20:58:04.156001 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:04.155973    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd43f2c9998219eadf34d68ce7c7d50dc.slice/crio-6a0e52fd6bf5a6cec24509ead90980c0ad3457fb2f11b433af76db7f161ed311 WatchSource:0}: Error finding container 6a0e52fd6bf5a6cec24509ead90980c0ad3457fb2f11b433af76db7f161ed311: Status 404 returned error can't find the container with id 6a0e52fd6bf5a6cec24509ead90980c0ad3457fb2f11b433af76db7f161ed311
Sep 10 20:58:04.159883 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.159862    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 20:58:04.221149 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:04.221116    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-165.ec2.internal\" not found"
Sep 10 20:58:04.246149 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.246124    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 20:58:04.332008 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.331985    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-165.ec2.internal"
Sep 10 20:58:04.347425 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.347406    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 20:58:04.348466 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.348454    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-165.ec2.internal"
Sep 10 20:58:04.363883 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.363861    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 20:58:04.710420 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.710389    2581 apiserver.go:52] "Watching apiserver"
Sep 10 20:58:04.719507 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.719474    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 20:58:04.720635 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.720606    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-lmwz4","openshift-network-operator/iptables-alerter-v9h74","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2l7n","openshift-dns/node-resolver-h4md4","openshift-image-registry/node-ca-mthkf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-165.ec2.internal","openshift-network-diagnostics/network-check-target-gvw9h","openshift-ovn-kubernetes/ovnkube-node-brrtd","kube-system/global-pull-secret-syncer-mgx5w","kube-system/konnectivity-agent-l2gkq","kube-system/kube-apiserver-proxy-ip-10-0-139-165.ec2.internal","openshift-cluster-node-tuning-operator/tuned-z4glj","openshift-multus/multus-additional-cni-plugins-5rbwt","openshift-multus/multus-qrbt5"]
Sep 10 20:58:04.723458 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.723434    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l2gkq"
Sep 10 20:58:04.726390 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.726370    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5rbwt"
Sep 10 20:58:04.727313 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.727084    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4k722\""
Sep 10 20:58:04.732124 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.731216    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 20:58:04.732124 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.731781    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.734092 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.732733    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 20:58:04.734092 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.732744    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 20:58:04.734092 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.732986    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7674z\""
Sep 10 20:58:04.734092 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.732993    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 20:58:04.734092 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.733226    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 20:58:04.734092 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.733434    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 20:58:04.734092 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.733661    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 20:58:04.734092 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.733886    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 20:58:04.734773 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.734715    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-z4glj"
Sep 10 20:58:04.737258 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.736798    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mthkf"
Sep 10 20:58:04.739584 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.739563    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lmwz4"
Sep 10 20:58:04.739681 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:04.739639    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lmwz4" podUID="781245a4-8475-4031-9137-dd43ecd1c260"
Sep 10 20:58:04.741606 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.741580    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nhl9\" (UniqueName: \"kubernetes.io/projected/432f3e3d-207e-4732-97f7-43182d9582b9-kube-api-access-6nhl9\") pod \"multus-additional-cni-plugins-5rbwt\" (UID: \"432f3e3d-207e-4732-97f7-43182d9582b9\") " pod="openshift-multus/multus-additional-cni-plugins-5rbwt"
Sep 10 20:58:04.741712 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.741623    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/705a88df-c5e0-4082-b22e-e1aad389cd03-etc-modprobe-d\") pod \"tuned-z4glj\" (UID: \"705a88df-c5e0-4082-b22e-e1aad389cd03\") " pod="openshift-cluster-node-tuning-operator/tuned-z4glj"
Sep 10 20:58:04.741712 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.741650    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/705a88df-c5e0-4082-b22e-e1aad389cd03-sys\") pod \"tuned-z4glj\" (UID: \"705a88df-c5e0-4082-b22e-e1aad389cd03\") " pod="openshift-cluster-node-tuning-operator/tuned-z4glj"
Sep 10 20:58:04.741712 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.741692    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/05ef07fc-6c16-4be8-8d44-4dfa99384a03-multus-socket-dir-parent\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.741890 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.741720    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/05ef07fc-6c16-4be8-8d44-4dfa99384a03-multus-conf-dir\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.741890 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.741752    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/05ef07fc-6c16-4be8-8d44-4dfa99384a03-multus-daemon-config\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.741890 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.741794    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/05ef07fc-6c16-4be8-8d44-4dfa99384a03-host-run-multus-certs\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.741890 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.741832    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/705a88df-c5e0-4082-b22e-e1aad389cd03-etc-sysctl-d\") pod \"tuned-z4glj\" (UID: \"705a88df-c5e0-4082-b22e-e1aad389cd03\") " pod="openshift-cluster-node-tuning-operator/tuned-z4glj"
Sep 10 20:58:04.741890 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.741867    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/705a88df-c5e0-4082-b22e-e1aad389cd03-etc-tuned\") pod \"tuned-z4glj\" (UID: \"705a88df-c5e0-4082-b22e-e1aad389cd03\") " pod="openshift-cluster-node-tuning-operator/tuned-z4glj"
Sep 10 20:58:04.742143 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.741893    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05ef07fc-6c16-4be8-8d44-4dfa99384a03-host-var-lib-kubelet\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.742143 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.741918    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwfpp\" (UniqueName: \"kubernetes.io/projected/05ef07fc-6c16-4be8-8d44-4dfa99384a03-kube-api-access-xwfpp\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.742143 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.741941    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/705a88df-c5e0-4082-b22e-e1aad389cd03-host\") pod \"tuned-z4glj\" (UID: \"705a88df-c5e0-4082-b22e-e1aad389cd03\") " pod="openshift-cluster-node-tuning-operator/tuned-z4glj"
Sep 10 20:58:04.742143 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.742011    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/432f3e3d-207e-4732-97f7-43182d9582b9-system-cni-dir\") pod \"multus-additional-cni-plugins-5rbwt\" (UID: \"432f3e3d-207e-4732-97f7-43182d9582b9\") " pod="openshift-multus/multus-additional-cni-plugins-5rbwt"
Sep 10 20:58:04.742143 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.742044    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/432f3e3d-207e-4732-97f7-43182d9582b9-cni-binary-copy\") pod \"multus-additional-cni-plugins-5rbwt\" (UID: \"432f3e3d-207e-4732-97f7-43182d9582b9\") " pod="openshift-multus/multus-additional-cni-plugins-5rbwt"
Sep 10 20:58:04.742143 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.742069    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/05ef07fc-6c16-4be8-8d44-4dfa99384a03-cnibin\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.742143 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.742094    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/05ef07fc-6c16-4be8-8d44-4dfa99384a03-hostroot\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.742143 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.742118    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/432f3e3d-207e-4732-97f7-43182d9582b9-cnibin\") pod \"multus-additional-cni-plugins-5rbwt\" (UID: \"432f3e3d-207e-4732-97f7-43182d9582b9\") " pod="openshift-multus/multus-additional-cni-plugins-5rbwt"
Sep 10 20:58:04.742540 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.742160    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/432f3e3d-207e-4732-97f7-43182d9582b9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5rbwt\" (UID: \"432f3e3d-207e-4732-97f7-43182d9582b9\") " pod="openshift-multus/multus-additional-cni-plugins-5rbwt"
Sep 10 20:58:04.742540 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.742189    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/705a88df-c5e0-4082-b22e-e1aad389cd03-lib-modules\") pod \"tuned-z4glj\" (UID: \"705a88df-c5e0-4082-b22e-e1aad389cd03\") " pod="openshift-cluster-node-tuning-operator/tuned-z4glj"
Sep 10 20:58:04.742627 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.742594    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/705a88df-c5e0-4082-b22e-e1aad389cd03-etc-sysconfig\") pod \"tuned-z4glj\" (UID: \"705a88df-c5e0-4082-b22e-e1aad389cd03\") " pod="openshift-cluster-node-tuning-operator/tuned-z4glj"
Sep 10 20:58:04.742684 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.742664    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/705a88df-c5e0-4082-b22e-e1aad389cd03-etc-sysctl-conf\") pod \"tuned-z4glj\" (UID: \"705a88df-c5e0-4082-b22e-e1aad389cd03\") " pod="openshift-cluster-node-tuning-operator/tuned-z4glj"
Sep 10 20:58:04.742757 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.742710    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/705a88df-c5e0-4082-b22e-e1aad389cd03-tmp\") pod \"tuned-z4glj\" (UID: \"705a88df-c5e0-4082-b22e-e1aad389cd03\") " pod="openshift-cluster-node-tuning-operator/tuned-z4glj"
Sep 10 20:58:04.742812 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.742774    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/407d670d-5138-4d8b-aba7-96bdd91d9b19-agent-certs\") pod \"konnectivity-agent-l2gkq\" (UID: \"407d670d-5138-4d8b-aba7-96bdd91d9b19\") " pod="kube-system/konnectivity-agent-l2gkq"
Sep 10 20:58:04.742867 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.742809    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/05ef07fc-6c16-4be8-8d44-4dfa99384a03-cni-binary-copy\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.742867 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.742842    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/705a88df-c5e0-4082-b22e-e1aad389cd03-etc-kubernetes\") pod \"tuned-z4glj\" (UID: \"705a88df-c5e0-4082-b22e-e1aad389cd03\") " pod="openshift-cluster-node-tuning-operator/tuned-z4glj"
Sep 10 20:58:04.742971 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.742875    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g45kh\" (UniqueName: \"kubernetes.io/projected/705a88df-c5e0-4082-b22e-e1aad389cd03-kube-api-access-g45kh\") pod \"tuned-z4glj\" (UID: \"705a88df-c5e0-4082-b22e-e1aad389cd03\") " pod="openshift-cluster-node-tuning-operator/tuned-z4glj"
Sep 10 20:58:04.742971 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.742914    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/432f3e3d-207e-4732-97f7-43182d9582b9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5rbwt\" (UID: \"432f3e3d-207e-4732-97f7-43182d9582b9\") " pod="openshift-multus/multus-additional-cni-plugins-5rbwt"
Sep 10 20:58:04.742971 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.742952    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/432f3e3d-207e-4732-97f7-43182d9582b9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5rbwt\" (UID: \"432f3e3d-207e-4732-97f7-43182d9582b9\") " pod="openshift-multus/multus-additional-cni-plugins-5rbwt"
Sep 10 20:58:04.743121 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.742978    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05ef07fc-6c16-4be8-8d44-4dfa99384a03-system-cni-dir\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.743121 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.742996    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05ef07fc-6c16-4be8-8d44-4dfa99384a03-multus-cni-dir\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.743121 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.743012    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/05ef07fc-6c16-4be8-8d44-4dfa99384a03-host-var-lib-cni-multus\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.743121 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.743045    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/705a88df-c5e0-4082-b22e-e1aad389cd03-var-lib-kubelet\") pod \"tuned-z4glj\" (UID: \"705a88df-c5e0-4082-b22e-e1aad389cd03\") " pod="openshift-cluster-node-tuning-operator/tuned-z4glj"
Sep 10 20:58:04.743121 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.743063    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/705a88df-c5e0-4082-b22e-e1aad389cd03-run\") pod \"tuned-z4glj\" (UID: \"705a88df-c5e0-4082-b22e-e1aad389cd03\") " pod="openshift-cluster-node-tuning-operator/tuned-z4glj"
Sep 10 20:58:04.743121 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.743113    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/407d670d-5138-4d8b-aba7-96bdd91d9b19-konnectivity-ca\") pod \"konnectivity-agent-l2gkq\" (UID: \"407d670d-5138-4d8b-aba7-96bdd91d9b19\") " pod="kube-system/konnectivity-agent-l2gkq"
Sep 10 20:58:04.743455 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.743134    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/05ef07fc-6c16-4be8-8d44-4dfa99384a03-host-run-k8s-cni-cncf-io\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.743455 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.743150    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05ef07fc-6c16-4be8-8d44-4dfa99384a03-etc-kubernetes\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.743455 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.743170    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/705a88df-c5e0-4082-b22e-e1aad389cd03-etc-systemd\") pod \"tuned-z4glj\" (UID: \"705a88df-c5e0-4082-b22e-e1aad389cd03\") " pod="openshift-cluster-node-tuning-operator/tuned-z4glj"
Sep 10 20:58:04.743455 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.743194    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05ef07fc-6c16-4be8-8d44-4dfa99384a03-host-run-netns\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.743455 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.743214    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05ef07fc-6c16-4be8-8d44-4dfa99384a03-host-var-lib-cni-bin\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.743774 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.743751    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 20:58:04.747273 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.746897    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vv2nt\""
Sep 10 20:58:04.748819 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.747943    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gvw9h"
Sep 10 20:58:04.748819 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.748043    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6pfzv\""
Sep 10 20:58:04.748819 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:04.748100    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gvw9h" podUID="3bfd0b36-183c-43c3-9fcf-2d6f7839223e"
Sep 10 20:58:04.748819 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.748295    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 20:58:04.748819 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.748306    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 20:58:04.748819 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.748505    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 20:58:04.748819 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.748566    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 20:58:04.749988 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.749970    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mgx5w"
Sep 10 20:58:04.750094 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:04.750041    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mgx5w" podUID="6e1f126a-f90d-4cf6-b863-d6130c01a622"
Sep 10 20:58:04.752627 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.752596    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v9h74"
Sep 10 20:58:04.755663 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.755643    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7xkcp\""
Sep 10 20:58:04.755936 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.755901    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 20:58:04.756155 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.756135    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-v6bl9\""
Sep 10 20:58:04.756242 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.756176    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 20:58:04.756403 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.756362    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 20:58:04.756653 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.756635    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 20:58:04.756968 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.756950    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h4md4"
Sep 10 20:58:04.759646 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.759624    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.762997 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.762976    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2l7n"
Sep 10 20:58:04.765180 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.765158    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 20:58:04.765286 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.765197    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 20:58:04.765678 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.765659    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 20:58:04.765830 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.765816    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5b689\""
Sep 10 20:58:04.765994 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.765980    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 20:58:04.766115 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.766102    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 20:58:04.766713 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.766692    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 20:58:04.766857 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.766841    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vqdb7\""
Sep 10 20:58:04.766975 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.766962    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 20:58:04.767269 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.767249    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 20:58:04.770955 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.770936    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 20:58:04.771160 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.771140    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jjq2l\""
Sep 10 20:58:04.771226 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.771193    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 20:58:04.771428 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.771395    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 20:58:04.773670 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.773629    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 20:58:04.811040 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.811006    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-nzcgz"
Sep 10 20:58:04.832843 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.832820    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 20:58:04.837278 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.837212    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-nzcgz"
Sep 10 20:58:04.844435 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.844410    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-var-lib-openvswitch\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.844542 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.844455    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05ef07fc-6c16-4be8-8d44-4dfa99384a03-etc-kubernetes\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.844542 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.844482    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-node-log\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.844542 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.844521    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/19d8d7ba-45a1-4013-a935-5670abe12b8a-ovnkube-script-lib\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.844706 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.844549    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05ef07fc-6c16-4be8-8d44-4dfa99384a03-host-var-lib-cni-bin\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.844706 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.844584    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-run-ovn\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.844706 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.844612    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/19d8d7ba-45a1-4013-a935-5670abe12b8a-ovnkube-config\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.844706 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.844648    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nhl9\" (UniqueName: \"kubernetes.io/projected/432f3e3d-207e-4732-97f7-43182d9582b9-kube-api-access-6nhl9\") pod \"multus-additional-cni-plugins-5rbwt\" (UID: \"432f3e3d-207e-4732-97f7-43182d9582b9\") " pod="openshift-multus/multus-additional-cni-plugins-5rbwt"
Sep 10 20:58:04.844706 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.844681    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/705a88df-c5e0-4082-b22e-e1aad389cd03-sys\") pod \"tuned-z4glj\" (UID: \"705a88df-c5e0-4082-b22e-e1aad389cd03\") " pod="openshift-cluster-node-tuning-operator/tuned-z4glj"
Sep 10 20:58:04.844971 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.844715    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/05ef07fc-6c16-4be8-8d44-4dfa99384a03-multus-socket-dir-parent\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.844971 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.844738    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/05ef07fc-6c16-4be8-8d44-4dfa99384a03-multus-conf-dir\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.844971 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.844772    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-etc-openvswitch\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.844971 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.844796    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-host-cni-netd\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.844971 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.844821    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce-socket-dir\") pod \"aws-ebs-csi-driver-node-k2l7n\" (UID: \"8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2l7n"
Sep 10 20:58:04.844971 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.844854    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/705a88df-c5e0-4082-b22e-e1aad389cd03-etc-tuned\") pod \"tuned-z4glj\" (UID: \"705a88df-c5e0-4082-b22e-e1aad389cd03\") " pod="openshift-cluster-node-tuning-operator/tuned-z4glj"
Sep 10 20:58:04.844971 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.844885    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwfpp\" (UniqueName: \"kubernetes.io/projected/05ef07fc-6c16-4be8-8d44-4dfa99384a03-kube-api-access-xwfpp\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.844971 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.844904    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7fc96d36-95bc-4159-b504-7fad608e5819-host\") pod \"node-ca-mthkf\" (UID: \"7fc96d36-95bc-4159-b504-7fad608e5819\") " pod="openshift-image-registry/node-ca-mthkf"
Sep 10 20:58:04.844971 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.844920    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce-device-dir\") pod \"aws-ebs-csi-driver-node-k2l7n\" (UID: \"8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2l7n"
Sep 10 20:58:04.844971 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.844933    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-systemd-units\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.845473 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.844963    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/432f3e3d-207e-4732-97f7-43182d9582b9-cni-binary-copy\") pod \"multus-additional-cni-plugins-5rbwt\" (UID: \"432f3e3d-207e-4732-97f7-43182d9582b9\") " pod="openshift-multus/multus-additional-cni-plugins-5rbwt"
Sep 10 20:58:04.845473 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.844994    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/05ef07fc-6c16-4be8-8d44-4dfa99384a03-hostroot\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.845473 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.845009    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dmm9\" (UniqueName: \"kubernetes.io/projected/781245a4-8475-4031-9137-dd43ecd1c260-kube-api-access-2dmm9\") pod \"network-metrics-daemon-lmwz4\" (UID: \"781245a4-8475-4031-9137-dd43ecd1c260\") " pod="openshift-multus/network-metrics-daemon-lmwz4"
Sep 10 20:58:04.845473 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.845026    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-run-openvswitch\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.845473 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.845052    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-log-socket\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.845473 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.845066    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltdbj\" (UniqueName: \"kubernetes.io/projected/7fc96d36-95bc-4159-b504-7fad608e5819-kube-api-access-ltdbj\") pod \"node-ca-mthkf\" (UID: \"7fc96d36-95bc-4159-b504-7fad608e5819\") " pod="openshift-image-registry/node-ca-mthkf"
Sep 10 20:58:04.845473 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.845080    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f68613e-8086-4f5f-8a47-af8dcd8e3a76-tmp-dir\") pod \"node-resolver-h4md4\" (UID: \"0f68613e-8086-4f5f-8a47-af8dcd8e3a76\") " pod="openshift-dns/node-resolver-h4md4"
Sep 10 20:58:04.845473 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.845095    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/432f3e3d-207e-4732-97f7-43182d9582b9-cnibin\") pod \"multus-additional-cni-plugins-5rbwt\" (UID: \"432f3e3d-207e-4732-97f7-43182d9582b9\") " pod="openshift-multus/multus-additional-cni-plugins-5rbwt"
Sep 10 20:58:04.845473 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.845110    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/705a88df-c5e0-4082-b22e-e1aad389cd03-etc-sysconfig\") pod \"tuned-z4glj\" (UID: \"705a88df-c5e0-4082-b22e-e1aad389cd03\") " pod="openshift-cluster-node-tuning-operator/tuned-z4glj"
Sep 10 20:58:04.845473 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.845128    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/407d670d-5138-4d8b-aba7-96bdd91d9b19-agent-certs\") pod \"konnectivity-agent-l2gkq\" (UID: \"407d670d-5138-4d8b-aba7-96bdd91d9b19\") " pod="kube-system/konnectivity-agent-l2gkq"
Sep 10 20:58:04.845473 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.845156    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/19d8d7ba-45a1-4013-a935-5670abe12b8a-env-overrides\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.845473 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.845182    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/705a88df-c5e0-4082-b22e-e1aad389cd03-etc-kubernetes\") pod \"tuned-z4glj\" (UID: \"705a88df-c5e0-4082-b22e-e1aad389cd03\") " pod="openshift-cluster-node-tuning-operator/tuned-z4glj"
Sep 10 20:58:04.845473 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.845198    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g45kh\" (UniqueName: \"kubernetes.io/projected/705a88df-c5e0-4082-b22e-e1aad389cd03-kube-api-access-g45kh\") pod \"tuned-z4glj\" (UID: \"705a88df-c5e0-4082-b22e-e1aad389cd03\") " pod="openshift-cluster-node-tuning-operator/tuned-z4glj"
Sep 10 20:58:04.845473 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.845217    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/432f3e3d-207e-4732-97f7-43182d9582b9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5rbwt\" (UID: \"432f3e3d-207e-4732-97f7-43182d9582b9\") " pod="openshift-multus/multus-additional-cni-plugins-5rbwt"
Sep 10 20:58:04.845473 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.845236    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/781245a4-8475-4031-9137-dd43ecd1c260-metrics-certs\") pod \"network-metrics-daemon-lmwz4\" (UID: \"781245a4-8475-4031-9137-dd43ecd1c260\") " pod="openshift-multus/network-metrics-daemon-lmwz4"
Sep 10 20:58:04.845473 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.845275    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/749647ce-cddd-4e0b-ac39-b6682295073c-host-slash\") pod \"iptables-alerter-v9h74\" (UID: \"749647ce-cddd-4e0b-ac39-b6682295073c\") " pod="openshift-network-operator/iptables-alerter-v9h74"
Sep 10 20:58:04.845473 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.845301    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce-registration-dir\") pod \"aws-ebs-csi-driver-node-k2l7n\" (UID: \"8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2l7n"
Sep 10 20:58:04.846217 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.845339    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce-sys-fs\") pod \"aws-ebs-csi-driver-node-k2l7n\" (UID: \"8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2l7n"
Sep 10 20:58:04.846217 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.845373    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr5t2\" (UniqueName: \"kubernetes.io/projected/19d8d7ba-45a1-4013-a935-5670abe12b8a-kube-api-access-sr5t2\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.846217 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.845422    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/705a88df-c5e0-4082-b22e-e1aad389cd03-var-lib-kubelet\") pod \"tuned-z4glj\" (UID: \"705a88df-c5e0-4082-b22e-e1aad389cd03\") " pod="openshift-cluster-node-tuning-operator/tuned-z4glj"
Sep 10 20:58:04.846217 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.845448    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/705a88df-c5e0-4082-b22e-e1aad389cd03-run\") pod \"tuned-z4glj\" (UID: \"705a88df-c5e0-4082-b22e-e1aad389cd03\") " pod="openshift-cluster-node-tuning-operator/tuned-z4glj"
Sep 10 20:58:04.846217 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.845529    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/407d670d-5138-4d8b-aba7-96bdd91d9b19-konnectivity-ca\") pod \"konnectivity-agent-l2gkq\" (UID: \"407d670d-5138-4d8b-aba7-96bdd91d9b19\") " pod="kube-system/konnectivity-agent-l2gkq"
Sep 10 20:58:04.846217 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.845556    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/05ef07fc-6c16-4be8-8d44-4dfa99384a03-host-run-k8s-cni-cncf-io\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.846217 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.845585    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7fc96d36-95bc-4159-b504-7fad608e5819-serviceca\") pod \"node-ca-mthkf\" (UID: \"7fc96d36-95bc-4159-b504-7fad608e5819\") " pod="openshift-image-registry/node-ca-mthkf"
Sep 10 20:58:04.846217 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.845605    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e1f126a-f90d-4cf6-b863-d6130c01a622-original-pull-secret\") pod \"global-pull-secret-syncer-mgx5w\" (UID: \"6e1f126a-f90d-4cf6-b863-d6130c01a622\") " pod="kube-system/global-pull-secret-syncer-mgx5w"
Sep 10 20:58:04.846217 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.845626    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/19d8d7ba-45a1-4013-a935-5670abe12b8a-ovn-node-metrics-cert\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.846217 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.845641    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/705a88df-c5e0-4082-b22e-e1aad389cd03-etc-systemd\") pod \"tuned-z4glj\" (UID: \"705a88df-c5e0-4082-b22e-e1aad389cd03\") " pod="openshift-cluster-node-tuning-operator/tuned-z4glj"
Sep 10 20:58:04.846217 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.845658    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05ef07fc-6c16-4be8-8d44-4dfa99384a03-host-run-netns\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.846217 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.845673    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/749647ce-cddd-4e0b-ac39-b6682295073c-iptables-alerter-script\") pod \"iptables-alerter-v9h74\" (UID: \"749647ce-cddd-4e0b-ac39-b6682295073c\") " pod="openshift-network-operator/iptables-alerter-v9h74"
Sep 10 20:58:04.846217 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.845689    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce-etc-selinux\") pod \"aws-ebs-csi-driver-node-k2l7n\" (UID: \"8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2l7n"
Sep 10 20:58:04.846217 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.845722    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/705a88df-c5e0-4082-b22e-e1aad389cd03-etc-modprobe-d\") pod \"tuned-z4glj\" (UID: \"705a88df-c5e0-4082-b22e-e1aad389cd03\") " pod="openshift-cluster-node-tuning-operator/tuned-z4glj"
Sep 10 20:58:04.846217 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.845741    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/05ef07fc-6c16-4be8-8d44-4dfa99384a03-multus-daemon-config\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.846217 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.845780    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/05ef07fc-6c16-4be8-8d44-4dfa99384a03-host-run-multus-certs\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.846217 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.845796    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-host-slash\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.847094 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.845821    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc2nv\" (UniqueName: \"kubernetes.io/projected/0f68613e-8086-4f5f-8a47-af8dcd8e3a76-kube-api-access-bc2nv\") pod \"node-resolver-h4md4\" (UID: \"0f68613e-8086-4f5f-8a47-af8dcd8e3a76\") " pod="openshift-dns/node-resolver-h4md4"
Sep 10 20:58:04.847094 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.845834    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-host-kubelet\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.847094 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.845860    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-run-systemd\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.847094 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.845878    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-host-cni-bin\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.847094 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.845893    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/705a88df-c5e0-4082-b22e-e1aad389cd03-etc-sysctl-d\") pod \"tuned-z4glj\" (UID: \"705a88df-c5e0-4082-b22e-e1aad389cd03\") " pod="openshift-cluster-node-tuning-operator/tuned-z4glj"
Sep 10 20:58:04.847094 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.845909    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05ef07fc-6c16-4be8-8d44-4dfa99384a03-host-var-lib-kubelet\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.847094 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.845924    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7n7n\" (UniqueName: \"kubernetes.io/projected/749647ce-cddd-4e0b-ac39-b6682295073c-kube-api-access-l7n7n\") pod \"iptables-alerter-v9h74\" (UID: \"749647ce-cddd-4e0b-ac39-b6682295073c\") " pod="openshift-network-operator/iptables-alerter-v9h74"
Sep 10 20:58:04.847094 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.845938    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0f68613e-8086-4f5f-8a47-af8dcd8e3a76-hosts-file\") pod \"node-resolver-h4md4\" (UID: \"0f68613e-8086-4f5f-8a47-af8dcd8e3a76\") " pod="openshift-dns/node-resolver-h4md4"
Sep 10 20:58:04.847094 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.845957    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k2l7n\" (UID: \"8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2l7n"
Sep 10 20:58:04.847094 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.845996    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwrq9\" (UniqueName: \"kubernetes.io/projected/8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce-kube-api-access-jwrq9\") pod \"aws-ebs-csi-driver-node-k2l7n\" (UID: \"8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2l7n"
Sep 10 20:58:04.847094 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.846018    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-host-run-ovn-kubernetes\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.847094 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.846039    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/705a88df-c5e0-4082-b22e-e1aad389cd03-host\") pod \"tuned-z4glj\" (UID: \"705a88df-c5e0-4082-b22e-e1aad389cd03\") " pod="openshift-cluster-node-tuning-operator/tuned-z4glj"
Sep 10 20:58:04.847094 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.846060    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/432f3e3d-207e-4732-97f7-43182d9582b9-system-cni-dir\") pod \"multus-additional-cni-plugins-5rbwt\" (UID: \"432f3e3d-207e-4732-97f7-43182d9582b9\") " pod="openshift-multus/multus-additional-cni-plugins-5rbwt"
Sep 10 20:58:04.847094 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.846080    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/05ef07fc-6c16-4be8-8d44-4dfa99384a03-cnibin\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.847094 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.846136    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.847094 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.846192    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/432f3e3d-207e-4732-97f7-43182d9582b9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5rbwt\" (UID: \"432f3e3d-207e-4732-97f7-43182d9582b9\") " pod="openshift-multus/multus-additional-cni-plugins-5rbwt"
Sep 10 20:58:04.847936 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.846220    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/705a88df-c5e0-4082-b22e-e1aad389cd03-lib-modules\") pod \"tuned-z4glj\" (UID: \"705a88df-c5e0-4082-b22e-e1aad389cd03\") " pod="openshift-cluster-node-tuning-operator/tuned-z4glj"
Sep 10 20:58:04.847936 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.846246    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/705a88df-c5e0-4082-b22e-e1aad389cd03-etc-sysctl-conf\") pod \"tuned-z4glj\" (UID: \"705a88df-c5e0-4082-b22e-e1aad389cd03\") " pod="openshift-cluster-node-tuning-operator/tuned-z4glj"
Sep 10 20:58:04.847936 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.846277    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/705a88df-c5e0-4082-b22e-e1aad389cd03-tmp\") pod \"tuned-z4glj\" (UID: \"705a88df-c5e0-4082-b22e-e1aad389cd03\") " pod="openshift-cluster-node-tuning-operator/tuned-z4glj"
Sep 10 20:58:04.847936 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.846303    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/05ef07fc-6c16-4be8-8d44-4dfa99384a03-cni-binary-copy\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.847936 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.846336    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6e1f126a-f90d-4cf6-b863-d6130c01a622-dbus\") pod \"global-pull-secret-syncer-mgx5w\" (UID: \"6e1f126a-f90d-4cf6-b863-d6130c01a622\") " pod="kube-system/global-pull-secret-syncer-mgx5w"
Sep 10 20:58:04.847936 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.846384    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/432f3e3d-207e-4732-97f7-43182d9582b9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5rbwt\" (UID: \"432f3e3d-207e-4732-97f7-43182d9582b9\") " pod="openshift-multus/multus-additional-cni-plugins-5rbwt"
Sep 10 20:58:04.847936 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.846421    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05ef07fc-6c16-4be8-8d44-4dfa99384a03-system-cni-dir\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.847936 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.846454    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05ef07fc-6c16-4be8-8d44-4dfa99384a03-multus-cni-dir\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.847936 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.846479    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/05ef07fc-6c16-4be8-8d44-4dfa99384a03-host-var-lib-cni-multus\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.847936 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.846501    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8nv4\" (UniqueName: \"kubernetes.io/projected/3bfd0b36-183c-43c3-9fcf-2d6f7839223e-kube-api-access-g8nv4\") pod \"network-check-target-gvw9h\" (UID: \"3bfd0b36-183c-43c3-9fcf-2d6f7839223e\") " pod="openshift-network-diagnostics/network-check-target-gvw9h"
Sep 10 20:58:04.847936 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.846517    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6e1f126a-f90d-4cf6-b863-d6130c01a622-kubelet-config\") pod \"global-pull-secret-syncer-mgx5w\" (UID: \"6e1f126a-f90d-4cf6-b863-d6130c01a622\") " pod="kube-system/global-pull-secret-syncer-mgx5w"
Sep 10 20:58:04.847936 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.846554    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-host-run-netns\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.847936 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.846644    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05ef07fc-6c16-4be8-8d44-4dfa99384a03-etc-kubernetes\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.847936 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.846685    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05ef07fc-6c16-4be8-8d44-4dfa99384a03-host-var-lib-cni-bin\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.847936 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.847017    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/705a88df-c5e0-4082-b22e-e1aad389cd03-sys\") pod \"tuned-z4glj\" (UID: \"705a88df-c5e0-4082-b22e-e1aad389cd03\") " pod="openshift-cluster-node-tuning-operator/tuned-z4glj"
Sep 10 20:58:04.847936 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.847098    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/05ef07fc-6c16-4be8-8d44-4dfa99384a03-multus-socket-dir-parent\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.847936 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.847134    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/05ef07fc-6c16-4be8-8d44-4dfa99384a03-multus-conf-dir\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.847936 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.847651    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 20:58:04.849740 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.848583    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/705a88df-c5e0-4082-b22e-e1aad389cd03-var-lib-kubelet\") pod \"tuned-z4glj\" (UID: \"705a88df-c5e0-4082-b22e-e1aad389cd03\") " pod="openshift-cluster-node-tuning-operator/tuned-z4glj"
Sep 10 20:58:04.849740 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.848620    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/705a88df-c5e0-4082-b22e-e1aad389cd03-run\") pod \"tuned-z4glj\" (UID: \"705a88df-c5e0-4082-b22e-e1aad389cd03\") " pod="openshift-cluster-node-tuning-operator/tuned-z4glj"
Sep 10 20:58:04.849740 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.848694    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/05ef07fc-6c16-4be8-8d44-4dfa99384a03-host-run-k8s-cni-cncf-io\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.849740 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.848767    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05ef07fc-6c16-4be8-8d44-4dfa99384a03-host-run-netns\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.849740 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.849088    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/432f3e3d-207e-4732-97f7-43182d9582b9-cni-binary-copy\") pod \"multus-additional-cni-plugins-5rbwt\" (UID: \"432f3e3d-207e-4732-97f7-43182d9582b9\") " pod="openshift-multus/multus-additional-cni-plugins-5rbwt"
Sep 10 20:58:04.849740 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.849309    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/407d670d-5138-4d8b-aba7-96bdd91d9b19-konnectivity-ca\") pod \"konnectivity-agent-l2gkq\" (UID: \"407d670d-5138-4d8b-aba7-96bdd91d9b19\") " pod="kube-system/konnectivity-agent-l2gkq"
Sep 10 20:58:04.849740 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.849473    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/05ef07fc-6c16-4be8-8d44-4dfa99384a03-multus-daemon-config\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.849740 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.849487    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/432f3e3d-207e-4732-97f7-43182d9582b9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5rbwt\" (UID: \"432f3e3d-207e-4732-97f7-43182d9582b9\") " pod="openshift-multus/multus-additional-cni-plugins-5rbwt"
Sep 10 20:58:04.849740 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.849542    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/705a88df-c5e0-4082-b22e-e1aad389cd03-etc-systemd\") pod \"tuned-z4glj\" (UID: \"705a88df-c5e0-4082-b22e-e1aad389cd03\") " pod="openshift-cluster-node-tuning-operator/tuned-z4glj"
Sep 10 20:58:04.849740 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.849555    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/05ef07fc-6c16-4be8-8d44-4dfa99384a03-hostroot\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.849740 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.849602    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/05ef07fc-6c16-4be8-8d44-4dfa99384a03-host-run-multus-certs\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.849740 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.849657    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/705a88df-c5e0-4082-b22e-e1aad389cd03-etc-modprobe-d\") pod \"tuned-z4glj\" (UID: \"705a88df-c5e0-4082-b22e-e1aad389cd03\") " pod="openshift-cluster-node-tuning-operator/tuned-z4glj"
Sep 10 20:58:04.849740 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.849665    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/05ef07fc-6c16-4be8-8d44-4dfa99384a03-host-var-lib-cni-multus\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.849740 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.849722    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/432f3e3d-207e-4732-97f7-43182d9582b9-system-cni-dir\") pod \"multus-additional-cni-plugins-5rbwt\" (UID: \"432f3e3d-207e-4732-97f7-43182d9582b9\") " pod="openshift-multus/multus-additional-cni-plugins-5rbwt"
Sep 10 20:58:04.850466 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.849800    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/705a88df-c5e0-4082-b22e-e1aad389cd03-etc-sysctl-d\") pod \"tuned-z4glj\" (UID: \"705a88df-c5e0-4082-b22e-e1aad389cd03\") " pod="openshift-cluster-node-tuning-operator/tuned-z4glj"
Sep 10 20:58:04.850466 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.849848    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05ef07fc-6c16-4be8-8d44-4dfa99384a03-host-var-lib-kubelet\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.850466 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.849859    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/705a88df-c5e0-4082-b22e-e1aad389cd03-host\") pod \"tuned-z4glj\" (UID: \"705a88df-c5e0-4082-b22e-e1aad389cd03\") " pod="openshift-cluster-node-tuning-operator/tuned-z4glj"
Sep 10 20:58:04.850466 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.849957    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/705a88df-c5e0-4082-b22e-e1aad389cd03-etc-sysconfig\") pod \"tuned-z4glj\" (UID: \"705a88df-c5e0-4082-b22e-e1aad389cd03\") " pod="openshift-cluster-node-tuning-operator/tuned-z4glj"
Sep 10 20:58:04.850466 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.849965    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/705a88df-c5e0-4082-b22e-e1aad389cd03-etc-kubernetes\") pod \"tuned-z4glj\" (UID: \"705a88df-c5e0-4082-b22e-e1aad389cd03\") " pod="openshift-cluster-node-tuning-operator/tuned-z4glj"
Sep 10 20:58:04.850466 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.850020    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/432f3e3d-207e-4732-97f7-43182d9582b9-cnibin\") pod \"multus-additional-cni-plugins-5rbwt\" (UID: \"432f3e3d-207e-4732-97f7-43182d9582b9\") " pod="openshift-multus/multus-additional-cni-plugins-5rbwt"
Sep 10 20:58:04.850466 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.850089    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05ef07fc-6c16-4be8-8d44-4dfa99384a03-multus-cni-dir\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.850466 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.850145    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05ef07fc-6c16-4be8-8d44-4dfa99384a03-system-cni-dir\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.850466 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.850366    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/432f3e3d-207e-4732-97f7-43182d9582b9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5rbwt\" (UID: \"432f3e3d-207e-4732-97f7-43182d9582b9\") " pod="openshift-multus/multus-additional-cni-plugins-5rbwt"
Sep 10 20:58:04.850853 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.850650    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/05ef07fc-6c16-4be8-8d44-4dfa99384a03-cnibin\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.850853 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.850655    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/432f3e3d-207e-4732-97f7-43182d9582b9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5rbwt\" (UID: \"432f3e3d-207e-4732-97f7-43182d9582b9\") " pod="openshift-multus/multus-additional-cni-plugins-5rbwt"
Sep 10 20:58:04.850853 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.850748    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/705a88df-c5e0-4082-b22e-e1aad389cd03-lib-modules\") pod \"tuned-z4glj\" (UID: \"705a88df-c5e0-4082-b22e-e1aad389cd03\") " pod="openshift-cluster-node-tuning-operator/tuned-z4glj"
Sep 10 20:58:04.850853 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.850824    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/05ef07fc-6c16-4be8-8d44-4dfa99384a03-cni-binary-copy\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.857547 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.854682    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/705a88df-c5e0-4082-b22e-e1aad389cd03-etc-sysctl-conf\") pod \"tuned-z4glj\" (UID: \"705a88df-c5e0-4082-b22e-e1aad389cd03\") " pod="openshift-cluster-node-tuning-operator/tuned-z4glj"
Sep 10 20:58:04.857547 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.854923    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/705a88df-c5e0-4082-b22e-e1aad389cd03-etc-tuned\") pod \"tuned-z4glj\" (UID: \"705a88df-c5e0-4082-b22e-e1aad389cd03\") " pod="openshift-cluster-node-tuning-operator/tuned-z4glj"
Sep 10 20:58:04.857547 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.855173    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/705a88df-c5e0-4082-b22e-e1aad389cd03-tmp\") pod \"tuned-z4glj\" (UID: \"705a88df-c5e0-4082-b22e-e1aad389cd03\") " pod="openshift-cluster-node-tuning-operator/tuned-z4glj"
Sep 10 20:58:04.857547 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.855345    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/407d670d-5138-4d8b-aba7-96bdd91d9b19-agent-certs\") pod \"konnectivity-agent-l2gkq\" (UID: \"407d670d-5138-4d8b-aba7-96bdd91d9b19\") " pod="kube-system/konnectivity-agent-l2gkq"
Sep 10 20:58:04.869652 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.869599    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-165.ec2.internal" event={"ID":"d43f2c9998219eadf34d68ce7c7d50dc","Type":"ContainerStarted","Data":"6a0e52fd6bf5a6cec24509ead90980c0ad3457fb2f11b433af76db7f161ed311"}
Sep 10 20:58:04.872227 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.872194    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-165.ec2.internal" event={"ID":"65f1e16d97785601ead433dc27092e06","Type":"ContainerStarted","Data":"8c9e3d8510cb75573d34cb85eeca5e26dba194ec47fa2346b696bb1c795ef9da"}
Sep 10 20:58:04.881256 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.881193    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nhl9\" (UniqueName: \"kubernetes.io/projected/432f3e3d-207e-4732-97f7-43182d9582b9-kube-api-access-6nhl9\") pod \"multus-additional-cni-plugins-5rbwt\" (UID: \"432f3e3d-207e-4732-97f7-43182d9582b9\") " pod="openshift-multus/multus-additional-cni-plugins-5rbwt"
Sep 10 20:58:04.881480 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.881460    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwfpp\" (UniqueName: \"kubernetes.io/projected/05ef07fc-6c16-4be8-8d44-4dfa99384a03-kube-api-access-xwfpp\") pod \"multus-qrbt5\" (UID: \"05ef07fc-6c16-4be8-8d44-4dfa99384a03\") " pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:04.883433 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.883397    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g45kh\" (UniqueName: \"kubernetes.io/projected/705a88df-c5e0-4082-b22e-e1aad389cd03-kube-api-access-g45kh\") pod \"tuned-z4glj\" (UID: \"705a88df-c5e0-4082-b22e-e1aad389cd03\") " pod="openshift-cluster-node-tuning-operator/tuned-z4glj"
Sep 10 20:58:04.949400 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.947708    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/19d8d7ba-45a1-4013-a935-5670abe12b8a-env-overrides\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.949400 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.947755    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/781245a4-8475-4031-9137-dd43ecd1c260-metrics-certs\") pod \"network-metrics-daemon-lmwz4\" (UID: \"781245a4-8475-4031-9137-dd43ecd1c260\") " pod="openshift-multus/network-metrics-daemon-lmwz4"
Sep 10 20:58:04.949400 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.947781    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/749647ce-cddd-4e0b-ac39-b6682295073c-host-slash\") pod \"iptables-alerter-v9h74\" (UID: \"749647ce-cddd-4e0b-ac39-b6682295073c\") " pod="openshift-network-operator/iptables-alerter-v9h74"
Sep 10 20:58:04.949400 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.947808    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce-registration-dir\") pod \"aws-ebs-csi-driver-node-k2l7n\" (UID: \"8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2l7n"
Sep 10 20:58:04.949400 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.947834    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce-sys-fs\") pod \"aws-ebs-csi-driver-node-k2l7n\" (UID: \"8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2l7n"
Sep 10 20:58:04.949400 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.947861    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr5t2\" (UniqueName: \"kubernetes.io/projected/19d8d7ba-45a1-4013-a935-5670abe12b8a-kube-api-access-sr5t2\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.949400 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.947879    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7fc96d36-95bc-4159-b504-7fad608e5819-serviceca\") pod \"node-ca-mthkf\" (UID: \"7fc96d36-95bc-4159-b504-7fad608e5819\") " pod="openshift-image-registry/node-ca-mthkf"
Sep 10 20:58:04.949400 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.947905    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e1f126a-f90d-4cf6-b863-d6130c01a622-original-pull-secret\") pod \"global-pull-secret-syncer-mgx5w\" (UID: \"6e1f126a-f90d-4cf6-b863-d6130c01a622\") " pod="kube-system/global-pull-secret-syncer-mgx5w"
Sep 10 20:58:04.949400 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.947926    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/19d8d7ba-45a1-4013-a935-5670abe12b8a-ovn-node-metrics-cert\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.949400 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.947956    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/749647ce-cddd-4e0b-ac39-b6682295073c-iptables-alerter-script\") pod \"iptables-alerter-v9h74\" (UID: \"749647ce-cddd-4e0b-ac39-b6682295073c\") " pod="openshift-network-operator/iptables-alerter-v9h74"
Sep 10 20:58:04.949400 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.947979    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce-etc-selinux\") pod \"aws-ebs-csi-driver-node-k2l7n\" (UID: \"8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2l7n"
Sep 10 20:58:04.949400 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.948016    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-host-slash\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.949400 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.948043    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc2nv\" (UniqueName: \"kubernetes.io/projected/0f68613e-8086-4f5f-8a47-af8dcd8e3a76-kube-api-access-bc2nv\") pod \"node-resolver-h4md4\" (UID: \"0f68613e-8086-4f5f-8a47-af8dcd8e3a76\") " pod="openshift-dns/node-resolver-h4md4"
Sep 10 20:58:04.949400 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.948067    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-host-kubelet\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.949400 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.948092    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-run-systemd\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.949400 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.948118    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-host-cni-bin\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.949400 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.948143    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7n7n\" (UniqueName: \"kubernetes.io/projected/749647ce-cddd-4e0b-ac39-b6682295073c-kube-api-access-l7n7n\") pod \"iptables-alerter-v9h74\" (UID: \"749647ce-cddd-4e0b-ac39-b6682295073c\") " pod="openshift-network-operator/iptables-alerter-v9h74"
Sep 10 20:58:04.950392 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.948168    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0f68613e-8086-4f5f-8a47-af8dcd8e3a76-hosts-file\") pod \"node-resolver-h4md4\" (UID: \"0f68613e-8086-4f5f-8a47-af8dcd8e3a76\") " pod="openshift-dns/node-resolver-h4md4"
Sep 10 20:58:04.950392 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.948192    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k2l7n\" (UID: \"8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2l7n"
Sep 10 20:58:04.950392 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.948219    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwrq9\" (UniqueName: \"kubernetes.io/projected/8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce-kube-api-access-jwrq9\") pod \"aws-ebs-csi-driver-node-k2l7n\" (UID: \"8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2l7n"
Sep 10 20:58:04.950392 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.948240    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-host-run-ovn-kubernetes\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.950392 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.948265    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.950392 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.948296    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6e1f126a-f90d-4cf6-b863-d6130c01a622-dbus\") pod \"global-pull-secret-syncer-mgx5w\" (UID: \"6e1f126a-f90d-4cf6-b863-d6130c01a622\") " pod="kube-system/global-pull-secret-syncer-mgx5w"
Sep 10 20:58:04.950392 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.948328    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8nv4\" (UniqueName: \"kubernetes.io/projected/3bfd0b36-183c-43c3-9fcf-2d6f7839223e-kube-api-access-g8nv4\") pod \"network-check-target-gvw9h\" (UID: \"3bfd0b36-183c-43c3-9fcf-2d6f7839223e\") " pod="openshift-network-diagnostics/network-check-target-gvw9h"
Sep 10 20:58:04.950392 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.948346    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6e1f126a-f90d-4cf6-b863-d6130c01a622-kubelet-config\") pod \"global-pull-secret-syncer-mgx5w\" (UID: \"6e1f126a-f90d-4cf6-b863-d6130c01a622\") " pod="kube-system/global-pull-secret-syncer-mgx5w"
Sep 10 20:58:04.950392 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.948390    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-host-run-netns\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.950392 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.948419    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-var-lib-openvswitch\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.950392 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.948445    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-node-log\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.950392 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.948465    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/19d8d7ba-45a1-4013-a935-5670abe12b8a-ovnkube-script-lib\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.950392 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.948481    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-run-ovn\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.950392 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.948495    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/19d8d7ba-45a1-4013-a935-5670abe12b8a-ovnkube-config\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.950392 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.948518    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-etc-openvswitch\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.950392 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.948544    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-host-cni-netd\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.950392 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.948577    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce-socket-dir\") pod \"aws-ebs-csi-driver-node-k2l7n\" (UID: \"8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2l7n"
Sep 10 20:58:04.951235 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.948607    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7fc96d36-95bc-4159-b504-7fad608e5819-host\") pod \"node-ca-mthkf\" (UID: \"7fc96d36-95bc-4159-b504-7fad608e5819\") " pod="openshift-image-registry/node-ca-mthkf"
Sep 10 20:58:04.951235 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.948632    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce-device-dir\") pod \"aws-ebs-csi-driver-node-k2l7n\" (UID: \"8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2l7n"
Sep 10 20:58:04.951235 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.948656    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-systemd-units\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.951235 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.948679    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dmm9\" (UniqueName: \"kubernetes.io/projected/781245a4-8475-4031-9137-dd43ecd1c260-kube-api-access-2dmm9\") pod \"network-metrics-daemon-lmwz4\" (UID: \"781245a4-8475-4031-9137-dd43ecd1c260\") " pod="openshift-multus/network-metrics-daemon-lmwz4"
Sep 10 20:58:04.951235 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.948694    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-run-openvswitch\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.951235 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.948710    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-log-socket\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.951235 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.948727    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltdbj\" (UniqueName: \"kubernetes.io/projected/7fc96d36-95bc-4159-b504-7fad608e5819-kube-api-access-ltdbj\") pod \"node-ca-mthkf\" (UID: \"7fc96d36-95bc-4159-b504-7fad608e5819\") " pod="openshift-image-registry/node-ca-mthkf"
Sep 10 20:58:04.951235 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.948745    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f68613e-8086-4f5f-8a47-af8dcd8e3a76-tmp-dir\") pod \"node-resolver-h4md4\" (UID: \"0f68613e-8086-4f5f-8a47-af8dcd8e3a76\") " pod="openshift-dns/node-resolver-h4md4"
Sep 10 20:58:04.951235 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.949103    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f68613e-8086-4f5f-8a47-af8dcd8e3a76-tmp-dir\") pod \"node-resolver-h4md4\" (UID: \"0f68613e-8086-4f5f-8a47-af8dcd8e3a76\") " pod="openshift-dns/node-resolver-h4md4"
Sep 10 20:58:04.951235 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.949381    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-host-run-ovn-kubernetes\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.951235 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:04.949487    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:58:04.951235 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:04.949556    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/781245a4-8475-4031-9137-dd43ecd1c260-metrics-certs podName:781245a4-8475-4031-9137-dd43ecd1c260 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:05.449533535 +0000 UTC m=+3.106442302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/781245a4-8475-4031-9137-dd43ecd1c260-metrics-certs") pod "network-metrics-daemon-lmwz4" (UID: "781245a4-8475-4031-9137-dd43ecd1c260") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:58:04.951235 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.949781    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/749647ce-cddd-4e0b-ac39-b6682295073c-host-slash\") pod \"iptables-alerter-v9h74\" (UID: \"749647ce-cddd-4e0b-ac39-b6682295073c\") " pod="openshift-network-operator/iptables-alerter-v9h74"
Sep 10 20:58:04.951235 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.949847    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce-registration-dir\") pod \"aws-ebs-csi-driver-node-k2l7n\" (UID: \"8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2l7n"
Sep 10 20:58:04.951235 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.949901    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce-sys-fs\") pod \"aws-ebs-csi-driver-node-k2l7n\" (UID: \"8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2l7n"
Sep 10 20:58:04.951235 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.950470    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-host-kubelet\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.951235 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.950574    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7fc96d36-95bc-4159-b504-7fad608e5819-serviceca\") pod \"node-ca-mthkf\" (UID: \"7fc96d36-95bc-4159-b504-7fad608e5819\") " pod="openshift-image-registry/node-ca-mthkf"
Sep 10 20:58:04.951235 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:04.950647    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:58:04.952095 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.950664    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce-device-dir\") pod \"aws-ebs-csi-driver-node-k2l7n\" (UID: \"8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2l7n"
Sep 10 20:58:04.952095 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:04.950699    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e1f126a-f90d-4cf6-b863-d6130c01a622-original-pull-secret podName:6e1f126a-f90d-4cf6-b863-d6130c01a622 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:05.45068171 +0000 UTC m=+3.107590478 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e1f126a-f90d-4cf6-b863-d6130c01a622-original-pull-secret") pod "global-pull-secret-syncer-mgx5w" (UID: "6e1f126a-f90d-4cf6-b863-d6130c01a622") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:58:04.952095 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.950718    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-etc-openvswitch\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.952095 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.950749    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce-etc-selinux\") pod \"aws-ebs-csi-driver-node-k2l7n\" (UID: \"8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2l7n"
Sep 10 20:58:04.952095 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.950766    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-host-cni-netd\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.952095 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.950880    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce-socket-dir\") pod \"aws-ebs-csi-driver-node-k2l7n\" (UID: \"8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2l7n"
Sep 10 20:58:04.952095 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.950889    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6e1f126a-f90d-4cf6-b863-d6130c01a622-kubelet-config\") pod \"global-pull-secret-syncer-mgx5w\" (UID: \"6e1f126a-f90d-4cf6-b863-d6130c01a622\") " pod="kube-system/global-pull-secret-syncer-mgx5w"
Sep 10 20:58:04.952095 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.950930    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7fc96d36-95bc-4159-b504-7fad608e5819-host\") pod \"node-ca-mthkf\" (UID: \"7fc96d36-95bc-4159-b504-7fad608e5819\") " pod="openshift-image-registry/node-ca-mthkf"
Sep 10 20:58:04.952095 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.950944    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-host-run-netns\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.952095 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.950644    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6e1f126a-f90d-4cf6-b863-d6130c01a622-dbus\") pod \"global-pull-secret-syncer-mgx5w\" (UID: \"6e1f126a-f90d-4cf6-b863-d6130c01a622\") " pod="kube-system/global-pull-secret-syncer-mgx5w"
Sep 10 20:58:04.952095 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.950996    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-var-lib-openvswitch\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.952095 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.951040    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-node-log\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.952095 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.951116    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-run-ovn\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.952095 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.951167    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-run-openvswitch\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.952095 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.951212    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-log-socket\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.952095 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.951274    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-systemd-units\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.952095 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.951483    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0f68613e-8086-4f5f-8a47-af8dcd8e3a76-hosts-file\") pod \"node-resolver-h4md4\" (UID: \"0f68613e-8086-4f5f-8a47-af8dcd8e3a76\") " pod="openshift-dns/node-resolver-h4md4"
Sep 10 20:58:04.952921 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.951494    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/749647ce-cddd-4e0b-ac39-b6682295073c-iptables-alerter-script\") pod \"iptables-alerter-v9h74\" (UID: \"749647ce-cddd-4e0b-ac39-b6682295073c\") " pod="openshift-network-operator/iptables-alerter-v9h74"
Sep 10 20:58:04.952921 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.951524    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-run-systemd\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.952921 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.951569    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-host-slash\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.952921 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.951577    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-host-cni-bin\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.952921 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.951598    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/19d8d7ba-45a1-4013-a935-5670abe12b8a-ovnkube-script-lib\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.952921 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.951812    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k2l7n\" (UID: \"8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2l7n"
Sep 10 20:58:04.952921 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.951860    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19d8d7ba-45a1-4013-a935-5670abe12b8a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.952921 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.951949    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/19d8d7ba-45a1-4013-a935-5670abe12b8a-ovnkube-config\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.953504 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.953458    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/19d8d7ba-45a1-4013-a935-5670abe12b8a-env-overrides\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.954107 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.954081    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/19d8d7ba-45a1-4013-a935-5670abe12b8a-ovn-node-metrics-cert\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.966685 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.966602    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr5t2\" (UniqueName: \"kubernetes.io/projected/19d8d7ba-45a1-4013-a935-5670abe12b8a-kube-api-access-sr5t2\") pod \"ovnkube-node-brrtd\" (UID: \"19d8d7ba-45a1-4013-a935-5670abe12b8a\") " pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:04.969766 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.969700    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dmm9\" (UniqueName: \"kubernetes.io/projected/781245a4-8475-4031-9137-dd43ecd1c260-kube-api-access-2dmm9\") pod \"network-metrics-daemon-lmwz4\" (UID: \"781245a4-8475-4031-9137-dd43ecd1c260\") " pod="openshift-multus/network-metrics-daemon-lmwz4"
Sep 10 20:58:04.970268 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.970246    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltdbj\" (UniqueName: \"kubernetes.io/projected/7fc96d36-95bc-4159-b504-7fad608e5819-kube-api-access-ltdbj\") pod \"node-ca-mthkf\" (UID: \"7fc96d36-95bc-4159-b504-7fad608e5819\") " pod="openshift-image-registry/node-ca-mthkf"
Sep 10 20:58:04.970600 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:04.970581    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:58:04.970693 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:04.970606    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:58:04.970693 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:04.970620    2581 projected.go:196] Error preparing data for projected volume kube-api-access-g8nv4 for pod openshift-network-diagnostics/network-check-target-gvw9h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:58:04.970809 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:04.970692    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3bfd0b36-183c-43c3-9fcf-2d6f7839223e-kube-api-access-g8nv4 podName:3bfd0b36-183c-43c3-9fcf-2d6f7839223e nodeName:}" failed. No retries permitted until 2026-09-10 20:58:05.470673268 +0000 UTC m=+3.127582028 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-g8nv4" (UniqueName: "kubernetes.io/projected/3bfd0b36-183c-43c3-9fcf-2d6f7839223e-kube-api-access-g8nv4") pod "network-check-target-gvw9h" (UID: "3bfd0b36-183c-43c3-9fcf-2d6f7839223e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:58:04.973415 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.973390    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 20:58:04.973591 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.973567    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7n7n\" (UniqueName: \"kubernetes.io/projected/749647ce-cddd-4e0b-ac39-b6682295073c-kube-api-access-l7n7n\") pod \"iptables-alerter-v9h74\" (UID: \"749647ce-cddd-4e0b-ac39-b6682295073c\") " pod="openshift-network-operator/iptables-alerter-v9h74"
Sep 10 20:58:04.976822 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.976693    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwrq9\" (UniqueName: \"kubernetes.io/projected/8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce-kube-api-access-jwrq9\") pod \"aws-ebs-csi-driver-node-k2l7n\" (UID: \"8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2l7n"
Sep 10 20:58:04.979696 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:04.979650    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc2nv\" (UniqueName: \"kubernetes.io/projected/0f68613e-8086-4f5f-8a47-af8dcd8e3a76-kube-api-access-bc2nv\") pod \"node-resolver-h4md4\" (UID: \"0f68613e-8086-4f5f-8a47-af8dcd8e3a76\") " pod="openshift-dns/node-resolver-h4md4"
Sep 10 20:58:05.038654 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:05.038625    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l2gkq"
Sep 10 20:58:05.051355 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:05.051335    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5rbwt"
Sep 10 20:58:05.051771 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:05.051742    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod407d670d_5138_4d8b_aba7_96bdd91d9b19.slice/crio-d2fbd66e8911fa87fc439ddfcb5aac1abc4595994c63d3dbd9ae6e71ca90e96e WatchSource:0}: Error finding container d2fbd66e8911fa87fc439ddfcb5aac1abc4595994c63d3dbd9ae6e71ca90e96e: Status 404 returned error can't find the container with id d2fbd66e8911fa87fc439ddfcb5aac1abc4595994c63d3dbd9ae6e71ca90e96e
Sep 10 20:58:05.064911 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:05.064877    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod432f3e3d_207e_4732_97f7_43182d9582b9.slice/crio-f7b487a90adeeab3f3242a52712c0b0bfb8b2b3f7a42debb4ea2359419c4097a WatchSource:0}: Error finding container f7b487a90adeeab3f3242a52712c0b0bfb8b2b3f7a42debb4ea2359419c4097a: Status 404 returned error can't find the container with id f7b487a90adeeab3f3242a52712c0b0bfb8b2b3f7a42debb4ea2359419c4097a
Sep 10 20:58:05.065025 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:05.065010    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qrbt5"
Sep 10 20:58:05.073644 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:05.073595    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-z4glj"
Sep 10 20:58:05.075275 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:05.075250    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05ef07fc_6c16_4be8_8d44_4dfa99384a03.slice/crio-269f4a129eb9a46557005f27be6e3a228fca10d6b91bc9f4084f4272886e0af0 WatchSource:0}: Error finding container 269f4a129eb9a46557005f27be6e3a228fca10d6b91bc9f4084f4272886e0af0: Status 404 returned error can't find the container with id 269f4a129eb9a46557005f27be6e3a228fca10d6b91bc9f4084f4272886e0af0
Sep 10 20:58:05.082214 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:05.082193    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mthkf"
Sep 10 20:58:05.083444 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:05.083415    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod705a88df_c5e0_4082_b22e_e1aad389cd03.slice/crio-6c5b116627e10a066d8c785fe50824124fe68c17ad8325c5cee45ddaf8a1d983 WatchSource:0}: Error finding container 6c5b116627e10a066d8c785fe50824124fe68c17ad8325c5cee45ddaf8a1d983: Status 404 returned error can't find the container with id 6c5b116627e10a066d8c785fe50824124fe68c17ad8325c5cee45ddaf8a1d983
Sep 10 20:58:05.092327 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:05.092288    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fc96d36_95bc_4159_b504_7fad608e5819.slice/crio-0c892a13fa3fbe7ba500f68110bf333afd722b809a107586a9fc6da778f91297 WatchSource:0}: Error finding container 0c892a13fa3fbe7ba500f68110bf333afd722b809a107586a9fc6da778f91297: Status 404 returned error can't find the container with id 0c892a13fa3fbe7ba500f68110bf333afd722b809a107586a9fc6da778f91297
Sep 10 20:58:05.093229 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:05.093208    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v9h74"
Sep 10 20:58:05.099975 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:05.099954    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-h4md4"
Sep 10 20:58:05.102362 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:05.102334    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod749647ce_cddd_4e0b_ac39_b6682295073c.slice/crio-84864f4f2c34e9aae91335b4d17f9f1365ccab6b094f4b0d25634e478d83af75 WatchSource:0}: Error finding container 84864f4f2c34e9aae91335b4d17f9f1365ccab6b094f4b0d25634e478d83af75: Status 404 returned error can't find the container with id 84864f4f2c34e9aae91335b4d17f9f1365ccab6b094f4b0d25634e478d83af75
Sep 10 20:58:05.113476 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:05.109893    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:05.114436 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:05.113905    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f68613e_8086_4f5f_8a47_af8dcd8e3a76.slice/crio-b0be950a7a381b10ca503ef349b5bfe35f2347aa8d4ff381f77fc256c9715e8d WatchSource:0}: Error finding container b0be950a7a381b10ca503ef349b5bfe35f2347aa8d4ff381f77fc256c9715e8d: Status 404 returned error can't find the container with id b0be950a7a381b10ca503ef349b5bfe35f2347aa8d4ff381f77fc256c9715e8d
Sep 10 20:58:05.118091 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:05.117903    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2l7n"
Sep 10 20:58:05.123832 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:05.123806    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19d8d7ba_45a1_4013_a935_5670abe12b8a.slice/crio-9928f5f2fe9a2d50e13f7a89877fda1855c7497d253886038d7711f391f9ecbe WatchSource:0}: Error finding container 9928f5f2fe9a2d50e13f7a89877fda1855c7497d253886038d7711f391f9ecbe: Status 404 returned error can't find the container with id 9928f5f2fe9a2d50e13f7a89877fda1855c7497d253886038d7711f391f9ecbe
Sep 10 20:58:05.130415 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:05.130389    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cbd4c76_ca0c_4f1b_87e0_26a98dfa48ce.slice/crio-230858d46964f32f350be4dbb392e84637f4873a087972ad361f59bf13071140 WatchSource:0}: Error finding container 230858d46964f32f350be4dbb392e84637f4873a087972ad361f59bf13071140: Status 404 returned error can't find the container with id 230858d46964f32f350be4dbb392e84637f4873a087972ad361f59bf13071140
Sep 10 20:58:05.452729 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:05.452641    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/781245a4-8475-4031-9137-dd43ecd1c260-metrics-certs\") pod \"network-metrics-daemon-lmwz4\" (UID: \"781245a4-8475-4031-9137-dd43ecd1c260\") " pod="openshift-multus/network-metrics-daemon-lmwz4"
Sep 10 20:58:05.452729 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:05.452690    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e1f126a-f90d-4cf6-b863-d6130c01a622-original-pull-secret\") pod \"global-pull-secret-syncer-mgx5w\" (UID: \"6e1f126a-f90d-4cf6-b863-d6130c01a622\") " pod="kube-system/global-pull-secret-syncer-mgx5w"
Sep 10 20:58:05.452946 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:05.452804    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:58:05.452946 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:05.452860    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e1f126a-f90d-4cf6-b863-d6130c01a622-original-pull-secret podName:6e1f126a-f90d-4cf6-b863-d6130c01a622 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:06.452846388 +0000 UTC m=+4.109755153 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e1f126a-f90d-4cf6-b863-d6130c01a622-original-pull-secret") pod "global-pull-secret-syncer-mgx5w" (UID: "6e1f126a-f90d-4cf6-b863-d6130c01a622") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:58:05.452946 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:05.452802    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:58:05.452946 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:05.452937    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/781245a4-8475-4031-9137-dd43ecd1c260-metrics-certs podName:781245a4-8475-4031-9137-dd43ecd1c260 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:06.452925084 +0000 UTC m=+4.109833850 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/781245a4-8475-4031-9137-dd43ecd1c260-metrics-certs") pod "network-metrics-daemon-lmwz4" (UID: "781245a4-8475-4031-9137-dd43ecd1c260") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:58:05.553736 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:05.553698    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8nv4\" (UniqueName: \"kubernetes.io/projected/3bfd0b36-183c-43c3-9fcf-2d6f7839223e-kube-api-access-g8nv4\") pod \"network-check-target-gvw9h\" (UID: \"3bfd0b36-183c-43c3-9fcf-2d6f7839223e\") " pod="openshift-network-diagnostics/network-check-target-gvw9h"
Sep 10 20:58:05.553894 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:05.553845    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:58:05.553894 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:05.553869    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:58:05.553894 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:05.553881    2581 projected.go:196] Error preparing data for projected volume kube-api-access-g8nv4 for pod openshift-network-diagnostics/network-check-target-gvw9h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:58:05.554042 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:05.553949    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3bfd0b36-183c-43c3-9fcf-2d6f7839223e-kube-api-access-g8nv4 podName:3bfd0b36-183c-43c3-9fcf-2d6f7839223e nodeName:}" failed. No retries permitted until 2026-09-10 20:58:06.553927458 +0000 UTC m=+4.210836214 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-g8nv4" (UniqueName: "kubernetes.io/projected/3bfd0b36-183c-43c3-9fcf-2d6f7839223e-kube-api-access-g8nv4") pod "network-check-target-gvw9h" (UID: "3bfd0b36-183c-43c3-9fcf-2d6f7839223e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:58:05.841034 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:05.840921    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 20:53:05 +0000 UTC" deadline="2028-05-03 15:33:13.052354346 +0000 UTC"
Sep 10 20:58:05.841034 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:05.840986    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14418h35m7.211373221s"
Sep 10 20:58:05.862836 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:05.862802    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lmwz4"
Sep 10 20:58:05.862984 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:05.862855    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mgx5w"
Sep 10 20:58:05.862984 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:05.862961    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lmwz4" podUID="781245a4-8475-4031-9137-dd43ecd1c260"
Sep 10 20:58:05.863111 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:05.863084    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mgx5w" podUID="6e1f126a-f90d-4cf6-b863-d6130c01a622"
Sep 10 20:58:05.875476 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:05.875367    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-brrtd" event={"ID":"19d8d7ba-45a1-4013-a935-5670abe12b8a","Type":"ContainerStarted","Data":"9928f5f2fe9a2d50e13f7a89877fda1855c7497d253886038d7711f391f9ecbe"}
Sep 10 20:58:05.876951 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:05.876921    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h4md4" event={"ID":"0f68613e-8086-4f5f-8a47-af8dcd8e3a76","Type":"ContainerStarted","Data":"b0be950a7a381b10ca503ef349b5bfe35f2347aa8d4ff381f77fc256c9715e8d"}
Sep 10 20:58:05.878211 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:05.878186    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mthkf" event={"ID":"7fc96d36-95bc-4159-b504-7fad608e5819","Type":"ContainerStarted","Data":"0c892a13fa3fbe7ba500f68110bf333afd722b809a107586a9fc6da778f91297"}
Sep 10 20:58:05.879476 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:05.879448    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-z4glj" event={"ID":"705a88df-c5e0-4082-b22e-e1aad389cd03","Type":"ContainerStarted","Data":"6c5b116627e10a066d8c785fe50824124fe68c17ad8325c5cee45ddaf8a1d983"}
Sep 10 20:58:05.881590 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:05.881568    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qrbt5" event={"ID":"05ef07fc-6c16-4be8-8d44-4dfa99384a03","Type":"ContainerStarted","Data":"269f4a129eb9a46557005f27be6e3a228fca10d6b91bc9f4084f4272886e0af0"}
Sep 10 20:58:05.885697 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:05.885671    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5rbwt" event={"ID":"432f3e3d-207e-4732-97f7-43182d9582b9","Type":"ContainerStarted","Data":"f7b487a90adeeab3f3242a52712c0b0bfb8b2b3f7a42debb4ea2359419c4097a"}
Sep 10 20:58:05.887024 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:05.886972    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2l7n" event={"ID":"8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce","Type":"ContainerStarted","Data":"230858d46964f32f350be4dbb392e84637f4873a087972ad361f59bf13071140"}
Sep 10 20:58:05.888418 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:05.888391    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v9h74" event={"ID":"749647ce-cddd-4e0b-ac39-b6682295073c","Type":"ContainerStarted","Data":"84864f4f2c34e9aae91335b4d17f9f1365ccab6b094f4b0d25634e478d83af75"}
Sep 10 20:58:05.889811 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:05.889762    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l2gkq" event={"ID":"407d670d-5138-4d8b-aba7-96bdd91d9b19","Type":"ContainerStarted","Data":"d2fbd66e8911fa87fc439ddfcb5aac1abc4595994c63d3dbd9ae6e71ca90e96e"}
Sep 10 20:58:06.460791 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:06.459955    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/781245a4-8475-4031-9137-dd43ecd1c260-metrics-certs\") pod \"network-metrics-daemon-lmwz4\" (UID: \"781245a4-8475-4031-9137-dd43ecd1c260\") " pod="openshift-multus/network-metrics-daemon-lmwz4"
Sep 10 20:58:06.460791 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:06.460010    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e1f126a-f90d-4cf6-b863-d6130c01a622-original-pull-secret\") pod \"global-pull-secret-syncer-mgx5w\" (UID: \"6e1f126a-f90d-4cf6-b863-d6130c01a622\") " pod="kube-system/global-pull-secret-syncer-mgx5w"
Sep 10 20:58:06.460791 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:06.460176    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:58:06.460791 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:06.460241    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e1f126a-f90d-4cf6-b863-d6130c01a622-original-pull-secret podName:6e1f126a-f90d-4cf6-b863-d6130c01a622 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:08.460221847 +0000 UTC m=+6.117130622 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e1f126a-f90d-4cf6-b863-d6130c01a622-original-pull-secret") pod "global-pull-secret-syncer-mgx5w" (UID: "6e1f126a-f90d-4cf6-b863-d6130c01a622") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:58:06.460791 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:06.460696    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:58:06.460791 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:06.460746    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/781245a4-8475-4031-9137-dd43ecd1c260-metrics-certs podName:781245a4-8475-4031-9137-dd43ecd1c260 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:08.460730963 +0000 UTC m=+6.117639721 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/781245a4-8475-4031-9137-dd43ecd1c260-metrics-certs") pod "network-metrics-daemon-lmwz4" (UID: "781245a4-8475-4031-9137-dd43ecd1c260") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:58:06.478684 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:06.478316    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 20:58:06.561416 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:06.561372    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8nv4\" (UniqueName: \"kubernetes.io/projected/3bfd0b36-183c-43c3-9fcf-2d6f7839223e-kube-api-access-g8nv4\") pod \"network-check-target-gvw9h\" (UID: \"3bfd0b36-183c-43c3-9fcf-2d6f7839223e\") " pod="openshift-network-diagnostics/network-check-target-gvw9h"
Sep 10 20:58:06.561589 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:06.561577    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:58:06.561677 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:06.561603    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:58:06.561677 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:06.561617    2581 projected.go:196] Error preparing data for projected volume kube-api-access-g8nv4 for pod openshift-network-diagnostics/network-check-target-gvw9h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:58:06.561788 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:06.561682    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3bfd0b36-183c-43c3-9fcf-2d6f7839223e-kube-api-access-g8nv4 podName:3bfd0b36-183c-43c3-9fcf-2d6f7839223e nodeName:}" failed. No retries permitted until 2026-09-10 20:58:08.561660977 +0000 UTC m=+6.218569747 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-g8nv4" (UniqueName: "kubernetes.io/projected/3bfd0b36-183c-43c3-9fcf-2d6f7839223e-kube-api-access-g8nv4") pod "network-check-target-gvw9h" (UID: "3bfd0b36-183c-43c3-9fcf-2d6f7839223e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:58:06.841441 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:06.841340    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 20:53:05 +0000 UTC" deadline="2028-06-12 22:35:29.999201452 +0000 UTC"
Sep 10 20:58:06.841441 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:06.841377    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15385h37m23.15782891s"
Sep 10 20:58:06.864497 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:06.863959    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gvw9h"
Sep 10 20:58:06.864497 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:06.864083    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gvw9h" podUID="3bfd0b36-183c-43c3-9fcf-2d6f7839223e"
Sep 10 20:58:07.862689 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:07.862611    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mgx5w"
Sep 10 20:58:07.863368 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:07.862749    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mgx5w" podUID="6e1f126a-f90d-4cf6-b863-d6130c01a622"
Sep 10 20:58:07.863368 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:07.862829    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lmwz4"
Sep 10 20:58:07.863368 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:07.862931    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lmwz4" podUID="781245a4-8475-4031-9137-dd43ecd1c260"
Sep 10 20:58:07.906572 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:07.906534    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-165.ec2.internal" event={"ID":"d43f2c9998219eadf34d68ce7c7d50dc","Type":"ContainerStarted","Data":"0c430c15286f281c3c5e2b18097c1869d34c9c1a489af5077eb799578b9b86c0"}
Sep 10 20:58:07.910745 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:07.910715    2581 generic.go:356] "Generic (PLEG): container finished" podID="65f1e16d97785601ead433dc27092e06" containerID="1a389531057c025b66e824a916a817b29c4113b9f25d7fd51db0d0b9fe6b6efe" exitCode=0
Sep 10 20:58:07.910908 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:07.910768    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-165.ec2.internal" event={"ID":"65f1e16d97785601ead433dc27092e06","Type":"ContainerDied","Data":"1a389531057c025b66e824a916a817b29c4113b9f25d7fd51db0d0b9fe6b6efe"}
Sep 10 20:58:07.934933 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:07.934876    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-165.ec2.internal" podStartSLOduration=2.934855184 podStartE2EDuration="2.934855184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:58:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:07.933818592 +0000 UTC m=+5.590727367" watchObservedRunningTime="2026-09-10 20:58:07.934855184 +0000 UTC m=+5.591763960"
Sep 10 20:58:08.481204 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:08.481164    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/781245a4-8475-4031-9137-dd43ecd1c260-metrics-certs\") pod \"network-metrics-daemon-lmwz4\" (UID: \"781245a4-8475-4031-9137-dd43ecd1c260\") " pod="openshift-multus/network-metrics-daemon-lmwz4"
Sep 10 20:58:08.481465 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:08.481222    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e1f126a-f90d-4cf6-b863-d6130c01a622-original-pull-secret\") pod \"global-pull-secret-syncer-mgx5w\" (UID: \"6e1f126a-f90d-4cf6-b863-d6130c01a622\") " pod="kube-system/global-pull-secret-syncer-mgx5w"
Sep 10 20:58:08.481465 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:08.481375    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:58:08.481465 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:08.481437    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e1f126a-f90d-4cf6-b863-d6130c01a622-original-pull-secret podName:6e1f126a-f90d-4cf6-b863-d6130c01a622 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:12.481418235 +0000 UTC m=+10.138327005 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e1f126a-f90d-4cf6-b863-d6130c01a622-original-pull-secret") pod "global-pull-secret-syncer-mgx5w" (UID: "6e1f126a-f90d-4cf6-b863-d6130c01a622") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:58:08.481672 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:08.481587    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:58:08.481672 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:08.481626    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/781245a4-8475-4031-9137-dd43ecd1c260-metrics-certs podName:781245a4-8475-4031-9137-dd43ecd1c260 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:12.48161213 +0000 UTC m=+10.138520883 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/781245a4-8475-4031-9137-dd43ecd1c260-metrics-certs") pod "network-metrics-daemon-lmwz4" (UID: "781245a4-8475-4031-9137-dd43ecd1c260") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:58:08.582606 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:08.582569    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8nv4\" (UniqueName: \"kubernetes.io/projected/3bfd0b36-183c-43c3-9fcf-2d6f7839223e-kube-api-access-g8nv4\") pod \"network-check-target-gvw9h\" (UID: \"3bfd0b36-183c-43c3-9fcf-2d6f7839223e\") " pod="openshift-network-diagnostics/network-check-target-gvw9h"
Sep 10 20:58:08.582796 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:08.582737    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:58:08.582796 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:08.582760    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:58:08.582796 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:08.582772    2581 projected.go:196] Error preparing data for projected volume kube-api-access-g8nv4 for pod openshift-network-diagnostics/network-check-target-gvw9h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:58:08.582957 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:08.582835    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3bfd0b36-183c-43c3-9fcf-2d6f7839223e-kube-api-access-g8nv4 podName:3bfd0b36-183c-43c3-9fcf-2d6f7839223e nodeName:}" failed. No retries permitted until 2026-09-10 20:58:12.582816203 +0000 UTC m=+10.239724962 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-g8nv4" (UniqueName: "kubernetes.io/projected/3bfd0b36-183c-43c3-9fcf-2d6f7839223e-kube-api-access-g8nv4") pod "network-check-target-gvw9h" (UID: "3bfd0b36-183c-43c3-9fcf-2d6f7839223e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:58:08.863910 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:08.863422    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gvw9h"
Sep 10 20:58:08.863910 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:08.863587    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gvw9h" podUID="3bfd0b36-183c-43c3-9fcf-2d6f7839223e"
Sep 10 20:58:09.862776 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:09.862740    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mgx5w"
Sep 10 20:58:09.862983 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:09.862888    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mgx5w" podUID="6e1f126a-f90d-4cf6-b863-d6130c01a622"
Sep 10 20:58:09.862983 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:09.862958    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lmwz4"
Sep 10 20:58:09.863109 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:09.863046    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lmwz4" podUID="781245a4-8475-4031-9137-dd43ecd1c260"
Sep 10 20:58:10.862950 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:10.862910    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gvw9h"
Sep 10 20:58:10.863503 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:10.863081    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gvw9h" podUID="3bfd0b36-183c-43c3-9fcf-2d6f7839223e"
Sep 10 20:58:11.862500 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:11.862410    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mgx5w"
Sep 10 20:58:11.862663 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:11.862420    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lmwz4"
Sep 10 20:58:11.862663 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:11.862554    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mgx5w" podUID="6e1f126a-f90d-4cf6-b863-d6130c01a622"
Sep 10 20:58:11.862663 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:11.862636    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lmwz4" podUID="781245a4-8475-4031-9137-dd43ecd1c260"
Sep 10 20:58:12.516411 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:12.516336    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/781245a4-8475-4031-9137-dd43ecd1c260-metrics-certs\") pod \"network-metrics-daemon-lmwz4\" (UID: \"781245a4-8475-4031-9137-dd43ecd1c260\") " pod="openshift-multus/network-metrics-daemon-lmwz4"
Sep 10 20:58:12.516411 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:12.516397    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e1f126a-f90d-4cf6-b863-d6130c01a622-original-pull-secret\") pod \"global-pull-secret-syncer-mgx5w\" (UID: \"6e1f126a-f90d-4cf6-b863-d6130c01a622\") " pod="kube-system/global-pull-secret-syncer-mgx5w"
Sep 10 20:58:12.517115 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:12.516553    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:58:12.517115 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:12.516616    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e1f126a-f90d-4cf6-b863-d6130c01a622-original-pull-secret podName:6e1f126a-f90d-4cf6-b863-d6130c01a622 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:20.516595963 +0000 UTC m=+18.173504733 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e1f126a-f90d-4cf6-b863-d6130c01a622-original-pull-secret") pod "global-pull-secret-syncer-mgx5w" (UID: "6e1f126a-f90d-4cf6-b863-d6130c01a622") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:58:12.517115 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:12.517027    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:58:12.517115 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:12.517085    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/781245a4-8475-4031-9137-dd43ecd1c260-metrics-certs podName:781245a4-8475-4031-9137-dd43ecd1c260 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:20.517066571 +0000 UTC m=+18.173975326 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/781245a4-8475-4031-9137-dd43ecd1c260-metrics-certs") pod "network-metrics-daemon-lmwz4" (UID: "781245a4-8475-4031-9137-dd43ecd1c260") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:58:12.617614 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:12.617479    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8nv4\" (UniqueName: \"kubernetes.io/projected/3bfd0b36-183c-43c3-9fcf-2d6f7839223e-kube-api-access-g8nv4\") pod \"network-check-target-gvw9h\" (UID: \"3bfd0b36-183c-43c3-9fcf-2d6f7839223e\") " pod="openshift-network-diagnostics/network-check-target-gvw9h"
Sep 10 20:58:12.617794 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:12.617704    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:58:12.617794 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:12.617736    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:58:12.617794 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:12.617750    2581 projected.go:196] Error preparing data for projected volume kube-api-access-g8nv4 for pod openshift-network-diagnostics/network-check-target-gvw9h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:58:12.618005 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:12.617826    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3bfd0b36-183c-43c3-9fcf-2d6f7839223e-kube-api-access-g8nv4 podName:3bfd0b36-183c-43c3-9fcf-2d6f7839223e nodeName:}" failed. No retries permitted until 2026-09-10 20:58:20.617802594 +0000 UTC m=+18.274711373 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-g8nv4" (UniqueName: "kubernetes.io/projected/3bfd0b36-183c-43c3-9fcf-2d6f7839223e-kube-api-access-g8nv4") pod "network-check-target-gvw9h" (UID: "3bfd0b36-183c-43c3-9fcf-2d6f7839223e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:58:12.867135 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:12.866575    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gvw9h"
Sep 10 20:58:12.867135 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:12.866710    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gvw9h" podUID="3bfd0b36-183c-43c3-9fcf-2d6f7839223e"
Sep 10 20:58:13.861983 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:13.861939    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mgx5w"
Sep 10 20:58:13.862409 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:13.861950    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lmwz4"
Sep 10 20:58:13.862409 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:13.862090    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mgx5w" podUID="6e1f126a-f90d-4cf6-b863-d6130c01a622"
Sep 10 20:58:13.862409 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:13.862179    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lmwz4" podUID="781245a4-8475-4031-9137-dd43ecd1c260"
Sep 10 20:58:14.862446 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:14.862412    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gvw9h"
Sep 10 20:58:14.862961 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:14.862549    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gvw9h" podUID="3bfd0b36-183c-43c3-9fcf-2d6f7839223e"
Sep 10 20:58:15.862340 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:15.862305    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lmwz4"
Sep 10 20:58:15.862491 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:15.862343    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mgx5w"
Sep 10 20:58:15.862491 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:15.862451    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lmwz4" podUID="781245a4-8475-4031-9137-dd43ecd1c260"
Sep 10 20:58:15.862869 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:15.862574    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mgx5w" podUID="6e1f126a-f90d-4cf6-b863-d6130c01a622"
Sep 10 20:58:16.862717 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:16.862623    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gvw9h"
Sep 10 20:58:16.863133 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:16.862738    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gvw9h" podUID="3bfd0b36-183c-43c3-9fcf-2d6f7839223e"
Sep 10 20:58:17.862459 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:17.862404    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lmwz4"
Sep 10 20:58:17.862459 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:17.862453    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mgx5w"
Sep 10 20:58:17.862662 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:17.862554    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lmwz4" podUID="781245a4-8475-4031-9137-dd43ecd1c260"
Sep 10 20:58:17.862738 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:17.862680    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mgx5w" podUID="6e1f126a-f90d-4cf6-b863-d6130c01a622"
Sep 10 20:58:18.862867 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:18.862838    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gvw9h"
Sep 10 20:58:18.863269 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:18.862961    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gvw9h" podUID="3bfd0b36-183c-43c3-9fcf-2d6f7839223e"
Sep 10 20:58:19.862074 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:19.862042    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mgx5w"
Sep 10 20:58:19.862278 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:19.862052    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lmwz4"
Sep 10 20:58:19.862278 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:19.862172    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mgx5w" podUID="6e1f126a-f90d-4cf6-b863-d6130c01a622"
Sep 10 20:58:19.862278 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:19.862231    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lmwz4" podUID="781245a4-8475-4031-9137-dd43ecd1c260"
Sep 10 20:58:20.576577 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:20.576539    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/781245a4-8475-4031-9137-dd43ecd1c260-metrics-certs\") pod \"network-metrics-daemon-lmwz4\" (UID: \"781245a4-8475-4031-9137-dd43ecd1c260\") " pod="openshift-multus/network-metrics-daemon-lmwz4"
Sep 10 20:58:20.576577 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:20.576594    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e1f126a-f90d-4cf6-b863-d6130c01a622-original-pull-secret\") pod \"global-pull-secret-syncer-mgx5w\" (UID: \"6e1f126a-f90d-4cf6-b863-d6130c01a622\") " pod="kube-system/global-pull-secret-syncer-mgx5w"
Sep 10 20:58:20.577205 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:20.576722    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:58:20.577205 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:20.576748    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:58:20.577205 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:20.576811    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e1f126a-f90d-4cf6-b863-d6130c01a622-original-pull-secret podName:6e1f126a-f90d-4cf6-b863-d6130c01a622 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:36.576789941 +0000 UTC m=+34.233698691 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e1f126a-f90d-4cf6-b863-d6130c01a622-original-pull-secret") pod "global-pull-secret-syncer-mgx5w" (UID: "6e1f126a-f90d-4cf6-b863-d6130c01a622") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:58:20.577205 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:20.576845    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/781245a4-8475-4031-9137-dd43ecd1c260-metrics-certs podName:781245a4-8475-4031-9137-dd43ecd1c260 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:36.576823505 +0000 UTC m=+34.233732287 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/781245a4-8475-4031-9137-dd43ecd1c260-metrics-certs") pod "network-metrics-daemon-lmwz4" (UID: "781245a4-8475-4031-9137-dd43ecd1c260") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:58:20.677252 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:20.677218    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8nv4\" (UniqueName: \"kubernetes.io/projected/3bfd0b36-183c-43c3-9fcf-2d6f7839223e-kube-api-access-g8nv4\") pod \"network-check-target-gvw9h\" (UID: \"3bfd0b36-183c-43c3-9fcf-2d6f7839223e\") " pod="openshift-network-diagnostics/network-check-target-gvw9h"
Sep 10 20:58:20.677468 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:20.677385    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:58:20.677468 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:20.677405    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:58:20.677468 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:20.677415    2581 projected.go:196] Error preparing data for projected volume kube-api-access-g8nv4 for pod openshift-network-diagnostics/network-check-target-gvw9h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:58:20.677468 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:20.677466    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3bfd0b36-183c-43c3-9fcf-2d6f7839223e-kube-api-access-g8nv4 podName:3bfd0b36-183c-43c3-9fcf-2d6f7839223e nodeName:}" failed. No retries permitted until 2026-09-10 20:58:36.67745366 +0000 UTC m=+34.334362415 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-g8nv4" (UniqueName: "kubernetes.io/projected/3bfd0b36-183c-43c3-9fcf-2d6f7839223e-kube-api-access-g8nv4") pod "network-check-target-gvw9h" (UID: "3bfd0b36-183c-43c3-9fcf-2d6f7839223e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:58:20.862825 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:20.862729    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gvw9h"
Sep 10 20:58:20.862958 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:20.862855    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gvw9h" podUID="3bfd0b36-183c-43c3-9fcf-2d6f7839223e"
Sep 10 20:58:21.862802 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:21.862760    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lmwz4"
Sep 10 20:58:21.863180 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:21.862778    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mgx5w"
Sep 10 20:58:21.863180 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:21.862892    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lmwz4" podUID="781245a4-8475-4031-9137-dd43ecd1c260"
Sep 10 20:58:21.863180 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:21.862928    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mgx5w" podUID="6e1f126a-f90d-4cf6-b863-d6130c01a622"
Sep 10 20:58:22.863034 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:22.862998    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gvw9h"
Sep 10 20:58:22.863490 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:22.863101    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gvw9h" podUID="3bfd0b36-183c-43c3-9fcf-2d6f7839223e"
Sep 10 20:58:23.862441 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:23.862411    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mgx5w"
Sep 10 20:58:23.862602 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:23.862415    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lmwz4"
Sep 10 20:58:23.862602 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:23.862504    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mgx5w" podUID="6e1f126a-f90d-4cf6-b863-d6130c01a622"
Sep 10 20:58:23.862602 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:23.862590    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lmwz4" podUID="781245a4-8475-4031-9137-dd43ecd1c260"
Sep 10 20:58:24.862895 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:24.862589    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gvw9h"
Sep 10 20:58:24.863501 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:24.862966    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gvw9h" podUID="3bfd0b36-183c-43c3-9fcf-2d6f7839223e"
Sep 10 20:58:24.947330 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:24.947236    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2l7n" event={"ID":"8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce","Type":"ContainerStarted","Data":"3eaad15b5ce29564cada7412c5fcfdc24bdf373a38631c40b13ae87a2b600c04"}
Sep 10 20:58:24.948930 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:24.948905    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l2gkq" event={"ID":"407d670d-5138-4d8b-aba7-96bdd91d9b19","Type":"ContainerStarted","Data":"81eced4b52beaeefde039d8c12de03d24151d9bf4aeb99a8eeda57aa42208dde"}
Sep 10 20:58:24.950962 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:24.950932    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-165.ec2.internal" event={"ID":"65f1e16d97785601ead433dc27092e06","Type":"ContainerStarted","Data":"239566c90bbbee86b6161926e4362d360d39f52261a7ff047ae02fb4c3a320ba"}
Sep 10 20:58:24.954068 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:24.954050    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brrtd_19d8d7ba-45a1-4013-a935-5670abe12b8a/ovn-acl-logging/0.log"
Sep 10 20:58:24.954442 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:24.954421    2581 generic.go:356] "Generic (PLEG): container finished" podID="19d8d7ba-45a1-4013-a935-5670abe12b8a" containerID="616b2d955f5cc07b82b6ee7af0e821e43e57393a9ee6ccc8abccb91fabb571de" exitCode=1
Sep 10 20:58:24.954525 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:24.954472    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-brrtd" event={"ID":"19d8d7ba-45a1-4013-a935-5670abe12b8a","Type":"ContainerStarted","Data":"88320fff71682d547ea43ecd7b7c0a6137a47e2edb32c8ac7cc6a2635cffa786"}
Sep 10 20:58:24.954571 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:24.954529    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-brrtd" event={"ID":"19d8d7ba-45a1-4013-a935-5670abe12b8a","Type":"ContainerStarted","Data":"141853f7e9991ff2d2f2952fcf66c32a8060de0b666420ec787b4af04eb04a23"}
Sep 10 20:58:24.954571 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:24.954564    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-brrtd" event={"ID":"19d8d7ba-45a1-4013-a935-5670abe12b8a","Type":"ContainerStarted","Data":"50ed9b53a74fb8863c51f11377b199cd96a6a48cf68a3e768a6c326be63dce2e"}
Sep 10 20:58:24.954656 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:24.954603    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-brrtd" event={"ID":"19d8d7ba-45a1-4013-a935-5670abe12b8a","Type":"ContainerStarted","Data":"2137789dd7ca6d3f7c684235dd1ab81799248e476cd072d0bd77a7131bf8177f"}
Sep 10 20:58:24.954656 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:24.954630    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-brrtd" event={"ID":"19d8d7ba-45a1-4013-a935-5670abe12b8a","Type":"ContainerDied","Data":"616b2d955f5cc07b82b6ee7af0e821e43e57393a9ee6ccc8abccb91fabb571de"}
Sep 10 20:58:24.954747 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:24.954657    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-brrtd" event={"ID":"19d8d7ba-45a1-4013-a935-5670abe12b8a","Type":"ContainerStarted","Data":"f1fcc5964ae56e98a01ebd49f64ce2b66019d7c8edb40248224c2015860a8cd7"}
Sep 10 20:58:24.956001 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:24.955979    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-h4md4" event={"ID":"0f68613e-8086-4f5f-8a47-af8dcd8e3a76","Type":"ContainerStarted","Data":"3ffb9eabaa1485c69f6e23637be9ab2f254189f4dc20ed1986385120a20d98e6"}
Sep 10 20:58:24.960215 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:24.960191    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mthkf" event={"ID":"7fc96d36-95bc-4159-b504-7fad608e5819","Type":"ContainerStarted","Data":"40af44a4fe0b0310152137a1366362a1d38fc22ae1fa90cbc01f3467539b8a90"}
Sep 10 20:58:24.962785 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:24.962761    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-z4glj" event={"ID":"705a88df-c5e0-4082-b22e-e1aad389cd03","Type":"ContainerStarted","Data":"bdcd15309a4839afc1c5669d1396ba690fb0adbddcb0eee09cc7a3a8c5359184"}
Sep 10 20:58:24.964382 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:24.964361    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qrbt5" event={"ID":"05ef07fc-6c16-4be8-8d44-4dfa99384a03","Type":"ContainerStarted","Data":"5adf36eae7b432ff0ebdfa6c72806497abc837328e52518ab796948a7d689224"}
Sep 10 20:58:24.966027 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:24.966008    2581 generic.go:356] "Generic (PLEG): container finished" podID="432f3e3d-207e-4732-97f7-43182d9582b9" containerID="8fa8905848266c86033bad59ed3e79d15455a507e2b870d852a001e6ae34942f" exitCode=0
Sep 10 20:58:24.966112 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:24.966035    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5rbwt" event={"ID":"432f3e3d-207e-4732-97f7-43182d9582b9","Type":"ContainerDied","Data":"8fa8905848266c86033bad59ed3e79d15455a507e2b870d852a001e6ae34942f"}
Sep 10 20:58:24.973889 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:24.973844    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-l2gkq" podStartSLOduration=2.97343814 podStartE2EDuration="21.973828974s" totalImagesPullingTime="19.000390834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:24.973278871 +0000 UTC m=+22.630187644" watchObservedRunningTime="2026-09-10 20:58:24.973828974 +0000 UTC m=+22.630737747"
Sep 10 20:58:24.996402 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:24.996355    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-z4glj" podStartSLOduration=3.027616439 podStartE2EDuration="21.996342084s" totalImagesPullingTime="18.968725645s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:24.995927858 +0000 UTC m=+22.652836631" watchObservedRunningTime="2026-09-10 20:58:24.996342084 +0000 UTC m=+22.653250857"
Sep 10 20:58:25.038829 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:25.038784    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-h4md4" podStartSLOduration=2.100434008 podStartE2EDuration="21.038771865s" totalImagesPullingTime="18.938337857s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:25.017342999 +0000 UTC m=+22.674251768" watchObservedRunningTime="2026-09-10 20:58:25.038771865 +0000 UTC m=+22.695680637"
Sep 10 20:58:25.038955 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:25.038910    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-165.ec2.internal" podStartSLOduration=20.038900682 podStartE2EDuration="20.038900682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:58:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:25.038897072 +0000 UTC m=+22.695805826" watchObservedRunningTime="2026-09-10 20:58:25.038900682 +0000 UTC m=+22.695809458"
Sep 10 20:58:25.116749 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:25.116702    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mthkf" podStartSLOduration=3.15667963 podStartE2EDuration="22.116685258s" totalImagesPullingTime="18.960005628s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:25.087745199 +0000 UTC m=+22.744653971" watchObservedRunningTime="2026-09-10 20:58:25.116685258 +0000 UTC m=+22.773594024"
Sep 10 20:58:25.505850 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:25.505812    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 20:58:25.788820 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:25.788685    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T20:58:25.505832158Z","UUID":"04d3ea3e-0d02-48a0-948b-a6d5dd203604","Handler":null,"Name":"","Endpoint":""}
Sep 10 20:58:25.791621 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:25.791274    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 20:58:25.791621 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:25.791331    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 20:58:25.861936 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:25.861909    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mgx5w"
Sep 10 20:58:25.862089 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:25.861909    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lmwz4"
Sep 10 20:58:25.862089 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:25.862033    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mgx5w" podUID="6e1f126a-f90d-4cf6-b863-d6130c01a622"
Sep 10 20:58:25.862223 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:25.862108    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lmwz4" podUID="781245a4-8475-4031-9137-dd43ecd1c260"
Sep 10 20:58:25.971324 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:25.971232    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2l7n" event={"ID":"8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce","Type":"ContainerStarted","Data":"b0d3b5c327a28a54f2450241e57112655eb0a67dc4f7736c6c9dc6d1b455de21"}
Sep 10 20:58:25.974365 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:25.974283    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v9h74" event={"ID":"749647ce-cddd-4e0b-ac39-b6682295073c","Type":"ContainerStarted","Data":"03a9747413b91568ebce971121d4e8863e8e340fde72e40cd2a7bff1a335947c"}
Sep 10 20:58:25.995437 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:25.995389    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-qrbt5" podStartSLOduration=3.681396267 podStartE2EDuration="22.9953708s" totalImagesPullingTime="19.313974533s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:25.123668088 +0000 UTC m=+22.780576860" watchObservedRunningTime="2026-09-10 20:58:25.9953708 +0000 UTC m=+23.652279575"
Sep 10 20:58:25.995610 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:25.995583    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-v9h74" podStartSLOduration=3.041614002 podStartE2EDuration="21.99557482s" totalImagesPullingTime="18.953960818s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:25.995027274 +0000 UTC m=+23.651936049" watchObservedRunningTime="2026-09-10 20:58:25.99557482 +0000 UTC m=+23.652483592"
Sep 10 20:58:26.862519 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:26.862443    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gvw9h"
Sep 10 20:58:26.862653 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:26.862575    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gvw9h" podUID="3bfd0b36-183c-43c3-9fcf-2d6f7839223e"
Sep 10 20:58:26.979864 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:26.979839    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brrtd_19d8d7ba-45a1-4013-a935-5670abe12b8a/ovn-acl-logging/0.log"
Sep 10 20:58:26.980378 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:26.980345    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-brrtd" event={"ID":"19d8d7ba-45a1-4013-a935-5670abe12b8a","Type":"ContainerStarted","Data":"ef11ef0858d9d229095e304e48d8409a0ed75597134e76c774005110e69f8155"}
Sep 10 20:58:26.982676 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:26.982634    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2l7n" event={"ID":"8cbd4c76-ca0c-4f1b-87e0-26a98dfa48ce","Type":"ContainerStarted","Data":"3e22db951e4d6412b81d59f2c43d512ad2b25b6b7d7dfd01beebd523d934eea4"}
Sep 10 20:58:27.862155 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:27.862132    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lmwz4"
Sep 10 20:58:27.862155 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:27.862140    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mgx5w"
Sep 10 20:58:27.862363 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:27.862249    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lmwz4" podUID="781245a4-8475-4031-9137-dd43ecd1c260"
Sep 10 20:58:27.862428 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:27.862386    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mgx5w" podUID="6e1f126a-f90d-4cf6-b863-d6130c01a622"
Sep 10 20:58:28.160929 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:28.160834    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-l2gkq"
Sep 10 20:58:28.161655 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:28.161635    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-l2gkq"
Sep 10 20:58:28.189870 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:28.189822    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2l7n" podStartSLOduration=10.368879917 podStartE2EDuration="24.189808693s" totalImagesPullingTime="13.820928776s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:58:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:27.016748107 +0000 UTC m=+24.673656882" watchObservedRunningTime="2026-09-10 20:58:28.189808693 +0000 UTC m=+25.846717487"
Sep 10 20:58:28.862695 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:28.862674    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gvw9h"
Sep 10 20:58:28.862782 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:28.862768    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gvw9h" podUID="3bfd0b36-183c-43c3-9fcf-2d6f7839223e"
Sep 10 20:58:28.991823 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:28.991797    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brrtd_19d8d7ba-45a1-4013-a935-5670abe12b8a/ovn-acl-logging/0.log"
Sep 10 20:58:28.992224 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:28.992190    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-brrtd" event={"ID":"19d8d7ba-45a1-4013-a935-5670abe12b8a","Type":"ContainerStarted","Data":"9f2b588c7ea8dc37b9ffb135eb0a1467d5059a524354d61bb69dafdb241e2673"}
Sep 10 20:58:28.992698 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:28.992666    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-l2gkq"
Sep 10 20:58:28.992803 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:28.992702    2581 scope.go:117] "RemoveContainer" containerID="616b2d955f5cc07b82b6ee7af0e821e43e57393a9ee6ccc8abccb91fabb571de"
Sep 10 20:58:28.992923 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:28.992907    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-l2gkq"
Sep 10 20:58:29.862535 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:29.862366    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lmwz4"
Sep 10 20:58:29.862930 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:29.862407    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mgx5w"
Sep 10 20:58:29.862930 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:29.862605    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lmwz4" podUID="781245a4-8475-4031-9137-dd43ecd1c260"
Sep 10 20:58:29.862930 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:29.862694    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mgx5w" podUID="6e1f126a-f90d-4cf6-b863-d6130c01a622"
Sep 10 20:58:29.995601 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:29.995568    2581 generic.go:356] "Generic (PLEG): container finished" podID="432f3e3d-207e-4732-97f7-43182d9582b9" containerID="868e37341334f641d52f3bddd0b8fb575fe9a25003a4a1ea97d4707889d1fa84" exitCode=0
Sep 10 20:58:29.995813 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:29.995650    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5rbwt" event={"ID":"432f3e3d-207e-4732-97f7-43182d9582b9","Type":"ContainerDied","Data":"868e37341334f641d52f3bddd0b8fb575fe9a25003a4a1ea97d4707889d1fa84"}
Sep 10 20:58:29.999477 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:29.999459    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brrtd_19d8d7ba-45a1-4013-a935-5670abe12b8a/ovn-acl-logging/0.log"
Sep 10 20:58:29.999838 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:29.999814    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-brrtd" event={"ID":"19d8d7ba-45a1-4013-a935-5670abe12b8a","Type":"ContainerStarted","Data":"93d94648accb5d91cdb96316df94a20b031b4067ba868761b3ef392559472785"}
Sep 10 20:58:29.999981 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:29.999960    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:30.000094 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:30.000019    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:30.000200 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:30.000186    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:30.015260 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:30.015236    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:30.015450 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:30.015393    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:58:30.072619 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:30.072522    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-brrtd" podStartSLOduration=7.107237767 podStartE2EDuration="26.072504379s" totalImagesPullingTime="18.965266612s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:30.072021771 +0000 UTC m=+27.728930542" watchObservedRunningTime="2026-09-10 20:58:30.072504379 +0000 UTC m=+27.729413150"
Sep 10 20:58:30.862745 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:30.862720    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gvw9h"
Sep 10 20:58:30.863410 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:30.862819    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gvw9h" podUID="3bfd0b36-183c-43c3-9fcf-2d6f7839223e"
Sep 10 20:58:30.982092 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:30.982026    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lmwz4"]
Sep 10 20:58:30.982309 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:30.982176    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lmwz4"
Sep 10 20:58:30.982401 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:30.982309    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lmwz4" podUID="781245a4-8475-4031-9137-dd43ecd1c260"
Sep 10 20:58:30.983336 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:30.983313    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gvw9h"]
Sep 10 20:58:30.983980 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:30.983962    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mgx5w"]
Sep 10 20:58:30.984054 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:30.984042    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mgx5w"
Sep 10 20:58:30.984132 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:30.984117    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mgx5w" podUID="6e1f126a-f90d-4cf6-b863-d6130c01a622"
Sep 10 20:58:31.005019 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:31.004995    2581 generic.go:356] "Generic (PLEG): container finished" podID="432f3e3d-207e-4732-97f7-43182d9582b9" containerID="5ecb50b17090ef51ac16434e18319cb39b17f68294cd0672cb4e040bc7140f3c" exitCode=0
Sep 10 20:58:31.005144 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:31.005057    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gvw9h"
Sep 10 20:58:31.005144 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:31.005080    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5rbwt" event={"ID":"432f3e3d-207e-4732-97f7-43182d9582b9","Type":"ContainerDied","Data":"5ecb50b17090ef51ac16434e18319cb39b17f68294cd0672cb4e040bc7140f3c"}
Sep 10 20:58:31.005258 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:31.005220    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gvw9h" podUID="3bfd0b36-183c-43c3-9fcf-2d6f7839223e"
Sep 10 20:58:32.010133 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:32.009930    2581 generic.go:356] "Generic (PLEG): container finished" podID="432f3e3d-207e-4732-97f7-43182d9582b9" containerID="a9d10203ec95ca76df4d1970cc6e178861cd606458b4f96eda88a25685642d95" exitCode=0
Sep 10 20:58:32.010489 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:32.010002    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5rbwt" event={"ID":"432f3e3d-207e-4732-97f7-43182d9582b9","Type":"ContainerDied","Data":"a9d10203ec95ca76df4d1970cc6e178861cd606458b4f96eda88a25685642d95"}
Sep 10 20:58:32.863246 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:32.863211    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mgx5w"
Sep 10 20:58:32.863459 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:32.863345    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mgx5w" podUID="6e1f126a-f90d-4cf6-b863-d6130c01a622"
Sep 10 20:58:32.863459 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:32.863387    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lmwz4"
Sep 10 20:58:32.863586 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:32.863521    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lmwz4" podUID="781245a4-8475-4031-9137-dd43ecd1c260"
Sep 10 20:58:32.863586 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:32.863536    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gvw9h"
Sep 10 20:58:32.863685 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:32.863632    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gvw9h" podUID="3bfd0b36-183c-43c3-9fcf-2d6f7839223e"
Sep 10 20:58:34.862695 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:34.862662    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lmwz4"
Sep 10 20:58:34.862695 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:34.862678    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mgx5w"
Sep 10 20:58:34.863227 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:34.862791    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lmwz4" podUID="781245a4-8475-4031-9137-dd43ecd1c260"
Sep 10 20:58:34.863227 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:34.862863    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mgx5w" podUID="6e1f126a-f90d-4cf6-b863-d6130c01a622"
Sep 10 20:58:34.863227 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:34.862880    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gvw9h"
Sep 10 20:58:34.863227 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:34.862977    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gvw9h" podUID="3bfd0b36-183c-43c3-9fcf-2d6f7839223e"
Sep 10 20:58:36.592682 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:36.592645    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/781245a4-8475-4031-9137-dd43ecd1c260-metrics-certs\") pod \"network-metrics-daemon-lmwz4\" (UID: \"781245a4-8475-4031-9137-dd43ecd1c260\") " pod="openshift-multus/network-metrics-daemon-lmwz4"
Sep 10 20:58:36.592682 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:36.592684    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e1f126a-f90d-4cf6-b863-d6130c01a622-original-pull-secret\") pod \"global-pull-secret-syncer-mgx5w\" (UID: \"6e1f126a-f90d-4cf6-b863-d6130c01a622\") " pod="kube-system/global-pull-secret-syncer-mgx5w"
Sep 10 20:58:36.593118 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:36.592778    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:58:36.593118 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:36.592801    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:58:36.593118 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:36.592826    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e1f126a-f90d-4cf6-b863-d6130c01a622-original-pull-secret podName:6e1f126a-f90d-4cf6-b863-d6130c01a622 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:08.592814051 +0000 UTC m=+66.249722800 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e1f126a-f90d-4cf6-b863-d6130c01a622-original-pull-secret") pod "global-pull-secret-syncer-mgx5w" (UID: "6e1f126a-f90d-4cf6-b863-d6130c01a622") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:58:36.593118 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:36.592857    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/781245a4-8475-4031-9137-dd43ecd1c260-metrics-certs podName:781245a4-8475-4031-9137-dd43ecd1c260 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:08.592844063 +0000 UTC m=+66.249752813 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/781245a4-8475-4031-9137-dd43ecd1c260-metrics-certs") pod "network-metrics-daemon-lmwz4" (UID: "781245a4-8475-4031-9137-dd43ecd1c260") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:58:36.693526 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:36.693482    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8nv4\" (UniqueName: \"kubernetes.io/projected/3bfd0b36-183c-43c3-9fcf-2d6f7839223e-kube-api-access-g8nv4\") pod \"network-check-target-gvw9h\" (UID: \"3bfd0b36-183c-43c3-9fcf-2d6f7839223e\") " pod="openshift-network-diagnostics/network-check-target-gvw9h"
Sep 10 20:58:36.693713 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:36.693675    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:58:36.693713 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:36.693701    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:58:36.693713 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:36.693712    2581 projected.go:196] Error preparing data for projected volume kube-api-access-g8nv4 for pod openshift-network-diagnostics/network-check-target-gvw9h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:58:36.693871 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:36.693778    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3bfd0b36-183c-43c3-9fcf-2d6f7839223e-kube-api-access-g8nv4 podName:3bfd0b36-183c-43c3-9fcf-2d6f7839223e nodeName:}" failed. No retries permitted until 2026-09-10 20:59:08.693759537 +0000 UTC m=+66.350668293 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-g8nv4" (UniqueName: "kubernetes.io/projected/3bfd0b36-183c-43c3-9fcf-2d6f7839223e-kube-api-access-g8nv4") pod "network-check-target-gvw9h" (UID: "3bfd0b36-183c-43c3-9fcf-2d6f7839223e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:58:36.862798 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:36.862730    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mgx5w"
Sep 10 20:58:36.862920 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:36.862730    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lmwz4"
Sep 10 20:58:36.862920 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:36.862836    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mgx5w" podUID="6e1f126a-f90d-4cf6-b863-d6130c01a622"
Sep 10 20:58:36.862920 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:36.862847    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gvw9h"
Sep 10 20:58:36.862920 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:36.862904    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lmwz4" podUID="781245a4-8475-4031-9137-dd43ecd1c260"
Sep 10 20:58:36.863128 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:36.862948    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gvw9h" podUID="3bfd0b36-183c-43c3-9fcf-2d6f7839223e"
Sep 10 20:58:37.024314 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:37.024273    2581 generic.go:356] "Generic (PLEG): container finished" podID="432f3e3d-207e-4732-97f7-43182d9582b9" containerID="929c6a3bf346d99705b175c4545d1e32679d9c6e3b992694644b102ad1d39f64" exitCode=0
Sep 10 20:58:37.024471 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:37.024359    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5rbwt" event={"ID":"432f3e3d-207e-4732-97f7-43182d9582b9","Type":"ContainerDied","Data":"929c6a3bf346d99705b175c4545d1e32679d9c6e3b992694644b102ad1d39f64"}
Sep 10 20:58:37.065210 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:37.065183    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-165.ec2.internal" event="NodeReady"
Sep 10 20:58:37.065341 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:37.065324    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 20:58:37.143107 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:37.142934    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wfsbr"]
Sep 10 20:58:37.165189 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:37.165157    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4bsk5"]
Sep 10 20:58:37.165528 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:37.165258    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wfsbr"
Sep 10 20:58:37.169908 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:37.169674    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9hkk2\""
Sep 10 20:58:37.169908 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:37.169735    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 20:58:37.170063 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:37.169923    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 20:58:37.191407 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:37.191385    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wfsbr"]
Sep 10 20:58:37.191512 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:37.191431    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4bsk5"]
Sep 10 20:58:37.191576 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:37.191510    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4bsk5"
Sep 10 20:58:37.195781 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:37.195764    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 20:58:37.195983 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:37.195962    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 20:58:37.196719 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:37.196699    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-278wb\""
Sep 10 20:58:37.196805 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:37.196704    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 20:58:37.297026 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:37.296990    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/443d28b9-b165-4d98-b86a-ea4924d54962-cert\") pod \"ingress-canary-4bsk5\" (UID: \"443d28b9-b165-4d98-b86a-ea4924d54962\") " pod="openshift-ingress-canary/ingress-canary-4bsk5"
Sep 10 20:58:37.297188 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:37.297048    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/378700dd-8e73-4a3e-9c10-35b09ebbdd68-metrics-tls\") pod \"dns-default-wfsbr\" (UID: \"378700dd-8e73-4a3e-9c10-35b09ebbdd68\") " pod="openshift-dns/dns-default-wfsbr"
Sep 10 20:58:37.297188 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:37.297094    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/378700dd-8e73-4a3e-9c10-35b09ebbdd68-config-volume\") pod \"dns-default-wfsbr\" (UID: \"378700dd-8e73-4a3e-9c10-35b09ebbdd68\") " pod="openshift-dns/dns-default-wfsbr"
Sep 10 20:58:37.297188 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:37.297116    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-667lj\" (UniqueName: \"kubernetes.io/projected/443d28b9-b165-4d98-b86a-ea4924d54962-kube-api-access-667lj\") pod \"ingress-canary-4bsk5\" (UID: \"443d28b9-b165-4d98-b86a-ea4924d54962\") " pod="openshift-ingress-canary/ingress-canary-4bsk5"
Sep 10 20:58:37.297188 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:37.297157    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fjsm\" (UniqueName: \"kubernetes.io/projected/378700dd-8e73-4a3e-9c10-35b09ebbdd68-kube-api-access-7fjsm\") pod \"dns-default-wfsbr\" (UID: \"378700dd-8e73-4a3e-9c10-35b09ebbdd68\") " pod="openshift-dns/dns-default-wfsbr"
Sep 10 20:58:37.297368 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:37.297204    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/378700dd-8e73-4a3e-9c10-35b09ebbdd68-tmp-dir\") pod \"dns-default-wfsbr\" (UID: \"378700dd-8e73-4a3e-9c10-35b09ebbdd68\") " pod="openshift-dns/dns-default-wfsbr"
Sep 10 20:58:37.397755 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:37.397694    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/378700dd-8e73-4a3e-9c10-35b09ebbdd68-config-volume\") pod \"dns-default-wfsbr\" (UID: \"378700dd-8e73-4a3e-9c10-35b09ebbdd68\") " pod="openshift-dns/dns-default-wfsbr"
Sep 10 20:58:37.397755 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:37.397729    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-667lj\" (UniqueName: \"kubernetes.io/projected/443d28b9-b165-4d98-b86a-ea4924d54962-kube-api-access-667lj\") pod \"ingress-canary-4bsk5\" (UID: \"443d28b9-b165-4d98-b86a-ea4924d54962\") " pod="openshift-ingress-canary/ingress-canary-4bsk5"
Sep 10 20:58:37.397755 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:37.397753    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fjsm\" (UniqueName: \"kubernetes.io/projected/378700dd-8e73-4a3e-9c10-35b09ebbdd68-kube-api-access-7fjsm\") pod \"dns-default-wfsbr\" (UID: \"378700dd-8e73-4a3e-9c10-35b09ebbdd68\") " pod="openshift-dns/dns-default-wfsbr"
Sep 10 20:58:37.397952 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:37.397790    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/378700dd-8e73-4a3e-9c10-35b09ebbdd68-tmp-dir\") pod \"dns-default-wfsbr\" (UID: \"378700dd-8e73-4a3e-9c10-35b09ebbdd68\") " pod="openshift-dns/dns-default-wfsbr"
Sep 10 20:58:37.397952 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:37.397812    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/443d28b9-b165-4d98-b86a-ea4924d54962-cert\") pod \"ingress-canary-4bsk5\" (UID: \"443d28b9-b165-4d98-b86a-ea4924d54962\") " pod="openshift-ingress-canary/ingress-canary-4bsk5"
Sep 10 20:58:37.397952 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:37.397841    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/378700dd-8e73-4a3e-9c10-35b09ebbdd68-metrics-tls\") pod \"dns-default-wfsbr\" (UID: \"378700dd-8e73-4a3e-9c10-35b09ebbdd68\") " pod="openshift-dns/dns-default-wfsbr"
Sep 10 20:58:37.397952 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:37.397931    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:58:37.398110 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:37.397989    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/378700dd-8e73-4a3e-9c10-35b09ebbdd68-metrics-tls podName:378700dd-8e73-4a3e-9c10-35b09ebbdd68 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:37.897973831 +0000 UTC m=+35.554882601 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/378700dd-8e73-4a3e-9c10-35b09ebbdd68-metrics-tls") pod "dns-default-wfsbr" (UID: "378700dd-8e73-4a3e-9c10-35b09ebbdd68") : secret "dns-default-metrics-tls" not found
Sep 10 20:58:37.398350 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:37.398334    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/378700dd-8e73-4a3e-9c10-35b09ebbdd68-config-volume\") pod \"dns-default-wfsbr\" (UID: \"378700dd-8e73-4a3e-9c10-35b09ebbdd68\") " pod="openshift-dns/dns-default-wfsbr"
Sep 10 20:58:37.398389 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:37.398353    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/378700dd-8e73-4a3e-9c10-35b09ebbdd68-tmp-dir\") pod \"dns-default-wfsbr\" (UID: \"378700dd-8e73-4a3e-9c10-35b09ebbdd68\") " pod="openshift-dns/dns-default-wfsbr"
Sep 10 20:58:37.401958 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:37.401941    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/443d28b9-b165-4d98-b86a-ea4924d54962-cert\") pod \"ingress-canary-4bsk5\" (UID: \"443d28b9-b165-4d98-b86a-ea4924d54962\") " pod="openshift-ingress-canary/ingress-canary-4bsk5"
Sep 10 20:58:37.407066 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:37.407039    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fjsm\" (UniqueName: \"kubernetes.io/projected/378700dd-8e73-4a3e-9c10-35b09ebbdd68-kube-api-access-7fjsm\") pod \"dns-default-wfsbr\" (UID: \"378700dd-8e73-4a3e-9c10-35b09ebbdd68\") " pod="openshift-dns/dns-default-wfsbr"
Sep 10 20:58:37.407396 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:37.407379    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-667lj\" (UniqueName: \"kubernetes.io/projected/443d28b9-b165-4d98-b86a-ea4924d54962-kube-api-access-667lj\") pod \"ingress-canary-4bsk5\" (UID: \"443d28b9-b165-4d98-b86a-ea4924d54962\") " pod="openshift-ingress-canary/ingress-canary-4bsk5"
Sep 10 20:58:37.499434 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:37.499411    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4bsk5"
Sep 10 20:58:37.814480 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:37.814447    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4bsk5"]
Sep 10 20:58:37.818367 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:37.818341    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod443d28b9_b165_4d98_b86a_ea4924d54962.slice/crio-6ee29cba0b2175d84c2065114935dcc2afae9cc3192e365291a87d30ad6c204e WatchSource:0}: Error finding container 6ee29cba0b2175d84c2065114935dcc2afae9cc3192e365291a87d30ad6c204e: Status 404 returned error can't find the container with id 6ee29cba0b2175d84c2065114935dcc2afae9cc3192e365291a87d30ad6c204e
Sep 10 20:58:37.902321 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:37.902267    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/378700dd-8e73-4a3e-9c10-35b09ebbdd68-metrics-tls\") pod \"dns-default-wfsbr\" (UID: \"378700dd-8e73-4a3e-9c10-35b09ebbdd68\") " pod="openshift-dns/dns-default-wfsbr"
Sep 10 20:58:37.902483 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:37.902432    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:58:37.902524 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:37.902505    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/378700dd-8e73-4a3e-9c10-35b09ebbdd68-metrics-tls podName:378700dd-8e73-4a3e-9c10-35b09ebbdd68 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:38.902483285 +0000 UTC m=+36.559392037 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/378700dd-8e73-4a3e-9c10-35b09ebbdd68-metrics-tls") pod "dns-default-wfsbr" (UID: "378700dd-8e73-4a3e-9c10-35b09ebbdd68") : secret "dns-default-metrics-tls" not found
Sep 10 20:58:38.030115 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:38.030080    2581 generic.go:356] "Generic (PLEG): container finished" podID="432f3e3d-207e-4732-97f7-43182d9582b9" containerID="d5ef7b2eac75cda46735010ff8631575af5aef042af28f4a43570bb9b7b7377c" exitCode=0
Sep 10 20:58:38.030270 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:38.030156    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5rbwt" event={"ID":"432f3e3d-207e-4732-97f7-43182d9582b9","Type":"ContainerDied","Data":"d5ef7b2eac75cda46735010ff8631575af5aef042af28f4a43570bb9b7b7377c"}
Sep 10 20:58:38.031201 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:38.031175    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4bsk5" event={"ID":"443d28b9-b165-4d98-b86a-ea4924d54962","Type":"ContainerStarted","Data":"6ee29cba0b2175d84c2065114935dcc2afae9cc3192e365291a87d30ad6c204e"}
Sep 10 20:58:38.862730 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:38.862687    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lmwz4"
Sep 10 20:58:38.863251 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:38.862734    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mgx5w"
Sep 10 20:58:38.863251 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:38.862696    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gvw9h"
Sep 10 20:58:38.865750 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:38.865729    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 20:58:38.865933 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:38.865912    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tjcr5\""
Sep 10 20:58:38.866526 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:38.866364    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 20:58:38.866683 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:38.866665    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 20:58:38.866768 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:38.866665    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ll8tt\""
Sep 10 20:58:38.866768 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:38.866712    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 20:58:38.910572 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:38.910555    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/378700dd-8e73-4a3e-9c10-35b09ebbdd68-metrics-tls\") pod \"dns-default-wfsbr\" (UID: \"378700dd-8e73-4a3e-9c10-35b09ebbdd68\") " pod="openshift-dns/dns-default-wfsbr"
Sep 10 20:58:38.910709 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:38.910683    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:58:38.910794 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:38.910748    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/378700dd-8e73-4a3e-9c10-35b09ebbdd68-metrics-tls podName:378700dd-8e73-4a3e-9c10-35b09ebbdd68 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:40.910729543 +0000 UTC m=+38.567638293 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/378700dd-8e73-4a3e-9c10-35b09ebbdd68-metrics-tls") pod "dns-default-wfsbr" (UID: "378700dd-8e73-4a3e-9c10-35b09ebbdd68") : secret "dns-default-metrics-tls" not found
Sep 10 20:58:39.037885 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:39.037710    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5rbwt" event={"ID":"432f3e3d-207e-4732-97f7-43182d9582b9","Type":"ContainerStarted","Data":"87629a2b4570428c20d216c43e7261ef16e2cb576c2d732a32c8e9f8fa4f9d93"}
Sep 10 20:58:39.063865 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:39.063802    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5rbwt" podStartSLOduration=7.375039332 podStartE2EDuration="36.063782719s" totalImagesPullingTime="28.688743387s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:58:03 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:39.06271492 +0000 UTC m=+36.719623693" watchObservedRunningTime="2026-09-10 20:58:39.063782719 +0000 UTC m=+36.720691492"
Sep 10 20:58:40.493913 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.493885    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-9d2gj"]
Sep 10 20:58:40.510257 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.510237    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-9d2gj"
Sep 10 20:58:40.514029 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.514006    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-hznk9\""
Sep 10 20:58:40.514593 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.514572    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 10 20:58:40.514690 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.514600    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 20:58:40.514964 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.514948    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 20:58:40.515040 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.515030    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 10 20:58:40.519397 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.519379    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-9d2gj"]
Sep 10 20:58:40.523493 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.523473    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 10 20:58:40.583527 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.583499    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-64c8fc8856-tfwzb"]
Sep 10 20:58:40.592440 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.592423    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-64c8fc8856-tfwzb"
Sep 10 20:58:40.595932 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.595910    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 20:58:40.596417 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.596394    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 10 20:58:40.596521 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.596509    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 20:58:40.596633 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.596568    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 10 20:58:40.597776 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.597289    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 10 20:58:40.597776 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.597378    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 10 20:58:40.597776 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.597442    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-rksqh\""
Sep 10 20:58:40.598081 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.598059    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-bvh97"]
Sep 10 20:58:40.604211 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.604193    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-z6858"]
Sep 10 20:58:40.604323 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.604310    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bvh97"
Sep 10 20:58:40.607099 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.607081    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 20:58:40.608463 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.608445    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-64c8fc8856-tfwzb"]
Sep 10 20:58:40.608571 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.608514    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-z6858"
Sep 10 20:58:40.609095 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.609076    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 20:58:40.609186 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.609114    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 10 20:58:40.609186 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.609070    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 10 20:58:40.609317 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.609286    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-57j94\""
Sep 10 20:58:40.610603 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.610582    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 10 20:58:40.610698 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.610590    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-7lp5s\""
Sep 10 20:58:40.611022 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.611002    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 10 20:58:40.611408 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.611390    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 10 20:58:40.612586 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.612568    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 10 20:58:40.616482 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.616462    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 10 20:58:40.621046 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.621029    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-bvh97"]
Sep 10 20:58:40.623721 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.623682    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-z6858"]
Sep 10 20:58:40.625896 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.625878    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08f9e62b-ae0b-426a-a306-39f8c4fdb7ff-trusted-ca-bundle\") pod \"insights-operator-5844b856db-9d2gj\" (UID: \"08f9e62b-ae0b-426a-a306-39f8c4fdb7ff\") " pod="openshift-insights/insights-operator-5844b856db-9d2gj"
Sep 10 20:58:40.625989 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.625905    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08f9e62b-ae0b-426a-a306-39f8c4fdb7ff-tmp\") pod \"insights-operator-5844b856db-9d2gj\" (UID: \"08f9e62b-ae0b-426a-a306-39f8c4fdb7ff\") " pod="openshift-insights/insights-operator-5844b856db-9d2gj"
Sep 10 20:58:40.625989 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.625923    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08f9e62b-ae0b-426a-a306-39f8c4fdb7ff-service-ca-bundle\") pod \"insights-operator-5844b856db-9d2gj\" (UID: \"08f9e62b-ae0b-426a-a306-39f8c4fdb7ff\") " pod="openshift-insights/insights-operator-5844b856db-9d2gj"
Sep 10 20:58:40.625989 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.625945    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/08f9e62b-ae0b-426a-a306-39f8c4fdb7ff-snapshots\") pod \"insights-operator-5844b856db-9d2gj\" (UID: \"08f9e62b-ae0b-426a-a306-39f8c4fdb7ff\") " pod="openshift-insights/insights-operator-5844b856db-9d2gj"
Sep 10 20:58:40.625989 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.625978    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/08f9e62b-ae0b-426a-a306-39f8c4fdb7ff-serving-cert\") pod \"insights-operator-5844b856db-9d2gj\" (UID: \"08f9e62b-ae0b-426a-a306-39f8c4fdb7ff\") " pod="openshift-insights/insights-operator-5844b856db-9d2gj"
Sep 10 20:58:40.626159 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.626018    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8pnp\" (UniqueName: \"kubernetes.io/projected/08f9e62b-ae0b-426a-a306-39f8c4fdb7ff-kube-api-access-s8pnp\") pod \"insights-operator-5844b856db-9d2gj\" (UID: \"08f9e62b-ae0b-426a-a306-39f8c4fdb7ff\") " pod="openshift-insights/insights-operator-5844b856db-9d2gj"
Sep 10 20:58:40.726726 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.726697    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/08f9e62b-ae0b-426a-a306-39f8c4fdb7ff-snapshots\") pod \"insights-operator-5844b856db-9d2gj\" (UID: \"08f9e62b-ae0b-426a-a306-39f8c4fdb7ff\") " pod="openshift-insights/insights-operator-5844b856db-9d2gj"
Sep 10 20:58:40.726726 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.726729    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64a30645-a3fb-4670-89af-a044b3dd00a0-service-ca-bundle\") pod \"router-default-64c8fc8856-tfwzb\" (UID: \"64a30645-a3fb-4670-89af-a044b3dd00a0\") " pod="openshift-ingress/router-default-64c8fc8856-tfwzb"
Sep 10 20:58:40.726896 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.726857    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/da65604c-6689-4ef5-9cdf-0507cbeba512-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-bvh97\" (UID: \"da65604c-6689-4ef5-9cdf-0507cbeba512\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bvh97"
Sep 10 20:58:40.726941 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.726895    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzknd\" (UniqueName: \"kubernetes.io/projected/c1e06766-b48c-429e-92f1-196bb083360b-kube-api-access-fzknd\") pod \"console-operator-696884fb9c-z6858\" (UID: \"c1e06766-b48c-429e-92f1-196bb083360b\") " pod="openshift-console-operator/console-operator-696884fb9c-z6858"
Sep 10 20:58:40.726941 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.726916    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8686b\" (UniqueName: \"kubernetes.io/projected/64a30645-a3fb-4670-89af-a044b3dd00a0-kube-api-access-8686b\") pod \"router-default-64c8fc8856-tfwzb\" (UID: \"64a30645-a3fb-4670-89af-a044b3dd00a0\") " pod="openshift-ingress/router-default-64c8fc8856-tfwzb"
Sep 10 20:58:40.727015 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.726946    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/08f9e62b-ae0b-426a-a306-39f8c4fdb7ff-serving-cert\") pod \"insights-operator-5844b856db-9d2gj\" (UID: \"08f9e62b-ae0b-426a-a306-39f8c4fdb7ff\") " pod="openshift-insights/insights-operator-5844b856db-9d2gj"
Sep 10 20:58:40.727015 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.726966    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8pnp\" (UniqueName: \"kubernetes.io/projected/08f9e62b-ae0b-426a-a306-39f8c4fdb7ff-kube-api-access-s8pnp\") pod \"insights-operator-5844b856db-9d2gj\" (UID: \"08f9e62b-ae0b-426a-a306-39f8c4fdb7ff\") " pod="openshift-insights/insights-operator-5844b856db-9d2gj"
Sep 10 20:58:40.727015 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.726991    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c1e06766-b48c-429e-92f1-196bb083360b-config\") pod \"console-operator-696884fb9c-z6858\" (UID: \"c1e06766-b48c-429e-92f1-196bb083360b\") " pod="openshift-console-operator/console-operator-696884fb9c-z6858"
Sep 10 20:58:40.727113 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.727092    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c1e06766-b48c-429e-92f1-196bb083360b-trusted-ca\") pod \"console-operator-696884fb9c-z6858\" (UID: \"c1e06766-b48c-429e-92f1-196bb083360b\") " pod="openshift-console-operator/console-operator-696884fb9c-z6858"
Sep 10 20:58:40.727285 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.727132    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mknkx\" (UniqueName: \"kubernetes.io/projected/da65604c-6689-4ef5-9cdf-0507cbeba512-kube-api-access-mknkx\") pod \"cluster-monitoring-operator-6cd954b897-bvh97\" (UID: \"da65604c-6689-4ef5-9cdf-0507cbeba512\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bvh97"
Sep 10 20:58:40.727285 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.727155    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c1e06766-b48c-429e-92f1-196bb083360b-serving-cert\") pod \"console-operator-696884fb9c-z6858\" (UID: \"c1e06766-b48c-429e-92f1-196bb083360b\") " pod="openshift-console-operator/console-operator-696884fb9c-z6858"
Sep 10 20:58:40.727285 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.727172    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/64a30645-a3fb-4670-89af-a044b3dd00a0-default-certificate\") pod \"router-default-64c8fc8856-tfwzb\" (UID: \"64a30645-a3fb-4670-89af-a044b3dd00a0\") " pod="openshift-ingress/router-default-64c8fc8856-tfwzb"
Sep 10 20:58:40.727285 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.727221    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08f9e62b-ae0b-426a-a306-39f8c4fdb7ff-trusted-ca-bundle\") pod \"insights-operator-5844b856db-9d2gj\" (UID: \"08f9e62b-ae0b-426a-a306-39f8c4fdb7ff\") " pod="openshift-insights/insights-operator-5844b856db-9d2gj"
Sep 10 20:58:40.727285 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.727260    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08f9e62b-ae0b-426a-a306-39f8c4fdb7ff-tmp\") pod \"insights-operator-5844b856db-9d2gj\" (UID: \"08f9e62b-ae0b-426a-a306-39f8c4fdb7ff\") " pod="openshift-insights/insights-operator-5844b856db-9d2gj"
Sep 10 20:58:40.727285 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.727278    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08f9e62b-ae0b-426a-a306-39f8c4fdb7ff-service-ca-bundle\") pod \"insights-operator-5844b856db-9d2gj\" (UID: \"08f9e62b-ae0b-426a-a306-39f8c4fdb7ff\") " pod="openshift-insights/insights-operator-5844b856db-9d2gj"
Sep 10 20:58:40.727531 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.727292    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/da65604c-6689-4ef5-9cdf-0507cbeba512-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-bvh97\" (UID: \"da65604c-6689-4ef5-9cdf-0507cbeba512\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bvh97"
Sep 10 20:58:40.727531 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.727336    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64a30645-a3fb-4670-89af-a044b3dd00a0-metrics-certs\") pod \"router-default-64c8fc8856-tfwzb\" (UID: \"64a30645-a3fb-4670-89af-a044b3dd00a0\") " pod="openshift-ingress/router-default-64c8fc8856-tfwzb"
Sep 10 20:58:40.727531 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.727360    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/08f9e62b-ae0b-426a-a306-39f8c4fdb7ff-snapshots\") pod \"insights-operator-5844b856db-9d2gj\" (UID: \"08f9e62b-ae0b-426a-a306-39f8c4fdb7ff\") " pod="openshift-insights/insights-operator-5844b856db-9d2gj"
Sep 10 20:58:40.727531 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.727367    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/64a30645-a3fb-4670-89af-a044b3dd00a0-stats-auth\") pod \"router-default-64c8fc8856-tfwzb\" (UID: \"64a30645-a3fb-4670-89af-a044b3dd00a0\") " pod="openshift-ingress/router-default-64c8fc8856-tfwzb"
Sep 10 20:58:40.727691 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.727584    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08f9e62b-ae0b-426a-a306-39f8c4fdb7ff-tmp\") pod \"insights-operator-5844b856db-9d2gj\" (UID: \"08f9e62b-ae0b-426a-a306-39f8c4fdb7ff\") " pod="openshift-insights/insights-operator-5844b856db-9d2gj"
Sep 10 20:58:40.727819 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.727802    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08f9e62b-ae0b-426a-a306-39f8c4fdb7ff-service-ca-bundle\") pod \"insights-operator-5844b856db-9d2gj\" (UID: \"08f9e62b-ae0b-426a-a306-39f8c4fdb7ff\") " pod="openshift-insights/insights-operator-5844b856db-9d2gj"
Sep 10 20:58:40.727985 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.727968    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08f9e62b-ae0b-426a-a306-39f8c4fdb7ff-trusted-ca-bundle\") pod \"insights-operator-5844b856db-9d2gj\" (UID: \"08f9e62b-ae0b-426a-a306-39f8c4fdb7ff\") " pod="openshift-insights/insights-operator-5844b856db-9d2gj"
Sep 10 20:58:40.730259 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.730242    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/08f9e62b-ae0b-426a-a306-39f8c4fdb7ff-serving-cert\") pod \"insights-operator-5844b856db-9d2gj\" (UID: \"08f9e62b-ae0b-426a-a306-39f8c4fdb7ff\") " pod="openshift-insights/insights-operator-5844b856db-9d2gj"
Sep 10 20:58:40.735842 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.735820    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8pnp\" (UniqueName: \"kubernetes.io/projected/08f9e62b-ae0b-426a-a306-39f8c4fdb7ff-kube-api-access-s8pnp\") pod \"insights-operator-5844b856db-9d2gj\" (UID: \"08f9e62b-ae0b-426a-a306-39f8c4fdb7ff\") " pod="openshift-insights/insights-operator-5844b856db-9d2gj"
Sep 10 20:58:40.820021 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.819990    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-9d2gj"
Sep 10 20:58:40.828275 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.828252    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c1e06766-b48c-429e-92f1-196bb083360b-config\") pod \"console-operator-696884fb9c-z6858\" (UID: \"c1e06766-b48c-429e-92f1-196bb083360b\") " pod="openshift-console-operator/console-operator-696884fb9c-z6858"
Sep 10 20:58:40.828355 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.828282    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c1e06766-b48c-429e-92f1-196bb083360b-trusted-ca\") pod \"console-operator-696884fb9c-z6858\" (UID: \"c1e06766-b48c-429e-92f1-196bb083360b\") " pod="openshift-console-operator/console-operator-696884fb9c-z6858"
Sep 10 20:58:40.828355 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.828314    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mknkx\" (UniqueName: \"kubernetes.io/projected/da65604c-6689-4ef5-9cdf-0507cbeba512-kube-api-access-mknkx\") pod \"cluster-monitoring-operator-6cd954b897-bvh97\" (UID: \"da65604c-6689-4ef5-9cdf-0507cbeba512\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bvh97"
Sep 10 20:58:40.828476 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.828456    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c1e06766-b48c-429e-92f1-196bb083360b-serving-cert\") pod \"console-operator-696884fb9c-z6858\" (UID: \"c1e06766-b48c-429e-92f1-196bb083360b\") " pod="openshift-console-operator/console-operator-696884fb9c-z6858"
Sep 10 20:58:40.828529 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.828496    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/64a30645-a3fb-4670-89af-a044b3dd00a0-default-certificate\") pod \"router-default-64c8fc8856-tfwzb\" (UID: \"64a30645-a3fb-4670-89af-a044b3dd00a0\") " pod="openshift-ingress/router-default-64c8fc8856-tfwzb"
Sep 10 20:58:40.828596 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.828580    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/da65604c-6689-4ef5-9cdf-0507cbeba512-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-bvh97\" (UID: \"da65604c-6689-4ef5-9cdf-0507cbeba512\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bvh97"
Sep 10 20:58:40.828652 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.828619    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64a30645-a3fb-4670-89af-a044b3dd00a0-metrics-certs\") pod \"router-default-64c8fc8856-tfwzb\" (UID: \"64a30645-a3fb-4670-89af-a044b3dd00a0\") " pod="openshift-ingress/router-default-64c8fc8856-tfwzb"
Sep 10 20:58:40.828710 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.828658    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/64a30645-a3fb-4670-89af-a044b3dd00a0-stats-auth\") pod \"router-default-64c8fc8856-tfwzb\" (UID: \"64a30645-a3fb-4670-89af-a044b3dd00a0\") " pod="openshift-ingress/router-default-64c8fc8856-tfwzb"
Sep 10 20:58:40.828710 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.828694    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64a30645-a3fb-4670-89af-a044b3dd00a0-service-ca-bundle\") pod \"router-default-64c8fc8856-tfwzb\" (UID: \"64a30645-a3fb-4670-89af-a044b3dd00a0\") " pod="openshift-ingress/router-default-64c8fc8856-tfwzb"
Sep 10 20:58:40.828803 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:40.828747    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 20:58:40.828851 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:40.828839    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64a30645-a3fb-4670-89af-a044b3dd00a0-metrics-certs podName:64a30645-a3fb-4670-89af-a044b3dd00a0 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:41.328791254 +0000 UTC m=+38.985700014 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64a30645-a3fb-4670-89af-a044b3dd00a0-metrics-certs") pod "router-default-64c8fc8856-tfwzb" (UID: "64a30645-a3fb-4670-89af-a044b3dd00a0") : secret "router-metrics-certs-default" not found
Sep 10 20:58:40.828907 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:40.828862    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/64a30645-a3fb-4670-89af-a044b3dd00a0-service-ca-bundle podName:64a30645-a3fb-4670-89af-a044b3dd00a0 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:41.328850688 +0000 UTC m=+38.985759443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/64a30645-a3fb-4670-89af-a044b3dd00a0-service-ca-bundle") pod "router-default-64c8fc8856-tfwzb" (UID: "64a30645-a3fb-4670-89af-a044b3dd00a0") : configmap references non-existent config key: service-ca.crt
Sep 10 20:58:40.828963 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.828915    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/da65604c-6689-4ef5-9cdf-0507cbeba512-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-bvh97\" (UID: \"da65604c-6689-4ef5-9cdf-0507cbeba512\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bvh97"
Sep 10 20:58:40.828963 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.828947    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzknd\" (UniqueName: \"kubernetes.io/projected/c1e06766-b48c-429e-92f1-196bb083360b-kube-api-access-fzknd\") pod \"console-operator-696884fb9c-z6858\" (UID: \"c1e06766-b48c-429e-92f1-196bb083360b\") " pod="openshift-console-operator/console-operator-696884fb9c-z6858"
Sep 10 20:58:40.829052 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.828971    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8686b\" (UniqueName: \"kubernetes.io/projected/64a30645-a3fb-4670-89af-a044b3dd00a0-kube-api-access-8686b\") pod \"router-default-64c8fc8856-tfwzb\" (UID: \"64a30645-a3fb-4670-89af-a044b3dd00a0\") " pod="openshift-ingress/router-default-64c8fc8856-tfwzb"
Sep 10 20:58:40.829113 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:40.829073    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 20:58:40.829162 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:40.829126    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da65604c-6689-4ef5-9cdf-0507cbeba512-cluster-monitoring-operator-tls podName:da65604c-6689-4ef5-9cdf-0507cbeba512 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:41.3291096 +0000 UTC m=+38.986018370 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/da65604c-6689-4ef5-9cdf-0507cbeba512-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-bvh97" (UID: "da65604c-6689-4ef5-9cdf-0507cbeba512") : secret "cluster-monitoring-operator-tls" not found
Sep 10 20:58:40.829162 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.829070    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c1e06766-b48c-429e-92f1-196bb083360b-config\") pod \"console-operator-696884fb9c-z6858\" (UID: \"c1e06766-b48c-429e-92f1-196bb083360b\") " pod="openshift-console-operator/console-operator-696884fb9c-z6858"
Sep 10 20:58:40.829634 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.829610    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/da65604c-6689-4ef5-9cdf-0507cbeba512-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-bvh97\" (UID: \"da65604c-6689-4ef5-9cdf-0507cbeba512\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bvh97"
Sep 10 20:58:40.829724 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.829701    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c1e06766-b48c-429e-92f1-196bb083360b-trusted-ca\") pod \"console-operator-696884fb9c-z6858\" (UID: \"c1e06766-b48c-429e-92f1-196bb083360b\") " pod="openshift-console-operator/console-operator-696884fb9c-z6858"
Sep 10 20:58:40.830912 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.830896    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c1e06766-b48c-429e-92f1-196bb083360b-serving-cert\") pod \"console-operator-696884fb9c-z6858\" (UID: \"c1e06766-b48c-429e-92f1-196bb083360b\") " pod="openshift-console-operator/console-operator-696884fb9c-z6858"
Sep 10 20:58:40.831209 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.831192    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/64a30645-a3fb-4670-89af-a044b3dd00a0-stats-auth\") pod \"router-default-64c8fc8856-tfwzb\" (UID: \"64a30645-a3fb-4670-89af-a044b3dd00a0\") " pod="openshift-ingress/router-default-64c8fc8856-tfwzb"
Sep 10 20:58:40.831496 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.831480    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/64a30645-a3fb-4670-89af-a044b3dd00a0-default-certificate\") pod \"router-default-64c8fc8856-tfwzb\" (UID: \"64a30645-a3fb-4670-89af-a044b3dd00a0\") " pod="openshift-ingress/router-default-64c8fc8856-tfwzb"
Sep 10 20:58:40.844177 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.844129    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mknkx\" (UniqueName: \"kubernetes.io/projected/da65604c-6689-4ef5-9cdf-0507cbeba512-kube-api-access-mknkx\") pod \"cluster-monitoring-operator-6cd954b897-bvh97\" (UID: \"da65604c-6689-4ef5-9cdf-0507cbeba512\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bvh97"
Sep 10 20:58:40.861181 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.861160    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzknd\" (UniqueName: \"kubernetes.io/projected/c1e06766-b48c-429e-92f1-196bb083360b-kube-api-access-fzknd\") pod \"console-operator-696884fb9c-z6858\" (UID: \"c1e06766-b48c-429e-92f1-196bb083360b\") " pod="openshift-console-operator/console-operator-696884fb9c-z6858"
Sep 10 20:58:40.864157 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.864136    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8686b\" (UniqueName: \"kubernetes.io/projected/64a30645-a3fb-4670-89af-a044b3dd00a0-kube-api-access-8686b\") pod \"router-default-64c8fc8856-tfwzb\" (UID: \"64a30645-a3fb-4670-89af-a044b3dd00a0\") " pod="openshift-ingress/router-default-64c8fc8856-tfwzb"
Sep 10 20:58:40.919835 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.919804    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-z6858"
Sep 10 20:58:40.930101 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:40.930077    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/378700dd-8e73-4a3e-9c10-35b09ebbdd68-metrics-tls\") pod \"dns-default-wfsbr\" (UID: \"378700dd-8e73-4a3e-9c10-35b09ebbdd68\") " pod="openshift-dns/dns-default-wfsbr"
Sep 10 20:58:40.930204 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:40.930195    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:58:40.930268 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:40.930248    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/378700dd-8e73-4a3e-9c10-35b09ebbdd68-metrics-tls podName:378700dd-8e73-4a3e-9c10-35b09ebbdd68 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:44.930235145 +0000 UTC m=+42.587143899 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/378700dd-8e73-4a3e-9c10-35b09ebbdd68-metrics-tls") pod "dns-default-wfsbr" (UID: "378700dd-8e73-4a3e-9c10-35b09ebbdd68") : secret "dns-default-metrics-tls" not found
Sep 10 20:58:41.043602 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:41.043569    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4bsk5" event={"ID":"443d28b9-b165-4d98-b86a-ea4924d54962","Type":"ContainerStarted","Data":"6b051cb09a9bfeab084ad2e8863b6e14f7d9fa882a98edea01f5e79ef0a97f82"}
Sep 10 20:58:41.064420 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:41.064368    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4bsk5" podStartSLOduration=1.596878582 podStartE2EDuration="4.064352603s" totalImagesPullingTime="2.467474021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:41.064190995 +0000 UTC m=+38.721099766" watchObservedRunningTime="2026-09-10 20:58:41.064352603 +0000 UTC m=+38.721261366"
Sep 10 20:58:41.116862 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:41.115841    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-9d2gj"]
Sep 10 20:58:41.123079 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:41.123052    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08f9e62b_ae0b_426a_a306_39f8c4fdb7ff.slice/crio-9681f48ddccab90e7f6fdb9363e566573864616d29d6e20a80e947d9b6a015a5 WatchSource:0}: Error finding container 9681f48ddccab90e7f6fdb9363e566573864616d29d6e20a80e947d9b6a015a5: Status 404 returned error can't find the container with id 9681f48ddccab90e7f6fdb9363e566573864616d29d6e20a80e947d9b6a015a5
Sep 10 20:58:41.213556 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:41.213197    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-z6858"]
Sep 10 20:58:41.215996 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:41.215971    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1e06766_b48c_429e_92f1_196bb083360b.slice/crio-a004e1e9edd76e0884be8488e6752ec80ca6451be35ddb7d2afd9757ca2931ae WatchSource:0}: Error finding container a004e1e9edd76e0884be8488e6752ec80ca6451be35ddb7d2afd9757ca2931ae: Status 404 returned error can't find the container with id a004e1e9edd76e0884be8488e6752ec80ca6451be35ddb7d2afd9757ca2931ae
Sep 10 20:58:41.333713 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:41.333686    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64a30645-a3fb-4670-89af-a044b3dd00a0-metrics-certs\") pod \"router-default-64c8fc8856-tfwzb\" (UID: \"64a30645-a3fb-4670-89af-a044b3dd00a0\") " pod="openshift-ingress/router-default-64c8fc8856-tfwzb"
Sep 10 20:58:41.333860 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:41.333724    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64a30645-a3fb-4670-89af-a044b3dd00a0-service-ca-bundle\") pod \"router-default-64c8fc8856-tfwzb\" (UID: \"64a30645-a3fb-4670-89af-a044b3dd00a0\") " pod="openshift-ingress/router-default-64c8fc8856-tfwzb"
Sep 10 20:58:41.333860 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:41.333762    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/da65604c-6689-4ef5-9cdf-0507cbeba512-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-bvh97\" (UID: \"da65604c-6689-4ef5-9cdf-0507cbeba512\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bvh97"
Sep 10 20:58:41.333860 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:41.333829    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 20:58:41.333997 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:41.333879    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 20:58:41.333997 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:41.333896    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64a30645-a3fb-4670-89af-a044b3dd00a0-metrics-certs podName:64a30645-a3fb-4670-89af-a044b3dd00a0 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:42.3338809 +0000 UTC m=+39.990789655 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64a30645-a3fb-4670-89af-a044b3dd00a0-metrics-certs") pod "router-default-64c8fc8856-tfwzb" (UID: "64a30645-a3fb-4670-89af-a044b3dd00a0") : secret "router-metrics-certs-default" not found
Sep 10 20:58:41.333997 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:41.333932    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da65604c-6689-4ef5-9cdf-0507cbeba512-cluster-monitoring-operator-tls podName:da65604c-6689-4ef5-9cdf-0507cbeba512 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:42.333921094 +0000 UTC m=+39.990829844 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/da65604c-6689-4ef5-9cdf-0507cbeba512-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-bvh97" (UID: "da65604c-6689-4ef5-9cdf-0507cbeba512") : secret "cluster-monitoring-operator-tls" not found
Sep 10 20:58:41.333997 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:41.333949    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/64a30645-a3fb-4670-89af-a044b3dd00a0-service-ca-bundle podName:64a30645-a3fb-4670-89af-a044b3dd00a0 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:42.333939648 +0000 UTC m=+39.990848398 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/64a30645-a3fb-4670-89af-a044b3dd00a0-service-ca-bundle") pod "router-default-64c8fc8856-tfwzb" (UID: "64a30645-a3fb-4670-89af-a044b3dd00a0") : configmap references non-existent config key: service-ca.crt
Sep 10 20:58:42.049146 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:42.049108    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-z6858" event={"ID":"c1e06766-b48c-429e-92f1-196bb083360b","Type":"ContainerStarted","Data":"a004e1e9edd76e0884be8488e6752ec80ca6451be35ddb7d2afd9757ca2931ae"}
Sep 10 20:58:42.051976 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:42.051945    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-9d2gj" event={"ID":"08f9e62b-ae0b-426a-a306-39f8c4fdb7ff","Type":"ContainerStarted","Data":"9681f48ddccab90e7f6fdb9363e566573864616d29d6e20a80e947d9b6a015a5"}
Sep 10 20:58:42.342468 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:42.342389    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64a30645-a3fb-4670-89af-a044b3dd00a0-metrics-certs\") pod \"router-default-64c8fc8856-tfwzb\" (UID: \"64a30645-a3fb-4670-89af-a044b3dd00a0\") " pod="openshift-ingress/router-default-64c8fc8856-tfwzb"
Sep 10 20:58:42.342468 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:42.342435    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64a30645-a3fb-4670-89af-a044b3dd00a0-service-ca-bundle\") pod \"router-default-64c8fc8856-tfwzb\" (UID: \"64a30645-a3fb-4670-89af-a044b3dd00a0\") " pod="openshift-ingress/router-default-64c8fc8856-tfwzb"
Sep 10 20:58:42.342468 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:42.342469    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/da65604c-6689-4ef5-9cdf-0507cbeba512-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-bvh97\" (UID: \"da65604c-6689-4ef5-9cdf-0507cbeba512\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bvh97"
Sep 10 20:58:42.342717 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:42.342550    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 20:58:42.342717 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:42.342582    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/64a30645-a3fb-4670-89af-a044b3dd00a0-service-ca-bundle podName:64a30645-a3fb-4670-89af-a044b3dd00a0 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:44.342564937 +0000 UTC m=+41.999473691 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/64a30645-a3fb-4670-89af-a044b3dd00a0-service-ca-bundle") pod "router-default-64c8fc8856-tfwzb" (UID: "64a30645-a3fb-4670-89af-a044b3dd00a0") : configmap references non-existent config key: service-ca.crt
Sep 10 20:58:42.342717 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:42.342604    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64a30645-a3fb-4670-89af-a044b3dd00a0-metrics-certs podName:64a30645-a3fb-4670-89af-a044b3dd00a0 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:44.342598213 +0000 UTC m=+41.999506963 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64a30645-a3fb-4670-89af-a044b3dd00a0-metrics-certs") pod "router-default-64c8fc8856-tfwzb" (UID: "64a30645-a3fb-4670-89af-a044b3dd00a0") : secret "router-metrics-certs-default" not found
Sep 10 20:58:42.342717 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:42.342556    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 20:58:42.342717 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:42.342694    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da65604c-6689-4ef5-9cdf-0507cbeba512-cluster-monitoring-operator-tls podName:da65604c-6689-4ef5-9cdf-0507cbeba512 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:44.342673414 +0000 UTC m=+41.999582165 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/da65604c-6689-4ef5-9cdf-0507cbeba512-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-bvh97" (UID: "da65604c-6689-4ef5-9cdf-0507cbeba512") : secret "cluster-monitoring-operator-tls" not found
Sep 10 20:58:44.358422 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:44.358393    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64a30645-a3fb-4670-89af-a044b3dd00a0-metrics-certs\") pod \"router-default-64c8fc8856-tfwzb\" (UID: \"64a30645-a3fb-4670-89af-a044b3dd00a0\") " pod="openshift-ingress/router-default-64c8fc8856-tfwzb"
Sep 10 20:58:44.358747 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:44.358435    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64a30645-a3fb-4670-89af-a044b3dd00a0-service-ca-bundle\") pod \"router-default-64c8fc8856-tfwzb\" (UID: \"64a30645-a3fb-4670-89af-a044b3dd00a0\") " pod="openshift-ingress/router-default-64c8fc8856-tfwzb"
Sep 10 20:58:44.358747 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:44.358485    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/da65604c-6689-4ef5-9cdf-0507cbeba512-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-bvh97\" (UID: \"da65604c-6689-4ef5-9cdf-0507cbeba512\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bvh97"
Sep 10 20:58:44.358747 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:44.358660    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/64a30645-a3fb-4670-89af-a044b3dd00a0-service-ca-bundle podName:64a30645-a3fb-4670-89af-a044b3dd00a0 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:48.358638824 +0000 UTC m=+46.015547591 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/64a30645-a3fb-4670-89af-a044b3dd00a0-service-ca-bundle") pod "router-default-64c8fc8856-tfwzb" (UID: "64a30645-a3fb-4670-89af-a044b3dd00a0") : configmap references non-existent config key: service-ca.crt
Sep 10 20:58:44.358977 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:44.358962    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 20:58:44.359019 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:44.358982    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 20:58:44.359019 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:44.359013    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64a30645-a3fb-4670-89af-a044b3dd00a0-metrics-certs podName:64a30645-a3fb-4670-89af-a044b3dd00a0 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:48.35900049 +0000 UTC m=+46.015909253 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64a30645-a3fb-4670-89af-a044b3dd00a0-metrics-certs") pod "router-default-64c8fc8856-tfwzb" (UID: "64a30645-a3fb-4670-89af-a044b3dd00a0") : secret "router-metrics-certs-default" not found
Sep 10 20:58:44.359111 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:44.359044    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da65604c-6689-4ef5-9cdf-0507cbeba512-cluster-monitoring-operator-tls podName:da65604c-6689-4ef5-9cdf-0507cbeba512 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:48.359027065 +0000 UTC m=+46.015935837 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/da65604c-6689-4ef5-9cdf-0507cbeba512-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-bvh97" (UID: "da65604c-6689-4ef5-9cdf-0507cbeba512") : secret "cluster-monitoring-operator-tls" not found
Sep 10 20:58:44.963836 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:44.963799    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/378700dd-8e73-4a3e-9c10-35b09ebbdd68-metrics-tls\") pod \"dns-default-wfsbr\" (UID: \"378700dd-8e73-4a3e-9c10-35b09ebbdd68\") " pod="openshift-dns/dns-default-wfsbr"
Sep 10 20:58:44.964018 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:44.963966    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:58:44.964086 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:44.964054    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/378700dd-8e73-4a3e-9c10-35b09ebbdd68-metrics-tls podName:378700dd-8e73-4a3e-9c10-35b09ebbdd68 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:52.964033929 +0000 UTC m=+50.620942685 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/378700dd-8e73-4a3e-9c10-35b09ebbdd68-metrics-tls") pod "dns-default-wfsbr" (UID: "378700dd-8e73-4a3e-9c10-35b09ebbdd68") : secret "dns-default-metrics-tls" not found
Sep 10 20:58:45.064826 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:45.064793    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-z6858_c1e06766-b48c-429e-92f1-196bb083360b/console-operator/0.log"
Sep 10 20:58:45.064988 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:45.064839    2581 generic.go:356] "Generic (PLEG): container finished" podID="c1e06766-b48c-429e-92f1-196bb083360b" containerID="351f8e84b2fedf7013c0b2891c975f85fceb1da9f5eb8527ada167d8ef34af00" exitCode=255
Sep 10 20:58:45.064988 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:45.064920    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-z6858" event={"ID":"c1e06766-b48c-429e-92f1-196bb083360b","Type":"ContainerDied","Data":"351f8e84b2fedf7013c0b2891c975f85fceb1da9f5eb8527ada167d8ef34af00"}
Sep 10 20:58:45.065201 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:45.065183    2581 scope.go:117] "RemoveContainer" containerID="351f8e84b2fedf7013c0b2891c975f85fceb1da9f5eb8527ada167d8ef34af00"
Sep 10 20:58:45.066573 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:45.066550    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-9d2gj" event={"ID":"08f9e62b-ae0b-426a-a306-39f8c4fdb7ff","Type":"ContainerStarted","Data":"2cfa69169e703ebf8dbd2f80089ffc453ec5f610c52bc1c205167059319841a9"}
Sep 10 20:58:45.117337 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:45.117261    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-9d2gj" podStartSLOduration=2.194773369 podStartE2EDuration="5.117240971s" totalImagesPullingTime="2.922467602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:45.115737511 +0000 UTC m=+42.772646283" watchObservedRunningTime="2026-09-10 20:58:45.117240971 +0000 UTC m=+42.774149742"
Sep 10 20:58:45.541500 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:45.541416    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-sxgtk"]
Sep 10 20:58:45.557855 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:45.557825    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sxgtk"
Sep 10 20:58:45.560596 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:45.560576    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 20:58:45.560744 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:45.560644    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k7jqh\""
Sep 10 20:58:45.560820 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:45.560746    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 20:58:45.565666 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:45.565648    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sxgtk"]
Sep 10 20:58:45.669112 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:45.669083    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/895ac1f0-9420-486f-8a54-6de076e5ae32-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxgtk\" (UID: \"895ac1f0-9420-486f-8a54-6de076e5ae32\") " pod="openshift-insights/insights-runtime-extractor-sxgtk"
Sep 10 20:58:45.669242 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:45.669127    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79s4v\" (UniqueName: \"kubernetes.io/projected/895ac1f0-9420-486f-8a54-6de076e5ae32-kube-api-access-79s4v\") pod \"insights-runtime-extractor-sxgtk\" (UID: \"895ac1f0-9420-486f-8a54-6de076e5ae32\") " pod="openshift-insights/insights-runtime-extractor-sxgtk"
Sep 10 20:58:45.669242 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:45.669212    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/895ac1f0-9420-486f-8a54-6de076e5ae32-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sxgtk\" (UID: \"895ac1f0-9420-486f-8a54-6de076e5ae32\") " pod="openshift-insights/insights-runtime-extractor-sxgtk"
Sep 10 20:58:45.669341 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:45.669262    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/895ac1f0-9420-486f-8a54-6de076e5ae32-crio-socket\") pod \"insights-runtime-extractor-sxgtk\" (UID: \"895ac1f0-9420-486f-8a54-6de076e5ae32\") " pod="openshift-insights/insights-runtime-extractor-sxgtk"
Sep 10 20:58:45.669385 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:45.669338    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/895ac1f0-9420-486f-8a54-6de076e5ae32-data-volume\") pod \"insights-runtime-extractor-sxgtk\" (UID: \"895ac1f0-9420-486f-8a54-6de076e5ae32\") " pod="openshift-insights/insights-runtime-extractor-sxgtk"
Sep 10 20:58:45.769798 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:45.769765    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/895ac1f0-9420-486f-8a54-6de076e5ae32-crio-socket\") pod \"insights-runtime-extractor-sxgtk\" (UID: \"895ac1f0-9420-486f-8a54-6de076e5ae32\") " pod="openshift-insights/insights-runtime-extractor-sxgtk"
Sep 10 20:58:45.769947 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:45.769808    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/895ac1f0-9420-486f-8a54-6de076e5ae32-data-volume\") pod \"insights-runtime-extractor-sxgtk\" (UID: \"895ac1f0-9420-486f-8a54-6de076e5ae32\") " pod="openshift-insights/insights-runtime-extractor-sxgtk"
Sep 10 20:58:45.769947 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:45.769889    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/895ac1f0-9420-486f-8a54-6de076e5ae32-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxgtk\" (UID: \"895ac1f0-9420-486f-8a54-6de076e5ae32\") " pod="openshift-insights/insights-runtime-extractor-sxgtk"
Sep 10 20:58:45.769947 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:45.769940    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79s4v\" (UniqueName: \"kubernetes.io/projected/895ac1f0-9420-486f-8a54-6de076e5ae32-kube-api-access-79s4v\") pod \"insights-runtime-extractor-sxgtk\" (UID: \"895ac1f0-9420-486f-8a54-6de076e5ae32\") " pod="openshift-insights/insights-runtime-extractor-sxgtk"
Sep 10 20:58:45.770098 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:45.769973    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/895ac1f0-9420-486f-8a54-6de076e5ae32-crio-socket\") pod \"insights-runtime-extractor-sxgtk\" (UID: \"895ac1f0-9420-486f-8a54-6de076e5ae32\") " pod="openshift-insights/insights-runtime-extractor-sxgtk"
Sep 10 20:58:45.770098 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:45.769988    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/895ac1f0-9420-486f-8a54-6de076e5ae32-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sxgtk\" (UID: \"895ac1f0-9420-486f-8a54-6de076e5ae32\") " pod="openshift-insights/insights-runtime-extractor-sxgtk"
Sep 10 20:58:45.770098 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:45.770059    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 20:58:45.770098 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:45.770072    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 20:58:45.770250 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:45.770111    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/895ac1f0-9420-486f-8a54-6de076e5ae32-data-volume\") pod \"insights-runtime-extractor-sxgtk\" (UID: \"895ac1f0-9420-486f-8a54-6de076e5ae32\") " pod="openshift-insights/insights-runtime-extractor-sxgtk"
Sep 10 20:58:45.770250 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:45.770123    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/895ac1f0-9420-486f-8a54-6de076e5ae32-kube-rbac-proxy-cm podName:895ac1f0-9420-486f-8a54-6de076e5ae32 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:46.270105976 +0000 UTC m=+43.927014749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/895ac1f0-9420-486f-8a54-6de076e5ae32-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sxgtk" (UID: "895ac1f0-9420-486f-8a54-6de076e5ae32") : configmap "kube-rbac-proxy" not found
Sep 10 20:58:45.770250 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:45.770141    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/895ac1f0-9420-486f-8a54-6de076e5ae32-insights-runtime-extractor-tls podName:895ac1f0-9420-486f-8a54-6de076e5ae32 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:46.270132706 +0000 UTC m=+43.927041472 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/895ac1f0-9420-486f-8a54-6de076e5ae32-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sxgtk" (UID: "895ac1f0-9420-486f-8a54-6de076e5ae32") : secret "insights-runtime-extractor-tls" not found
Sep 10 20:58:45.780480 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:45.780461    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79s4v\" (UniqueName: \"kubernetes.io/projected/895ac1f0-9420-486f-8a54-6de076e5ae32-kube-api-access-79s4v\") pod \"insights-runtime-extractor-sxgtk\" (UID: \"895ac1f0-9420-486f-8a54-6de076e5ae32\") " pod="openshift-insights/insights-runtime-extractor-sxgtk"
Sep 10 20:58:46.070940 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:46.070911    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-z6858_c1e06766-b48c-429e-92f1-196bb083360b/console-operator/1.log"
Sep 10 20:58:46.071346 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:46.071329    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-z6858_c1e06766-b48c-429e-92f1-196bb083360b/console-operator/0.log"
Sep 10 20:58:46.071458 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:46.071369    2581 generic.go:356] "Generic (PLEG): container finished" podID="c1e06766-b48c-429e-92f1-196bb083360b" containerID="bfc746521aa3e8f97083a5507352b086eb08d1212839732dffe53946dfd57b44" exitCode=255
Sep 10 20:58:46.071505 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:46.071451    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-z6858" event={"ID":"c1e06766-b48c-429e-92f1-196bb083360b","Type":"ContainerDied","Data":"bfc746521aa3e8f97083a5507352b086eb08d1212839732dffe53946dfd57b44"}
Sep 10 20:58:46.071505 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:46.071500    2581 scope.go:117] "RemoveContainer" containerID="351f8e84b2fedf7013c0b2891c975f85fceb1da9f5eb8527ada167d8ef34af00"
Sep 10 20:58:46.071727 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:46.071706    2581 scope.go:117] "RemoveContainer" containerID="bfc746521aa3e8f97083a5507352b086eb08d1212839732dffe53946dfd57b44"
Sep 10 20:58:46.071914 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:46.071887    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-z6858_openshift-console-operator(c1e06766-b48c-429e-92f1-196bb083360b)\"" pod="openshift-console-operator/console-operator-696884fb9c-z6858" podUID="c1e06766-b48c-429e-92f1-196bb083360b"
Sep 10 20:58:46.274152 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:46.274123    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/895ac1f0-9420-486f-8a54-6de076e5ae32-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sxgtk\" (UID: \"895ac1f0-9420-486f-8a54-6de076e5ae32\") " pod="openshift-insights/insights-runtime-extractor-sxgtk"
Sep 10 20:58:46.274270 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:46.274193    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/895ac1f0-9420-486f-8a54-6de076e5ae32-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxgtk\" (UID: \"895ac1f0-9420-486f-8a54-6de076e5ae32\") " pod="openshift-insights/insights-runtime-extractor-sxgtk"
Sep 10 20:58:46.274270 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:46.274246    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 20:58:46.274270 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:46.274265    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 20:58:46.274423 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:46.274331    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/895ac1f0-9420-486f-8a54-6de076e5ae32-kube-rbac-proxy-cm podName:895ac1f0-9420-486f-8a54-6de076e5ae32 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:47.274312636 +0000 UTC m=+44.931221401 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/895ac1f0-9420-486f-8a54-6de076e5ae32-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sxgtk" (UID: "895ac1f0-9420-486f-8a54-6de076e5ae32") : configmap "kube-rbac-proxy" not found
Sep 10 20:58:46.274423 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:46.274350    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/895ac1f0-9420-486f-8a54-6de076e5ae32-insights-runtime-extractor-tls podName:895ac1f0-9420-486f-8a54-6de076e5ae32 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:47.274341816 +0000 UTC m=+44.931250581 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/895ac1f0-9420-486f-8a54-6de076e5ae32-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sxgtk" (UID: "895ac1f0-9420-486f-8a54-6de076e5ae32") : secret "insights-runtime-extractor-tls" not found
Sep 10 20:58:46.359750 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:46.359693    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-vqc26"]
Sep 10 20:58:46.389666 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:46.389643    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-vqc26"]
Sep 10 20:58:46.389751 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:46.389726    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vqc26"
Sep 10 20:58:46.392081 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:46.392058    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 10 20:58:46.392214 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:46.392195    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 10 20:58:46.392362 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:46.392343    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 10 20:58:46.392454 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:46.392393    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 10 20:58:46.392516 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:46.392465    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-85lh2\""
Sep 10 20:58:46.576337 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:46.576292    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ad941378-2761-47a0-9f64-6201c964276c-serving-cert\") pod \"service-ca-operator-cf7c776b7-vqc26\" (UID: \"ad941378-2761-47a0-9f64-6201c964276c\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vqc26"
Sep 10 20:58:46.576700 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:46.576374    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfc2r\" (UniqueName: \"kubernetes.io/projected/ad941378-2761-47a0-9f64-6201c964276c-kube-api-access-xfc2r\") pod \"service-ca-operator-cf7c776b7-vqc26\" (UID: \"ad941378-2761-47a0-9f64-6201c964276c\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vqc26"
Sep 10 20:58:46.576700 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:46.576407    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ad941378-2761-47a0-9f64-6201c964276c-config\") pod \"service-ca-operator-cf7c776b7-vqc26\" (UID: \"ad941378-2761-47a0-9f64-6201c964276c\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vqc26"
Sep 10 20:58:46.677678 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:46.677597    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ad941378-2761-47a0-9f64-6201c964276c-serving-cert\") pod \"service-ca-operator-cf7c776b7-vqc26\" (UID: \"ad941378-2761-47a0-9f64-6201c964276c\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vqc26"
Sep 10 20:58:46.677817 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:46.677678    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfc2r\" (UniqueName: \"kubernetes.io/projected/ad941378-2761-47a0-9f64-6201c964276c-kube-api-access-xfc2r\") pod \"service-ca-operator-cf7c776b7-vqc26\" (UID: \"ad941378-2761-47a0-9f64-6201c964276c\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vqc26"
Sep 10 20:58:46.677817 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:46.677725    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ad941378-2761-47a0-9f64-6201c964276c-config\") pod \"service-ca-operator-cf7c776b7-vqc26\" (UID: \"ad941378-2761-47a0-9f64-6201c964276c\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vqc26"
Sep 10 20:58:46.678259 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:46.678236    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ad941378-2761-47a0-9f64-6201c964276c-config\") pod \"service-ca-operator-cf7c776b7-vqc26\" (UID: \"ad941378-2761-47a0-9f64-6201c964276c\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vqc26"
Sep 10 20:58:46.679713 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:46.679690    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ad941378-2761-47a0-9f64-6201c964276c-serving-cert\") pod \"service-ca-operator-cf7c776b7-vqc26\" (UID: \"ad941378-2761-47a0-9f64-6201c964276c\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vqc26"
Sep 10 20:58:46.686695 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:46.686673    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfc2r\" (UniqueName: \"kubernetes.io/projected/ad941378-2761-47a0-9f64-6201c964276c-kube-api-access-xfc2r\") pod \"service-ca-operator-cf7c776b7-vqc26\" (UID: \"ad941378-2761-47a0-9f64-6201c964276c\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vqc26"
Sep 10 20:58:46.698704 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:46.698686    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vqc26"
Sep 10 20:58:47.011276 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:47.011208    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-vqc26"]
Sep 10 20:58:47.014387 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:58:47.014359    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad941378_2761_47a0_9f64_6201c964276c.slice/crio-02bbd42a5a1023145cdf968b16504e86dc56f428e6e6f0720c89f66ca0e36c14 WatchSource:0}: Error finding container 02bbd42a5a1023145cdf968b16504e86dc56f428e6e6f0720c89f66ca0e36c14: Status 404 returned error can't find the container with id 02bbd42a5a1023145cdf968b16504e86dc56f428e6e6f0720c89f66ca0e36c14
Sep 10 20:58:47.075968 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:47.075948    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-z6858_c1e06766-b48c-429e-92f1-196bb083360b/console-operator/1.log"
Sep 10 20:58:47.076325 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:47.076284    2581 scope.go:117] "RemoveContainer" containerID="bfc746521aa3e8f97083a5507352b086eb08d1212839732dffe53946dfd57b44"
Sep 10 20:58:47.076484 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:47.076461    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-z6858_openshift-console-operator(c1e06766-b48c-429e-92f1-196bb083360b)\"" pod="openshift-console-operator/console-operator-696884fb9c-z6858" podUID="c1e06766-b48c-429e-92f1-196bb083360b"
Sep 10 20:58:47.077281 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:47.077259    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vqc26" event={"ID":"ad941378-2761-47a0-9f64-6201c964276c","Type":"ContainerStarted","Data":"02bbd42a5a1023145cdf968b16504e86dc56f428e6e6f0720c89f66ca0e36c14"}
Sep 10 20:58:47.229931 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:47.229906    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h4md4_0f68613e-8086-4f5f-8a47-af8dcd8e3a76/dns-node-resolver/0.log"
Sep 10 20:58:47.282882 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:47.282821    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/895ac1f0-9420-486f-8a54-6de076e5ae32-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxgtk\" (UID: \"895ac1f0-9420-486f-8a54-6de076e5ae32\") " pod="openshift-insights/insights-runtime-extractor-sxgtk"
Sep 10 20:58:47.282992 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:47.282916    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/895ac1f0-9420-486f-8a54-6de076e5ae32-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sxgtk\" (UID: \"895ac1f0-9420-486f-8a54-6de076e5ae32\") " pod="openshift-insights/insights-runtime-extractor-sxgtk"
Sep 10 20:58:47.282992 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:47.282946    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 20:58:47.283077 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:47.282995    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/895ac1f0-9420-486f-8a54-6de076e5ae32-insights-runtime-extractor-tls podName:895ac1f0-9420-486f-8a54-6de076e5ae32 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:49.282981234 +0000 UTC m=+46.939889988 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/895ac1f0-9420-486f-8a54-6de076e5ae32-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sxgtk" (UID: "895ac1f0-9420-486f-8a54-6de076e5ae32") : secret "insights-runtime-extractor-tls" not found
Sep 10 20:58:47.283077 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:47.283018    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 20:58:47.283077 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:47.283066    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/895ac1f0-9420-486f-8a54-6de076e5ae32-kube-rbac-proxy-cm podName:895ac1f0-9420-486f-8a54-6de076e5ae32 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:49.283051241 +0000 UTC m=+46.939959993 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/895ac1f0-9420-486f-8a54-6de076e5ae32-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sxgtk" (UID: "895ac1f0-9420-486f-8a54-6de076e5ae32") : configmap "kube-rbac-proxy" not found
Sep 10 20:58:48.393821 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:48.393787    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64a30645-a3fb-4670-89af-a044b3dd00a0-metrics-certs\") pod \"router-default-64c8fc8856-tfwzb\" (UID: \"64a30645-a3fb-4670-89af-a044b3dd00a0\") " pod="openshift-ingress/router-default-64c8fc8856-tfwzb"
Sep 10 20:58:48.394192 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:48.393842    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/da65604c-6689-4ef5-9cdf-0507cbeba512-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-bvh97\" (UID: \"da65604c-6689-4ef5-9cdf-0507cbeba512\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bvh97"
Sep 10 20:58:48.394192 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:48.393919    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64a30645-a3fb-4670-89af-a044b3dd00a0-service-ca-bundle\") pod \"router-default-64c8fc8856-tfwzb\" (UID: \"64a30645-a3fb-4670-89af-a044b3dd00a0\") " pod="openshift-ingress/router-default-64c8fc8856-tfwzb"
Sep 10 20:58:48.394192 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:48.393947    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 20:58:48.394192 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:48.394013    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/64a30645-a3fb-4670-89af-a044b3dd00a0-service-ca-bundle podName:64a30645-a3fb-4670-89af-a044b3dd00a0 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:56.393997161 +0000 UTC m=+54.050905911 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/64a30645-a3fb-4670-89af-a044b3dd00a0-service-ca-bundle") pod "router-default-64c8fc8856-tfwzb" (UID: "64a30645-a3fb-4670-89af-a044b3dd00a0") : configmap references non-existent config key: service-ca.crt
Sep 10 20:58:48.394192 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:48.394023    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 20:58:48.394192 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:48.394030    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64a30645-a3fb-4670-89af-a044b3dd00a0-metrics-certs podName:64a30645-a3fb-4670-89af-a044b3dd00a0 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:56.394022193 +0000 UTC m=+54.050930943 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64a30645-a3fb-4670-89af-a044b3dd00a0-metrics-certs") pod "router-default-64c8fc8856-tfwzb" (UID: "64a30645-a3fb-4670-89af-a044b3dd00a0") : secret "router-metrics-certs-default" not found
Sep 10 20:58:48.394192 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:48.394160    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da65604c-6689-4ef5-9cdf-0507cbeba512-cluster-monitoring-operator-tls podName:da65604c-6689-4ef5-9cdf-0507cbeba512 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:56.394147007 +0000 UTC m=+54.051055757 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/da65604c-6689-4ef5-9cdf-0507cbeba512-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-bvh97" (UID: "da65604c-6689-4ef5-9cdf-0507cbeba512") : secret "cluster-monitoring-operator-tls" not found
Sep 10 20:58:48.431483 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:48.431453    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mthkf_7fc96d36-95bc-4159-b504-7fad608e5819/node-ca/0.log"
Sep 10 20:58:49.030023 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:49.030001    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4bsk5_443d28b9-b165-4d98-b86a-ea4924d54962/serve-healthcheck-canary/0.log"
Sep 10 20:58:49.300606 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:49.300577    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/895ac1f0-9420-486f-8a54-6de076e5ae32-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxgtk\" (UID: \"895ac1f0-9420-486f-8a54-6de076e5ae32\") " pod="openshift-insights/insights-runtime-extractor-sxgtk"
Sep 10 20:58:49.300764 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:49.300653    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/895ac1f0-9420-486f-8a54-6de076e5ae32-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sxgtk\" (UID: \"895ac1f0-9420-486f-8a54-6de076e5ae32\") " pod="openshift-insights/insights-runtime-extractor-sxgtk"
Sep 10 20:58:49.300764 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:49.300730    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 20:58:49.300764 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:49.300740    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 20:58:49.300893 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:49.300785    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/895ac1f0-9420-486f-8a54-6de076e5ae32-kube-rbac-proxy-cm podName:895ac1f0-9420-486f-8a54-6de076e5ae32 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:53.300766753 +0000 UTC m=+50.957675519 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/895ac1f0-9420-486f-8a54-6de076e5ae32-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sxgtk" (UID: "895ac1f0-9420-486f-8a54-6de076e5ae32") : configmap "kube-rbac-proxy" not found
Sep 10 20:58:49.300893 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:49.300803    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/895ac1f0-9420-486f-8a54-6de076e5ae32-insights-runtime-extractor-tls podName:895ac1f0-9420-486f-8a54-6de076e5ae32 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:53.300794186 +0000 UTC m=+50.957702941 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/895ac1f0-9420-486f-8a54-6de076e5ae32-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sxgtk" (UID: "895ac1f0-9420-486f-8a54-6de076e5ae32") : secret "insights-runtime-extractor-tls" not found
Sep 10 20:58:50.087000 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:50.086970    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vqc26" event={"ID":"ad941378-2761-47a0-9f64-6201c964276c","Type":"ContainerStarted","Data":"5cccc18c27495ee96d1fa8e42cac16b671749f29847136538e78bf9c8a0a9ef7"}
Sep 10 20:58:50.105660 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:50.105614    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vqc26" podStartSLOduration=1.81722422 podStartE2EDuration="4.105599637s" totalImagesPullingTime="2.288375417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:50.105140696 +0000 UTC m=+47.762049468" watchObservedRunningTime="2026-09-10 20:58:50.105599637 +0000 UTC m=+47.762508409"
Sep 10 20:58:50.920508 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:50.920475    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-z6858"
Sep 10 20:58:50.920663 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:50.920522    2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-z6858"
Sep 10 20:58:50.920815 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:50.920803    2581 scope.go:117] "RemoveContainer" containerID="bfc746521aa3e8f97083a5507352b086eb08d1212839732dffe53946dfd57b44"
Sep 10 20:58:50.920953 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:50.920935    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-z6858_openshift-console-operator(c1e06766-b48c-429e-92f1-196bb083360b)\"" pod="openshift-console-operator/console-operator-696884fb9c-z6858" podUID="c1e06766-b48c-429e-92f1-196bb083360b"
Sep 10 20:58:53.025634 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:53.025586    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/378700dd-8e73-4a3e-9c10-35b09ebbdd68-metrics-tls\") pod \"dns-default-wfsbr\" (UID: \"378700dd-8e73-4a3e-9c10-35b09ebbdd68\") " pod="openshift-dns/dns-default-wfsbr"
Sep 10 20:58:53.026035 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:53.025719    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:58:53.026035 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:53.025781    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/378700dd-8e73-4a3e-9c10-35b09ebbdd68-metrics-tls podName:378700dd-8e73-4a3e-9c10-35b09ebbdd68 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:09.025766022 +0000 UTC m=+66.682674771 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/378700dd-8e73-4a3e-9c10-35b09ebbdd68-metrics-tls") pod "dns-default-wfsbr" (UID: "378700dd-8e73-4a3e-9c10-35b09ebbdd68") : secret "dns-default-metrics-tls" not found
Sep 10 20:58:53.326877 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:53.326843    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/895ac1f0-9420-486f-8a54-6de076e5ae32-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sxgtk\" (UID: \"895ac1f0-9420-486f-8a54-6de076e5ae32\") " pod="openshift-insights/insights-runtime-extractor-sxgtk"
Sep 10 20:58:53.327028 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:53.326925    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/895ac1f0-9420-486f-8a54-6de076e5ae32-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxgtk\" (UID: \"895ac1f0-9420-486f-8a54-6de076e5ae32\") " pod="openshift-insights/insights-runtime-extractor-sxgtk"
Sep 10 20:58:53.327028 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:53.326956    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 20:58:53.327028 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:53.327022    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/895ac1f0-9420-486f-8a54-6de076e5ae32-kube-rbac-proxy-cm podName:895ac1f0-9420-486f-8a54-6de076e5ae32 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:01.327007538 +0000 UTC m=+58.983916287 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/895ac1f0-9420-486f-8a54-6de076e5ae32-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sxgtk" (UID: "895ac1f0-9420-486f-8a54-6de076e5ae32") : configmap "kube-rbac-proxy" not found
Sep 10 20:58:53.327028 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:53.327025    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 20:58:53.327164 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:53.327064    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/895ac1f0-9420-486f-8a54-6de076e5ae32-insights-runtime-extractor-tls podName:895ac1f0-9420-486f-8a54-6de076e5ae32 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:01.327053715 +0000 UTC m=+58.983962477 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/895ac1f0-9420-486f-8a54-6de076e5ae32-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sxgtk" (UID: "895ac1f0-9420-486f-8a54-6de076e5ae32") : secret "insights-runtime-extractor-tls" not found
Sep 10 20:58:56.450593 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:56.450555    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64a30645-a3fb-4670-89af-a044b3dd00a0-metrics-certs\") pod \"router-default-64c8fc8856-tfwzb\" (UID: \"64a30645-a3fb-4670-89af-a044b3dd00a0\") " pod="openshift-ingress/router-default-64c8fc8856-tfwzb"
Sep 10 20:58:56.450950 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:56.450604    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/da65604c-6689-4ef5-9cdf-0507cbeba512-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-bvh97\" (UID: \"da65604c-6689-4ef5-9cdf-0507cbeba512\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bvh97"
Sep 10 20:58:56.450950 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:58:56.450678    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64a30645-a3fb-4670-89af-a044b3dd00a0-service-ca-bundle\") pod \"router-default-64c8fc8856-tfwzb\" (UID: \"64a30645-a3fb-4670-89af-a044b3dd00a0\") " pod="openshift-ingress/router-default-64c8fc8856-tfwzb"
Sep 10 20:58:56.450950 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:56.450721    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 20:58:56.450950 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:56.450771    2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 20:58:56.450950 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:56.450785    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64a30645-a3fb-4670-89af-a044b3dd00a0-metrics-certs podName:64a30645-a3fb-4670-89af-a044b3dd00a0 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:12.450765989 +0000 UTC m=+70.107674762 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64a30645-a3fb-4670-89af-a044b3dd00a0-metrics-certs") pod "router-default-64c8fc8856-tfwzb" (UID: "64a30645-a3fb-4670-89af-a044b3dd00a0") : secret "router-metrics-certs-default" not found
Sep 10 20:58:56.450950 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:56.450825    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da65604c-6689-4ef5-9cdf-0507cbeba512-cluster-monitoring-operator-tls podName:da65604c-6689-4ef5-9cdf-0507cbeba512 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:12.45081092 +0000 UTC m=+70.107719670 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/da65604c-6689-4ef5-9cdf-0507cbeba512-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-bvh97" (UID: "da65604c-6689-4ef5-9cdf-0507cbeba512") : secret "cluster-monitoring-operator-tls" not found
Sep 10 20:58:56.450950 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:58:56.450837    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/64a30645-a3fb-4670-89af-a044b3dd00a0-service-ca-bundle podName:64a30645-a3fb-4670-89af-a044b3dd00a0 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:12.450831677 +0000 UTC m=+70.107740426 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/64a30645-a3fb-4670-89af-a044b3dd00a0-service-ca-bundle") pod "router-default-64c8fc8856-tfwzb" (UID: "64a30645-a3fb-4670-89af-a044b3dd00a0") : configmap references non-existent config key: service-ca.crt
Sep 10 20:59:01.390872 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:01.390833    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/895ac1f0-9420-486f-8a54-6de076e5ae32-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxgtk\" (UID: \"895ac1f0-9420-486f-8a54-6de076e5ae32\") " pod="openshift-insights/insights-runtime-extractor-sxgtk"
Sep 10 20:59:01.391375 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:01.390918    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/895ac1f0-9420-486f-8a54-6de076e5ae32-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sxgtk\" (UID: \"895ac1f0-9420-486f-8a54-6de076e5ae32\") " pod="openshift-insights/insights-runtime-extractor-sxgtk"
Sep 10 20:59:01.391375 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:59:01.390987    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 20:59:01.391375 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:59:01.391058    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/895ac1f0-9420-486f-8a54-6de076e5ae32-insights-runtime-extractor-tls podName:895ac1f0-9420-486f-8a54-6de076e5ae32 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:17.391040716 +0000 UTC m=+75.047949470 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/895ac1f0-9420-486f-8a54-6de076e5ae32-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sxgtk" (UID: "895ac1f0-9420-486f-8a54-6de076e5ae32") : secret "insights-runtime-extractor-tls" not found
Sep 10 20:59:01.391527 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:01.391438    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/895ac1f0-9420-486f-8a54-6de076e5ae32-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sxgtk\" (UID: \"895ac1f0-9420-486f-8a54-6de076e5ae32\") " pod="openshift-insights/insights-runtime-extractor-sxgtk"
Sep 10 20:59:02.020470 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:02.020443    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-brrtd"
Sep 10 20:59:02.864018 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:02.863989    2581 scope.go:117] "RemoveContainer" containerID="bfc746521aa3e8f97083a5507352b086eb08d1212839732dffe53946dfd57b44"
Sep 10 20:59:03.121932 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:03.121856    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-z6858_c1e06766-b48c-429e-92f1-196bb083360b/console-operator/1.log"
Sep 10 20:59:03.122073 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:03.121942    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-z6858" event={"ID":"c1e06766-b48c-429e-92f1-196bb083360b","Type":"ContainerStarted","Data":"d1fe72ccb6af9337deb1b8ec7c95338814722aa9376eaa9bc6f49dbf0e0d312e"}
Sep 10 20:59:03.122234 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:03.122212    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-z6858"
Sep 10 20:59:03.853412 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:03.853378    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-z6858"
Sep 10 20:59:03.878402 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:03.878112    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-z6858" podStartSLOduration=21.042698635 podStartE2EDuration="23.878090058s" totalImagesPullingTime="2.835391423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:03.147700681 +0000 UTC m=+60.804609452" watchObservedRunningTime="2026-09-10 20:59:03.878090058 +0000 UTC m=+61.534998831"
Sep 10 20:59:08.652608 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:08.652569    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e1f126a-f90d-4cf6-b863-d6130c01a622-original-pull-secret\") pod \"global-pull-secret-syncer-mgx5w\" (UID: \"6e1f126a-f90d-4cf6-b863-d6130c01a622\") " pod="kube-system/global-pull-secret-syncer-mgx5w"
Sep 10 20:59:08.653253 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:08.652696    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/781245a4-8475-4031-9137-dd43ecd1c260-metrics-certs\") pod \"network-metrics-daemon-lmwz4\" (UID: \"781245a4-8475-4031-9137-dd43ecd1c260\") " pod="openshift-multus/network-metrics-daemon-lmwz4"
Sep 10 20:59:08.655705 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:08.655683    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 20:59:08.655822 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:08.655785    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 20:59:08.665441 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:08.665421    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e1f126a-f90d-4cf6-b863-d6130c01a622-original-pull-secret\") pod \"global-pull-secret-syncer-mgx5w\" (UID: \"6e1f126a-f90d-4cf6-b863-d6130c01a622\") " pod="kube-system/global-pull-secret-syncer-mgx5w"
Sep 10 20:59:08.665533 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:08.665440    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/781245a4-8475-4031-9137-dd43ecd1c260-metrics-certs\") pod \"network-metrics-daemon-lmwz4\" (UID: \"781245a4-8475-4031-9137-dd43ecd1c260\") " pod="openshift-multus/network-metrics-daemon-lmwz4"
Sep 10 20:59:08.753571 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:08.753542    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8nv4\" (UniqueName: \"kubernetes.io/projected/3bfd0b36-183c-43c3-9fcf-2d6f7839223e-kube-api-access-g8nv4\") pod \"network-check-target-gvw9h\" (UID: \"3bfd0b36-183c-43c3-9fcf-2d6f7839223e\") " pod="openshift-network-diagnostics/network-check-target-gvw9h"
Sep 10 20:59:08.756187 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:08.756169    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 20:59:08.767374 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:08.767350    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 20:59:08.776847 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:08.776822    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8nv4\" (UniqueName: \"kubernetes.io/projected/3bfd0b36-183c-43c3-9fcf-2d6f7839223e-kube-api-access-g8nv4\") pod \"network-check-target-gvw9h\" (UID: \"3bfd0b36-183c-43c3-9fcf-2d6f7839223e\") " pod="openshift-network-diagnostics/network-check-target-gvw9h"
Sep 10 20:59:08.877576 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:08.877550    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tjcr5\""
Sep 10 20:59:08.884614 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:08.884596    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lmwz4"
Sep 10 20:59:08.886392 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:08.886378    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ll8tt\""
Sep 10 20:59:08.888109 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:08.888091    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mgx5w"
Sep 10 20:59:08.892949 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:08.892927    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gvw9h"
Sep 10 20:59:09.057032 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:09.056987    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/378700dd-8e73-4a3e-9c10-35b09ebbdd68-metrics-tls\") pod \"dns-default-wfsbr\" (UID: \"378700dd-8e73-4a3e-9c10-35b09ebbdd68\") " pod="openshift-dns/dns-default-wfsbr"
Sep 10 20:59:09.059222 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:09.059193    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/378700dd-8e73-4a3e-9c10-35b09ebbdd68-metrics-tls\") pod \"dns-default-wfsbr\" (UID: \"378700dd-8e73-4a3e-9c10-35b09ebbdd68\") " pod="openshift-dns/dns-default-wfsbr"
Sep 10 20:59:09.205621 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:09.205541    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gvw9h"]
Sep 10 20:59:09.213061 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:09.213027    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mgx5w"]
Sep 10 20:59:09.213197 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:09.213135    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lmwz4"]
Sep 10 20:59:09.216456 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:59:09.216429    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e1f126a_f90d_4cf6_b863_d6130c01a622.slice/crio-374955aa6eb68cfe322396cb68a05b14b06f2190d89ef690fd77cc21165c058b WatchSource:0}: Error finding container 374955aa6eb68cfe322396cb68a05b14b06f2190d89ef690fd77cc21165c058b: Status 404 returned error can't find the container with id 374955aa6eb68cfe322396cb68a05b14b06f2190d89ef690fd77cc21165c058b
Sep 10 20:59:09.217148 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:59:09.217125    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod781245a4_8475_4031_9137_dd43ecd1c260.slice/crio-b7a162ca6661e5c7b64952f67f3d07911ebb3ce0d04a4981468190938b952bc8 WatchSource:0}: Error finding container b7a162ca6661e5c7b64952f67f3d07911ebb3ce0d04a4981468190938b952bc8: Status 404 returned error can't find the container with id b7a162ca6661e5c7b64952f67f3d07911ebb3ce0d04a4981468190938b952bc8
Sep 10 20:59:09.281215 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:09.281181    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9hkk2\""
Sep 10 20:59:09.287493 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:09.287469    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wfsbr"
Sep 10 20:59:09.568128 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:09.568019    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wfsbr"]
Sep 10 20:59:09.570848 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:59:09.570820    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod378700dd_8e73_4a3e_9c10_35b09ebbdd68.slice/crio-6a201cdfec3ff110d2c92b334907079ad87503ded8cafa041186ea3646646f86 WatchSource:0}: Error finding container 6a201cdfec3ff110d2c92b334907079ad87503ded8cafa041186ea3646646f86: Status 404 returned error can't find the container with id 6a201cdfec3ff110d2c92b334907079ad87503ded8cafa041186ea3646646f86
Sep 10 20:59:10.150214 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:10.150171    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lmwz4" event={"ID":"781245a4-8475-4031-9137-dd43ecd1c260","Type":"ContainerStarted","Data":"b7a162ca6661e5c7b64952f67f3d07911ebb3ce0d04a4981468190938b952bc8"}
Sep 10 20:59:10.152098 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:10.152022    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gvw9h" event={"ID":"3bfd0b36-183c-43c3-9fcf-2d6f7839223e","Type":"ContainerStarted","Data":"b1b9f172dd45026ed9b8f52b4a4f1a04c48812403a0dbe12f76d1670bd9e8dc2"}
Sep 10 20:59:10.154029 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:10.153998    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mgx5w" event={"ID":"6e1f126a-f90d-4cf6-b863-d6130c01a622","Type":"ContainerStarted","Data":"374955aa6eb68cfe322396cb68a05b14b06f2190d89ef690fd77cc21165c058b"}
Sep 10 20:59:10.155374 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:10.155348    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wfsbr" event={"ID":"378700dd-8e73-4a3e-9c10-35b09ebbdd68","Type":"ContainerStarted","Data":"6a201cdfec3ff110d2c92b334907079ad87503ded8cafa041186ea3646646f86"}
Sep 10 20:59:11.162935 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:11.162897    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lmwz4" event={"ID":"781245a4-8475-4031-9137-dd43ecd1c260","Type":"ContainerStarted","Data":"b726df972c5a774f369d9667c0260a118d0c6af1b79ccf10213c40bfa6fedabc"}
Sep 10 20:59:11.163329 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:11.162963    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lmwz4" event={"ID":"781245a4-8475-4031-9137-dd43ecd1c260","Type":"ContainerStarted","Data":"81a9ae62300b05420f1bd5b05ff05da19bd3a5069b0cae2dbaf918109c19a030"}
Sep 10 20:59:11.188581 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:11.188520    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lmwz4" podStartSLOduration=67.102335674 podStartE2EDuration="1m8.188500337s" totalImagesPullingTime="1.086164663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:11.186795953 +0000 UTC m=+68.843704725" watchObservedRunningTime="2026-09-10 20:59:11.188500337 +0000 UTC m=+68.845409113"
Sep 10 20:59:12.169108 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:12.169071    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wfsbr" event={"ID":"378700dd-8e73-4a3e-9c10-35b09ebbdd68","Type":"ContainerStarted","Data":"c620038a2203a8083cb382b5c0339b4f257849396964d971d862e98546285dcd"}
Sep 10 20:59:12.486779 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:12.486690    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64a30645-a3fb-4670-89af-a044b3dd00a0-metrics-certs\") pod \"router-default-64c8fc8856-tfwzb\" (UID: \"64a30645-a3fb-4670-89af-a044b3dd00a0\") " pod="openshift-ingress/router-default-64c8fc8856-tfwzb"
Sep 10 20:59:12.486779 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:12.486755    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/da65604c-6689-4ef5-9cdf-0507cbeba512-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-bvh97\" (UID: \"da65604c-6689-4ef5-9cdf-0507cbeba512\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bvh97"
Sep 10 20:59:12.487009 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:12.486826    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64a30645-a3fb-4670-89af-a044b3dd00a0-service-ca-bundle\") pod \"router-default-64c8fc8856-tfwzb\" (UID: \"64a30645-a3fb-4670-89af-a044b3dd00a0\") " pod="openshift-ingress/router-default-64c8fc8856-tfwzb"
Sep 10 20:59:12.487602 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:12.487572    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64a30645-a3fb-4670-89af-a044b3dd00a0-service-ca-bundle\") pod \"router-default-64c8fc8856-tfwzb\" (UID: \"64a30645-a3fb-4670-89af-a044b3dd00a0\") " pod="openshift-ingress/router-default-64c8fc8856-tfwzb"
Sep 10 20:59:12.489423 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:12.489401    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64a30645-a3fb-4670-89af-a044b3dd00a0-metrics-certs\") pod \"router-default-64c8fc8856-tfwzb\" (UID: \"64a30645-a3fb-4670-89af-a044b3dd00a0\") " pod="openshift-ingress/router-default-64c8fc8856-tfwzb"
Sep 10 20:59:12.489557 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:12.489470    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/da65604c-6689-4ef5-9cdf-0507cbeba512-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-bvh97\" (UID: \"da65604c-6689-4ef5-9cdf-0507cbeba512\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bvh97"
Sep 10 20:59:12.704002 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:12.703973    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-rksqh\""
Sep 10 20:59:12.712247 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:12.712223    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-64c8fc8856-tfwzb"
Sep 10 20:59:12.718972 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:12.718947    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-57j94\""
Sep 10 20:59:12.725994 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:12.725976    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bvh97"
Sep 10 20:59:14.614170 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:14.614005    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-bvh97"]
Sep 10 20:59:14.614938 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:14.614915    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-64c8fc8856-tfwzb"]
Sep 10 20:59:14.617679 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:59:14.617649    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda65604c_6689_4ef5_9cdf_0507cbeba512.slice/crio-5aac7481a350b3aa3b3ff13d50c6a0f087099ccbcfdc126fabbdf96759126cc6 WatchSource:0}: Error finding container 5aac7481a350b3aa3b3ff13d50c6a0f087099ccbcfdc126fabbdf96759126cc6: Status 404 returned error can't find the container with id 5aac7481a350b3aa3b3ff13d50c6a0f087099ccbcfdc126fabbdf96759126cc6
Sep 10 20:59:14.618212 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:59:14.618190    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64a30645_a3fb_4670_89af_a044b3dd00a0.slice/crio-48ce405c83cd2f35e16c10f6faa20309e001611f55de4f10e335afa7bba70dc9 WatchSource:0}: Error finding container 48ce405c83cd2f35e16c10f6faa20309e001611f55de4f10e335afa7bba70dc9: Status 404 returned error can't find the container with id 48ce405c83cd2f35e16c10f6faa20309e001611f55de4f10e335afa7bba70dc9
Sep 10 20:59:15.180680 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:15.180638    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gvw9h" event={"ID":"3bfd0b36-183c-43c3-9fcf-2d6f7839223e","Type":"ContainerStarted","Data":"d63d36cb3c75e3f669493c8dc02b02d9a626b155068deb7ff7108e8b36905b7b"}
Sep 10 20:59:15.180940 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:15.180728    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gvw9h"
Sep 10 20:59:15.182439 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:15.182411    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mgx5w" event={"ID":"6e1f126a-f90d-4cf6-b863-d6130c01a622","Type":"ContainerStarted","Data":"edb6a001616192f95eb9ba6921f9eb15381072bc55a03bb10e405537132fa8ab"}
Sep 10 20:59:15.184537 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:15.184510    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wfsbr" event={"ID":"378700dd-8e73-4a3e-9c10-35b09ebbdd68","Type":"ContainerStarted","Data":"15647d7e2a7d5a987a21948bc7772bf1da4fa85e2c4e992cb0c3c5e860b05548"}
Sep 10 20:59:15.184713 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:15.184651    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wfsbr"
Sep 10 20:59:15.186314 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:15.186265    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bvh97" event={"ID":"da65604c-6689-4ef5-9cdf-0507cbeba512","Type":"ContainerStarted","Data":"5aac7481a350b3aa3b3ff13d50c6a0f087099ccbcfdc126fabbdf96759126cc6"}
Sep 10 20:59:15.190187 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:15.190162    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-64c8fc8856-tfwzb" event={"ID":"64a30645-a3fb-4670-89af-a044b3dd00a0","Type":"ContainerStarted","Data":"2caa9b38b0ac3b62e9f05f2fe3f155598250272a0ffef6bbd2817f59cc8f367e"}
Sep 10 20:59:15.190356 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:15.190205    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-64c8fc8856-tfwzb" event={"ID":"64a30645-a3fb-4670-89af-a044b3dd00a0","Type":"ContainerStarted","Data":"48ce405c83cd2f35e16c10f6faa20309e001611f55de4f10e335afa7bba70dc9"}
Sep 10 20:59:15.248157 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:15.248103    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-64c8fc8856-tfwzb" podStartSLOduration=35.248085739 podStartE2EDuration="35.248085739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:15.247505488 +0000 UTC m=+72.904414259" watchObservedRunningTime="2026-09-10 20:59:15.248085739 +0000 UTC m=+72.904994513"
Sep 10 20:59:15.248861 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:15.248831    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gvw9h" podStartSLOduration=67.140296399 podStartE2EDuration="1m12.248821925s" totalImagesPullingTime="5.108525526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:15.213287225 +0000 UTC m=+72.870195999" watchObservedRunningTime="2026-09-10 20:59:15.248821925 +0000 UTC m=+72.905730718"
Sep 10 20:59:15.271143 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:15.271092    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mgx5w" podStartSLOduration=66.156862522 podStartE2EDuration="1m11.271081472s" totalImagesPullingTime="5.11421895s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:15.270536143 +0000 UTC m=+72.927444916" watchObservedRunningTime="2026-09-10 20:59:15.271081472 +0000 UTC m=+72.927990245"
Sep 10 20:59:15.297480 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:15.297436    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-wfsbr" podStartSLOduration=36.346495454 podStartE2EDuration="38.2974237s" totalImagesPullingTime="1.950928246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:15.296770811 +0000 UTC m=+72.953679586" watchObservedRunningTime="2026-09-10 20:59:15.2974237 +0000 UTC m=+72.954332530"
Sep 10 20:59:15.712692 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:15.712656    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-64c8fc8856-tfwzb"
Sep 10 20:59:15.715928 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:15.715899    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-64c8fc8856-tfwzb"
Sep 10 20:59:15.802524 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:15.802489    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-9h26g"]
Sep 10 20:59:15.806208 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:15.806183    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d77577fb-f5vr6"]
Sep 10 20:59:15.806355 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:15.806338    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-9h26g"
Sep 10 20:59:15.808009 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:15.807983    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d77577fb-f5vr6"
Sep 10 20:59:15.812589 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:15.812566    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 10 20:59:15.812991 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:15.812962    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 10 20:59:15.813097 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:15.813061    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 10 20:59:15.813177 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:15.813162    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 10 20:59:15.815476 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:15.815455    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 10 20:59:15.815585 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:15.815482    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 10 20:59:15.815585 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:15.815461    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-9z85c\""
Sep 10 20:59:15.827059 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:15.827005    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-9h26g"]
Sep 10 20:59:15.827984 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:15.827961    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d77577fb-f5vr6"]
Sep 10 20:59:15.914087 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:15.914056    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f9b5291-c0be-4959-a2b5-dfb539e576e6-tmp\") pod \"klusterlet-addon-workmgr-74d77577fb-f5vr6\" (UID: \"2f9b5291-c0be-4959-a2b5-dfb539e576e6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d77577fb-f5vr6"
Sep 10 20:59:15.914087 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:15.914096    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8vf5\" (UniqueName: \"kubernetes.io/projected/2f9b5291-c0be-4959-a2b5-dfb539e576e6-kube-api-access-x8vf5\") pod \"klusterlet-addon-workmgr-74d77577fb-f5vr6\" (UID: \"2f9b5291-c0be-4959-a2b5-dfb539e576e6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d77577fb-f5vr6"
Sep 10 20:59:15.914332 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:15.914126    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1a92d9ed-9fff-4373-8513-be11d25b6a37-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-9h26g\" (UID: \"1a92d9ed-9fff-4373-8513-be11d25b6a37\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-9h26g"
Sep 10 20:59:15.914332 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:15.914200    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1a92d9ed-9fff-4373-8513-be11d25b6a37-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-9h26g\" (UID: \"1a92d9ed-9fff-4373-8513-be11d25b6a37\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-9h26g"
Sep 10 20:59:15.914430 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:15.914337    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2f9b5291-c0be-4959-a2b5-dfb539e576e6-klusterlet-config\") pod \"klusterlet-addon-workmgr-74d77577fb-f5vr6\" (UID: \"2f9b5291-c0be-4959-a2b5-dfb539e576e6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d77577fb-f5vr6"
Sep 10 20:59:15.926071 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:15.926041    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-757878586c-489dg"]
Sep 10 20:59:15.928291 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:15.928261    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-757878586c-489dg"
Sep 10 20:59:15.931613 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:15.931587    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 10 20:59:15.932446 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:15.932429    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 10 20:59:15.932933 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:15.932918    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8bbrk\""
Sep 10 20:59:15.934201 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:15.934174    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 10 20:59:15.959243 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:15.959219    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-757878586c-489dg"]
Sep 10 20:59:15.976274 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:15.976200    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 10 20:59:16.014928 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:16.014896    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9da9490c-a5bc-40b9-ae71-37fcb67f73ee-bound-sa-token\") pod \"image-registry-757878586c-489dg\" (UID: \"9da9490c-a5bc-40b9-ae71-37fcb67f73ee\") " pod="openshift-image-registry/image-registry-757878586c-489dg"
Sep 10 20:59:16.015087 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:16.014934    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9da9490c-a5bc-40b9-ae71-37fcb67f73ee-ca-trust-extracted\") pod \"image-registry-757878586c-489dg\" (UID: \"9da9490c-a5bc-40b9-ae71-37fcb67f73ee\") " pod="openshift-image-registry/image-registry-757878586c-489dg"
Sep 10 20:59:16.015087 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:16.014963    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9da9490c-a5bc-40b9-ae71-37fcb67f73ee-trusted-ca\") pod \"image-registry-757878586c-489dg\" (UID: \"9da9490c-a5bc-40b9-ae71-37fcb67f73ee\") " pod="openshift-image-registry/image-registry-757878586c-489dg"
Sep 10 20:59:16.015087 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:16.015030    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9da9490c-a5bc-40b9-ae71-37fcb67f73ee-installation-pull-secrets\") pod \"image-registry-757878586c-489dg\" (UID: \"9da9490c-a5bc-40b9-ae71-37fcb67f73ee\") " pod="openshift-image-registry/image-registry-757878586c-489dg"
Sep 10 20:59:16.015245 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:16.015113    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f9b5291-c0be-4959-a2b5-dfb539e576e6-tmp\") pod \"klusterlet-addon-workmgr-74d77577fb-f5vr6\" (UID: \"2f9b5291-c0be-4959-a2b5-dfb539e576e6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d77577fb-f5vr6"
Sep 10 20:59:16.015245 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:16.015175    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9da9490c-a5bc-40b9-ae71-37fcb67f73ee-registry-tls\") pod \"image-registry-757878586c-489dg\" (UID: \"9da9490c-a5bc-40b9-ae71-37fcb67f73ee\") " pod="openshift-image-registry/image-registry-757878586c-489dg"
Sep 10 20:59:16.015245 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:16.015207    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8vf5\" (UniqueName: \"kubernetes.io/projected/2f9b5291-c0be-4959-a2b5-dfb539e576e6-kube-api-access-x8vf5\") pod \"klusterlet-addon-workmgr-74d77577fb-f5vr6\" (UID: \"2f9b5291-c0be-4959-a2b5-dfb539e576e6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d77577fb-f5vr6"
Sep 10 20:59:16.015245 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:16.015232    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpmkg\" (UniqueName: \"kubernetes.io/projected/9da9490c-a5bc-40b9-ae71-37fcb67f73ee-kube-api-access-kpmkg\") pod \"image-registry-757878586c-489dg\" (UID: \"9da9490c-a5bc-40b9-ae71-37fcb67f73ee\") " pod="openshift-image-registry/image-registry-757878586c-489dg"
Sep 10 20:59:16.015479 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:16.015268    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1a92d9ed-9fff-4373-8513-be11d25b6a37-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-9h26g\" (UID: \"1a92d9ed-9fff-4373-8513-be11d25b6a37\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-9h26g"
Sep 10 20:59:16.015479 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:16.015323    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1a92d9ed-9fff-4373-8513-be11d25b6a37-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-9h26g\" (UID: \"1a92d9ed-9fff-4373-8513-be11d25b6a37\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-9h26g"
Sep 10 20:59:16.015479 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:16.015350    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9da9490c-a5bc-40b9-ae71-37fcb67f73ee-image-registry-private-configuration\") pod \"image-registry-757878586c-489dg\" (UID: \"9da9490c-a5bc-40b9-ae71-37fcb67f73ee\") " pod="openshift-image-registry/image-registry-757878586c-489dg"
Sep 10 20:59:16.015479 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:16.015397    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9da9490c-a5bc-40b9-ae71-37fcb67f73ee-registry-certificates\") pod \"image-registry-757878586c-489dg\" (UID: \"9da9490c-a5bc-40b9-ae71-37fcb67f73ee\") " pod="openshift-image-registry/image-registry-757878586c-489dg"
Sep 10 20:59:16.015479 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:16.015424    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2f9b5291-c0be-4959-a2b5-dfb539e576e6-klusterlet-config\") pod \"klusterlet-addon-workmgr-74d77577fb-f5vr6\" (UID: \"2f9b5291-c0be-4959-a2b5-dfb539e576e6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d77577fb-f5vr6"
Sep 10 20:59:16.015713 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:16.015615    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f9b5291-c0be-4959-a2b5-dfb539e576e6-tmp\") pod \"klusterlet-addon-workmgr-74d77577fb-f5vr6\" (UID: \"2f9b5291-c0be-4959-a2b5-dfb539e576e6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d77577fb-f5vr6"
Sep 10 20:59:16.016102 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:16.016078    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1a92d9ed-9fff-4373-8513-be11d25b6a37-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-9h26g\" (UID: \"1a92d9ed-9fff-4373-8513-be11d25b6a37\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-9h26g"
Sep 10 20:59:16.018107 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:16.018085    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1a92d9ed-9fff-4373-8513-be11d25b6a37-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-9h26g\" (UID: \"1a92d9ed-9fff-4373-8513-be11d25b6a37\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-9h26g"
Sep 10 20:59:16.018206 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:16.018143    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2f9b5291-c0be-4959-a2b5-dfb539e576e6-klusterlet-config\") pod \"klusterlet-addon-workmgr-74d77577fb-f5vr6\" (UID: \"2f9b5291-c0be-4959-a2b5-dfb539e576e6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d77577fb-f5vr6"
Sep 10 20:59:16.028386 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:16.028360    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8vf5\" (UniqueName: \"kubernetes.io/projected/2f9b5291-c0be-4959-a2b5-dfb539e576e6-kube-api-access-x8vf5\") pod \"klusterlet-addon-workmgr-74d77577fb-f5vr6\" (UID: \"2f9b5291-c0be-4959-a2b5-dfb539e576e6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d77577fb-f5vr6"
Sep 10 20:59:16.116338 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:16.116284    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9da9490c-a5bc-40b9-ae71-37fcb67f73ee-registry-certificates\") pod \"image-registry-757878586c-489dg\" (UID: \"9da9490c-a5bc-40b9-ae71-37fcb67f73ee\") " pod="openshift-image-registry/image-registry-757878586c-489dg"
Sep 10 20:59:16.116520 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:16.116381    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9da9490c-a5bc-40b9-ae71-37fcb67f73ee-bound-sa-token\") pod \"image-registry-757878586c-489dg\" (UID: \"9da9490c-a5bc-40b9-ae71-37fcb67f73ee\") " pod="openshift-image-registry/image-registry-757878586c-489dg"
Sep 10 20:59:16.116520 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:16.116408    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9da9490c-a5bc-40b9-ae71-37fcb67f73ee-ca-trust-extracted\") pod \"image-registry-757878586c-489dg\" (UID: \"9da9490c-a5bc-40b9-ae71-37fcb67f73ee\") " pod="openshift-image-registry/image-registry-757878586c-489dg"
Sep 10 20:59:16.116520 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:16.116436    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9da9490c-a5bc-40b9-ae71-37fcb67f73ee-trusted-ca\") pod \"image-registry-757878586c-489dg\" (UID: \"9da9490c-a5bc-40b9-ae71-37fcb67f73ee\") " pod="openshift-image-registry/image-registry-757878586c-489dg"
Sep 10 20:59:16.116520 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:16.116471    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9da9490c-a5bc-40b9-ae71-37fcb67f73ee-installation-pull-secrets\") pod \"image-registry-757878586c-489dg\" (UID: \"9da9490c-a5bc-40b9-ae71-37fcb67f73ee\") " pod="openshift-image-registry/image-registry-757878586c-489dg"
Sep 10 20:59:16.116520 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:16.116506    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9da9490c-a5bc-40b9-ae71-37fcb67f73ee-registry-tls\") pod \"image-registry-757878586c-489dg\" (UID: \"9da9490c-a5bc-40b9-ae71-37fcb67f73ee\") " pod="openshift-image-registry/image-registry-757878586c-489dg"
Sep 10 20:59:16.116787 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:16.116749    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9da9490c-a5bc-40b9-ae71-37fcb67f73ee-ca-trust-extracted\") pod \"image-registry-757878586c-489dg\" (UID: \"9da9490c-a5bc-40b9-ae71-37fcb67f73ee\") " pod="openshift-image-registry/image-registry-757878586c-489dg"
Sep 10 20:59:16.116847 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:16.116792    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpmkg\" (UniqueName: \"kubernetes.io/projected/9da9490c-a5bc-40b9-ae71-37fcb67f73ee-kube-api-access-kpmkg\") pod \"image-registry-757878586c-489dg\" (UID: \"9da9490c-a5bc-40b9-ae71-37fcb67f73ee\") " pod="openshift-image-registry/image-registry-757878586c-489dg"
Sep 10 20:59:16.116847 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:16.116828    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9da9490c-a5bc-40b9-ae71-37fcb67f73ee-image-registry-private-configuration\") pod \"image-registry-757878586c-489dg\" (UID: \"9da9490c-a5bc-40b9-ae71-37fcb67f73ee\") " pod="openshift-image-registry/image-registry-757878586c-489dg"
Sep 10 20:59:16.117215 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:16.117168    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9da9490c-a5bc-40b9-ae71-37fcb67f73ee-registry-certificates\") pod \"image-registry-757878586c-489dg\" (UID: \"9da9490c-a5bc-40b9-ae71-37fcb67f73ee\") " pod="openshift-image-registry/image-registry-757878586c-489dg"
Sep 10 20:59:16.117414 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:16.117358    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9da9490c-a5bc-40b9-ae71-37fcb67f73ee-trusted-ca\") pod \"image-registry-757878586c-489dg\" (UID: \"9da9490c-a5bc-40b9-ae71-37fcb67f73ee\") " pod="openshift-image-registry/image-registry-757878586c-489dg"
Sep 10 20:59:16.118215 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:16.118181    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-9h26g"
Sep 10 20:59:16.119594 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:16.119431    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9da9490c-a5bc-40b9-ae71-37fcb67f73ee-installation-pull-secrets\") pod \"image-registry-757878586c-489dg\" (UID: \"9da9490c-a5bc-40b9-ae71-37fcb67f73ee\") " pod="openshift-image-registry/image-registry-757878586c-489dg"
Sep 10 20:59:16.119594 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:16.119439    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9da9490c-a5bc-40b9-ae71-37fcb67f73ee-image-registry-private-configuration\") pod \"image-registry-757878586c-489dg\" (UID: \"9da9490c-a5bc-40b9-ae71-37fcb67f73ee\") " pod="openshift-image-registry/image-registry-757878586c-489dg"
Sep 10 20:59:16.119594 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:16.119562    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9da9490c-a5bc-40b9-ae71-37fcb67f73ee-registry-tls\") pod \"image-registry-757878586c-489dg\" (UID: \"9da9490c-a5bc-40b9-ae71-37fcb67f73ee\") " pod="openshift-image-registry/image-registry-757878586c-489dg"
Sep 10 20:59:16.124985 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:16.124964    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d77577fb-f5vr6"
Sep 10 20:59:16.128683 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:16.128660    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpmkg\" (UniqueName: \"kubernetes.io/projected/9da9490c-a5bc-40b9-ae71-37fcb67f73ee-kube-api-access-kpmkg\") pod \"image-registry-757878586c-489dg\" (UID: \"9da9490c-a5bc-40b9-ae71-37fcb67f73ee\") " pod="openshift-image-registry/image-registry-757878586c-489dg"
Sep 10 20:59:16.130069 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:16.130027    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9da9490c-a5bc-40b9-ae71-37fcb67f73ee-bound-sa-token\") pod \"image-registry-757878586c-489dg\" (UID: \"9da9490c-a5bc-40b9-ae71-37fcb67f73ee\") " pod="openshift-image-registry/image-registry-757878586c-489dg"
Sep 10 20:59:16.195184 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:16.195134    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-64c8fc8856-tfwzb"
Sep 10 20:59:16.196094 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:16.196066    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-64c8fc8856-tfwzb"
Sep 10 20:59:16.239212 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:16.238932    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-757878586c-489dg"
Sep 10 20:59:16.435477 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:16.435442    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d77577fb-f5vr6"]
Sep 10 20:59:16.438376 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:59:16.438347    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f9b5291_c0be_4959_a2b5_dfb539e576e6.slice/crio-86285289882cb3179e8035eb47f8d5012a7e3591bee6128d23ec40886f01deff WatchSource:0}: Error finding container 86285289882cb3179e8035eb47f8d5012a7e3591bee6128d23ec40886f01deff: Status 404 returned error can't find the container with id 86285289882cb3179e8035eb47f8d5012a7e3591bee6128d23ec40886f01deff
Sep 10 20:59:16.449061 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:16.449041    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-9h26g"]
Sep 10 20:59:16.452009 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:59:16.451984    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a92d9ed_9fff_4373_8513_be11d25b6a37.slice/crio-eb25791e0fad2c5063e7b861d5dabf0c45d82fe22afedb8f22c1563da2198386 WatchSource:0}: Error finding container eb25791e0fad2c5063e7b861d5dabf0c45d82fe22afedb8f22c1563da2198386: Status 404 returned error can't find the container with id eb25791e0fad2c5063e7b861d5dabf0c45d82fe22afedb8f22c1563da2198386
Sep 10 20:59:16.563628 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:16.563593    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-757878586c-489dg"]
Sep 10 20:59:16.566724 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:59:16.566695    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9da9490c_a5bc_40b9_ae71_37fcb67f73ee.slice/crio-6ace1b8dbbdcdc27b0095fab3f1ffb16a357aad7078705475e2257880a664070 WatchSource:0}: Error finding container 6ace1b8dbbdcdc27b0095fab3f1ffb16a357aad7078705475e2257880a664070: Status 404 returned error can't find the container with id 6ace1b8dbbdcdc27b0095fab3f1ffb16a357aad7078705475e2257880a664070
Sep 10 20:59:17.199678 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:17.199609    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-9h26g" event={"ID":"1a92d9ed-9fff-4373-8513-be11d25b6a37","Type":"ContainerStarted","Data":"eb25791e0fad2c5063e7b861d5dabf0c45d82fe22afedb8f22c1563da2198386"}
Sep 10 20:59:17.201050 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:17.200980    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d77577fb-f5vr6" event={"ID":"2f9b5291-c0be-4959-a2b5-dfb539e576e6","Type":"ContainerStarted","Data":"86285289882cb3179e8035eb47f8d5012a7e3591bee6128d23ec40886f01deff"}
Sep 10 20:59:17.202762 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:17.202733    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bvh97" event={"ID":"da65604c-6689-4ef5-9cdf-0507cbeba512","Type":"ContainerStarted","Data":"0ba9342949ea30fcc1ecb97f36197043a8ce9b1f37418f601ea06478b4e3bb8c"}
Sep 10 20:59:17.204567 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:17.204538    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-757878586c-489dg" event={"ID":"9da9490c-a5bc-40b9-ae71-37fcb67f73ee","Type":"ContainerStarted","Data":"37ec12dcf4c596b0314fe43abd7a3dfaee5b3fd11cab1a7c79adca00aa89e460"}
Sep 10 20:59:17.204688 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:17.204581    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-757878586c-489dg" event={"ID":"9da9490c-a5bc-40b9-ae71-37fcb67f73ee","Type":"ContainerStarted","Data":"6ace1b8dbbdcdc27b0095fab3f1ffb16a357aad7078705475e2257880a664070"}
Sep 10 20:59:17.227916 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:17.227858    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bvh97" podStartSLOduration=35.662167824 podStartE2EDuration="37.22784437s" totalImagesPullingTime="1.565676546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:17.226670957 +0000 UTC m=+74.883579733" watchObservedRunningTime="2026-09-10 20:59:17.22784437 +0000 UTC m=+74.884753141"
Sep 10 20:59:17.253116 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:17.253052    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-757878586c-489dg" podStartSLOduration=2.253034061 podStartE2EDuration="2.253034061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:17.252522469 +0000 UTC m=+74.909431242" watchObservedRunningTime="2026-09-10 20:59:17.253034061 +0000 UTC m=+74.909942829"
Sep 10 20:59:17.431598 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:17.431496    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/895ac1f0-9420-486f-8a54-6de076e5ae32-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxgtk\" (UID: \"895ac1f0-9420-486f-8a54-6de076e5ae32\") " pod="openshift-insights/insights-runtime-extractor-sxgtk"
Sep 10 20:59:17.443636 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:17.443575    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/895ac1f0-9420-486f-8a54-6de076e5ae32-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sxgtk\" (UID: \"895ac1f0-9420-486f-8a54-6de076e5ae32\") " pod="openshift-insights/insights-runtime-extractor-sxgtk"
Sep 10 20:59:17.669827 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:17.669784    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k7jqh\""
Sep 10 20:59:17.677792 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:17.677766    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sxgtk"
Sep 10 20:59:18.000713 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:18.000638    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sxgtk"]
Sep 10 20:59:18.004204 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:59:18.004176    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod895ac1f0_9420_486f_8a54_6de076e5ae32.slice/crio-3c0287c35451b0865a0753ee466788b641879b2456b03f6a0b74cefeb4603167 WatchSource:0}: Error finding container 3c0287c35451b0865a0753ee466788b641879b2456b03f6a0b74cefeb4603167: Status 404 returned error can't find the container with id 3c0287c35451b0865a0753ee466788b641879b2456b03f6a0b74cefeb4603167
Sep 10 20:59:18.215384 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:18.215329    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sxgtk" event={"ID":"895ac1f0-9420-486f-8a54-6de076e5ae32","Type":"ContainerStarted","Data":"f4c5e7bc9d698b73c46fbedb1ee36f661e913c5cff06cb14116f8f60ae538a11"}
Sep 10 20:59:18.215807 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:18.215431    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sxgtk" event={"ID":"895ac1f0-9420-486f-8a54-6de076e5ae32","Type":"ContainerStarted","Data":"3c0287c35451b0865a0753ee466788b641879b2456b03f6a0b74cefeb4603167"}
Sep 10 20:59:18.217351 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:18.217314    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-9h26g" event={"ID":"1a92d9ed-9fff-4373-8513-be11d25b6a37","Type":"ContainerStarted","Data":"b705d54cb3c188464ff23c731347bc66f655670d3a5b3a1fb480d49f242249d3"}
Sep 10 20:59:18.217612 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:18.217590    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-757878586c-489dg"
Sep 10 20:59:19.223799 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:19.223765    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sxgtk" event={"ID":"895ac1f0-9420-486f-8a54-6de076e5ae32","Type":"ContainerStarted","Data":"e21f77095f3f5025d94f5314678cb3739b304892738eeadccdc7bac919187f88"}
Sep 10 20:59:21.233805 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:21.233762    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d77577fb-f5vr6" event={"ID":"2f9b5291-c0be-4959-a2b5-dfb539e576e6","Type":"ContainerStarted","Data":"2f282d10bf4d1fb6aef27ae456febf83778c9e688d4446a14b1a2cf102744292"}
Sep 10 20:59:21.234229 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:21.233987    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d77577fb-f5vr6"
Sep 10 20:59:21.235876 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:21.235855    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d77577fb-f5vr6"
Sep 10 20:59:21.252997 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:21.252950    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-9h26g" podStartSLOduration=4.957915027 podStartE2EDuration="6.252936092s" totalImagesPullingTime="1.295021065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:18.240133622 +0000 UTC m=+75.897042395" watchObservedRunningTime="2026-09-10 20:59:21.252936092 +0000 UTC m=+78.909844855"
Sep 10 20:59:21.253126 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:21.253025    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74d77577fb-f5vr6" podStartSLOduration=2.28167125 podStartE2EDuration="6.253021965s" totalImagesPullingTime="3.971350715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:21.251863398 +0000 UTC m=+78.908772171" watchObservedRunningTime="2026-09-10 20:59:21.253021965 +0000 UTC m=+78.909930736"
Sep 10 20:59:21.848626 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:21.848593    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-mjx9q"]
Sep 10 20:59:21.852825 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:21.852807    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-mjx9q"
Sep 10 20:59:21.856315 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:21.856279    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 10 20:59:21.856424 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:21.856289    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 10 20:59:21.856470 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:21.856288    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 20:59:21.856742 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:21.856726    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-b7xvj\""
Sep 10 20:59:21.865006 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:21.864986    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-mjx9q"]
Sep 10 20:59:21.972899 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:21.972844    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/18a79c16-9d5e-49e2-a4b4-d909f665c2b5-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-mjx9q\" (UID: \"18a79c16-9d5e-49e2-a4b4-d909f665c2b5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-mjx9q"
Sep 10 20:59:21.976162 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:21.976129    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqbqr\" (UniqueName: \"kubernetes.io/projected/18a79c16-9d5e-49e2-a4b4-d909f665c2b5-kube-api-access-tqbqr\") pod \"prometheus-operator-68bcd7f987-mjx9q\" (UID: \"18a79c16-9d5e-49e2-a4b4-d909f665c2b5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-mjx9q"
Sep 10 20:59:21.976327 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:21.976201    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/18a79c16-9d5e-49e2-a4b4-d909f665c2b5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-mjx9q\" (UID: \"18a79c16-9d5e-49e2-a4b4-d909f665c2b5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-mjx9q"
Sep 10 20:59:21.976327 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:21.976246    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18a79c16-9d5e-49e2-a4b4-d909f665c2b5-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-mjx9q\" (UID: \"18a79c16-9d5e-49e2-a4b4-d909f665c2b5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-mjx9q"
Sep 10 20:59:22.077403 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:22.077368    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18a79c16-9d5e-49e2-a4b4-d909f665c2b5-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-mjx9q\" (UID: \"18a79c16-9d5e-49e2-a4b4-d909f665c2b5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-mjx9q"
Sep 10 20:59:22.077543 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:22.077416    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/18a79c16-9d5e-49e2-a4b4-d909f665c2b5-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-mjx9q\" (UID: \"18a79c16-9d5e-49e2-a4b4-d909f665c2b5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-mjx9q"
Sep 10 20:59:22.077543 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:22.077456    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqbqr\" (UniqueName: \"kubernetes.io/projected/18a79c16-9d5e-49e2-a4b4-d909f665c2b5-kube-api-access-tqbqr\") pod \"prometheus-operator-68bcd7f987-mjx9q\" (UID: \"18a79c16-9d5e-49e2-a4b4-d909f665c2b5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-mjx9q"
Sep 10 20:59:22.077543 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:22.077486    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/18a79c16-9d5e-49e2-a4b4-d909f665c2b5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-mjx9q\" (UID: \"18a79c16-9d5e-49e2-a4b4-d909f665c2b5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-mjx9q"
Sep 10 20:59:22.078086 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:22.078059    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18a79c16-9d5e-49e2-a4b4-d909f665c2b5-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-mjx9q\" (UID: \"18a79c16-9d5e-49e2-a4b4-d909f665c2b5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-mjx9q"
Sep 10 20:59:22.079886 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:22.079865    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/18a79c16-9d5e-49e2-a4b4-d909f665c2b5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-mjx9q\" (UID: \"18a79c16-9d5e-49e2-a4b4-d909f665c2b5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-mjx9q"
Sep 10 20:59:22.080032 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:22.080012    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/18a79c16-9d5e-49e2-a4b4-d909f665c2b5-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-mjx9q\" (UID: \"18a79c16-9d5e-49e2-a4b4-d909f665c2b5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-mjx9q"
Sep 10 20:59:22.086987 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:22.086961    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqbqr\" (UniqueName: \"kubernetes.io/projected/18a79c16-9d5e-49e2-a4b4-d909f665c2b5-kube-api-access-tqbqr\") pod \"prometheus-operator-68bcd7f987-mjx9q\" (UID: \"18a79c16-9d5e-49e2-a4b4-d909f665c2b5\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-mjx9q"
Sep 10 20:59:22.161399 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:22.161322    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-mjx9q"
Sep 10 20:59:22.197138 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:22.197108    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wfsbr"
Sep 10 20:59:22.240554 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:22.240427    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sxgtk" event={"ID":"895ac1f0-9420-486f-8a54-6de076e5ae32","Type":"ContainerStarted","Data":"0fbed069dc0f37fb2226d84822486c0ae9979ea31d4f6a05803f3140b438385f"}
Sep 10 20:59:22.262325 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:22.262261    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-sxgtk" podStartSLOduration=33.947052899 podStartE2EDuration="37.262246657s" totalImagesPullingTime="3.315193758s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:58:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:22.261018523 +0000 UTC m=+79.917927297" watchObservedRunningTime="2026-09-10 20:59:22.262246657 +0000 UTC m=+79.919155428"
Sep 10 20:59:22.432996 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:22.432918    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-mjx9q"]
Sep 10 20:59:22.435780 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:59:22.435752    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18a79c16_9d5e_49e2_a4b4_d909f665c2b5.slice/crio-0c90c45a2707e7e92d22b00a21e5c3a22e4833283dfae99adbc376e53730736e WatchSource:0}: Error finding container 0c90c45a2707e7e92d22b00a21e5c3a22e4833283dfae99adbc376e53730736e: Status 404 returned error can't find the container with id 0c90c45a2707e7e92d22b00a21e5c3a22e4833283dfae99adbc376e53730736e
Sep 10 20:59:23.245854 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:23.245813    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-mjx9q" event={"ID":"18a79c16-9d5e-49e2-a4b4-d909f665c2b5","Type":"ContainerStarted","Data":"0c90c45a2707e7e92d22b00a21e5c3a22e4833283dfae99adbc376e53730736e"}
Sep 10 20:59:24.250733 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:24.250695    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-mjx9q" event={"ID":"18a79c16-9d5e-49e2-a4b4-d909f665c2b5","Type":"ContainerStarted","Data":"34edf0231cec4c0d24a46d5d222f77df81b58fb443dd0fe8a4b95b5d5a78527f"}
Sep 10 20:59:24.251196 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:24.250755    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-mjx9q" event={"ID":"18a79c16-9d5e-49e2-a4b4-d909f665c2b5","Type":"ContainerStarted","Data":"04d64f64a39d40e8de4db0402a3320401954e30bd30098e8cfe7f17313f99378"}
Sep 10 20:59:24.270575 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:24.270527    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-mjx9q" podStartSLOduration=2.093419503 podStartE2EDuration="3.270510915s" totalImagesPullingTime="1.177091412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:24.268940539 +0000 UTC m=+81.925849310" watchObservedRunningTime="2026-09-10 20:59:24.270510915 +0000 UTC m=+81.927419688"
Sep 10 20:59:26.267257 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.267129    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-t4vqh"]
Sep 10 20:59:26.272052 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.272028    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-t4vqh"
Sep 10 20:59:26.283353 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.283333    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-lng7b\""
Sep 10 20:59:26.284009 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.283989    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 10 20:59:26.284097 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.284041    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 10 20:59:26.296876 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.296850    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-t4vqh"]
Sep 10 20:59:26.328702 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.328669    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rp4kl"]
Sep 10 20:59:26.331907 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.331881    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rp4kl"
Sep 10 20:59:26.334555 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.334533    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 20:59:26.334647 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.334627    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 20:59:26.334754 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.334736    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jp262\""
Sep 10 20:59:26.335054 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.335039    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 20:59:26.412045 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.412013    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ed90097e-3c6b-476e-ad90-84a73b81a994-node-exporter-accelerators-collector-config\") pod \"node-exporter-rp4kl\" (UID: \"ed90097e-3c6b-476e-ad90-84a73b81a994\") " pod="openshift-monitoring/node-exporter-rp4kl"
Sep 10 20:59:26.412173 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.412050    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d3e7ffe4-fc2f-4782-a8dc-867199ee980c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-t4vqh\" (UID: \"d3e7ffe4-fc2f-4782-a8dc-867199ee980c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-t4vqh"
Sep 10 20:59:26.412173 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.412072    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ed90097e-3c6b-476e-ad90-84a73b81a994-node-exporter-wtmp\") pod \"node-exporter-rp4kl\" (UID: \"ed90097e-3c6b-476e-ad90-84a73b81a994\") " pod="openshift-monitoring/node-exporter-rp4kl"
Sep 10 20:59:26.412173 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.412104    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ed90097e-3c6b-476e-ad90-84a73b81a994-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rp4kl\" (UID: \"ed90097e-3c6b-476e-ad90-84a73b81a994\") " pod="openshift-monitoring/node-exporter-rp4kl"
Sep 10 20:59:26.412173 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.412157    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ed90097e-3c6b-476e-ad90-84a73b81a994-node-exporter-tls\") pod \"node-exporter-rp4kl\" (UID: \"ed90097e-3c6b-476e-ad90-84a73b81a994\") " pod="openshift-monitoring/node-exporter-rp4kl"
Sep 10 20:59:26.412351 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.412189    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ed90097e-3c6b-476e-ad90-84a73b81a994-root\") pod \"node-exporter-rp4kl\" (UID: \"ed90097e-3c6b-476e-ad90-84a73b81a994\") " pod="openshift-monitoring/node-exporter-rp4kl"
Sep 10 20:59:26.412351 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.412215    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3e7ffe4-fc2f-4782-a8dc-867199ee980c-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-t4vqh\" (UID: \"d3e7ffe4-fc2f-4782-a8dc-867199ee980c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-t4vqh"
Sep 10 20:59:26.412351 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.412259    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b44ft\" (UniqueName: \"kubernetes.io/projected/ed90097e-3c6b-476e-ad90-84a73b81a994-kube-api-access-b44ft\") pod \"node-exporter-rp4kl\" (UID: \"ed90097e-3c6b-476e-ad90-84a73b81a994\") " pod="openshift-monitoring/node-exporter-rp4kl"
Sep 10 20:59:26.412351 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.412279    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d3e7ffe4-fc2f-4782-a8dc-867199ee980c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-t4vqh\" (UID: \"d3e7ffe4-fc2f-4782-a8dc-867199ee980c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-t4vqh"
Sep 10 20:59:26.412351 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.412327    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxjlt\" (UniqueName: \"kubernetes.io/projected/d3e7ffe4-fc2f-4782-a8dc-867199ee980c-kube-api-access-lxjlt\") pod \"openshift-state-metrics-67dcf8c979-t4vqh\" (UID: \"d3e7ffe4-fc2f-4782-a8dc-867199ee980c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-t4vqh"
Sep 10 20:59:26.412351 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.412345    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed90097e-3c6b-476e-ad90-84a73b81a994-sys\") pod \"node-exporter-rp4kl\" (UID: \"ed90097e-3c6b-476e-ad90-84a73b81a994\") " pod="openshift-monitoring/node-exporter-rp4kl"
Sep 10 20:59:26.412544 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.412369    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ed90097e-3c6b-476e-ad90-84a73b81a994-node-exporter-textfile\") pod \"node-exporter-rp4kl\" (UID: \"ed90097e-3c6b-476e-ad90-84a73b81a994\") " pod="openshift-monitoring/node-exporter-rp4kl"
Sep 10 20:59:26.412544 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.412394    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed90097e-3c6b-476e-ad90-84a73b81a994-metrics-client-ca\") pod \"node-exporter-rp4kl\" (UID: \"ed90097e-3c6b-476e-ad90-84a73b81a994\") " pod="openshift-monitoring/node-exporter-rp4kl"
Sep 10 20:59:26.513153 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.513105    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3e7ffe4-fc2f-4782-a8dc-867199ee980c-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-t4vqh\" (UID: \"d3e7ffe4-fc2f-4782-a8dc-867199ee980c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-t4vqh"
Sep 10 20:59:26.513153 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.513153    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b44ft\" (UniqueName: \"kubernetes.io/projected/ed90097e-3c6b-476e-ad90-84a73b81a994-kube-api-access-b44ft\") pod \"node-exporter-rp4kl\" (UID: \"ed90097e-3c6b-476e-ad90-84a73b81a994\") " pod="openshift-monitoring/node-exporter-rp4kl"
Sep 10 20:59:26.513384 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.513184    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d3e7ffe4-fc2f-4782-a8dc-867199ee980c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-t4vqh\" (UID: \"d3e7ffe4-fc2f-4782-a8dc-867199ee980c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-t4vqh"
Sep 10 20:59:26.513384 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.513216    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxjlt\" (UniqueName: \"kubernetes.io/projected/d3e7ffe4-fc2f-4782-a8dc-867199ee980c-kube-api-access-lxjlt\") pod \"openshift-state-metrics-67dcf8c979-t4vqh\" (UID: \"d3e7ffe4-fc2f-4782-a8dc-867199ee980c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-t4vqh"
Sep 10 20:59:26.513384 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.513236    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed90097e-3c6b-476e-ad90-84a73b81a994-sys\") pod \"node-exporter-rp4kl\" (UID: \"ed90097e-3c6b-476e-ad90-84a73b81a994\") " pod="openshift-monitoring/node-exporter-rp4kl"
Sep 10 20:59:26.513384 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.513267    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ed90097e-3c6b-476e-ad90-84a73b81a994-node-exporter-textfile\") pod \"node-exporter-rp4kl\" (UID: \"ed90097e-3c6b-476e-ad90-84a73b81a994\") " pod="openshift-monitoring/node-exporter-rp4kl"
Sep 10 20:59:26.513384 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.513365    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed90097e-3c6b-476e-ad90-84a73b81a994-sys\") pod \"node-exporter-rp4kl\" (UID: \"ed90097e-3c6b-476e-ad90-84a73b81a994\") " pod="openshift-monitoring/node-exporter-rp4kl"
Sep 10 20:59:26.513618 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.513430    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed90097e-3c6b-476e-ad90-84a73b81a994-metrics-client-ca\") pod \"node-exporter-rp4kl\" (UID: \"ed90097e-3c6b-476e-ad90-84a73b81a994\") " pod="openshift-monitoring/node-exporter-rp4kl"
Sep 10 20:59:26.513618 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:59:26.513499    2581 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 10 20:59:26.513618 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:59:26.513574    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3e7ffe4-fc2f-4782-a8dc-867199ee980c-openshift-state-metrics-tls podName:d3e7ffe4-fc2f-4782-a8dc-867199ee980c nodeName:}" failed. No retries permitted until 2026-09-10 20:59:27.013553469 +0000 UTC m=+84.670462240 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/d3e7ffe4-fc2f-4782-a8dc-867199ee980c-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-t4vqh" (UID: "d3e7ffe4-fc2f-4782-a8dc-867199ee980c") : secret "openshift-state-metrics-tls" not found
Sep 10 20:59:26.513756 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.513503    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ed90097e-3c6b-476e-ad90-84a73b81a994-node-exporter-accelerators-collector-config\") pod \"node-exporter-rp4kl\" (UID: \"ed90097e-3c6b-476e-ad90-84a73b81a994\") " pod="openshift-monitoring/node-exporter-rp4kl"
Sep 10 20:59:26.513756 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.513643    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ed90097e-3c6b-476e-ad90-84a73b81a994-node-exporter-textfile\") pod \"node-exporter-rp4kl\" (UID: \"ed90097e-3c6b-476e-ad90-84a73b81a994\") " pod="openshift-monitoring/node-exporter-rp4kl"
Sep 10 20:59:26.513756 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.513678    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d3e7ffe4-fc2f-4782-a8dc-867199ee980c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-t4vqh\" (UID: \"d3e7ffe4-fc2f-4782-a8dc-867199ee980c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-t4vqh"
Sep 10 20:59:26.513756 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.513714    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ed90097e-3c6b-476e-ad90-84a73b81a994-node-exporter-wtmp\") pod \"node-exporter-rp4kl\" (UID: \"ed90097e-3c6b-476e-ad90-84a73b81a994\") " pod="openshift-monitoring/node-exporter-rp4kl"
Sep 10 20:59:26.513954 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.513761    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ed90097e-3c6b-476e-ad90-84a73b81a994-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rp4kl\" (UID: \"ed90097e-3c6b-476e-ad90-84a73b81a994\") " pod="openshift-monitoring/node-exporter-rp4kl"
Sep 10 20:59:26.513954 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.513817    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ed90097e-3c6b-476e-ad90-84a73b81a994-node-exporter-tls\") pod \"node-exporter-rp4kl\" (UID: \"ed90097e-3c6b-476e-ad90-84a73b81a994\") " pod="openshift-monitoring/node-exporter-rp4kl"
Sep 10 20:59:26.513954 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.513853    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ed90097e-3c6b-476e-ad90-84a73b81a994-root\") pod \"node-exporter-rp4kl\" (UID: \"ed90097e-3c6b-476e-ad90-84a73b81a994\") " pod="openshift-monitoring/node-exporter-rp4kl"
Sep 10 20:59:26.513954 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.513892    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ed90097e-3c6b-476e-ad90-84a73b81a994-node-exporter-wtmp\") pod \"node-exporter-rp4kl\" (UID: \"ed90097e-3c6b-476e-ad90-84a73b81a994\") " pod="openshift-monitoring/node-exporter-rp4kl"
Sep 10 20:59:26.514145 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.513954    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ed90097e-3c6b-476e-ad90-84a73b81a994-root\") pod \"node-exporter-rp4kl\" (UID: \"ed90097e-3c6b-476e-ad90-84a73b81a994\") " pod="openshift-monitoring/node-exporter-rp4kl"
Sep 10 20:59:26.514145 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:59:26.513981    2581 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 10 20:59:26.514145 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:59:26.514032    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed90097e-3c6b-476e-ad90-84a73b81a994-node-exporter-tls podName:ed90097e-3c6b-476e-ad90-84a73b81a994 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:27.014013975 +0000 UTC m=+84.670922728 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/ed90097e-3c6b-476e-ad90-84a73b81a994-node-exporter-tls") pod "node-exporter-rp4kl" (UID: "ed90097e-3c6b-476e-ad90-84a73b81a994") : secret "node-exporter-tls" not found
Sep 10 20:59:26.514145 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.514067    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3e7ffe4-fc2f-4782-a8dc-867199ee980c-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-t4vqh\" (UID: \"d3e7ffe4-fc2f-4782-a8dc-867199ee980c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-t4vqh"
Sep 10 20:59:26.514145 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.514106    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ed90097e-3c6b-476e-ad90-84a73b81a994-metrics-client-ca\") pod \"node-exporter-rp4kl\" (UID: \"ed90097e-3c6b-476e-ad90-84a73b81a994\") " pod="openshift-monitoring/node-exporter-rp4kl"
Sep 10 20:59:26.514347 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.514168    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ed90097e-3c6b-476e-ad90-84a73b81a994-node-exporter-accelerators-collector-config\") pod \"node-exporter-rp4kl\" (UID: \"ed90097e-3c6b-476e-ad90-84a73b81a994\") " pod="openshift-monitoring/node-exporter-rp4kl"
Sep 10 20:59:26.516094 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.516073    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ed90097e-3c6b-476e-ad90-84a73b81a994-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rp4kl\" (UID: \"ed90097e-3c6b-476e-ad90-84a73b81a994\") " pod="openshift-monitoring/node-exporter-rp4kl"
Sep 10 20:59:26.516173 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.516104    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d3e7ffe4-fc2f-4782-a8dc-867199ee980c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-t4vqh\" (UID: \"d3e7ffe4-fc2f-4782-a8dc-867199ee980c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-t4vqh"
Sep 10 20:59:26.536486 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.536429    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b44ft\" (UniqueName: \"kubernetes.io/projected/ed90097e-3c6b-476e-ad90-84a73b81a994-kube-api-access-b44ft\") pod \"node-exporter-rp4kl\" (UID: \"ed90097e-3c6b-476e-ad90-84a73b81a994\") " pod="openshift-monitoring/node-exporter-rp4kl"
Sep 10 20:59:26.550437 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:26.550415    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxjlt\" (UniqueName: \"kubernetes.io/projected/d3e7ffe4-fc2f-4782-a8dc-867199ee980c-kube-api-access-lxjlt\") pod \"openshift-state-metrics-67dcf8c979-t4vqh\" (UID: \"d3e7ffe4-fc2f-4782-a8dc-867199ee980c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-t4vqh"
Sep 10 20:59:27.017646 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:27.017612    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ed90097e-3c6b-476e-ad90-84a73b81a994-node-exporter-tls\") pod \"node-exporter-rp4kl\" (UID: \"ed90097e-3c6b-476e-ad90-84a73b81a994\") " pod="openshift-monitoring/node-exporter-rp4kl"
Sep 10 20:59:27.017793 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:27.017665    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d3e7ffe4-fc2f-4782-a8dc-867199ee980c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-t4vqh\" (UID: \"d3e7ffe4-fc2f-4782-a8dc-867199ee980c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-t4vqh"
Sep 10 20:59:27.019898 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:27.019865    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ed90097e-3c6b-476e-ad90-84a73b81a994-node-exporter-tls\") pod \"node-exporter-rp4kl\" (UID: \"ed90097e-3c6b-476e-ad90-84a73b81a994\") " pod="openshift-monitoring/node-exporter-rp4kl"
Sep 10 20:59:27.020146 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:27.020122    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d3e7ffe4-fc2f-4782-a8dc-867199ee980c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-t4vqh\" (UID: \"d3e7ffe4-fc2f-4782-a8dc-867199ee980c\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-t4vqh"
Sep 10 20:59:27.180785 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:27.180752    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-t4vqh"
Sep 10 20:59:27.241699 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:27.241666    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rp4kl"
Sep 10 20:59:27.258543 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:59:27.258478    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded90097e_3c6b_476e_ad90_84a73b81a994.slice/crio-55a1217b81a3d80d3e0a28dc24e9a51955bbda26a1d7503f87f7e71fd768f8a2 WatchSource:0}: Error finding container 55a1217b81a3d80d3e0a28dc24e9a51955bbda26a1d7503f87f7e71fd768f8a2: Status 404 returned error can't find the container with id 55a1217b81a3d80d3e0a28dc24e9a51955bbda26a1d7503f87f7e71fd768f8a2
Sep 10 20:59:27.462231 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:27.462199    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-t4vqh"]
Sep 10 20:59:27.465901 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:59:27.465872    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e7ffe4_fc2f_4782_a8dc_867199ee980c.slice/crio-bd707ce7e92c6576bf3e481ad9ed4cc4a27f9d9cf634c61e2590f7eb1efa7b34 WatchSource:0}: Error finding container bd707ce7e92c6576bf3e481ad9ed4cc4a27f9d9cf634c61e2590f7eb1efa7b34: Status 404 returned error can't find the container with id bd707ce7e92c6576bf3e481ad9ed4cc4a27f9d9cf634c61e2590f7eb1efa7b34
Sep 10 20:59:28.270240 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:28.270207    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rp4kl" event={"ID":"ed90097e-3c6b-476e-ad90-84a73b81a994","Type":"ContainerStarted","Data":"55a1217b81a3d80d3e0a28dc24e9a51955bbda26a1d7503f87f7e71fd768f8a2"}
Sep 10 20:59:28.272618 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:28.272595    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-t4vqh" event={"ID":"d3e7ffe4-fc2f-4782-a8dc-867199ee980c","Type":"ContainerStarted","Data":"716c110548bf3fe874c53ccabace3bb4b091f6528764310135a31409acb207a1"}
Sep 10 20:59:28.272728 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:28.272650    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-t4vqh" event={"ID":"d3e7ffe4-fc2f-4782-a8dc-867199ee980c","Type":"ContainerStarted","Data":"4a3b8fa819855c5067e2389b717e41979fa79d33174e2c9479be532b4a2ea83b"}
Sep 10 20:59:28.272728 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:28.272684    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-t4vqh" event={"ID":"d3e7ffe4-fc2f-4782-a8dc-867199ee980c","Type":"ContainerStarted","Data":"bd707ce7e92c6576bf3e481ad9ed4cc4a27f9d9cf634c61e2590f7eb1efa7b34"}
Sep 10 20:59:29.278221 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:29.278183    2581 generic.go:356] "Generic (PLEG): container finished" podID="ed90097e-3c6b-476e-ad90-84a73b81a994" containerID="799e4c55dbffe911b785d1c7dfdd81fa9406a44b5b30f56f6a67aed78d012b31" exitCode=0
Sep 10 20:59:29.278672 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:29.278278    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rp4kl" event={"ID":"ed90097e-3c6b-476e-ad90-84a73b81a994","Type":"ContainerDied","Data":"799e4c55dbffe911b785d1c7dfdd81fa9406a44b5b30f56f6a67aed78d012b31"}
Sep 10 20:59:29.280681 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:29.280655    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-t4vqh" event={"ID":"d3e7ffe4-fc2f-4782-a8dc-867199ee980c","Type":"ContainerStarted","Data":"dbe12905d8e4ecc1892e0f2e2036b05ee973de3a93960bbd9d78d6644083541b"}
Sep 10 20:59:29.326066 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:29.326020    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-t4vqh" podStartSLOduration=2.403781473 podStartE2EDuration="3.32600035s" totalImagesPullingTime="922.218877ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:29.324440074 +0000 UTC m=+86.981348848" watchObservedRunningTime="2026-09-10 20:59:29.32600035 +0000 UTC m=+86.982909122"
Sep 10 20:59:30.287197 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:30.287159    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rp4kl" event={"ID":"ed90097e-3c6b-476e-ad90-84a73b81a994","Type":"ContainerStarted","Data":"8bb277fe51f854ed1c5b7fa62cc43092e42bcfd87a231f550a6fa803f3bd402b"}
Sep 10 20:59:30.287676 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:30.287220    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rp4kl" event={"ID":"ed90097e-3c6b-476e-ad90-84a73b81a994","Type":"ContainerStarted","Data":"8dc44f98eec2fa0b060f11c3cd98caf7b9d935e08f7dbbb66540fa62bb36def0"}
Sep 10 20:59:30.332163 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:30.332116    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rp4kl" podStartSLOduration=3.282985706 podStartE2EDuration="4.332099941s" totalImagesPullingTime="981.578059ms" totalInitContainerRuntime="67.536176ms" isStatefulPod=true podCreationTimestamp="2026-09-10 20:59:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:30.331189309 +0000 UTC m=+87.988098082" watchObservedRunningTime="2026-09-10 20:59:30.332099941 +0000 UTC m=+87.989008712"
Sep 10 20:59:31.015641 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:31.015603    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-gf7q2"]
Sep 10 20:59:31.019009 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:31.018983    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-gf7q2"
Sep 10 20:59:31.021334 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:31.021287    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-jfrq5\""
Sep 10 20:59:31.021595 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:31.021577    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 10 20:59:31.034164 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:31.034142    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-gf7q2"]
Sep 10 20:59:31.151772 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:31.151740    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e546d5f4-6eb9-41d4-9244-46f8c9509f7e-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-gf7q2\" (UID: \"e546d5f4-6eb9-41d4-9244-46f8c9509f7e\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-gf7q2"
Sep 10 20:59:31.151951 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:31.151807    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx5nl\" (UniqueName: \"kubernetes.io/projected/e546d5f4-6eb9-41d4-9244-46f8c9509f7e-kube-api-access-bx5nl\") pod \"monitoring-plugin-5d85778b9d-gf7q2\" (UID: \"e546d5f4-6eb9-41d4-9244-46f8c9509f7e\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-gf7q2"
Sep 10 20:59:31.252274 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:31.252243    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx5nl\" (UniqueName: \"kubernetes.io/projected/e546d5f4-6eb9-41d4-9244-46f8c9509f7e-kube-api-access-bx5nl\") pod \"monitoring-plugin-5d85778b9d-gf7q2\" (UID: \"e546d5f4-6eb9-41d4-9244-46f8c9509f7e\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-gf7q2"
Sep 10 20:59:31.252431 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:31.252314    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e546d5f4-6eb9-41d4-9244-46f8c9509f7e-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-gf7q2\" (UID: \"e546d5f4-6eb9-41d4-9244-46f8c9509f7e\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-gf7q2"
Sep 10 20:59:31.252431 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:59:31.252401    2581 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 10 20:59:31.252508 ip-10-0-139-165 kubenswrapper[2581]: E0910 20:59:31.252453    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e546d5f4-6eb9-41d4-9244-46f8c9509f7e-monitoring-plugin-cert podName:e546d5f4-6eb9-41d4-9244-46f8c9509f7e nodeName:}" failed. No retries permitted until 2026-09-10 20:59:31.75243829 +0000 UTC m=+89.409347053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/e546d5f4-6eb9-41d4-9244-46f8c9509f7e-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-gf7q2" (UID: "e546d5f4-6eb9-41d4-9244-46f8c9509f7e") : secret "monitoring-plugin-cert" not found
Sep 10 20:59:31.269828 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:31.269773    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx5nl\" (UniqueName: \"kubernetes.io/projected/e546d5f4-6eb9-41d4-9244-46f8c9509f7e-kube-api-access-bx5nl\") pod \"monitoring-plugin-5d85778b9d-gf7q2\" (UID: \"e546d5f4-6eb9-41d4-9244-46f8c9509f7e\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-gf7q2"
Sep 10 20:59:31.756869 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:31.756834    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e546d5f4-6eb9-41d4-9244-46f8c9509f7e-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-gf7q2\" (UID: \"e546d5f4-6eb9-41d4-9244-46f8c9509f7e\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-gf7q2"
Sep 10 20:59:31.759274 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:31.759247    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e546d5f4-6eb9-41d4-9244-46f8c9509f7e-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-gf7q2\" (UID: \"e546d5f4-6eb9-41d4-9244-46f8c9509f7e\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-gf7q2"
Sep 10 20:59:31.928415 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:31.928382    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-gf7q2"
Sep 10 20:59:32.196644 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.196610    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-gf7q2"]
Sep 10 20:59:32.200525 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:59:32.200493    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode546d5f4_6eb9_41d4_9244_46f8c9509f7e.slice/crio-0a429294ef46d3152621d3c4c4d24f4369bb66d658081a59d7392185d0e17782 WatchSource:0}: Error finding container 0a429294ef46d3152621d3c4c4d24f4369bb66d658081a59d7392185d0e17782: Status 404 returned error can't find the container with id 0a429294ef46d3152621d3c4c4d24f4369bb66d658081a59d7392185d0e17782
Sep 10 20:59:32.296563 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.296519    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-gf7q2" event={"ID":"e546d5f4-6eb9-41d4-9244-46f8c9509f7e","Type":"ContainerStarted","Data":"0a429294ef46d3152621d3c4c4d24f4369bb66d658081a59d7392185d0e17782"}
Sep 10 20:59:32.459621 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.459540    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 20:59:32.470223 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.465414    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.471574 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.471549    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 10 20:59:32.472561 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.472479    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 10 20:59:32.472688 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.472626    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 10 20:59:32.472756 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.472713    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 10 20:59:32.472942 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.472905    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 10 20:59:32.474847 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.474826    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 10 20:59:32.475358 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.475341    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 10 20:59:32.475672 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.475651    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 10 20:59:32.476980 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.476925    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 10 20:59:32.476980 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.476967    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 10 20:59:32.477143 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.477078    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 10 20:59:32.477143 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.477126    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 10 20:59:32.477270 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.477204    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-90637mv87vj6u\""
Sep 10 20:59:32.477511 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.477479    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 10 20:59:32.477735 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.477712    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-h4wm8\""
Sep 10 20:59:32.477810 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.477753    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 10 20:59:32.478038 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.478015    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 10 20:59:32.489181 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.489157    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 20:59:32.562315 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.562263    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.562315 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.562314    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/04bd0140-0658-4be7-8926-37f09be530a0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.562547 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.562345    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/04bd0140-0658-4be7-8926-37f09be530a0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.562547 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.562365    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/04bd0140-0658-4be7-8926-37f09be530a0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.562547 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.562381    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-config\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.562547 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.562438    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/04bd0140-0658-4be7-8926-37f09be530a0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.562547 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.562478    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04bd0140-0658-4be7-8926-37f09be530a0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.562805 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.562563    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.562805 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.562591    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdc9j\" (UniqueName: \"kubernetes.io/projected/04bd0140-0658-4be7-8926-37f09be530a0-kube-api-access-qdc9j\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.562805 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.562618    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.562805 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.562666    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.562805 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.562692    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04bd0140-0658-4be7-8926-37f09be530a0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.562805 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.562717    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/04bd0140-0658-4be7-8926-37f09be530a0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.562805 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.562746    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.562805 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.562772    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.563153 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.562843    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.563153 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.562895    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/04bd0140-0658-4be7-8926-37f09be530a0-config-out\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.563153 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.562924    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04bd0140-0658-4be7-8926-37f09be530a0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.563153 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.562953    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04bd0140-0658-4be7-8926-37f09be530a0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.563153 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.563002    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-web-config\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.663565 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.663525    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-web-config\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.663742 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.663635    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.663742 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.663665    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/04bd0140-0658-4be7-8926-37f09be530a0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.663742 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.663711    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/04bd0140-0658-4be7-8926-37f09be530a0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.663742 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.663737    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/04bd0140-0658-4be7-8926-37f09be530a0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.663959 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.663772    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-config\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.663959 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.663804    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/04bd0140-0658-4be7-8926-37f09be530a0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.663959 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.663930    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04bd0140-0658-4be7-8926-37f09be530a0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.664113 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.663987    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.664113 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.664013    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdc9j\" (UniqueName: \"kubernetes.io/projected/04bd0140-0658-4be7-8926-37f09be530a0-kube-api-access-qdc9j\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.664113 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.664042    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.664113 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.664084    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.664319 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.664115    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04bd0140-0658-4be7-8926-37f09be530a0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.664319 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.664139    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/04bd0140-0658-4be7-8926-37f09be530a0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.664319 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.664170    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.664319 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.664200    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.664319 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.664240    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/04bd0140-0658-4be7-8926-37f09be530a0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.664319 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.664252    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.664631 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.664341    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/04bd0140-0658-4be7-8926-37f09be530a0-config-out\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.664631 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.664365    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04bd0140-0658-4be7-8926-37f09be530a0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.664631 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.664398    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04bd0140-0658-4be7-8926-37f09be530a0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.664781 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.664728    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/04bd0140-0658-4be7-8926-37f09be530a0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.666191 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.666105    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04bd0140-0658-4be7-8926-37f09be530a0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.666191 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.666563    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/04bd0140-0658-4be7-8926-37f09be530a0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.666191 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.666965    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-config\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.666191 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.666991    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04bd0140-0658-4be7-8926-37f09be530a0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.666191 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.667686    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04bd0140-0658-4be7-8926-37f09be530a0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.672755 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.669264    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04bd0140-0658-4be7-8926-37f09be530a0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.672755 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.671550    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.672755 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.672310    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.672755 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.672504    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-web-config\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.673179 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.673132    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.673648 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.673627    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/04bd0140-0658-4be7-8926-37f09be530a0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.673875 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.673839    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/04bd0140-0658-4be7-8926-37f09be530a0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.674088 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.674029    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.674424 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.674240    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.674424 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.674375    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.674677 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.674658    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.674875 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.674855    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/04bd0140-0658-4be7-8926-37f09be530a0-config-out\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.676722 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.676676    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdc9j\" (UniqueName: \"kubernetes.io/projected/04bd0140-0658-4be7-8926-37f09be530a0-kube-api-access-qdc9j\") pod \"prometheus-k8s-0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:32.782377 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:32.782291    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:33.089952 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:33.089822    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 20:59:33.347511 ip-10-0-139-165 kubenswrapper[2581]: W0910 20:59:33.347438    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04bd0140_0658_4be7_8926_37f09be530a0.slice/crio-a230ece3c308632efefa75882daee518f866ea63191e49492726f6654e71d233 WatchSource:0}: Error finding container a230ece3c308632efefa75882daee518f866ea63191e49492726f6654e71d233: Status 404 returned error can't find the container with id a230ece3c308632efefa75882daee518f866ea63191e49492726f6654e71d233
Sep 10 20:59:34.305311 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:34.305255    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"04bd0140-0658-4be7-8926-37f09be530a0","Type":"ContainerStarted","Data":"a230ece3c308632efefa75882daee518f866ea63191e49492726f6654e71d233"}
Sep 10 20:59:34.306980 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:34.306952    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-gf7q2" event={"ID":"e546d5f4-6eb9-41d4-9244-46f8c9509f7e","Type":"ContainerStarted","Data":"59ac5898fc8175393e50a4aec5ceead74d2fad03ca87c95385141afaf1024d74"}
Sep 10 20:59:34.307184 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:34.307164    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-gf7q2"
Sep 10 20:59:34.312147 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:34.312126    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-gf7q2"
Sep 10 20:59:34.338756 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:34.338717    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-gf7q2" podStartSLOduration=3.148559868 podStartE2EDuration="4.338702919s" totalImagesPullingTime="1.190143051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:34.336992865 +0000 UTC m=+91.993901634" watchObservedRunningTime="2026-09-10 20:59:34.338702919 +0000 UTC m=+91.995611749"
Sep 10 20:59:35.312278 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:35.312246    2581 generic.go:356] "Generic (PLEG): container finished" podID="04bd0140-0658-4be7-8926-37f09be530a0" containerID="1163069b102c08afcd73d5bc822af82a20cf80e3153015454ddbd74aec3f1ce3" exitCode=0
Sep 10 20:59:35.312682 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:35.312335    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"04bd0140-0658-4be7-8926-37f09be530a0","Type":"ContainerDied","Data":"1163069b102c08afcd73d5bc822af82a20cf80e3153015454ddbd74aec3f1ce3"}
Sep 10 20:59:39.228264 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:39.228235    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-757878586c-489dg"
Sep 10 20:59:39.334503 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:39.334448    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"04bd0140-0658-4be7-8926-37f09be530a0","Type":"ContainerStarted","Data":"3ce0884513d450db9e98b3958c201dc405e74cc6c6cbbb52928b624314d64476"}
Sep 10 20:59:39.334662 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:39.334550    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"04bd0140-0658-4be7-8926-37f09be530a0","Type":"ContainerStarted","Data":"76bda818f36504aa7858472b3bd5cb7117593f3c3f83e98bf1d284eb5306bc69"}
Sep 10 20:59:40.342129 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:40.342092    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"04bd0140-0658-4be7-8926-37f09be530a0","Type":"ContainerStarted","Data":"5d3e882227d80ffcdb90f6abdd2b553f8d9162ac382d7ab2d2c3ed6163a9982d"}
Sep 10 20:59:40.342604 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:40.342169    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"04bd0140-0658-4be7-8926-37f09be530a0","Type":"ContainerStarted","Data":"574bc64faec8824e6672618e4701aad31977c5832d24443deaa9f541f778782e"}
Sep 10 20:59:40.342604 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:40.342213    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"04bd0140-0658-4be7-8926-37f09be530a0","Type":"ContainerStarted","Data":"23f69eecd3716406612d91e9cb3ec918e7896b123c070110ed9fdc8f9ac69f74"}
Sep 10 20:59:40.342604 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:40.342260    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"04bd0140-0658-4be7-8926-37f09be530a0","Type":"ContainerStarted","Data":"a0952dd7e75290a421ac7adda78c7d8d513fe76191464169b31682c2d1e14ee1"}
Sep 10 20:59:40.387372 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:40.387329    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.856282637 podStartE2EDuration="8.387291961s" totalImagesPullingTime="5.513163312s" totalInitContainerRuntime="17.846012ms" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:40.386876285 +0000 UTC m=+98.043785057" watchObservedRunningTime="2026-09-10 20:59:40.387291961 +0000 UTC m=+98.044200733"
Sep 10 20:59:42.783195 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:42.783164    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 20:59:46.196755 ip-10-0-139-165 kubenswrapper[2581]: I0910 20:59:46.196725    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gvw9h"
Sep 10 21:00:15.477829 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:15.477795    2581 generic.go:356] "Generic (PLEG): container finished" podID="08f9e62b-ae0b-426a-a306-39f8c4fdb7ff" containerID="2cfa69169e703ebf8dbd2f80089ffc453ec5f610c52bc1c205167059319841a9" exitCode=0
Sep 10 21:00:15.478241 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:15.477876    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-9d2gj" event={"ID":"08f9e62b-ae0b-426a-a306-39f8c4fdb7ff","Type":"ContainerDied","Data":"2cfa69169e703ebf8dbd2f80089ffc453ec5f610c52bc1c205167059319841a9"}
Sep 10 21:00:15.478241 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:15.478206    2581 scope.go:117] "RemoveContainer" containerID="2cfa69169e703ebf8dbd2f80089ffc453ec5f610c52bc1c205167059319841a9"
Sep 10 21:00:16.483017 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:16.482979    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-9d2gj" event={"ID":"08f9e62b-ae0b-426a-a306-39f8c4fdb7ff","Type":"ContainerStarted","Data":"6ac47f5f2c612020261d0b240778d5ff99d0f09589de400f269f172fe9fd0760"}
Sep 10 21:00:21.503384 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:21.503348    2581 generic.go:356] "Generic (PLEG): container finished" podID="ad941378-2761-47a0-9f64-6201c964276c" containerID="5cccc18c27495ee96d1fa8e42cac16b671749f29847136538e78bf9c8a0a9ef7" exitCode=0
Sep 10 21:00:21.503791 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:21.503424    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vqc26" event={"ID":"ad941378-2761-47a0-9f64-6201c964276c","Type":"ContainerDied","Data":"5cccc18c27495ee96d1fa8e42cac16b671749f29847136538e78bf9c8a0a9ef7"}
Sep 10 21:00:21.503791 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:21.503713    2581 scope.go:117] "RemoveContainer" containerID="5cccc18c27495ee96d1fa8e42cac16b671749f29847136538e78bf9c8a0a9ef7"
Sep 10 21:00:22.509367 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:22.509330    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-vqc26" event={"ID":"ad941378-2761-47a0-9f64-6201c964276c","Type":"ContainerStarted","Data":"33d11ccca7526c69ba28d3a782970d2595fb8aee93656d14aebdd4ecd2b660ae"}
Sep 10 21:00:32.782893 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:32.782849    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:32.798420 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:32.798396    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:33.570740 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:33.570714    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:50.968452 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:50.968414    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 21:00:50.968898 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:50.968693    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="04bd0140-0658-4be7-8926-37f09be530a0" containerName="prometheus" containerID="cri-o://3ce0884513d450db9e98b3958c201dc405e74cc6c6cbbb52928b624314d64476" gracePeriod=600
Sep 10 21:00:50.968898 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:50.968711    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="04bd0140-0658-4be7-8926-37f09be530a0" containerName="kube-rbac-proxy" containerID="cri-o://574bc64faec8824e6672618e4701aad31977c5832d24443deaa9f541f778782e" gracePeriod=600
Sep 10 21:00:50.968898 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:50.968788    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="04bd0140-0658-4be7-8926-37f09be530a0" containerName="thanos-sidecar" containerID="cri-o://a0952dd7e75290a421ac7adda78c7d8d513fe76191464169b31682c2d1e14ee1" gracePeriod=600
Sep 10 21:00:50.968898 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:50.968820    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="04bd0140-0658-4be7-8926-37f09be530a0" containerName="kube-rbac-proxy-web" containerID="cri-o://23f69eecd3716406612d91e9cb3ec918e7896b123c070110ed9fdc8f9ac69f74" gracePeriod=600
Sep 10 21:00:50.968898 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:50.968765    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="04bd0140-0658-4be7-8926-37f09be530a0" containerName="kube-rbac-proxy-thanos" containerID="cri-o://5d3e882227d80ffcdb90f6abdd2b553f8d9162ac382d7ab2d2c3ed6163a9982d" gracePeriod=600
Sep 10 21:00:50.968898 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:50.968780    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="04bd0140-0658-4be7-8926-37f09be530a0" containerName="config-reloader" containerID="cri-o://76bda818f36504aa7858472b3bd5cb7117593f3c3f83e98bf1d284eb5306bc69" gracePeriod=600
Sep 10 21:00:51.627930 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:51.627899    2581 generic.go:356] "Generic (PLEG): container finished" podID="04bd0140-0658-4be7-8926-37f09be530a0" containerID="5d3e882227d80ffcdb90f6abdd2b553f8d9162ac382d7ab2d2c3ed6163a9982d" exitCode=0
Sep 10 21:00:51.627930 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:51.627923    2581 generic.go:356] "Generic (PLEG): container finished" podID="04bd0140-0658-4be7-8926-37f09be530a0" containerID="574bc64faec8824e6672618e4701aad31977c5832d24443deaa9f541f778782e" exitCode=0
Sep 10 21:00:51.627930 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:51.627931    2581 generic.go:356] "Generic (PLEG): container finished" podID="04bd0140-0658-4be7-8926-37f09be530a0" containerID="a0952dd7e75290a421ac7adda78c7d8d513fe76191464169b31682c2d1e14ee1" exitCode=0
Sep 10 21:00:51.628229 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:51.627941    2581 generic.go:356] "Generic (PLEG): container finished" podID="04bd0140-0658-4be7-8926-37f09be530a0" containerID="76bda818f36504aa7858472b3bd5cb7117593f3c3f83e98bf1d284eb5306bc69" exitCode=0
Sep 10 21:00:51.628229 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:51.627949    2581 generic.go:356] "Generic (PLEG): container finished" podID="04bd0140-0658-4be7-8926-37f09be530a0" containerID="3ce0884513d450db9e98b3958c201dc405e74cc6c6cbbb52928b624314d64476" exitCode=0
Sep 10 21:00:51.628229 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:51.627968    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"04bd0140-0658-4be7-8926-37f09be530a0","Type":"ContainerDied","Data":"5d3e882227d80ffcdb90f6abdd2b553f8d9162ac382d7ab2d2c3ed6163a9982d"}
Sep 10 21:00:51.628229 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:51.628029    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"04bd0140-0658-4be7-8926-37f09be530a0","Type":"ContainerDied","Data":"574bc64faec8824e6672618e4701aad31977c5832d24443deaa9f541f778782e"}
Sep 10 21:00:51.628229 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:51.628057    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"04bd0140-0658-4be7-8926-37f09be530a0","Type":"ContainerDied","Data":"a0952dd7e75290a421ac7adda78c7d8d513fe76191464169b31682c2d1e14ee1"}
Sep 10 21:00:51.628229 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:51.628081    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"04bd0140-0658-4be7-8926-37f09be530a0","Type":"ContainerDied","Data":"76bda818f36504aa7858472b3bd5cb7117593f3c3f83e98bf1d284eb5306bc69"}
Sep 10 21:00:51.628229 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:51.628107    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"04bd0140-0658-4be7-8926-37f09be530a0","Type":"ContainerDied","Data":"3ce0884513d450db9e98b3958c201dc405e74cc6c6cbbb52928b624314d64476"}
Sep 10 21:00:52.213179 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.213153    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.242699 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.242672    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04bd0140-0658-4be7-8926-37f09be530a0-configmap-kubelet-serving-ca-bundle\") pod \"04bd0140-0658-4be7-8926-37f09be530a0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") "
Sep 10 21:00:52.242817 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.242709    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-secret-kube-rbac-proxy\") pod \"04bd0140-0658-4be7-8926-37f09be530a0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") "
Sep 10 21:00:52.242817 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.242736    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"04bd0140-0658-4be7-8926-37f09be530a0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") "
Sep 10 21:00:52.242893 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.242856    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-secret-grpc-tls\") pod \"04bd0140-0658-4be7-8926-37f09be530a0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") "
Sep 10 21:00:52.242927 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.242892    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-secret-prometheus-k8s-tls\") pod \"04bd0140-0658-4be7-8926-37f09be530a0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") "
Sep 10 21:00:52.242927 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.242921    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04bd0140-0658-4be7-8926-37f09be530a0-configmap-metrics-client-ca\") pod \"04bd0140-0658-4be7-8926-37f09be530a0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") "
Sep 10 21:00:52.243022 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.242949    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"04bd0140-0658-4be7-8926-37f09be530a0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") "
Sep 10 21:00:52.243022 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.242976    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04bd0140-0658-4be7-8926-37f09be530a0-prometheus-trusted-ca-bundle\") pod \"04bd0140-0658-4be7-8926-37f09be530a0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") "
Sep 10 21:00:52.243022 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.243014    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/04bd0140-0658-4be7-8926-37f09be530a0-prometheus-k8s-rulefiles-0\") pod \"04bd0140-0658-4be7-8926-37f09be530a0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") "
Sep 10 21:00:52.243168 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.243042    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/04bd0140-0658-4be7-8926-37f09be530a0-config-out\") pod \"04bd0140-0658-4be7-8926-37f09be530a0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") "
Sep 10 21:00:52.243168 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.243088    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-thanos-prometheus-http-client-file\") pod \"04bd0140-0658-4be7-8926-37f09be530a0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") "
Sep 10 21:00:52.243168 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.243113    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04bd0140-0658-4be7-8926-37f09be530a0-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "04bd0140-0658-4be7-8926-37f09be530a0" (UID: "04bd0140-0658-4be7-8926-37f09be530a0"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:00:52.243168 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.243125    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/04bd0140-0658-4be7-8926-37f09be530a0-prometheus-k8s-rulefiles-1\") pod \"04bd0140-0658-4be7-8926-37f09be530a0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") "
Sep 10 21:00:52.243426 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.243258    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-secret-metrics-client-certs\") pod \"04bd0140-0658-4be7-8926-37f09be530a0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") "
Sep 10 21:00:52.243426 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.243287    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdc9j\" (UniqueName: \"kubernetes.io/projected/04bd0140-0658-4be7-8926-37f09be530a0-kube-api-access-qdc9j\") pod \"04bd0140-0658-4be7-8926-37f09be530a0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") "
Sep 10 21:00:52.243426 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.243345    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-web-config\") pod \"04bd0140-0658-4be7-8926-37f09be530a0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") "
Sep 10 21:00:52.243426 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.243379    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/04bd0140-0658-4be7-8926-37f09be530a0-prometheus-k8s-db\") pod \"04bd0140-0658-4be7-8926-37f09be530a0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") "
Sep 10 21:00:52.243426 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.243378    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04bd0140-0658-4be7-8926-37f09be530a0-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "04bd0140-0658-4be7-8926-37f09be530a0" (UID: "04bd0140-0658-4be7-8926-37f09be530a0"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:00:52.243426 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.243402    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/04bd0140-0658-4be7-8926-37f09be530a0-tls-assets\") pod \"04bd0140-0658-4be7-8926-37f09be530a0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") "
Sep 10 21:00:52.243724 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.243433    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04bd0140-0658-4be7-8926-37f09be530a0-configmap-serving-certs-ca-bundle\") pod \"04bd0140-0658-4be7-8926-37f09be530a0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") "
Sep 10 21:00:52.243724 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.243464    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-config\") pod \"04bd0140-0658-4be7-8926-37f09be530a0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") "
Sep 10 21:00:52.243724 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.243516    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/04bd0140-0658-4be7-8926-37f09be530a0-prometheus-k8s-rulefiles-2\") pod \"04bd0140-0658-4be7-8926-37f09be530a0\" (UID: \"04bd0140-0658-4be7-8926-37f09be530a0\") "
Sep 10 21:00:52.243724 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.243686    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04bd0140-0658-4be7-8926-37f09be530a0-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "04bd0140-0658-4be7-8926-37f09be530a0" (UID: "04bd0140-0658-4be7-8926-37f09be530a0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:00:52.243934 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.243803    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/04bd0140-0658-4be7-8926-37f09be530a0-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:00:52.243934 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.243824    2581 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04bd0140-0658-4be7-8926-37f09be530a0-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:00:52.243934 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.243837    2581 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04bd0140-0658-4be7-8926-37f09be530a0-configmap-metrics-client-ca\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:00:52.244095 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.244017    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04bd0140-0658-4be7-8926-37f09be530a0-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "04bd0140-0658-4be7-8926-37f09be530a0" (UID: "04bd0140-0658-4be7-8926-37f09be530a0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:00:52.244631 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.244417    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04bd0140-0658-4be7-8926-37f09be530a0-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "04bd0140-0658-4be7-8926-37f09be530a0" (UID: "04bd0140-0658-4be7-8926-37f09be530a0"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:00:52.244746 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.244687    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04bd0140-0658-4be7-8926-37f09be530a0-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "04bd0140-0658-4be7-8926-37f09be530a0" (UID: "04bd0140-0658-4be7-8926-37f09be530a0"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:00:52.245835 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.245714    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04bd0140-0658-4be7-8926-37f09be530a0-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "04bd0140-0658-4be7-8926-37f09be530a0" (UID: "04bd0140-0658-4be7-8926-37f09be530a0"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:00:52.245835 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.245789    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "04bd0140-0658-4be7-8926-37f09be530a0" (UID: "04bd0140-0658-4be7-8926-37f09be530a0"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:00:52.245835 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.245813    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "04bd0140-0658-4be7-8926-37f09be530a0" (UID: "04bd0140-0658-4be7-8926-37f09be530a0"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:00:52.246817 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.246778    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "04bd0140-0658-4be7-8926-37f09be530a0" (UID: "04bd0140-0658-4be7-8926-37f09be530a0"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:00:52.246915 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.246831    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04bd0140-0658-4be7-8926-37f09be530a0-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "04bd0140-0658-4be7-8926-37f09be530a0" (UID: "04bd0140-0658-4be7-8926-37f09be530a0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:00:52.246980 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.246915    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "04bd0140-0658-4be7-8926-37f09be530a0" (UID: "04bd0140-0658-4be7-8926-37f09be530a0"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:00:52.247287 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.247257    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04bd0140-0658-4be7-8926-37f09be530a0-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "04bd0140-0658-4be7-8926-37f09be530a0" (UID: "04bd0140-0658-4be7-8926-37f09be530a0"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:00:52.247902 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.247857    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04bd0140-0658-4be7-8926-37f09be530a0-kube-api-access-qdc9j" (OuterVolumeSpecName: "kube-api-access-qdc9j") pod "04bd0140-0658-4be7-8926-37f09be530a0" (UID: "04bd0140-0658-4be7-8926-37f09be530a0"). InnerVolumeSpecName "kube-api-access-qdc9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:00:52.248028 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.248005    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "04bd0140-0658-4be7-8926-37f09be530a0" (UID: "04bd0140-0658-4be7-8926-37f09be530a0"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:00:52.248134 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.248110    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-config" (OuterVolumeSpecName: "config") pod "04bd0140-0658-4be7-8926-37f09be530a0" (UID: "04bd0140-0658-4be7-8926-37f09be530a0"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:00:52.248355 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.248331    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "04bd0140-0658-4be7-8926-37f09be530a0" (UID: "04bd0140-0658-4be7-8926-37f09be530a0"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:00:52.248647 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.248629    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04bd0140-0658-4be7-8926-37f09be530a0-config-out" (OuterVolumeSpecName: "config-out") pod "04bd0140-0658-4be7-8926-37f09be530a0" (UID: "04bd0140-0658-4be7-8926-37f09be530a0"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:00:52.249463 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.249445    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "04bd0140-0658-4be7-8926-37f09be530a0" (UID: "04bd0140-0658-4be7-8926-37f09be530a0"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:00:52.258606 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.258578    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-web-config" (OuterVolumeSpecName: "web-config") pod "04bd0140-0658-4be7-8926-37f09be530a0" (UID: "04bd0140-0658-4be7-8926-37f09be530a0"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:00:52.345177 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.345096    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/04bd0140-0658-4be7-8926-37f09be530a0-prometheus-k8s-db\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:00:52.345177 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.345126    2581 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/04bd0140-0658-4be7-8926-37f09be530a0-tls-assets\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:00:52.345177 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.345139    2581 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04bd0140-0658-4be7-8926-37f09be530a0-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:00:52.345177 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.345150    2581 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-config\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:00:52.345177 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.345160    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/04bd0140-0658-4be7-8926-37f09be530a0-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:00:52.345177 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.345168    2581 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-secret-kube-rbac-proxy\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:00:52.345177 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.345177    2581 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:00:52.345516 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.345186    2581 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-secret-grpc-tls\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:00:52.345516 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.345194    2581 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-secret-prometheus-k8s-tls\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:00:52.345516 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.345202    2581 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:00:52.345516 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.345212    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/04bd0140-0658-4be7-8926-37f09be530a0-prometheus-trusted-ca-bundle\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:00:52.345516 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.345220    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/04bd0140-0658-4be7-8926-37f09be530a0-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:00:52.345516 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.345228    2581 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/04bd0140-0658-4be7-8926-37f09be530a0-config-out\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:00:52.345516 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.345236    2581 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:00:52.345516 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.345244    2581 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-secret-metrics-client-certs\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:00:52.345516 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.345253    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdc9j\" (UniqueName: \"kubernetes.io/projected/04bd0140-0658-4be7-8926-37f09be530a0-kube-api-access-qdc9j\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:00:52.345516 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.345262    2581 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/04bd0140-0658-4be7-8926-37f09be530a0-web-config\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:00:52.640220 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.640121    2581 generic.go:356] "Generic (PLEG): container finished" podID="04bd0140-0658-4be7-8926-37f09be530a0" containerID="23f69eecd3716406612d91e9cb3ec918e7896b123c070110ed9fdc8f9ac69f74" exitCode=0
Sep 10 21:00:52.640380 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.640209    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"04bd0140-0658-4be7-8926-37f09be530a0","Type":"ContainerDied","Data":"23f69eecd3716406612d91e9cb3ec918e7896b123c070110ed9fdc8f9ac69f74"}
Sep 10 21:00:52.640380 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.640230    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 21:00:52.640380 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.640311    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"04bd0140-0658-4be7-8926-37f09be530a0","Type":"ContainerDied","Data":"a230ece3c308632efefa75882daee518f866ea63191e49492726f6654e71d233"}
Sep 10 21:00:52.640380 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.640343    2581 scope.go:117] "RemoveContainer" containerID="5d3e882227d80ffcdb90f6abdd2b553f8d9162ac382d7ab2d2c3ed6163a9982d"
Sep 10 21:00:52.648970 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.648750    2581 scope.go:117] "RemoveContainer" containerID="574bc64faec8824e6672618e4701aad31977c5832d24443deaa9f541f778782e"
Sep 10 21:00:52.656519 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.656501    2581 scope.go:117] "RemoveContainer" containerID="23f69eecd3716406612d91e9cb3ec918e7896b123c070110ed9fdc8f9ac69f74"
Sep 10 21:00:52.663332 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.663311    2581 scope.go:117] "RemoveContainer" containerID="a0952dd7e75290a421ac7adda78c7d8d513fe76191464169b31682c2d1e14ee1"
Sep 10 21:00:52.666630 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.666466    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 21:00:52.671248 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.671224    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 21:00:52.671577 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.671564    2581 scope.go:117] "RemoveContainer" containerID="76bda818f36504aa7858472b3bd5cb7117593f3c3f83e98bf1d284eb5306bc69"
Sep 10 21:00:52.679901 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.679885    2581 scope.go:117] "RemoveContainer" containerID="3ce0884513d450db9e98b3958c201dc405e74cc6c6cbbb52928b624314d64476"
Sep 10 21:00:52.686932 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.686907    2581 scope.go:117] "RemoveContainer" containerID="1163069b102c08afcd73d5bc822af82a20cf80e3153015454ddbd74aec3f1ce3"
Sep 10 21:00:52.693588 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.693567    2581 scope.go:117] "RemoveContainer" containerID="5d3e882227d80ffcdb90f6abdd2b553f8d9162ac382d7ab2d2c3ed6163a9982d"
Sep 10 21:00:52.693865 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:00:52.693847    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d3e882227d80ffcdb90f6abdd2b553f8d9162ac382d7ab2d2c3ed6163a9982d\": container with ID starting with 5d3e882227d80ffcdb90f6abdd2b553f8d9162ac382d7ab2d2c3ed6163a9982d not found: ID does not exist" containerID="5d3e882227d80ffcdb90f6abdd2b553f8d9162ac382d7ab2d2c3ed6163a9982d"
Sep 10 21:00:52.693911 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.693873    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d3e882227d80ffcdb90f6abdd2b553f8d9162ac382d7ab2d2c3ed6163a9982d"} err="failed to get container status \"5d3e882227d80ffcdb90f6abdd2b553f8d9162ac382d7ab2d2c3ed6163a9982d\": rpc error: code = NotFound desc = could not find container \"5d3e882227d80ffcdb90f6abdd2b553f8d9162ac382d7ab2d2c3ed6163a9982d\": container with ID starting with 5d3e882227d80ffcdb90f6abdd2b553f8d9162ac382d7ab2d2c3ed6163a9982d not found: ID does not exist"
Sep 10 21:00:52.693911 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.693906    2581 scope.go:117] "RemoveContainer" containerID="574bc64faec8824e6672618e4701aad31977c5832d24443deaa9f541f778782e"
Sep 10 21:00:52.694173 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:00:52.694154    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"574bc64faec8824e6672618e4701aad31977c5832d24443deaa9f541f778782e\": container with ID starting with 574bc64faec8824e6672618e4701aad31977c5832d24443deaa9f541f778782e not found: ID does not exist" containerID="574bc64faec8824e6672618e4701aad31977c5832d24443deaa9f541f778782e"
Sep 10 21:00:52.694231 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.694179    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"574bc64faec8824e6672618e4701aad31977c5832d24443deaa9f541f778782e"} err="failed to get container status \"574bc64faec8824e6672618e4701aad31977c5832d24443deaa9f541f778782e\": rpc error: code = NotFound desc = could not find container \"574bc64faec8824e6672618e4701aad31977c5832d24443deaa9f541f778782e\": container with ID starting with 574bc64faec8824e6672618e4701aad31977c5832d24443deaa9f541f778782e not found: ID does not exist"
Sep 10 21:00:52.694231 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.694195    2581 scope.go:117] "RemoveContainer" containerID="23f69eecd3716406612d91e9cb3ec918e7896b123c070110ed9fdc8f9ac69f74"
Sep 10 21:00:52.694457 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:00:52.694441    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23f69eecd3716406612d91e9cb3ec918e7896b123c070110ed9fdc8f9ac69f74\": container with ID starting with 23f69eecd3716406612d91e9cb3ec918e7896b123c070110ed9fdc8f9ac69f74 not found: ID does not exist" containerID="23f69eecd3716406612d91e9cb3ec918e7896b123c070110ed9fdc8f9ac69f74"
Sep 10 21:00:52.694500 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.694462    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23f69eecd3716406612d91e9cb3ec918e7896b123c070110ed9fdc8f9ac69f74"} err="failed to get container status \"23f69eecd3716406612d91e9cb3ec918e7896b123c070110ed9fdc8f9ac69f74\": rpc error: code = NotFound desc = could not find container \"23f69eecd3716406612d91e9cb3ec918e7896b123c070110ed9fdc8f9ac69f74\": container with ID starting with 23f69eecd3716406612d91e9cb3ec918e7896b123c070110ed9fdc8f9ac69f74 not found: ID does not exist"
Sep 10 21:00:52.694500 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.694476    2581 scope.go:117] "RemoveContainer" containerID="a0952dd7e75290a421ac7adda78c7d8d513fe76191464169b31682c2d1e14ee1"
Sep 10 21:00:52.694717 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:00:52.694700    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0952dd7e75290a421ac7adda78c7d8d513fe76191464169b31682c2d1e14ee1\": container with ID starting with a0952dd7e75290a421ac7adda78c7d8d513fe76191464169b31682c2d1e14ee1 not found: ID does not exist" containerID="a0952dd7e75290a421ac7adda78c7d8d513fe76191464169b31682c2d1e14ee1"
Sep 10 21:00:52.694756 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.694722    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0952dd7e75290a421ac7adda78c7d8d513fe76191464169b31682c2d1e14ee1"} err="failed to get container status \"a0952dd7e75290a421ac7adda78c7d8d513fe76191464169b31682c2d1e14ee1\": rpc error: code = NotFound desc = could not find container \"a0952dd7e75290a421ac7adda78c7d8d513fe76191464169b31682c2d1e14ee1\": container with ID starting with a0952dd7e75290a421ac7adda78c7d8d513fe76191464169b31682c2d1e14ee1 not found: ID does not exist"
Sep 10 21:00:52.694756 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.694736    2581 scope.go:117] "RemoveContainer" containerID="76bda818f36504aa7858472b3bd5cb7117593f3c3f83e98bf1d284eb5306bc69"
Sep 10 21:00:52.694964 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:00:52.694949    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76bda818f36504aa7858472b3bd5cb7117593f3c3f83e98bf1d284eb5306bc69\": container with ID starting with 76bda818f36504aa7858472b3bd5cb7117593f3c3f83e98bf1d284eb5306bc69 not found: ID does not exist" containerID="76bda818f36504aa7858472b3bd5cb7117593f3c3f83e98bf1d284eb5306bc69"
Sep 10 21:00:52.695004 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.694968    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76bda818f36504aa7858472b3bd5cb7117593f3c3f83e98bf1d284eb5306bc69"} err="failed to get container status \"76bda818f36504aa7858472b3bd5cb7117593f3c3f83e98bf1d284eb5306bc69\": rpc error: code = NotFound desc = could not find container \"76bda818f36504aa7858472b3bd5cb7117593f3c3f83e98bf1d284eb5306bc69\": container with ID starting with 76bda818f36504aa7858472b3bd5cb7117593f3c3f83e98bf1d284eb5306bc69 not found: ID does not exist"
Sep 10 21:00:52.695004 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.694982    2581 scope.go:117] "RemoveContainer" containerID="3ce0884513d450db9e98b3958c201dc405e74cc6c6cbbb52928b624314d64476"
Sep 10 21:00:52.695177 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:00:52.695162    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ce0884513d450db9e98b3958c201dc405e74cc6c6cbbb52928b624314d64476\": container with ID starting with 3ce0884513d450db9e98b3958c201dc405e74cc6c6cbbb52928b624314d64476 not found: ID does not exist" containerID="3ce0884513d450db9e98b3958c201dc405e74cc6c6cbbb52928b624314d64476"
Sep 10 21:00:52.695216 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.695180    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ce0884513d450db9e98b3958c201dc405e74cc6c6cbbb52928b624314d64476"} err="failed to get container status \"3ce0884513d450db9e98b3958c201dc405e74cc6c6cbbb52928b624314d64476\": rpc error: code = NotFound desc = could not find container \"3ce0884513d450db9e98b3958c201dc405e74cc6c6cbbb52928b624314d64476\": container with ID starting with 3ce0884513d450db9e98b3958c201dc405e74cc6c6cbbb52928b624314d64476 not found: ID does not exist"
Sep 10 21:00:52.695216 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.695192    2581 scope.go:117] "RemoveContainer" containerID="1163069b102c08afcd73d5bc822af82a20cf80e3153015454ddbd74aec3f1ce3"
Sep 10 21:00:52.695488 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:00:52.695472    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1163069b102c08afcd73d5bc822af82a20cf80e3153015454ddbd74aec3f1ce3\": container with ID starting with 1163069b102c08afcd73d5bc822af82a20cf80e3153015454ddbd74aec3f1ce3 not found: ID does not exist" containerID="1163069b102c08afcd73d5bc822af82a20cf80e3153015454ddbd74aec3f1ce3"
Sep 10 21:00:52.695546 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.695491    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1163069b102c08afcd73d5bc822af82a20cf80e3153015454ddbd74aec3f1ce3"} err="failed to get container status \"1163069b102c08afcd73d5bc822af82a20cf80e3153015454ddbd74aec3f1ce3\": rpc error: code = NotFound desc = could not find container \"1163069b102c08afcd73d5bc822af82a20cf80e3153015454ddbd74aec3f1ce3\": container with ID starting with 1163069b102c08afcd73d5bc822af82a20cf80e3153015454ddbd74aec3f1ce3 not found: ID does not exist"
Sep 10 21:00:52.868200 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:00:52.868166    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04bd0140-0658-4be7-8926-37f09be530a0" path="/var/lib/kubelet/pods/04bd0140-0658-4be7-8926-37f09be530a0/volumes"
Sep 10 21:02:08.919986 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:08.919956    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-w8ph6"]
Sep 10 21:02:08.920561 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:08.920351    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04bd0140-0658-4be7-8926-37f09be530a0" containerName="kube-rbac-proxy-thanos"
Sep 10 21:02:08.920561 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:08.920368    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="04bd0140-0658-4be7-8926-37f09be530a0" containerName="kube-rbac-proxy-thanos"
Sep 10 21:02:08.920561 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:08.920380    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04bd0140-0658-4be7-8926-37f09be530a0" containerName="kube-rbac-proxy"
Sep 10 21:02:08.920561 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:08.920390    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="04bd0140-0658-4be7-8926-37f09be530a0" containerName="kube-rbac-proxy"
Sep 10 21:02:08.920561 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:08.920401    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04bd0140-0658-4be7-8926-37f09be530a0" containerName="config-reloader"
Sep 10 21:02:08.920561 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:08.920409    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="04bd0140-0658-4be7-8926-37f09be530a0" containerName="config-reloader"
Sep 10 21:02:08.920561 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:08.920427    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04bd0140-0658-4be7-8926-37f09be530a0" containerName="thanos-sidecar"
Sep 10 21:02:08.920561 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:08.920435    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="04bd0140-0658-4be7-8926-37f09be530a0" containerName="thanos-sidecar"
Sep 10 21:02:08.920561 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:08.920451    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04bd0140-0658-4be7-8926-37f09be530a0" containerName="kube-rbac-proxy-web"
Sep 10 21:02:08.920561 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:08.920461    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="04bd0140-0658-4be7-8926-37f09be530a0" containerName="kube-rbac-proxy-web"
Sep 10 21:02:08.920561 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:08.920472    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04bd0140-0658-4be7-8926-37f09be530a0" containerName="prometheus"
Sep 10 21:02:08.920561 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:08.920480    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="04bd0140-0658-4be7-8926-37f09be530a0" containerName="prometheus"
Sep 10 21:02:08.920561 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:08.920489    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04bd0140-0658-4be7-8926-37f09be530a0" containerName="init-config-reloader"
Sep 10 21:02:08.920561 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:08.920498    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="04bd0140-0658-4be7-8926-37f09be530a0" containerName="init-config-reloader"
Sep 10 21:02:08.921198 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:08.920574    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="04bd0140-0658-4be7-8926-37f09be530a0" containerName="config-reloader"
Sep 10 21:02:08.921198 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:08.920585    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="04bd0140-0658-4be7-8926-37f09be530a0" containerName="thanos-sidecar"
Sep 10 21:02:08.921198 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:08.920594    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="04bd0140-0658-4be7-8926-37f09be530a0" containerName="kube-rbac-proxy"
Sep 10 21:02:08.921198 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:08.920605    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="04bd0140-0658-4be7-8926-37f09be530a0" containerName="kube-rbac-proxy-web"
Sep 10 21:02:08.921198 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:08.920615    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="04bd0140-0658-4be7-8926-37f09be530a0" containerName="prometheus"
Sep 10 21:02:08.921198 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:08.920625    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="04bd0140-0658-4be7-8926-37f09be530a0" containerName="kube-rbac-proxy-thanos"
Sep 10 21:02:08.923693 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:08.923676    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-w8ph6"
Sep 10 21:02:08.926331 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:08.926274    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 10 21:02:08.926440 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:08.926379    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 10 21:02:08.926514 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:08.926500    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-6dbp4\""
Sep 10 21:02:08.935989 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:08.935968    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-w8ph6"]
Sep 10 21:02:09.030447 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:09.030412    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3c44baa-4250-4041-b381-9f54f3a63852-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-w8ph6\" (UID: \"b3c44baa-4250-4041-b381-9f54f3a63852\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-w8ph6"
Sep 10 21:02:09.030608 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:09.030456    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d9lt\" (UniqueName: \"kubernetes.io/projected/b3c44baa-4250-4041-b381-9f54f3a63852-kube-api-access-9d9lt\") pod \"cert-manager-operator-controller-manager-55c444bf86-w8ph6\" (UID: \"b3c44baa-4250-4041-b381-9f54f3a63852\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-w8ph6"
Sep 10 21:02:09.131726 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:09.131691    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3c44baa-4250-4041-b381-9f54f3a63852-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-w8ph6\" (UID: \"b3c44baa-4250-4041-b381-9f54f3a63852\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-w8ph6"
Sep 10 21:02:09.131882 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:09.131739    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d9lt\" (UniqueName: \"kubernetes.io/projected/b3c44baa-4250-4041-b381-9f54f3a63852-kube-api-access-9d9lt\") pod \"cert-manager-operator-controller-manager-55c444bf86-w8ph6\" (UID: \"b3c44baa-4250-4041-b381-9f54f3a63852\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-w8ph6"
Sep 10 21:02:09.132090 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:09.132070    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3c44baa-4250-4041-b381-9f54f3a63852-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-w8ph6\" (UID: \"b3c44baa-4250-4041-b381-9f54f3a63852\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-w8ph6"
Sep 10 21:02:09.140411 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:09.140380    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d9lt\" (UniqueName: \"kubernetes.io/projected/b3c44baa-4250-4041-b381-9f54f3a63852-kube-api-access-9d9lt\") pod \"cert-manager-operator-controller-manager-55c444bf86-w8ph6\" (UID: \"b3c44baa-4250-4041-b381-9f54f3a63852\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-w8ph6"
Sep 10 21:02:09.232465 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:09.232383    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-w8ph6"
Sep 10 21:02:09.508186 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:09.508163    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-w8ph6"]
Sep 10 21:02:09.511264 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:02:09.511234    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3c44baa_4250_4041_b381_9f54f3a63852.slice/crio-a4d9c8f88fb1ad49fd523c9e5471c7d30cbffa74d5a0f8c505e52c1e9e28a1a2 WatchSource:0}: Error finding container a4d9c8f88fb1ad49fd523c9e5471c7d30cbffa74d5a0f8c505e52c1e9e28a1a2: Status 404 returned error can't find the container with id a4d9c8f88fb1ad49fd523c9e5471c7d30cbffa74d5a0f8c505e52c1e9e28a1a2
Sep 10 21:02:09.921837 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:09.921800    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-w8ph6" event={"ID":"b3c44baa-4250-4041-b381-9f54f3a63852","Type":"ContainerStarted","Data":"a4d9c8f88fb1ad49fd523c9e5471c7d30cbffa74d5a0f8c505e52c1e9e28a1a2"}
Sep 10 21:02:12.938462 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:12.938422    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-w8ph6" event={"ID":"b3c44baa-4250-4041-b381-9f54f3a63852","Type":"ContainerStarted","Data":"20a41dd28e80f3be7a248e21fcf47b5fddd27e08bdc0ea6ae95d84f29ee84a62"}
Sep 10 21:02:12.963704 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:12.963653    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-w8ph6" podStartSLOduration=2.385418435 podStartE2EDuration="4.963635379s" totalImagesPullingTime="2.578216944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:02:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:02:12.961733319 +0000 UTC m=+250.618642092" watchObservedRunningTime="2026-09-10 21:02:12.963635379 +0000 UTC m=+250.620544150"
Sep 10 21:02:24.394250 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:24.394216    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-trl46"]
Sep 10 21:02:24.397438 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:24.397416    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-trl46"
Sep 10 21:02:24.399697 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:24.399675    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 10 21:02:24.399697 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:24.399691    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-c4gdb\""
Sep 10 21:02:24.400404 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:24.400385    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 10 21:02:24.408789 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:24.408766    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-trl46"]
Sep 10 21:02:24.561385 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:24.561351    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/38dbcfc6-4dc3-470d-a9c5-f0a83ca97aea-bound-sa-token\") pod \"cert-manager-55b4d954d5-trl46\" (UID: \"38dbcfc6-4dc3-470d-a9c5-f0a83ca97aea\") " pod="cert-manager/cert-manager-55b4d954d5-trl46"
Sep 10 21:02:24.561603 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:24.561424    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s8r6\" (UniqueName: \"kubernetes.io/projected/38dbcfc6-4dc3-470d-a9c5-f0a83ca97aea-kube-api-access-8s8r6\") pod \"cert-manager-55b4d954d5-trl46\" (UID: \"38dbcfc6-4dc3-470d-a9c5-f0a83ca97aea\") " pod="cert-manager/cert-manager-55b4d954d5-trl46"
Sep 10 21:02:24.661816 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:24.661729    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s8r6\" (UniqueName: \"kubernetes.io/projected/38dbcfc6-4dc3-470d-a9c5-f0a83ca97aea-kube-api-access-8s8r6\") pod \"cert-manager-55b4d954d5-trl46\" (UID: \"38dbcfc6-4dc3-470d-a9c5-f0a83ca97aea\") " pod="cert-manager/cert-manager-55b4d954d5-trl46"
Sep 10 21:02:24.661816 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:24.661787    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/38dbcfc6-4dc3-470d-a9c5-f0a83ca97aea-bound-sa-token\") pod \"cert-manager-55b4d954d5-trl46\" (UID: \"38dbcfc6-4dc3-470d-a9c5-f0a83ca97aea\") " pod="cert-manager/cert-manager-55b4d954d5-trl46"
Sep 10 21:02:24.670018 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:24.669995    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/38dbcfc6-4dc3-470d-a9c5-f0a83ca97aea-bound-sa-token\") pod \"cert-manager-55b4d954d5-trl46\" (UID: \"38dbcfc6-4dc3-470d-a9c5-f0a83ca97aea\") " pod="cert-manager/cert-manager-55b4d954d5-trl46"
Sep 10 21:02:24.670231 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:24.670209    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s8r6\" (UniqueName: \"kubernetes.io/projected/38dbcfc6-4dc3-470d-a9c5-f0a83ca97aea-kube-api-access-8s8r6\") pod \"cert-manager-55b4d954d5-trl46\" (UID: \"38dbcfc6-4dc3-470d-a9c5-f0a83ca97aea\") " pod="cert-manager/cert-manager-55b4d954d5-trl46"
Sep 10 21:02:24.716326 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:24.716265    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-trl46"
Sep 10 21:02:24.991156 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:24.991058    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-trl46"]
Sep 10 21:02:24.994387 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:02:24.994355    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38dbcfc6_4dc3_470d_a9c5_f0a83ca97aea.slice/crio-298770aa514b657e5347aac5bd53900481ba4015ad7fbe757cafd289bba4be48 WatchSource:0}: Error finding container 298770aa514b657e5347aac5bd53900481ba4015ad7fbe757cafd289bba4be48: Status 404 returned error can't find the container with id 298770aa514b657e5347aac5bd53900481ba4015ad7fbe757cafd289bba4be48
Sep 10 21:02:25.992237 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:25.992196    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-trl46" event={"ID":"38dbcfc6-4dc3-470d-a9c5-f0a83ca97aea","Type":"ContainerStarted","Data":"298770aa514b657e5347aac5bd53900481ba4015ad7fbe757cafd289bba4be48"}
Sep 10 21:02:28.002664 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:28.002627    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-trl46" event={"ID":"38dbcfc6-4dc3-470d-a9c5-f0a83ca97aea","Type":"ContainerStarted","Data":"046245c60558d106afe91ecf6cf8e8d766b8c3e4c158a1380f187182087137b6"}
Sep 10 21:02:28.020272 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:28.020222    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-trl46" podStartSLOduration=1.710329848 podStartE2EDuration="4.020206882s" totalImagesPullingTime="2.309877034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:02:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:02:28.01891534 +0000 UTC m=+265.675824111" watchObservedRunningTime="2026-09-10 21:02:28.020206882 +0000 UTC m=+265.677115653"
Sep 10 21:02:44.146935 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:44.146897    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d4c889df7-ck2h9"]
Sep 10 21:02:44.150617 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:44.150595    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-ck2h9"
Sep 10 21:02:44.153312 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:44.153269    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 21:02:44.158666 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:44.158639    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 21:02:44.158911 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:44.158890    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-t9kg8\""
Sep 10 21:02:44.158986 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:44.158890    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 21:02:44.159042 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:44.159023    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 21:02:44.169998 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:44.169972    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d4c889df7-ck2h9"]
Sep 10 21:02:44.221540 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:44.221504    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f146c084-a4fd-4b66-b74d-2b9887fedd3f-manifests\") pod \"opendatahub-operator-controller-manager-6d4c889df7-ck2h9\" (UID: \"f146c084-a4fd-4b66-b74d-2b9887fedd3f\") " pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-ck2h9"
Sep 10 21:02:44.221704 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:44.221566    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7d52\" (UniqueName: \"kubernetes.io/projected/f146c084-a4fd-4b66-b74d-2b9887fedd3f-kube-api-access-f7d52\") pod \"opendatahub-operator-controller-manager-6d4c889df7-ck2h9\" (UID: \"f146c084-a4fd-4b66-b74d-2b9887fedd3f\") " pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-ck2h9"
Sep 10 21:02:44.221704 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:44.221604    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f146c084-a4fd-4b66-b74d-2b9887fedd3f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d4c889df7-ck2h9\" (UID: \"f146c084-a4fd-4b66-b74d-2b9887fedd3f\") " pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-ck2h9"
Sep 10 21:02:44.221704 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:44.221625    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f146c084-a4fd-4b66-b74d-2b9887fedd3f-tmp\") pod \"opendatahub-operator-controller-manager-6d4c889df7-ck2h9\" (UID: \"f146c084-a4fd-4b66-b74d-2b9887fedd3f\") " pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-ck2h9"
Sep 10 21:02:44.221704 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:44.221693    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f146c084-a4fd-4b66-b74d-2b9887fedd3f-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d4c889df7-ck2h9\" (UID: \"f146c084-a4fd-4b66-b74d-2b9887fedd3f\") " pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-ck2h9"
Sep 10 21:02:44.322057 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:44.322020    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f146c084-a4fd-4b66-b74d-2b9887fedd3f-manifests\") pod \"opendatahub-operator-controller-manager-6d4c889df7-ck2h9\" (UID: \"f146c084-a4fd-4b66-b74d-2b9887fedd3f\") " pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-ck2h9"
Sep 10 21:02:44.322240 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:44.322082    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7d52\" (UniqueName: \"kubernetes.io/projected/f146c084-a4fd-4b66-b74d-2b9887fedd3f-kube-api-access-f7d52\") pod \"opendatahub-operator-controller-manager-6d4c889df7-ck2h9\" (UID: \"f146c084-a4fd-4b66-b74d-2b9887fedd3f\") " pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-ck2h9"
Sep 10 21:02:44.322240 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:44.322110    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f146c084-a4fd-4b66-b74d-2b9887fedd3f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d4c889df7-ck2h9\" (UID: \"f146c084-a4fd-4b66-b74d-2b9887fedd3f\") " pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-ck2h9"
Sep 10 21:02:44.322240 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:44.322138    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f146c084-a4fd-4b66-b74d-2b9887fedd3f-tmp\") pod \"opendatahub-operator-controller-manager-6d4c889df7-ck2h9\" (UID: \"f146c084-a4fd-4b66-b74d-2b9887fedd3f\") " pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-ck2h9"
Sep 10 21:02:44.322458 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:44.322280    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f146c084-a4fd-4b66-b74d-2b9887fedd3f-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d4c889df7-ck2h9\" (UID: \"f146c084-a4fd-4b66-b74d-2b9887fedd3f\") " pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-ck2h9"
Sep 10 21:02:44.322580 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:44.322511    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f146c084-a4fd-4b66-b74d-2b9887fedd3f-manifests\") pod \"opendatahub-operator-controller-manager-6d4c889df7-ck2h9\" (UID: \"f146c084-a4fd-4b66-b74d-2b9887fedd3f\") " pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-ck2h9"
Sep 10 21:02:44.324501 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:44.324474    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f146c084-a4fd-4b66-b74d-2b9887fedd3f-tmp\") pod \"opendatahub-operator-controller-manager-6d4c889df7-ck2h9\" (UID: \"f146c084-a4fd-4b66-b74d-2b9887fedd3f\") " pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-ck2h9"
Sep 10 21:02:44.324657 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:44.324636    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f146c084-a4fd-4b66-b74d-2b9887fedd3f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d4c889df7-ck2h9\" (UID: \"f146c084-a4fd-4b66-b74d-2b9887fedd3f\") " pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-ck2h9"
Sep 10 21:02:44.324741 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:44.324722    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f146c084-a4fd-4b66-b74d-2b9887fedd3f-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d4c889df7-ck2h9\" (UID: \"f146c084-a4fd-4b66-b74d-2b9887fedd3f\") " pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-ck2h9"
Sep 10 21:02:44.330681 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:44.330650    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7d52\" (UniqueName: \"kubernetes.io/projected/f146c084-a4fd-4b66-b74d-2b9887fedd3f-kube-api-access-f7d52\") pod \"opendatahub-operator-controller-manager-6d4c889df7-ck2h9\" (UID: \"f146c084-a4fd-4b66-b74d-2b9887fedd3f\") " pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-ck2h9"
Sep 10 21:02:44.465036 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:44.464949    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-ck2h9"
Sep 10 21:02:44.740249 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:44.740165    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d4c889df7-ck2h9"]
Sep 10 21:02:44.743953 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:02:44.743922    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf146c084_a4fd_4b66_b74d_2b9887fedd3f.slice/crio-e997f668da37ab1b94533bf207de3e6e2dc8867799b64e8a38759a5b9bce20a8 WatchSource:0}: Error finding container e997f668da37ab1b94533bf207de3e6e2dc8867799b64e8a38759a5b9bce20a8: Status 404 returned error can't find the container with id e997f668da37ab1b94533bf207de3e6e2dc8867799b64e8a38759a5b9bce20a8
Sep 10 21:02:45.072444 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:45.072407    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-ck2h9" event={"ID":"f146c084-a4fd-4b66-b74d-2b9887fedd3f","Type":"ContainerStarted","Data":"e997f668da37ab1b94533bf207de3e6e2dc8867799b64e8a38759a5b9bce20a8"}
Sep 10 21:02:48.089252 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:48.089212    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-ck2h9" event={"ID":"f146c084-a4fd-4b66-b74d-2b9887fedd3f","Type":"ContainerStarted","Data":"0ac0e98bb98919849e24dbfd50206cfca89f1c0244d62b42a9bebcbbf0dba2a6"}
Sep 10 21:02:49.095315 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:49.095274    2581 generic.go:356] "Generic (PLEG): container finished" podID="f146c084-a4fd-4b66-b74d-2b9887fedd3f" containerID="0ac0e98bb98919849e24dbfd50206cfca89f1c0244d62b42a9bebcbbf0dba2a6" exitCode=0
Sep 10 21:02:49.095759 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:49.095346    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-ck2h9" event={"ID":"f146c084-a4fd-4b66-b74d-2b9887fedd3f","Type":"ContainerDied","Data":"0ac0e98bb98919849e24dbfd50206cfca89f1c0244d62b42a9bebcbbf0dba2a6"}
Sep 10 21:02:50.102921 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:50.102876    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-ck2h9" event={"ID":"f146c084-a4fd-4b66-b74d-2b9887fedd3f","Type":"ContainerStarted","Data":"29590cbfc725af348a441e4a43087d6029d0615dde2d12f2957ec72720029072"}
Sep 10 21:02:50.103416 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:50.103038    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-ck2h9"
Sep 10 21:02:50.140839 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:02:50.140794    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-ck2h9" podStartSLOduration=1.886061446 podStartE2EDuration="6.140775839s" totalImagesPullingTime="3.174218617s" totalInitContainerRuntime="1.080495776s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:02:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:02:50.140741194 +0000 UTC m=+287.797649965" watchObservedRunningTime="2026-09-10 21:02:50.140775839 +0000 UTC m=+287.797684609"
Sep 10 21:03:01.108670 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:01.108636    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6d4c889df7-ck2h9"
Sep 10 21:03:02.751314 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:02.751262    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-z6858_c1e06766-b48c-429e-92f1-196bb083360b/console-operator/1.log"
Sep 10 21:03:02.755144 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:02.755117    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-z6858_c1e06766-b48c-429e-92f1-196bb083360b/console-operator/1.log"
Sep 10 21:03:02.756598 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:02.756571    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brrtd_19d8d7ba-45a1-4013-a935-5670abe12b8a/ovn-acl-logging/0.log"
Sep 10 21:03:02.759666 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:02.759642    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brrtd_19d8d7ba-45a1-4013-a935-5670abe12b8a/ovn-acl-logging/0.log"
Sep 10 21:03:02.763419 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:02.763399    2581 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 21:03:03.520457 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:03.520424    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d99b4bcb4-cdq2z"]
Sep 10 21:03:03.522360 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:03.522342    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5d99b4bcb4-cdq2z"
Sep 10 21:03:03.524897 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:03.524876    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 10 21:03:03.526208 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:03.526189    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 10 21:03:03.526356 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:03.526220    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 10 21:03:03.526356 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:03.526237    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 10 21:03:03.526356 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:03.526269    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-47vpz\""
Sep 10 21:03:03.526356 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:03.526182    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 10 21:03:03.540862 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:03.540829    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d99b4bcb4-cdq2z"]
Sep 10 21:03:03.575108 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:03.575062    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7282ee80-78f8-4de9-b3a4-d09f186772b7-manager-config\") pod \"lws-controller-manager-5d99b4bcb4-cdq2z\" (UID: \"7282ee80-78f8-4de9-b3a4-d09f186772b7\") " pod="openshift-lws-operator/lws-controller-manager-5d99b4bcb4-cdq2z"
Sep 10 21:03:03.575108 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:03.575107    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7282ee80-78f8-4de9-b3a4-d09f186772b7-metrics-cert\") pod \"lws-controller-manager-5d99b4bcb4-cdq2z\" (UID: \"7282ee80-78f8-4de9-b3a4-d09f186772b7\") " pod="openshift-lws-operator/lws-controller-manager-5d99b4bcb4-cdq2z"
Sep 10 21:03:03.575324 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:03.575134    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7282ee80-78f8-4de9-b3a4-d09f186772b7-cert\") pod \"lws-controller-manager-5d99b4bcb4-cdq2z\" (UID: \"7282ee80-78f8-4de9-b3a4-d09f186772b7\") " pod="openshift-lws-operator/lws-controller-manager-5d99b4bcb4-cdq2z"
Sep 10 21:03:03.575324 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:03.575210    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9qmv\" (UniqueName: \"kubernetes.io/projected/7282ee80-78f8-4de9-b3a4-d09f186772b7-kube-api-access-v9qmv\") pod \"lws-controller-manager-5d99b4bcb4-cdq2z\" (UID: \"7282ee80-78f8-4de9-b3a4-d09f186772b7\") " pod="openshift-lws-operator/lws-controller-manager-5d99b4bcb4-cdq2z"
Sep 10 21:03:03.676559 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:03.676520    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7282ee80-78f8-4de9-b3a4-d09f186772b7-manager-config\") pod \"lws-controller-manager-5d99b4bcb4-cdq2z\" (UID: \"7282ee80-78f8-4de9-b3a4-d09f186772b7\") " pod="openshift-lws-operator/lws-controller-manager-5d99b4bcb4-cdq2z"
Sep 10 21:03:03.676716 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:03.676568    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7282ee80-78f8-4de9-b3a4-d09f186772b7-metrics-cert\") pod \"lws-controller-manager-5d99b4bcb4-cdq2z\" (UID: \"7282ee80-78f8-4de9-b3a4-d09f186772b7\") " pod="openshift-lws-operator/lws-controller-manager-5d99b4bcb4-cdq2z"
Sep 10 21:03:03.676716 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:03.676681    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7282ee80-78f8-4de9-b3a4-d09f186772b7-cert\") pod \"lws-controller-manager-5d99b4bcb4-cdq2z\" (UID: \"7282ee80-78f8-4de9-b3a4-d09f186772b7\") " pod="openshift-lws-operator/lws-controller-manager-5d99b4bcb4-cdq2z"
Sep 10 21:03:03.676792 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:03.676720    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9qmv\" (UniqueName: \"kubernetes.io/projected/7282ee80-78f8-4de9-b3a4-d09f186772b7-kube-api-access-v9qmv\") pod \"lws-controller-manager-5d99b4bcb4-cdq2z\" (UID: \"7282ee80-78f8-4de9-b3a4-d09f186772b7\") " pod="openshift-lws-operator/lws-controller-manager-5d99b4bcb4-cdq2z"
Sep 10 21:03:03.677164 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:03.677137    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7282ee80-78f8-4de9-b3a4-d09f186772b7-manager-config\") pod \"lws-controller-manager-5d99b4bcb4-cdq2z\" (UID: \"7282ee80-78f8-4de9-b3a4-d09f186772b7\") " pod="openshift-lws-operator/lws-controller-manager-5d99b4bcb4-cdq2z"
Sep 10 21:03:03.679091 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:03.679067    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7282ee80-78f8-4de9-b3a4-d09f186772b7-metrics-cert\") pod \"lws-controller-manager-5d99b4bcb4-cdq2z\" (UID: \"7282ee80-78f8-4de9-b3a4-d09f186772b7\") " pod="openshift-lws-operator/lws-controller-manager-5d99b4bcb4-cdq2z"
Sep 10 21:03:03.679196 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:03.679156    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7282ee80-78f8-4de9-b3a4-d09f186772b7-cert\") pod \"lws-controller-manager-5d99b4bcb4-cdq2z\" (UID: \"7282ee80-78f8-4de9-b3a4-d09f186772b7\") " pod="openshift-lws-operator/lws-controller-manager-5d99b4bcb4-cdq2z"
Sep 10 21:03:03.688880 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:03.688849    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9qmv\" (UniqueName: \"kubernetes.io/projected/7282ee80-78f8-4de9-b3a4-d09f186772b7-kube-api-access-v9qmv\") pod \"lws-controller-manager-5d99b4bcb4-cdq2z\" (UID: \"7282ee80-78f8-4de9-b3a4-d09f186772b7\") " pod="openshift-lws-operator/lws-controller-manager-5d99b4bcb4-cdq2z"
Sep 10 21:03:03.833149 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:03.833106    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5d99b4bcb4-cdq2z"
Sep 10 21:03:04.147653 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:04.147624    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d99b4bcb4-cdq2z"]
Sep 10 21:03:04.149196 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:03:04.149158    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7282ee80_78f8_4de9_b3a4_d09f186772b7.slice/crio-c53ed817c56c1586043d5cde977d50d9908d30cdcf255c441b1723d5b50693d4 WatchSource:0}: Error finding container c53ed817c56c1586043d5cde977d50d9908d30cdcf255c441b1723d5b50693d4: Status 404 returned error can't find the container with id c53ed817c56c1586043d5cde977d50d9908d30cdcf255c441b1723d5b50693d4
Sep 10 21:03:04.160776 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:04.160733    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5d99b4bcb4-cdq2z" event={"ID":"7282ee80-78f8-4de9-b3a4-d09f186772b7","Type":"ContainerStarted","Data":"c53ed817c56c1586043d5cde977d50d9908d30cdcf255c441b1723d5b50693d4"}
Sep 10 21:03:07.176109 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:07.176068    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5d99b4bcb4-cdq2z" event={"ID":"7282ee80-78f8-4de9-b3a4-d09f186772b7","Type":"ContainerStarted","Data":"210daf1b5aff7d8572bb38fd41b9a39ca3a139a1520b591fdeff79438317f8b0"}
Sep 10 21:03:07.176628 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:07.176190    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5d99b4bcb4-cdq2z"
Sep 10 21:03:07.195751 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:07.195697    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5d99b4bcb4-cdq2z" podStartSLOduration=2.084959664 podStartE2EDuration="4.195681482s" totalImagesPullingTime="2.110721818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:03:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:03:07.195120833 +0000 UTC m=+304.852029605" watchObservedRunningTime="2026-09-10 21:03:07.195681482 +0000 UTC m=+304.852590254"
Sep 10 21:03:11.125328 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:11.125285    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfltqv"]
Sep 10 21:03:11.127601 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:11.127584    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfltqv"
Sep 10 21:03:11.130009 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:03:11.129980    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"data-science-gateway-data-science-gateway-class-68cf4d799cfltqv\" is forbidden: User \"system:node:ip-10-0-139-165.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-139-165.ec2.internal' and this object" podUID="d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfltqv"
Sep 10 21:03:11.130134 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:03:11.130011    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"data-science-gateway-data-science-gateway-class-dockercfg-264c6\" is forbidden: User \"system:node:ip-10-0-139-165.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-139-165.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-264c6\"" type="*v1.Secret"
Sep 10 21:03:11.130134 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:03:11.130011    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-gw-ca-root-cert\" is forbidden: User \"system:node:ip-10-0-139-165.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-139-165.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" type="*v1.ConfigMap"
Sep 10 21:03:11.146487 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:11.146463    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfltqv"]
Sep 10 21:03:11.243870 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:11.243831    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfltqv\" (UID: \"d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfltqv"
Sep 10 21:03:11.243870 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:11.243874    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfltqv\" (UID: \"d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfltqv"
Sep 10 21:03:11.244136 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:11.243900    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfltqv\" (UID: \"d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfltqv"
Sep 10 21:03:11.244136 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:11.243937    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgqsf\" (UniqueName: \"kubernetes.io/projected/d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f-kube-api-access-zgqsf\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfltqv\" (UID: \"d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfltqv"
Sep 10 21:03:11.244136 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:11.243960    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfltqv\" (UID: \"d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfltqv"
Sep 10 21:03:11.244136 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:11.244026    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfltqv\" (UID: \"d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfltqv"
Sep 10 21:03:11.244136 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:11.244061    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfltqv\" (UID: \"d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfltqv"
Sep 10 21:03:11.244136 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:11.244094    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfltqv\" (UID: \"d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfltqv"
Sep 10 21:03:11.244349 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:11.244160    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfltqv\" (UID: \"d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfltqv"
Sep 10 21:03:11.344754 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:11.344714    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfltqv\" (UID: \"d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfltqv"
Sep 10 21:03:11.344754 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:11.344755    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfltqv\" (UID: \"d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfltqv"
Sep 10 21:03:11.345008 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:11.344789    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfltqv\" (UID: \"d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfltqv"
Sep 10 21:03:11.345008 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:11.344818    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgqsf\" (UniqueName: \"kubernetes.io/projected/d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f-kube-api-access-zgqsf\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfltqv\" (UID: \"d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfltqv"
Sep 10 21:03:11.345008 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:11.344846    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfltqv\" (UID: \"d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfltqv"
Sep 10 21:03:11.345008 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:11.344950    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfltqv\" (UID: \"d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfltqv"
Sep 10 21:03:11.345008 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:11.344992    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfltqv\" (UID: \"d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfltqv"
Sep 10 21:03:11.345258 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:11.345034    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfltqv\" (UID: \"d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfltqv"
Sep 10 21:03:11.345258 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:11.345080    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfltqv\" (UID: \"d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfltqv"
Sep 10 21:03:11.345258 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:11.345218    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfltqv\" (UID: \"d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfltqv"
Sep 10 21:03:11.345424 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:11.345275    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfltqv\" (UID: \"d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfltqv"
Sep 10 21:03:11.345424 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:11.345409    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfltqv\" (UID: \"d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfltqv"
Sep 10 21:03:11.345493 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:11.345482    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfltqv\" (UID: \"d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfltqv"
Sep 10 21:03:11.347085 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:11.347055    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfltqv\" (UID: \"d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfltqv"
Sep 10 21:03:11.347433 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:11.347408    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfltqv\" (UID: \"d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfltqv"
Sep 10 21:03:11.355570 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:11.355541    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfltqv\" (UID: \"d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfltqv"
Sep 10 21:03:11.355961 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:11.355939    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgqsf\" (UniqueName: \"kubernetes.io/projected/d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f-kube-api-access-zgqsf\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfltqv\" (UID: \"d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfltqv"
Sep 10 21:03:12.288839 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:12.288803    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 21:03:12.296138 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:12.296114    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfltqv\" (UID: \"d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfltqv"
Sep 10 21:03:12.602821 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:12.602784    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-264c6\""
Sep 10 21:03:12.608484 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:12.608446    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfltqv"
Sep 10 21:03:12.902155 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:12.902129    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfltqv"]
Sep 10 21:03:12.904095 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:03:12.904061    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd04cd5a4_8327_4dae_bfd8_72dafdd2cc4f.slice/crio-2b69ff2e80e2e7a2271e592cee0fe024646da7c8660ce9710b5d6a878781b00e WatchSource:0}: Error finding container 2b69ff2e80e2e7a2271e592cee0fe024646da7c8660ce9710b5d6a878781b00e: Status 404 returned error can't find the container with id 2b69ff2e80e2e7a2271e592cee0fe024646da7c8660ce9710b5d6a878781b00e
Sep 10 21:03:12.906121 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:12.906103    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 21:03:13.206690 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:13.206608    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfltqv" event={"ID":"d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f","Type":"ContainerStarted","Data":"2b69ff2e80e2e7a2271e592cee0fe024646da7c8660ce9710b5d6a878781b00e"}
Sep 10 21:03:14.919655 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:14.919616    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-7blpg"]
Sep 10 21:03:14.925485 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:14.925452    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-7blpg"
Sep 10 21:03:14.928260 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:14.927994    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 10 21:03:14.928260 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:14.928072    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-ggmgv\""
Sep 10 21:03:14.934172 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:14.934141    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-7blpg"]
Sep 10 21:03:14.975117 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:14.975059    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfh4b\" (UniqueName: \"kubernetes.io/projected/e7b385e1-fa9f-47b8-87bf-006187f9d607-kube-api-access-hfh4b\") pod \"odh-model-controller-7998c67dc4-7blpg\" (UID: \"e7b385e1-fa9f-47b8-87bf-006187f9d607\") " pod="opendatahub/odh-model-controller-7998c67dc4-7blpg"
Sep 10 21:03:14.975346 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:14.975145    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e7b385e1-fa9f-47b8-87bf-006187f9d607-cert\") pod \"odh-model-controller-7998c67dc4-7blpg\" (UID: \"e7b385e1-fa9f-47b8-87bf-006187f9d607\") " pod="opendatahub/odh-model-controller-7998c67dc4-7blpg"
Sep 10 21:03:15.076030 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:15.075989    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfh4b\" (UniqueName: \"kubernetes.io/projected/e7b385e1-fa9f-47b8-87bf-006187f9d607-kube-api-access-hfh4b\") pod \"odh-model-controller-7998c67dc4-7blpg\" (UID: \"e7b385e1-fa9f-47b8-87bf-006187f9d607\") " pod="opendatahub/odh-model-controller-7998c67dc4-7blpg"
Sep 10 21:03:15.076217 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:15.076042    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e7b385e1-fa9f-47b8-87bf-006187f9d607-cert\") pod \"odh-model-controller-7998c67dc4-7blpg\" (UID: \"e7b385e1-fa9f-47b8-87bf-006187f9d607\") " pod="opendatahub/odh-model-controller-7998c67dc4-7blpg"
Sep 10 21:03:15.076280 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:03:15.076232    2581 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 10 21:03:15.076348 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:03:15.076331    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7b385e1-fa9f-47b8-87bf-006187f9d607-cert podName:e7b385e1-fa9f-47b8-87bf-006187f9d607 nodeName:}" failed. No retries permitted until 2026-09-10 21:03:15.576278786 +0000 UTC m=+313.233187539 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e7b385e1-fa9f-47b8-87bf-006187f9d607-cert") pod "odh-model-controller-7998c67dc4-7blpg" (UID: "e7b385e1-fa9f-47b8-87bf-006187f9d607") : secret "odh-model-controller-webhook-cert" not found
Sep 10 21:03:15.085039 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:15.085008    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfh4b\" (UniqueName: \"kubernetes.io/projected/e7b385e1-fa9f-47b8-87bf-006187f9d607-kube-api-access-hfh4b\") pod \"odh-model-controller-7998c67dc4-7blpg\" (UID: \"e7b385e1-fa9f-47b8-87bf-006187f9d607\") " pod="opendatahub/odh-model-controller-7998c67dc4-7blpg"
Sep 10 21:03:15.464288 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:15.464247    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 21:03:15.464383 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:15.464364    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 21:03:15.464422 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:15.464394    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 21:03:15.580130 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:15.580098    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e7b385e1-fa9f-47b8-87bf-006187f9d607-cert\") pod \"odh-model-controller-7998c67dc4-7blpg\" (UID: \"e7b385e1-fa9f-47b8-87bf-006187f9d607\") " pod="opendatahub/odh-model-controller-7998c67dc4-7blpg"
Sep 10 21:03:15.583279 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:15.583205    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e7b385e1-fa9f-47b8-87bf-006187f9d607-cert\") pod \"odh-model-controller-7998c67dc4-7blpg\" (UID: \"e7b385e1-fa9f-47b8-87bf-006187f9d607\") " pod="opendatahub/odh-model-controller-7998c67dc4-7blpg"
Sep 10 21:03:15.841655 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:15.841613    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-7blpg"
Sep 10 21:03:16.127599 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:16.127566    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-7blpg"]
Sep 10 21:03:16.130342 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:03:16.130314    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7b385e1_fa9f_47b8_87bf_006187f9d607.slice/crio-f6ccbb5b513b62a66276109c15a43ff8464e3e2a6a2f20c4a92d2ec51c2e8fe0 WatchSource:0}: Error finding container f6ccbb5b513b62a66276109c15a43ff8464e3e2a6a2f20c4a92d2ec51c2e8fe0: Status 404 returned error can't find the container with id f6ccbb5b513b62a66276109c15a43ff8464e3e2a6a2f20c4a92d2ec51c2e8fe0
Sep 10 21:03:16.223490 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:16.223244    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfltqv" event={"ID":"d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f","Type":"ContainerStarted","Data":"138c6f32ab30f626ae00c7f996ea97b5166a35e820fc8067f70794c1126343f4"}
Sep 10 21:03:16.224817 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:16.224787    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-7blpg" event={"ID":"e7b385e1-fa9f-47b8-87bf-006187f9d607","Type":"ContainerStarted","Data":"f6ccbb5b513b62a66276109c15a43ff8464e3e2a6a2f20c4a92d2ec51c2e8fe0"}
Sep 10 21:03:16.245975 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:16.245923    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfltqv" podStartSLOduration=2.688132675 podStartE2EDuration="5.245904034s" totalImagesPullingTime="2.557771359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:03:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:03:16.243944848 +0000 UTC m=+313.900853621" watchObservedRunningTime="2026-09-10 21:03:16.245904034 +0000 UTC m=+313.902812806"
Sep 10 21:03:16.609053 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:16.609012    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfltqv"
Sep 10 21:03:16.615197 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:16.615170    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfltqv"
Sep 10 21:03:17.230457 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:17.230343    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfltqv"
Sep 10 21:03:17.231676 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:17.231651    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfltqv"
Sep 10 21:03:18.183150 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:18.183115    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5d99b4bcb4-cdq2z"
Sep 10 21:03:19.240588 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:19.240552    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-7blpg" event={"ID":"e7b385e1-fa9f-47b8-87bf-006187f9d607","Type":"ContainerStarted","Data":"9bbe74b534de63633b7ef12ad9b6625803b013d07f99cb99bf8f05fd659f9a6a"}
Sep 10 21:03:19.259745 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:19.259693    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-7blpg" podStartSLOduration=2.272964559 podStartE2EDuration="5.259675112s" totalImagesPullingTime="2.986710553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:03:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:03:19.259097846 +0000 UTC m=+316.916006620" watchObservedRunningTime="2026-09-10 21:03:19.259675112 +0000 UTC m=+316.916583862"
Sep 10 21:03:20.244887 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:20.244860    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-7blpg"
Sep 10 21:03:22.238253 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:22.238218    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-fmnwn"]
Sep 10 21:03:22.241010 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:22.240982    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fmnwn"
Sep 10 21:03:22.244386 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:22.244367    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 10 21:03:22.244495 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:22.244430    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 10 21:03:22.244566 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:22.244506    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-t9zwq\""
Sep 10 21:03:22.244682 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:22.244670    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 10 21:03:22.269508 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:22.269474    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-fmnwn"]
Sep 10 21:03:22.339858 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:22.339823    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/20736e92-a91b-452c-a01f-5a981c3c629a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-fmnwn\" (UID: \"20736e92-a91b-452c-a01f-5a981c3c629a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fmnwn"
Sep 10 21:03:22.340027 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:22.339861    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20736e92-a91b-452c-a01f-5a981c3c629a-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-fmnwn\" (UID: \"20736e92-a91b-452c-a01f-5a981c3c629a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fmnwn"
Sep 10 21:03:22.340027 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:22.339889    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69qqn\" (UniqueName: \"kubernetes.io/projected/20736e92-a91b-452c-a01f-5a981c3c629a-kube-api-access-69qqn\") pod \"istiod-openshift-gateway-5677ccb49b-fmnwn\" (UID: \"20736e92-a91b-452c-a01f-5a981c3c629a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fmnwn"
Sep 10 21:03:22.340027 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:22.339954    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/20736e92-a91b-452c-a01f-5a981c3c629a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-fmnwn\" (UID: \"20736e92-a91b-452c-a01f-5a981c3c629a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fmnwn"
Sep 10 21:03:22.340027 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:22.340004    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/20736e92-a91b-452c-a01f-5a981c3c629a-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-fmnwn\" (UID: \"20736e92-a91b-452c-a01f-5a981c3c629a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fmnwn"
Sep 10 21:03:22.340159 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:22.340077    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/20736e92-a91b-452c-a01f-5a981c3c629a-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-fmnwn\" (UID: \"20736e92-a91b-452c-a01f-5a981c3c629a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fmnwn"
Sep 10 21:03:22.340159 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:22.340099    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/20736e92-a91b-452c-a01f-5a981c3c629a-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-fmnwn\" (UID: \"20736e92-a91b-452c-a01f-5a981c3c629a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fmnwn"
Sep 10 21:03:22.441118 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:22.441077    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/20736e92-a91b-452c-a01f-5a981c3c629a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-fmnwn\" (UID: \"20736e92-a91b-452c-a01f-5a981c3c629a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fmnwn"
Sep 10 21:03:22.441335 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:22.441131    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/20736e92-a91b-452c-a01f-5a981c3c629a-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-fmnwn\" (UID: \"20736e92-a91b-452c-a01f-5a981c3c629a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fmnwn"
Sep 10 21:03:22.441335 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:22.441178    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/20736e92-a91b-452c-a01f-5a981c3c629a-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-fmnwn\" (UID: \"20736e92-a91b-452c-a01f-5a981c3c629a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fmnwn"
Sep 10 21:03:22.441335 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:22.441195    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/20736e92-a91b-452c-a01f-5a981c3c629a-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-fmnwn\" (UID: \"20736e92-a91b-452c-a01f-5a981c3c629a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fmnwn"
Sep 10 21:03:22.441335 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:22.441237    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/20736e92-a91b-452c-a01f-5a981c3c629a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-fmnwn\" (UID: \"20736e92-a91b-452c-a01f-5a981c3c629a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fmnwn"
Sep 10 21:03:22.441335 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:22.441260    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20736e92-a91b-452c-a01f-5a981c3c629a-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-fmnwn\" (UID: \"20736e92-a91b-452c-a01f-5a981c3c629a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fmnwn"
Sep 10 21:03:22.441335 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:22.441274    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69qqn\" (UniqueName: \"kubernetes.io/projected/20736e92-a91b-452c-a01f-5a981c3c629a-kube-api-access-69qqn\") pod \"istiod-openshift-gateway-5677ccb49b-fmnwn\" (UID: \"20736e92-a91b-452c-a01f-5a981c3c629a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fmnwn"
Sep 10 21:03:22.442084 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:22.442050    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/20736e92-a91b-452c-a01f-5a981c3c629a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-fmnwn\" (UID: \"20736e92-a91b-452c-a01f-5a981c3c629a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fmnwn"
Sep 10 21:03:22.443681 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:22.443647    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/20736e92-a91b-452c-a01f-5a981c3c629a-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-fmnwn\" (UID: \"20736e92-a91b-452c-a01f-5a981c3c629a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fmnwn"
Sep 10 21:03:22.443780 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:22.443653    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/20736e92-a91b-452c-a01f-5a981c3c629a-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-fmnwn\" (UID: \"20736e92-a91b-452c-a01f-5a981c3c629a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fmnwn"
Sep 10 21:03:22.443780 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:22.443716    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/20736e92-a91b-452c-a01f-5a981c3c629a-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-fmnwn\" (UID: \"20736e92-a91b-452c-a01f-5a981c3c629a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fmnwn"
Sep 10 21:03:22.443939 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:22.443917    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/20736e92-a91b-452c-a01f-5a981c3c629a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-fmnwn\" (UID: \"20736e92-a91b-452c-a01f-5a981c3c629a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fmnwn"
Sep 10 21:03:22.459222 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:22.459194    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20736e92-a91b-452c-a01f-5a981c3c629a-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-fmnwn\" (UID: \"20736e92-a91b-452c-a01f-5a981c3c629a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fmnwn"
Sep 10 21:03:22.459469 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:22.459450    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69qqn\" (UniqueName: \"kubernetes.io/projected/20736e92-a91b-452c-a01f-5a981c3c629a-kube-api-access-69qqn\") pod \"istiod-openshift-gateway-5677ccb49b-fmnwn\" (UID: \"20736e92-a91b-452c-a01f-5a981c3c629a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fmnwn"
Sep 10 21:03:22.550542 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:22.550514    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fmnwn"
Sep 10 21:03:22.846733 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:03:22.845072    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20736e92_a91b_452c_a01f_5a981c3c629a.slice/crio-ea449211913386804c84c39d7a251fda48bbd2a6fee0e97cc13fe912e264d480 WatchSource:0}: Error finding container ea449211913386804c84c39d7a251fda48bbd2a6fee0e97cc13fe912e264d480: Status 404 returned error can't find the container with id ea449211913386804c84c39d7a251fda48bbd2a6fee0e97cc13fe912e264d480
Sep 10 21:03:22.847597 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:22.847473    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-fmnwn"]
Sep 10 21:03:23.258694 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:23.258610    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fmnwn" event={"ID":"20736e92-a91b-452c-a01f-5a981c3c629a","Type":"ContainerStarted","Data":"ea449211913386804c84c39d7a251fda48bbd2a6fee0e97cc13fe912e264d480"}
Sep 10 21:03:25.398501 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:25.398461    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 21:03:25.398873 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:25.398547    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 21:03:26.275919 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:26.275880    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fmnwn" event={"ID":"20736e92-a91b-452c-a01f-5a981c3c629a","Type":"ContainerStarted","Data":"a9ae49c0ab69bfc962f74f6c665bb49ed6d6329f331233fda5cc4b51e75660cb"}
Sep 10 21:03:26.276107 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:26.276038    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fmnwn"
Sep 10 21:03:26.277684 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:26.277662    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fmnwn"
Sep 10 21:03:26.306020 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:26.305963    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-fmnwn" podStartSLOduration=1.757439784 podStartE2EDuration="4.305941781s" totalImagesPullingTime="2.548501997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:03:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:03:26.303559025 +0000 UTC m=+323.960467797" watchObservedRunningTime="2026-09-10 21:03:26.305941781 +0000 UTC m=+323.962850554"
Sep 10 21:03:31.252162 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:03:31.252131    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-7blpg"
Sep 10 21:06:01.025673 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:01.025637    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-58c9bd97bf-gb5bj"]
Sep 10 21:06:01.027890 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:01.027874    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58c9bd97bf-gb5bj"
Sep 10 21:06:01.030155 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:01.030127    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 10 21:06:01.030155 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:01.030135    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 10 21:06:01.030503 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:01.030483    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2s7xb\""
Sep 10 21:06:01.030597 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:01.030514    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 21:06:01.040580 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:01.040556    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58c9bd97bf-gb5bj"]
Sep 10 21:06:01.087499 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:01.087449    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x795k\" (UniqueName: \"kubernetes.io/projected/29b029bb-2f06-49d7-86cf-1e472b06e16c-kube-api-access-x795k\") pod \"maas-controller-58c9bd97bf-gb5bj\" (UID: \"29b029bb-2f06-49d7-86cf-1e472b06e16c\") " pod="opendatahub/maas-controller-58c9bd97bf-gb5bj"
Sep 10 21:06:01.087668 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:01.087519    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29b029bb-2f06-49d7-86cf-1e472b06e16c-metrics-certs\") pod \"maas-controller-58c9bd97bf-gb5bj\" (UID: \"29b029bb-2f06-49d7-86cf-1e472b06e16c\") " pod="opendatahub/maas-controller-58c9bd97bf-gb5bj"
Sep 10 21:06:01.087668 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:01.087595    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/29b029bb-2f06-49d7-86cf-1e472b06e16c-webhook-cert\") pod \"maas-controller-58c9bd97bf-gb5bj\" (UID: \"29b029bb-2f06-49d7-86cf-1e472b06e16c\") " pod="opendatahub/maas-controller-58c9bd97bf-gb5bj"
Sep 10 21:06:01.153768 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:01.153731    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6d795c6fb-r5g2k"]
Sep 10 21:06:01.156003 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:01.155985    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d795c6fb-r5g2k"
Sep 10 21:06:01.183955 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:01.183920    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d795c6fb-r5g2k"]
Sep 10 21:06:01.188146 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:01.188119    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29b029bb-2f06-49d7-86cf-1e472b06e16c-metrics-certs\") pod \"maas-controller-58c9bd97bf-gb5bj\" (UID: \"29b029bb-2f06-49d7-86cf-1e472b06e16c\") " pod="opendatahub/maas-controller-58c9bd97bf-gb5bj"
Sep 10 21:06:01.188287 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:01.188170    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/29b029bb-2f06-49d7-86cf-1e472b06e16c-webhook-cert\") pod \"maas-controller-58c9bd97bf-gb5bj\" (UID: \"29b029bb-2f06-49d7-86cf-1e472b06e16c\") " pod="opendatahub/maas-controller-58c9bd97bf-gb5bj"
Sep 10 21:06:01.188287 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:01.188207    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x795k\" (UniqueName: \"kubernetes.io/projected/29b029bb-2f06-49d7-86cf-1e472b06e16c-kube-api-access-x795k\") pod \"maas-controller-58c9bd97bf-gb5bj\" (UID: \"29b029bb-2f06-49d7-86cf-1e472b06e16c\") " pod="opendatahub/maas-controller-58c9bd97bf-gb5bj"
Sep 10 21:06:01.188287 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:06:01.188258    2581 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 10 21:06:01.188498 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:06:01.188342    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29b029bb-2f06-49d7-86cf-1e472b06e16c-metrics-certs podName:29b029bb-2f06-49d7-86cf-1e472b06e16c nodeName:}" failed. No retries permitted until 2026-09-10 21:06:01.688324861 +0000 UTC m=+479.345233624 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29b029bb-2f06-49d7-86cf-1e472b06e16c-metrics-certs") pod "maas-controller-58c9bd97bf-gb5bj" (UID: "29b029bb-2f06-49d7-86cf-1e472b06e16c") : secret "maas-controller-metrics-tls" not found
Sep 10 21:06:01.190700 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:01.190673    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/29b029bb-2f06-49d7-86cf-1e472b06e16c-webhook-cert\") pod \"maas-controller-58c9bd97bf-gb5bj\" (UID: \"29b029bb-2f06-49d7-86cf-1e472b06e16c\") " pod="opendatahub/maas-controller-58c9bd97bf-gb5bj"
Sep 10 21:06:01.199231 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:01.199210    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x795k\" (UniqueName: \"kubernetes.io/projected/29b029bb-2f06-49d7-86cf-1e472b06e16c-kube-api-access-x795k\") pod \"maas-controller-58c9bd97bf-gb5bj\" (UID: \"29b029bb-2f06-49d7-86cf-1e472b06e16c\") " pod="opendatahub/maas-controller-58c9bd97bf-gb5bj"
Sep 10 21:06:01.289622 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:01.289552    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7dpn\" (UniqueName: \"kubernetes.io/projected/e9192d60-ff3d-486d-b34b-9de531284fb7-kube-api-access-p7dpn\") pod \"maas-controller-6d795c6fb-r5g2k\" (UID: \"e9192d60-ff3d-486d-b34b-9de531284fb7\") " pod="opendatahub/maas-controller-6d795c6fb-r5g2k"
Sep 10 21:06:01.289622 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:01.289587    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e9192d60-ff3d-486d-b34b-9de531284fb7-webhook-cert\") pod \"maas-controller-6d795c6fb-r5g2k\" (UID: \"e9192d60-ff3d-486d-b34b-9de531284fb7\") " pod="opendatahub/maas-controller-6d795c6fb-r5g2k"
Sep 10 21:06:01.289622 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:01.289613    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9192d60-ff3d-486d-b34b-9de531284fb7-metrics-certs\") pod \"maas-controller-6d795c6fb-r5g2k\" (UID: \"e9192d60-ff3d-486d-b34b-9de531284fb7\") " pod="opendatahub/maas-controller-6d795c6fb-r5g2k"
Sep 10 21:06:01.390972 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:01.390936    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7dpn\" (UniqueName: \"kubernetes.io/projected/e9192d60-ff3d-486d-b34b-9de531284fb7-kube-api-access-p7dpn\") pod \"maas-controller-6d795c6fb-r5g2k\" (UID: \"e9192d60-ff3d-486d-b34b-9de531284fb7\") " pod="opendatahub/maas-controller-6d795c6fb-r5g2k"
Sep 10 21:06:01.390972 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:01.390974    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e9192d60-ff3d-486d-b34b-9de531284fb7-webhook-cert\") pod \"maas-controller-6d795c6fb-r5g2k\" (UID: \"e9192d60-ff3d-486d-b34b-9de531284fb7\") " pod="opendatahub/maas-controller-6d795c6fb-r5g2k"
Sep 10 21:06:01.391164 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:01.390993    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9192d60-ff3d-486d-b34b-9de531284fb7-metrics-certs\") pod \"maas-controller-6d795c6fb-r5g2k\" (UID: \"e9192d60-ff3d-486d-b34b-9de531284fb7\") " pod="opendatahub/maas-controller-6d795c6fb-r5g2k"
Sep 10 21:06:01.391164 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:06:01.391095    2581 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 10 21:06:01.391164 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:06:01.391160    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9192d60-ff3d-486d-b34b-9de531284fb7-metrics-certs podName:e9192d60-ff3d-486d-b34b-9de531284fb7 nodeName:}" failed. No retries permitted until 2026-09-10 21:06:01.891140566 +0000 UTC m=+479.548049323 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e9192d60-ff3d-486d-b34b-9de531284fb7-metrics-certs") pod "maas-controller-6d795c6fb-r5g2k" (UID: "e9192d60-ff3d-486d-b34b-9de531284fb7") : secret "maas-controller-metrics-tls" not found
Sep 10 21:06:01.393492 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:01.393466    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e9192d60-ff3d-486d-b34b-9de531284fb7-webhook-cert\") pod \"maas-controller-6d795c6fb-r5g2k\" (UID: \"e9192d60-ff3d-486d-b34b-9de531284fb7\") " pod="opendatahub/maas-controller-6d795c6fb-r5g2k"
Sep 10 21:06:01.401869 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:01.401845    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7dpn\" (UniqueName: \"kubernetes.io/projected/e9192d60-ff3d-486d-b34b-9de531284fb7-kube-api-access-p7dpn\") pod \"maas-controller-6d795c6fb-r5g2k\" (UID: \"e9192d60-ff3d-486d-b34b-9de531284fb7\") " pod="opendatahub/maas-controller-6d795c6fb-r5g2k"
Sep 10 21:06:01.693234 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:01.693201    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29b029bb-2f06-49d7-86cf-1e472b06e16c-metrics-certs\") pod \"maas-controller-58c9bd97bf-gb5bj\" (UID: \"29b029bb-2f06-49d7-86cf-1e472b06e16c\") " pod="opendatahub/maas-controller-58c9bd97bf-gb5bj"
Sep 10 21:06:01.695606 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:01.695587    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29b029bb-2f06-49d7-86cf-1e472b06e16c-metrics-certs\") pod \"maas-controller-58c9bd97bf-gb5bj\" (UID: \"29b029bb-2f06-49d7-86cf-1e472b06e16c\") " pod="opendatahub/maas-controller-58c9bd97bf-gb5bj"
Sep 10 21:06:01.894718 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:01.894681    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9192d60-ff3d-486d-b34b-9de531284fb7-metrics-certs\") pod \"maas-controller-6d795c6fb-r5g2k\" (UID: \"e9192d60-ff3d-486d-b34b-9de531284fb7\") " pod="opendatahub/maas-controller-6d795c6fb-r5g2k"
Sep 10 21:06:01.897050 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:01.897030    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9192d60-ff3d-486d-b34b-9de531284fb7-metrics-certs\") pod \"maas-controller-6d795c6fb-r5g2k\" (UID: \"e9192d60-ff3d-486d-b34b-9de531284fb7\") " pod="opendatahub/maas-controller-6d795c6fb-r5g2k"
Sep 10 21:06:01.937969 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:01.937939    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58c9bd97bf-gb5bj"
Sep 10 21:06:02.066471 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:02.066435    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d795c6fb-r5g2k"
Sep 10 21:06:02.208989 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:02.208963    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58c9bd97bf-gb5bj"]
Sep 10 21:06:02.211352 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:06:02.211323    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29b029bb_2f06_49d7_86cf_1e472b06e16c.slice/crio-1929fc7d1245cb623b0afb4e31929651a44e08cdbc09e2b201286456d861eed7 WatchSource:0}: Error finding container 1929fc7d1245cb623b0afb4e31929651a44e08cdbc09e2b201286456d861eed7: Status 404 returned error can't find the container with id 1929fc7d1245cb623b0afb4e31929651a44e08cdbc09e2b201286456d861eed7
Sep 10 21:06:02.339412 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:02.339384    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d795c6fb-r5g2k"]
Sep 10 21:06:02.341617 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:06:02.341589    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9192d60_ff3d_486d_b34b_9de531284fb7.slice/crio-9fc57a3704c479c9749498662840b4ffb05778bb7c765a29c171802910b3a4ce WatchSource:0}: Error finding container 9fc57a3704c479c9749498662840b4ffb05778bb7c765a29c171802910b3a4ce: Status 404 returned error can't find the container with id 9fc57a3704c479c9749498662840b4ffb05778bb7c765a29c171802910b3a4ce
Sep 10 21:06:02.962373 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:02.962327    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58c9bd97bf-gb5bj" event={"ID":"29b029bb-2f06-49d7-86cf-1e472b06e16c","Type":"ContainerStarted","Data":"1929fc7d1245cb623b0afb4e31929651a44e08cdbc09e2b201286456d861eed7"}
Sep 10 21:06:02.963508 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:02.963478    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d795c6fb-r5g2k" event={"ID":"e9192d60-ff3d-486d-b34b-9de531284fb7","Type":"ContainerStarted","Data":"9fc57a3704c479c9749498662840b4ffb05778bb7c765a29c171802910b3a4ce"}
Sep 10 21:06:05.236452 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:05.236427    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 21:06:05.979741 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:05.979704    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58c9bd97bf-gb5bj" event={"ID":"29b029bb-2f06-49d7-86cf-1e472b06e16c","Type":"ContainerStarted","Data":"aec5258458115754bb74ebc0671f7a6704fd943b9a8a9dc8456d257d31db0b1d"}
Sep 10 21:06:05.979948 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:05.979874    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-58c9bd97bf-gb5bj"
Sep 10 21:06:05.981189 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:05.981168    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d795c6fb-r5g2k" event={"ID":"e9192d60-ff3d-486d-b34b-9de531284fb7","Type":"ContainerStarted","Data":"8661e8307f006c62989da69497b92d8397046a632a0c83133bf4ffa97eb3e2ec"}
Sep 10 21:06:05.981306 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:05.981287    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6d795c6fb-r5g2k"
Sep 10 21:06:06.000466 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:06.000417    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-58c9bd97bf-gb5bj" podStartSLOduration=2.973128459 podStartE2EDuration="6.000401895s" totalImagesPullingTime="3.027273436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:06:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:06:05.999911274 +0000 UTC m=+483.656820046" watchObservedRunningTime="2026-09-10 21:06:06.000401895 +0000 UTC m=+483.657310669"
Sep 10 21:06:06.020629 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:06.020579    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6d795c6fb-r5g2k" podStartSLOduration=2.130490553 podStartE2EDuration="5.020565377s" totalImagesPullingTime="2.890074824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:06:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:06:06.018490729 +0000 UTC m=+483.675399514" watchObservedRunningTime="2026-09-10 21:06:06.020565377 +0000 UTC m=+483.677474149"
Sep 10 21:06:16.988751 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:16.988718    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6d795c6fb-r5g2k"
Sep 10 21:06:16.989408 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:16.988791    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-58c9bd97bf-gb5bj"
Sep 10 21:06:17.060709 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:17.060672    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58c9bd97bf-gb5bj"]
Sep 10 21:06:17.060939 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:17.060891    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-58c9bd97bf-gb5bj" podUID="29b029bb-2f06-49d7-86cf-1e472b06e16c" containerName="manager" containerID="cri-o://aec5258458115754bb74ebc0671f7a6704fd943b9a8a9dc8456d257d31db0b1d" gracePeriod=30
Sep 10 21:06:17.307388 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:17.307365    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58c9bd97bf-gb5bj"
Sep 10 21:06:17.435191 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:17.435150    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/29b029bb-2f06-49d7-86cf-1e472b06e16c-webhook-cert\") pod \"29b029bb-2f06-49d7-86cf-1e472b06e16c\" (UID: \"29b029bb-2f06-49d7-86cf-1e472b06e16c\") "
Sep 10 21:06:17.435410 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:17.435200    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x795k\" (UniqueName: \"kubernetes.io/projected/29b029bb-2f06-49d7-86cf-1e472b06e16c-kube-api-access-x795k\") pod \"29b029bb-2f06-49d7-86cf-1e472b06e16c\" (UID: \"29b029bb-2f06-49d7-86cf-1e472b06e16c\") "
Sep 10 21:06:17.435410 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:17.435351    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29b029bb-2f06-49d7-86cf-1e472b06e16c-metrics-certs\") pod \"29b029bb-2f06-49d7-86cf-1e472b06e16c\" (UID: \"29b029bb-2f06-49d7-86cf-1e472b06e16c\") "
Sep 10 21:06:17.437635 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:17.437601    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29b029bb-2f06-49d7-86cf-1e472b06e16c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "29b029bb-2f06-49d7-86cf-1e472b06e16c" (UID: "29b029bb-2f06-49d7-86cf-1e472b06e16c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:06:17.437782 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:17.437633    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29b029bb-2f06-49d7-86cf-1e472b06e16c-kube-api-access-x795k" (OuterVolumeSpecName: "kube-api-access-x795k") pod "29b029bb-2f06-49d7-86cf-1e472b06e16c" (UID: "29b029bb-2f06-49d7-86cf-1e472b06e16c"). InnerVolumeSpecName "kube-api-access-x795k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:06:17.437862 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:17.437843    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29b029bb-2f06-49d7-86cf-1e472b06e16c-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "29b029bb-2f06-49d7-86cf-1e472b06e16c" (UID: "29b029bb-2f06-49d7-86cf-1e472b06e16c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:06:17.536608 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:17.536573    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29b029bb-2f06-49d7-86cf-1e472b06e16c-metrics-certs\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:06:17.536608 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:17.536601    2581 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/29b029bb-2f06-49d7-86cf-1e472b06e16c-webhook-cert\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:06:17.536608 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:17.536610    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x795k\" (UniqueName: \"kubernetes.io/projected/29b029bb-2f06-49d7-86cf-1e472b06e16c-kube-api-access-x795k\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:06:18.037709 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:18.037676    2581 generic.go:356] "Generic (PLEG): container finished" podID="29b029bb-2f06-49d7-86cf-1e472b06e16c" containerID="aec5258458115754bb74ebc0671f7a6704fd943b9a8a9dc8456d257d31db0b1d" exitCode=0
Sep 10 21:06:18.038114 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:18.037722    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58c9bd97bf-gb5bj"
Sep 10 21:06:18.038114 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:18.037759    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58c9bd97bf-gb5bj" event={"ID":"29b029bb-2f06-49d7-86cf-1e472b06e16c","Type":"ContainerDied","Data":"aec5258458115754bb74ebc0671f7a6704fd943b9a8a9dc8456d257d31db0b1d"}
Sep 10 21:06:18.038114 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:18.037811    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58c9bd97bf-gb5bj" event={"ID":"29b029bb-2f06-49d7-86cf-1e472b06e16c","Type":"ContainerDied","Data":"1929fc7d1245cb623b0afb4e31929651a44e08cdbc09e2b201286456d861eed7"}
Sep 10 21:06:18.038114 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:18.037834    2581 scope.go:117] "RemoveContainer" containerID="aec5258458115754bb74ebc0671f7a6704fd943b9a8a9dc8456d257d31db0b1d"
Sep 10 21:06:18.046657 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:18.046638    2581 scope.go:117] "RemoveContainer" containerID="aec5258458115754bb74ebc0671f7a6704fd943b9a8a9dc8456d257d31db0b1d"
Sep 10 21:06:18.046927 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:06:18.046897    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aec5258458115754bb74ebc0671f7a6704fd943b9a8a9dc8456d257d31db0b1d\": container with ID starting with aec5258458115754bb74ebc0671f7a6704fd943b9a8a9dc8456d257d31db0b1d not found: ID does not exist" containerID="aec5258458115754bb74ebc0671f7a6704fd943b9a8a9dc8456d257d31db0b1d"
Sep 10 21:06:18.046970 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:18.046935    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aec5258458115754bb74ebc0671f7a6704fd943b9a8a9dc8456d257d31db0b1d"} err="failed to get container status \"aec5258458115754bb74ebc0671f7a6704fd943b9a8a9dc8456d257d31db0b1d\": rpc error: code = NotFound desc = could not find container \"aec5258458115754bb74ebc0671f7a6704fd943b9a8a9dc8456d257d31db0b1d\": container with ID starting with aec5258458115754bb74ebc0671f7a6704fd943b9a8a9dc8456d257d31db0b1d not found: ID does not exist"
Sep 10 21:06:18.059066 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:18.059040    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58c9bd97bf-gb5bj"]
Sep 10 21:06:18.064603 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:18.064584    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-58c9bd97bf-gb5bj"]
Sep 10 21:06:18.867674 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:18.867645    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29b029bb-2f06-49d7-86cf-1e472b06e16c" path="/var/lib/kubelet/pods/29b029bb-2f06-49d7-86cf-1e472b06e16c/volumes"
Sep 10 21:06:40.245041 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:40.245006    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-chrtv"]
Sep 10 21:06:40.245544 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:40.245491    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29b029bb-2f06-49d7-86cf-1e472b06e16c" containerName="manager"
Sep 10 21:06:40.245544 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:40.245510    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="29b029bb-2f06-49d7-86cf-1e472b06e16c" containerName="manager"
Sep 10 21:06:40.245661 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:40.245601    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="29b029bb-2f06-49d7-86cf-1e472b06e16c" containerName="manager"
Sep 10 21:06:40.250147 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:40.250126    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-chrtv"
Sep 10 21:06:40.252576 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:40.252556    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 10 21:06:40.262814 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:40.262790    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-chrtv"]
Sep 10 21:06:40.340778 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:40.340747    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a87366b4-0438-4d4a-8b0c-6f4973b07de9-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-chrtv\" (UID: \"a87366b4-0438-4d4a-8b0c-6f4973b07de9\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-chrtv"
Sep 10 21:06:40.340923 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:40.340782    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp4bz\" (UniqueName: \"kubernetes.io/projected/a87366b4-0438-4d4a-8b0c-6f4973b07de9-kube-api-access-mp4bz\") pod \"payload-pre-processing-59fbc47bf8-chrtv\" (UID: \"a87366b4-0438-4d4a-8b0c-6f4973b07de9\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-chrtv"
Sep 10 21:06:40.441823 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:40.441782    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a87366b4-0438-4d4a-8b0c-6f4973b07de9-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-chrtv\" (UID: \"a87366b4-0438-4d4a-8b0c-6f4973b07de9\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-chrtv"
Sep 10 21:06:40.441964 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:40.441841    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp4bz\" (UniqueName: \"kubernetes.io/projected/a87366b4-0438-4d4a-8b0c-6f4973b07de9-kube-api-access-mp4bz\") pod \"payload-pre-processing-59fbc47bf8-chrtv\" (UID: \"a87366b4-0438-4d4a-8b0c-6f4973b07de9\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-chrtv"
Sep 10 21:06:40.442592 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:40.442572    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a87366b4-0438-4d4a-8b0c-6f4973b07de9-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-chrtv\" (UID: \"a87366b4-0438-4d4a-8b0c-6f4973b07de9\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-chrtv"
Sep 10 21:06:40.451369 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:40.451342    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp4bz\" (UniqueName: \"kubernetes.io/projected/a87366b4-0438-4d4a-8b0c-6f4973b07de9-kube-api-access-mp4bz\") pod \"payload-pre-processing-59fbc47bf8-chrtv\" (UID: \"a87366b4-0438-4d4a-8b0c-6f4973b07de9\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-chrtv"
Sep 10 21:06:40.560526 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:40.560486    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-chrtv"
Sep 10 21:06:40.832100 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:40.832064    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-chrtv"]
Sep 10 21:06:40.834348 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:06:40.834315    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda87366b4_0438_4d4a_8b0c_6f4973b07de9.slice/crio-8bfa3949fb78e236fa0fa89b2113af06770d1848ab15ad68df9141f47055a1db WatchSource:0}: Error finding container 8bfa3949fb78e236fa0fa89b2113af06770d1848ab15ad68df9141f47055a1db: Status 404 returned error can't find the container with id 8bfa3949fb78e236fa0fa89b2113af06770d1848ab15ad68df9141f47055a1db
Sep 10 21:06:41.139523 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:41.139437    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-chrtv" event={"ID":"a87366b4-0438-4d4a-8b0c-6f4973b07de9","Type":"ContainerStarted","Data":"8bfa3949fb78e236fa0fa89b2113af06770d1848ab15ad68df9141f47055a1db"}
Sep 10 21:06:43.150276 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:43.150244    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-chrtv" event={"ID":"a87366b4-0438-4d4a-8b0c-6f4973b07de9","Type":"ContainerStarted","Data":"7931fa6f24a4baa4c2aba18267a7484b1ea6d1322806362f780f14ae6b93ddb8"}
Sep 10 21:06:43.150737 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:43.150323    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-chrtv"
Sep 10 21:06:43.169033 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:43.168979    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-chrtv" podStartSLOduration=1.569680397 podStartE2EDuration="3.168965227s" totalImagesPullingTime="1.59928483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:06:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:06:43.167490381 +0000 UTC m=+520.824399154" watchObservedRunningTime="2026-09-10 21:06:43.168965227 +0000 UTC m=+520.825873998"
Sep 10 21:06:54.162440 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:06:54.162410    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-chrtv"
Sep 10 21:07:16.523130 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:16.523098    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85"]
Sep 10 21:07:16.526457 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:16.526430    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85"
Sep 10 21:07:16.529431 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:16.529406    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 21:07:16.529553 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:16.529408    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-mbf94\""
Sep 10 21:07:16.529553 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:16.529406    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 21:07:16.529553 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:16.529417    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 10 21:07:16.539004 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:16.538980    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85"]
Sep 10 21:07:16.559144 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:16.559113    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk98v\" (UniqueName: \"kubernetes.io/projected/8c1c5f73-c025-4c6c-b610-a55cd7fd5a9d-kube-api-access-jk98v\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85\" (UID: \"8c1c5f73-c025-4c6c-b610-a55cd7fd5a9d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85"
Sep 10 21:07:16.559261 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:16.559153    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c1c5f73-c025-4c6c-b610-a55cd7fd5a9d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85\" (UID: \"8c1c5f73-c025-4c6c-b610-a55cd7fd5a9d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85"
Sep 10 21:07:16.559261 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:16.559186    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c1c5f73-c025-4c6c-b610-a55cd7fd5a9d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85\" (UID: \"8c1c5f73-c025-4c6c-b610-a55cd7fd5a9d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85"
Sep 10 21:07:16.559261 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:16.559251    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c1c5f73-c025-4c6c-b610-a55cd7fd5a9d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85\" (UID: \"8c1c5f73-c025-4c6c-b610-a55cd7fd5a9d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85"
Sep 10 21:07:16.559391 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:16.559278    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c1c5f73-c025-4c6c-b610-a55cd7fd5a9d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85\" (UID: \"8c1c5f73-c025-4c6c-b610-a55cd7fd5a9d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85"
Sep 10 21:07:16.559391 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:16.559328    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c1c5f73-c025-4c6c-b610-a55cd7fd5a9d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85\" (UID: \"8c1c5f73-c025-4c6c-b610-a55cd7fd5a9d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85"
Sep 10 21:07:16.659737 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:16.659702    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c1c5f73-c025-4c6c-b610-a55cd7fd5a9d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85\" (UID: \"8c1c5f73-c025-4c6c-b610-a55cd7fd5a9d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85"
Sep 10 21:07:16.659737 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:16.659743    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c1c5f73-c025-4c6c-b610-a55cd7fd5a9d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85\" (UID: \"8c1c5f73-c025-4c6c-b610-a55cd7fd5a9d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85"
Sep 10 21:07:16.659976 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:16.659770    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c1c5f73-c025-4c6c-b610-a55cd7fd5a9d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85\" (UID: \"8c1c5f73-c025-4c6c-b610-a55cd7fd5a9d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85"
Sep 10 21:07:16.659976 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:16.659811    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk98v\" (UniqueName: \"kubernetes.io/projected/8c1c5f73-c025-4c6c-b610-a55cd7fd5a9d-kube-api-access-jk98v\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85\" (UID: \"8c1c5f73-c025-4c6c-b610-a55cd7fd5a9d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85"
Sep 10 21:07:16.659976 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:16.659840    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c1c5f73-c025-4c6c-b610-a55cd7fd5a9d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85\" (UID: \"8c1c5f73-c025-4c6c-b610-a55cd7fd5a9d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85"
Sep 10 21:07:16.659976 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:16.659893    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c1c5f73-c025-4c6c-b610-a55cd7fd5a9d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85\" (UID: \"8c1c5f73-c025-4c6c-b610-a55cd7fd5a9d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85"
Sep 10 21:07:16.660189 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:16.660126    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c1c5f73-c025-4c6c-b610-a55cd7fd5a9d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85\" (UID: \"8c1c5f73-c025-4c6c-b610-a55cd7fd5a9d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85"
Sep 10 21:07:16.660289 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:16.660271    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c1c5f73-c025-4c6c-b610-a55cd7fd5a9d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85\" (UID: \"8c1c5f73-c025-4c6c-b610-a55cd7fd5a9d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85"
Sep 10 21:07:16.660381 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:16.660321    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c1c5f73-c025-4c6c-b610-a55cd7fd5a9d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85\" (UID: \"8c1c5f73-c025-4c6c-b610-a55cd7fd5a9d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85"
Sep 10 21:07:16.662202 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:16.662182    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c1c5f73-c025-4c6c-b610-a55cd7fd5a9d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85\" (UID: \"8c1c5f73-c025-4c6c-b610-a55cd7fd5a9d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85"
Sep 10 21:07:16.662410 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:16.662393    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c1c5f73-c025-4c6c-b610-a55cd7fd5a9d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85\" (UID: \"8c1c5f73-c025-4c6c-b610-a55cd7fd5a9d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85"
Sep 10 21:07:16.670617 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:16.670597    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk98v\" (UniqueName: \"kubernetes.io/projected/8c1c5f73-c025-4c6c-b610-a55cd7fd5a9d-kube-api-access-jk98v\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85\" (UID: \"8c1c5f73-c025-4c6c-b610-a55cd7fd5a9d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85"
Sep 10 21:07:16.838068 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:16.838036    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85"
Sep 10 21:07:17.134330 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:17.134273    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85"]
Sep 10 21:07:17.136233 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:07:17.136207    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c1c5f73_c025_4c6c_b610_a55cd7fd5a9d.slice/crio-8aa223975a3107e75aea825589a2689c7fe27a9cb363bf189404096cabe5f530 WatchSource:0}: Error finding container 8aa223975a3107e75aea825589a2689c7fe27a9cb363bf189404096cabe5f530: Status 404 returned error can't find the container with id 8aa223975a3107e75aea825589a2689c7fe27a9cb363bf189404096cabe5f530
Sep 10 21:07:17.306083 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:17.306049    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85" event={"ID":"8c1c5f73-c025-4c6c-b610-a55cd7fd5a9d","Type":"ContainerStarted","Data":"8aa223975a3107e75aea825589a2689c7fe27a9cb363bf189404096cabe5f530"}
Sep 10 21:07:18.311672 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:18.311635    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85" event={"ID":"8c1c5f73-c025-4c6c-b610-a55cd7fd5a9d","Type":"ContainerStarted","Data":"6cdb25898de630cf521a6b4250691c2f2dc0066d4d2fde955a8e5f4249ebfc63"}
Sep 10 21:07:18.329857 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:18.329811    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85" podStartSLOduration=1.3826055529999999 podStartE2EDuration="2.329797489s" totalImagesPullingTime="947.191936ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:07:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:07:18.329228393 +0000 UTC m=+555.986137160" watchObservedRunningTime="2026-09-10 21:07:18.329797489 +0000 UTC m=+555.986706262"
Sep 10 21:07:26.838444 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:26.838403    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85"
Sep 10 21:07:26.838939 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:26.838484    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85"
Sep 10 21:07:26.850681 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:26.850656    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85"
Sep 10 21:07:27.361399 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:27.361373    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85"
Sep 10 21:07:29.397714 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:29.397681    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-xcbjr"]
Sep 10 21:07:29.401166 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:29.401145    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xcbjr"
Sep 10 21:07:29.403318 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:29.403287    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 10 21:07:29.412631 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:29.412604    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-xcbjr"]
Sep 10 21:07:29.455948 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:29.455917    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b57b2f45-34c2-4ca5-9ce8-148b66cb3b49-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xcbjr\" (UID: \"b57b2f45-34c2-4ca5-9ce8-148b66cb3b49\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xcbjr"
Sep 10 21:07:29.456079 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:29.455963    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpw2f\" (UniqueName: \"kubernetes.io/projected/b57b2f45-34c2-4ca5-9ce8-148b66cb3b49-kube-api-access-gpw2f\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xcbjr\" (UID: \"b57b2f45-34c2-4ca5-9ce8-148b66cb3b49\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xcbjr"
Sep 10 21:07:29.456079 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:29.456038    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b57b2f45-34c2-4ca5-9ce8-148b66cb3b49-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xcbjr\" (UID: \"b57b2f45-34c2-4ca5-9ce8-148b66cb3b49\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xcbjr"
Sep 10 21:07:29.456151 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:29.456094    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b57b2f45-34c2-4ca5-9ce8-148b66cb3b49-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xcbjr\" (UID: \"b57b2f45-34c2-4ca5-9ce8-148b66cb3b49\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xcbjr"
Sep 10 21:07:29.456151 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:29.456113    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b57b2f45-34c2-4ca5-9ce8-148b66cb3b49-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xcbjr\" (UID: \"b57b2f45-34c2-4ca5-9ce8-148b66cb3b49\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xcbjr"
Sep 10 21:07:29.456151 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:29.456134    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b57b2f45-34c2-4ca5-9ce8-148b66cb3b49-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xcbjr\" (UID: \"b57b2f45-34c2-4ca5-9ce8-148b66cb3b49\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xcbjr"
Sep 10 21:07:29.556491 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:29.556459    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b57b2f45-34c2-4ca5-9ce8-148b66cb3b49-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xcbjr\" (UID: \"b57b2f45-34c2-4ca5-9ce8-148b66cb3b49\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xcbjr"
Sep 10 21:07:29.556637 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:29.556502    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpw2f\" (UniqueName: \"kubernetes.io/projected/b57b2f45-34c2-4ca5-9ce8-148b66cb3b49-kube-api-access-gpw2f\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xcbjr\" (UID: \"b57b2f45-34c2-4ca5-9ce8-148b66cb3b49\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xcbjr"
Sep 10 21:07:29.556637 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:29.556526    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b57b2f45-34c2-4ca5-9ce8-148b66cb3b49-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xcbjr\" (UID: \"b57b2f45-34c2-4ca5-9ce8-148b66cb3b49\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xcbjr"
Sep 10 21:07:29.556637 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:29.556571    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b57b2f45-34c2-4ca5-9ce8-148b66cb3b49-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xcbjr\" (UID: \"b57b2f45-34c2-4ca5-9ce8-148b66cb3b49\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xcbjr"
Sep 10 21:07:29.556637 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:29.556586    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b57b2f45-34c2-4ca5-9ce8-148b66cb3b49-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xcbjr\" (UID: \"b57b2f45-34c2-4ca5-9ce8-148b66cb3b49\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xcbjr"
Sep 10 21:07:29.556637 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:29.556609    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b57b2f45-34c2-4ca5-9ce8-148b66cb3b49-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xcbjr\" (UID: \"b57b2f45-34c2-4ca5-9ce8-148b66cb3b49\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xcbjr"
Sep 10 21:07:29.557007 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:29.556980    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b57b2f45-34c2-4ca5-9ce8-148b66cb3b49-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xcbjr\" (UID: \"b57b2f45-34c2-4ca5-9ce8-148b66cb3b49\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xcbjr"
Sep 10 21:07:29.557007 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:29.556993    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b57b2f45-34c2-4ca5-9ce8-148b66cb3b49-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xcbjr\" (UID: \"b57b2f45-34c2-4ca5-9ce8-148b66cb3b49\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xcbjr"
Sep 10 21:07:29.557124 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:29.557039    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b57b2f45-34c2-4ca5-9ce8-148b66cb3b49-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xcbjr\" (UID: \"b57b2f45-34c2-4ca5-9ce8-148b66cb3b49\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xcbjr"
Sep 10 21:07:29.558736 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:29.558718    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b57b2f45-34c2-4ca5-9ce8-148b66cb3b49-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xcbjr\" (UID: \"b57b2f45-34c2-4ca5-9ce8-148b66cb3b49\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xcbjr"
Sep 10 21:07:29.558899 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:29.558883    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b57b2f45-34c2-4ca5-9ce8-148b66cb3b49-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xcbjr\" (UID: \"b57b2f45-34c2-4ca5-9ce8-148b66cb3b49\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xcbjr"
Sep 10 21:07:29.564675 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:29.564654    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpw2f\" (UniqueName: \"kubernetes.io/projected/b57b2f45-34c2-4ca5-9ce8-148b66cb3b49-kube-api-access-gpw2f\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xcbjr\" (UID: \"b57b2f45-34c2-4ca5-9ce8-148b66cb3b49\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xcbjr"
Sep 10 21:07:29.712687 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:29.712622    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xcbjr"
Sep 10 21:07:29.984950 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:29.984923    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-xcbjr"]
Sep 10 21:07:29.986926 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:07:29.986897    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb57b2f45_34c2_4ca5_9ce8_148b66cb3b49.slice/crio-1350af4d969b29c483084d5cf57408bd0178ac64193586ccd372100a379fcbd2 WatchSource:0}: Error finding container 1350af4d969b29c483084d5cf57408bd0178ac64193586ccd372100a379fcbd2: Status 404 returned error can't find the container with id 1350af4d969b29c483084d5cf57408bd0178ac64193586ccd372100a379fcbd2
Sep 10 21:07:30.370753 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:30.370722    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xcbjr" event={"ID":"b57b2f45-34c2-4ca5-9ce8-148b66cb3b49","Type":"ContainerStarted","Data":"a633e970ff6d988b06dc2a2b44a491099dc993deeb1f6ad26fcd1e3c247250f2"}
Sep 10 21:07:30.370931 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:30.370803    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xcbjr" event={"ID":"b57b2f45-34c2-4ca5-9ce8-148b66cb3b49","Type":"ContainerStarted","Data":"1350af4d969b29c483084d5cf57408bd0178ac64193586ccd372100a379fcbd2"}
Sep 10 21:07:30.391095 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:30.391046    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xcbjr" podStartSLOduration=1.160196514 podStartE2EDuration="1.391033944s" totalImagesPullingTime="230.83743ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:07:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:07:30.38955151 +0000 UTC m=+568.046460306" watchObservedRunningTime="2026-09-10 21:07:30.391033944 +0000 UTC m=+568.047942715"
Sep 10 21:07:39.712964 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:39.712915    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xcbjr"
Sep 10 21:07:39.713476 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:39.712989    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xcbjr"
Sep 10 21:07:39.725364 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:39.725339    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xcbjr"
Sep 10 21:07:40.425889 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:40.425861    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xcbjr"
Sep 10 21:07:48.714884 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:48.714847    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-6gpmj"]
Sep 10 21:07:48.719096 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:48.719077    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6gpmj"
Sep 10 21:07:48.721569 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:48.721545    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 10 21:07:48.727060 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:48.727028    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-6gpmj"]
Sep 10 21:07:48.822816 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:48.822778    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff227acf-0519-4216-abcf-ece643c1bc05-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6gpmj\" (UID: \"ff227acf-0519-4216-abcf-ece643c1bc05\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6gpmj"
Sep 10 21:07:48.822985 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:48.822827    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff227acf-0519-4216-abcf-ece643c1bc05-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6gpmj\" (UID: \"ff227acf-0519-4216-abcf-ece643c1bc05\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6gpmj"
Sep 10 21:07:48.822985 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:48.822909    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff227acf-0519-4216-abcf-ece643c1bc05-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6gpmj\" (UID: \"ff227acf-0519-4216-abcf-ece643c1bc05\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6gpmj"
Sep 10 21:07:48.822985 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:48.822938    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch89l\" (UniqueName: \"kubernetes.io/projected/ff227acf-0519-4216-abcf-ece643c1bc05-kube-api-access-ch89l\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6gpmj\" (UID: \"ff227acf-0519-4216-abcf-ece643c1bc05\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6gpmj"
Sep 10 21:07:48.822985 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:48.822970    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff227acf-0519-4216-abcf-ece643c1bc05-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6gpmj\" (UID: \"ff227acf-0519-4216-abcf-ece643c1bc05\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6gpmj"
Sep 10 21:07:48.823127 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:48.823016    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff227acf-0519-4216-abcf-ece643c1bc05-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6gpmj\" (UID: \"ff227acf-0519-4216-abcf-ece643c1bc05\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6gpmj"
Sep 10 21:07:48.923646 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:48.923614    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff227acf-0519-4216-abcf-ece643c1bc05-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6gpmj\" (UID: \"ff227acf-0519-4216-abcf-ece643c1bc05\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6gpmj"
Sep 10 21:07:48.923804 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:48.923661    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff227acf-0519-4216-abcf-ece643c1bc05-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6gpmj\" (UID: \"ff227acf-0519-4216-abcf-ece643c1bc05\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6gpmj"
Sep 10 21:07:48.923804 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:48.923789    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff227acf-0519-4216-abcf-ece643c1bc05-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6gpmj\" (UID: \"ff227acf-0519-4216-abcf-ece643c1bc05\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6gpmj"
Sep 10 21:07:48.923910 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:48.923853    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff227acf-0519-4216-abcf-ece643c1bc05-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6gpmj\" (UID: \"ff227acf-0519-4216-abcf-ece643c1bc05\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6gpmj"
Sep 10 21:07:48.923966 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:48.923943    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff227acf-0519-4216-abcf-ece643c1bc05-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6gpmj\" (UID: \"ff227acf-0519-4216-abcf-ece643c1bc05\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6gpmj"
Sep 10 21:07:48.924016 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:48.923972    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch89l\" (UniqueName: \"kubernetes.io/projected/ff227acf-0519-4216-abcf-ece643c1bc05-kube-api-access-ch89l\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6gpmj\" (UID: \"ff227acf-0519-4216-abcf-ece643c1bc05\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6gpmj"
Sep 10 21:07:48.924095 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:48.924066    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff227acf-0519-4216-abcf-ece643c1bc05-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6gpmj\" (UID: \"ff227acf-0519-4216-abcf-ece643c1bc05\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6gpmj"
Sep 10 21:07:48.924095 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:48.924084    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff227acf-0519-4216-abcf-ece643c1bc05-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6gpmj\" (UID: \"ff227acf-0519-4216-abcf-ece643c1bc05\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6gpmj"
Sep 10 21:07:48.924399 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:48.924377    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff227acf-0519-4216-abcf-ece643c1bc05-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6gpmj\" (UID: \"ff227acf-0519-4216-abcf-ece643c1bc05\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6gpmj"
Sep 10 21:07:48.926331 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:48.926285    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff227acf-0519-4216-abcf-ece643c1bc05-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6gpmj\" (UID: \"ff227acf-0519-4216-abcf-ece643c1bc05\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6gpmj"
Sep 10 21:07:48.926458 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:48.926444    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff227acf-0519-4216-abcf-ece643c1bc05-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6gpmj\" (UID: \"ff227acf-0519-4216-abcf-ece643c1bc05\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6gpmj"
Sep 10 21:07:48.932416 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:48.932396    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch89l\" (UniqueName: \"kubernetes.io/projected/ff227acf-0519-4216-abcf-ece643c1bc05-kube-api-access-ch89l\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6gpmj\" (UID: \"ff227acf-0519-4216-abcf-ece643c1bc05\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6gpmj"
Sep 10 21:07:49.032356 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:49.032249    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6gpmj"
Sep 10 21:07:49.300870 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:49.300771    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-6gpmj"]
Sep 10 21:07:49.302954 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:07:49.302924    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff227acf_0519_4216_abcf_ece643c1bc05.slice/crio-008084f22335e88af87b9efcd605852a841c946a924ebbe9d01278473a23f2ac WatchSource:0}: Error finding container 008084f22335e88af87b9efcd605852a841c946a924ebbe9d01278473a23f2ac: Status 404 returned error can't find the container with id 008084f22335e88af87b9efcd605852a841c946a924ebbe9d01278473a23f2ac
Sep 10 21:07:49.454632 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:49.454593    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6gpmj" event={"ID":"ff227acf-0519-4216-abcf-ece643c1bc05","Type":"ContainerStarted","Data":"008084f22335e88af87b9efcd605852a841c946a924ebbe9d01278473a23f2ac"}
Sep 10 21:07:50.460699 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:50.460662    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6gpmj" event={"ID":"ff227acf-0519-4216-abcf-ece643c1bc05","Type":"ContainerStarted","Data":"d1c629070148aa55e1355184dc978ac8a188a5faf67032bc19b1df9122331bd5"}
Sep 10 21:07:50.477311 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:50.477232    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6gpmj" podStartSLOduration=2.309901275 podStartE2EDuration="2.477218969s" totalImagesPullingTime="167.317694ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:07:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:07:50.476123938 +0000 UTC m=+588.133032710" watchObservedRunningTime="2026-09-10 21:07:50.477218969 +0000 UTC m=+588.134127741"
Sep 10 21:07:59.032784 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:59.032747    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6gpmj"
Sep 10 21:07:59.033175 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:59.032808    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6gpmj"
Sep 10 21:07:59.045379 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:59.045349    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6gpmj"
Sep 10 21:07:59.510340 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:07:59.510286    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6gpmj"
Sep 10 21:08:02.781120 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:08:02.781091    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-z6858_c1e06766-b48c-429e-92f1-196bb083360b/console-operator/1.log"
Sep 10 21:08:02.782938 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:08:02.782914    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-z6858_c1e06766-b48c-429e-92f1-196bb083360b/console-operator/1.log"
Sep 10 21:08:02.784924 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:08:02.784903    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brrtd_19d8d7ba-45a1-4013-a935-5670abe12b8a/ovn-acl-logging/0.log"
Sep 10 21:08:02.786741 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:08:02.786722    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brrtd_19d8d7ba-45a1-4013-a935-5670abe12b8a/ovn-acl-logging/0.log"
Sep 10 21:09:48.342767 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:48.342737    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-47f62f-66fbbbb67-hzspx"]
Sep 10 21:09:48.346078 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:48.346059    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-47f62f-66fbbbb67-hzspx"
Sep 10 21:09:48.348606 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:48.348583    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-47f62f-dockercfg-xp6j4\""
Sep 10 21:09:48.348719 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:48.348635    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-47f62f\""
Sep 10 21:09:48.361203 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:48.361174    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-47f62f-66fbbbb67-hzspx"]
Sep 10 21:09:48.378705 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:48.378680    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aec5e2ad-edad-4725-86e4-85ba96a70cbc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-47f62f-66fbbbb67-hzspx\" (UID: \"aec5e2ad-edad-4725-86e4-85ba96a70cbc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-47f62f-66fbbbb67-hzspx"
Sep 10 21:09:48.378809 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:48.378716    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr9h7\" (UniqueName: \"kubernetes.io/projected/aec5e2ad-edad-4725-86e4-85ba96a70cbc-kube-api-access-dr9h7\") pod \"payload-processing-e2e-inf-a-47f62f-66fbbbb67-hzspx\" (UID: \"aec5e2ad-edad-4725-86e4-85ba96a70cbc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-47f62f-66fbbbb67-hzspx"
Sep 10 21:09:48.401190 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:48.401162    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-47f62f-7656d8ffdb-xht9v"]
Sep 10 21:09:48.404503 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:48.404482    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-47f62f-7656d8ffdb-xht9v"
Sep 10 21:09:48.414895 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:48.414871    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-47f62f-7656d8ffdb-xht9v"]
Sep 10 21:09:48.479938 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:48.479897    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aec5e2ad-edad-4725-86e4-85ba96a70cbc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-47f62f-66fbbbb67-hzspx\" (UID: \"aec5e2ad-edad-4725-86e4-85ba96a70cbc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-47f62f-66fbbbb67-hzspx"
Sep 10 21:09:48.480111 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:48.479952    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr9h7\" (UniqueName: \"kubernetes.io/projected/aec5e2ad-edad-4725-86e4-85ba96a70cbc-kube-api-access-dr9h7\") pod \"payload-processing-e2e-inf-a-47f62f-66fbbbb67-hzspx\" (UID: \"aec5e2ad-edad-4725-86e4-85ba96a70cbc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-47f62f-66fbbbb67-hzspx"
Sep 10 21:09:48.480111 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:48.480056    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72088031-9f18-43ae-b436-0940d32a5492-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-47f62f-7656d8ffdb-xht9v\" (UID: \"72088031-9f18-43ae-b436-0940d32a5492\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-47f62f-7656d8ffdb-xht9v"
Sep 10 21:09:48.480235 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:48.480158    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpqzq\" (UniqueName: \"kubernetes.io/projected/72088031-9f18-43ae-b436-0940d32a5492-kube-api-access-hpqzq\") pod \"payload-pre-processing-e2e-inf-a-47f62f-7656d8ffdb-xht9v\" (UID: \"72088031-9f18-43ae-b436-0940d32a5492\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-47f62f-7656d8ffdb-xht9v"
Sep 10 21:09:48.480750 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:48.480720    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aec5e2ad-edad-4725-86e4-85ba96a70cbc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-47f62f-66fbbbb67-hzspx\" (UID: \"aec5e2ad-edad-4725-86e4-85ba96a70cbc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-47f62f-66fbbbb67-hzspx"
Sep 10 21:09:48.489103 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:48.489081    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr9h7\" (UniqueName: \"kubernetes.io/projected/aec5e2ad-edad-4725-86e4-85ba96a70cbc-kube-api-access-dr9h7\") pod \"payload-processing-e2e-inf-a-47f62f-66fbbbb67-hzspx\" (UID: \"aec5e2ad-edad-4725-86e4-85ba96a70cbc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-47f62f-66fbbbb67-hzspx"
Sep 10 21:09:48.581237 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:48.581190    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpqzq\" (UniqueName: \"kubernetes.io/projected/72088031-9f18-43ae-b436-0940d32a5492-kube-api-access-hpqzq\") pod \"payload-pre-processing-e2e-inf-a-47f62f-7656d8ffdb-xht9v\" (UID: \"72088031-9f18-43ae-b436-0940d32a5492\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-47f62f-7656d8ffdb-xht9v"
Sep 10 21:09:48.581425 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:48.581290    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72088031-9f18-43ae-b436-0940d32a5492-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-47f62f-7656d8ffdb-xht9v\" (UID: \"72088031-9f18-43ae-b436-0940d32a5492\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-47f62f-7656d8ffdb-xht9v"
Sep 10 21:09:48.581995 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:48.581971    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72088031-9f18-43ae-b436-0940d32a5492-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-47f62f-7656d8ffdb-xht9v\" (UID: \"72088031-9f18-43ae-b436-0940d32a5492\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-47f62f-7656d8ffdb-xht9v"
Sep 10 21:09:48.598803 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:48.598740    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpqzq\" (UniqueName: \"kubernetes.io/projected/72088031-9f18-43ae-b436-0940d32a5492-kube-api-access-hpqzq\") pod \"payload-pre-processing-e2e-inf-a-47f62f-7656d8ffdb-xht9v\" (UID: \"72088031-9f18-43ae-b436-0940d32a5492\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-47f62f-7656d8ffdb-xht9v"
Sep 10 21:09:48.657725 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:48.657696    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-47f62f-66fbbbb67-hzspx"
Sep 10 21:09:48.715658 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:48.715630    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-47f62f-7656d8ffdb-xht9v"
Sep 10 21:09:48.941856 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:48.941821    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-47f62f-66fbbbb67-hzspx"]
Sep 10 21:09:48.944143 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:09:48.944117    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaec5e2ad_edad_4725_86e4_85ba96a70cbc.slice/crio-21b503b4b31eb257bf3658a5959864ec8e11a2f141c33389eb6be47ea9279da7 WatchSource:0}: Error finding container 21b503b4b31eb257bf3658a5959864ec8e11a2f141c33389eb6be47ea9279da7: Status 404 returned error can't find the container with id 21b503b4b31eb257bf3658a5959864ec8e11a2f141c33389eb6be47ea9279da7
Sep 10 21:09:48.994588 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:48.994551    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-47f62f-66fbbbb67-hzspx" event={"ID":"aec5e2ad-edad-4725-86e4-85ba96a70cbc","Type":"ContainerStarted","Data":"21b503b4b31eb257bf3658a5959864ec8e11a2f141c33389eb6be47ea9279da7"}
Sep 10 21:09:48.994795 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:48.994771    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-47f62f-7656d8ffdb-xht9v"]
Sep 10 21:09:48.997216 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:09:48.997191    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72088031_9f18_43ae_b436_0940d32a5492.slice/crio-0c0e19d0f1f8ab2ed43af1ffbcebfaa8dd3844c06c116fa90f3e1d2bdcd65ba8 WatchSource:0}: Error finding container 0c0e19d0f1f8ab2ed43af1ffbcebfaa8dd3844c06c116fa90f3e1d2bdcd65ba8: Status 404 returned error can't find the container with id 0c0e19d0f1f8ab2ed43af1ffbcebfaa8dd3844c06c116fa90f3e1d2bdcd65ba8
Sep 10 21:09:50.001306 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:50.001245    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-47f62f-66fbbbb67-hzspx" event={"ID":"aec5e2ad-edad-4725-86e4-85ba96a70cbc","Type":"ContainerStarted","Data":"638f670c664674c43a6c840eac96e92dec76295e9a3af0323928d207bc7c8fa7"}
Sep 10 21:09:50.001756 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:50.001343    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-47f62f-66fbbbb67-hzspx"
Sep 10 21:09:50.003394 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:50.003361    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-47f62f-7656d8ffdb-xht9v" event={"ID":"72088031-9f18-43ae-b436-0940d32a5492","Type":"ContainerStarted","Data":"af71ec2e051504f7902d8921a0e34c7edb14a1b98ff580e863d0b6cadcf1628a"}
Sep 10 21:09:50.003506 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:50.003410    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-47f62f-7656d8ffdb-xht9v" event={"ID":"72088031-9f18-43ae-b436-0940d32a5492","Type":"ContainerStarted","Data":"0c0e19d0f1f8ab2ed43af1ffbcebfaa8dd3844c06c116fa90f3e1d2bdcd65ba8"}
Sep 10 21:09:50.003506 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:50.003444    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-47f62f-7656d8ffdb-xht9v"
Sep 10 21:09:50.024214 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:50.024169    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-47f62f-66fbbbb67-hzspx" podStartSLOduration=2.024155735 podStartE2EDuration="2.024155735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:09:50.021757214 +0000 UTC m=+707.678665985" watchObservedRunningTime="2026-09-10 21:09:50.024155735 +0000 UTC m=+707.681064556"
Sep 10 21:09:50.041701 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:50.041657    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-47f62f-7656d8ffdb-xht9v" podStartSLOduration=2.041643301 podStartE2EDuration="2.041643301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:09:50.038587861 +0000 UTC m=+707.695496633" watchObservedRunningTime="2026-09-10 21:09:50.041643301 +0000 UTC m=+707.698552073"
Sep 10 21:09:56.770992 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:56.770904    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1a398fc7-869656d44b-djr42"]
Sep 10 21:09:56.775542 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:56.775519    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a398fc7-869656d44b-djr42"
Sep 10 21:09:56.778723 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:56.778703    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1a398fc7-dockercfg-sx8pc\""
Sep 10 21:09:56.778816 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:56.778730    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1a398fc7\""
Sep 10 21:09:56.797370 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:56.797346    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1a398fc7-869656d44b-djr42"]
Sep 10 21:09:56.846400 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:56.846360    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hckh8\" (UniqueName: \"kubernetes.io/projected/3f6a0984-3589-4518-ba0c-e6f7a86d35dc-kube-api-access-hckh8\") pod \"payload-pre-processing-e2e-ait-1a398fc7-869656d44b-djr42\" (UID: \"3f6a0984-3589-4518-ba0c-e6f7a86d35dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1a398fc7-869656d44b-djr42"
Sep 10 21:09:56.846589 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:56.846420    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f6a0984-3589-4518-ba0c-e6f7a86d35dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1a398fc7-869656d44b-djr42\" (UID: \"3f6a0984-3589-4518-ba0c-e6f7a86d35dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1a398fc7-869656d44b-djr42"
Sep 10 21:09:56.875796 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:56.875767    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1a398fc7-658fb86699-8dzw7"]
Sep 10 21:09:56.879460 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:56.879439    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1a398fc7-658fb86699-8dzw7"
Sep 10 21:09:56.898082 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:56.898057    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1a398fc7-658fb86699-8dzw7"]
Sep 10 21:09:56.947117 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:56.947085    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hckh8\" (UniqueName: \"kubernetes.io/projected/3f6a0984-3589-4518-ba0c-e6f7a86d35dc-kube-api-access-hckh8\") pod \"payload-pre-processing-e2e-ait-1a398fc7-869656d44b-djr42\" (UID: \"3f6a0984-3589-4518-ba0c-e6f7a86d35dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1a398fc7-869656d44b-djr42"
Sep 10 21:09:56.947250 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:56.947121    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/037bdae4-9edb-4705-b2f2-a394364bb5c8-plugins-config-volume\") pod \"payload-processing-e2e-ait-1a398fc7-658fb86699-8dzw7\" (UID: \"037bdae4-9edb-4705-b2f2-a394364bb5c8\") " pod="openshift-ingress/payload-processing-e2e-ait-1a398fc7-658fb86699-8dzw7"
Sep 10 21:09:56.947250 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:56.947157    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f6a0984-3589-4518-ba0c-e6f7a86d35dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1a398fc7-869656d44b-djr42\" (UID: \"3f6a0984-3589-4518-ba0c-e6f7a86d35dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1a398fc7-869656d44b-djr42"
Sep 10 21:09:56.947362 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:56.947239    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vrrg\" (UniqueName: \"kubernetes.io/projected/037bdae4-9edb-4705-b2f2-a394364bb5c8-kube-api-access-8vrrg\") pod \"payload-processing-e2e-ait-1a398fc7-658fb86699-8dzw7\" (UID: \"037bdae4-9edb-4705-b2f2-a394364bb5c8\") " pod="openshift-ingress/payload-processing-e2e-ait-1a398fc7-658fb86699-8dzw7"
Sep 10 21:09:56.947806 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:56.947789    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f6a0984-3589-4518-ba0c-e6f7a86d35dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1a398fc7-869656d44b-djr42\" (UID: \"3f6a0984-3589-4518-ba0c-e6f7a86d35dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1a398fc7-869656d44b-djr42"
Sep 10 21:09:56.956412 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:56.956389    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hckh8\" (UniqueName: \"kubernetes.io/projected/3f6a0984-3589-4518-ba0c-e6f7a86d35dc-kube-api-access-hckh8\") pod \"payload-pre-processing-e2e-ait-1a398fc7-869656d44b-djr42\" (UID: \"3f6a0984-3589-4518-ba0c-e6f7a86d35dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1a398fc7-869656d44b-djr42"
Sep 10 21:09:57.048270 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:57.048237    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/037bdae4-9edb-4705-b2f2-a394364bb5c8-plugins-config-volume\") pod \"payload-processing-e2e-ait-1a398fc7-658fb86699-8dzw7\" (UID: \"037bdae4-9edb-4705-b2f2-a394364bb5c8\") " pod="openshift-ingress/payload-processing-e2e-ait-1a398fc7-658fb86699-8dzw7"
Sep 10 21:09:57.048441 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:57.048289    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vrrg\" (UniqueName: \"kubernetes.io/projected/037bdae4-9edb-4705-b2f2-a394364bb5c8-kube-api-access-8vrrg\") pod \"payload-processing-e2e-ait-1a398fc7-658fb86699-8dzw7\" (UID: \"037bdae4-9edb-4705-b2f2-a394364bb5c8\") " pod="openshift-ingress/payload-processing-e2e-ait-1a398fc7-658fb86699-8dzw7"
Sep 10 21:09:57.049043 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:57.049017    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/037bdae4-9edb-4705-b2f2-a394364bb5c8-plugins-config-volume\") pod \"payload-processing-e2e-ait-1a398fc7-658fb86699-8dzw7\" (UID: \"037bdae4-9edb-4705-b2f2-a394364bb5c8\") " pod="openshift-ingress/payload-processing-e2e-ait-1a398fc7-658fb86699-8dzw7"
Sep 10 21:09:57.063125 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:57.063096    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vrrg\" (UniqueName: \"kubernetes.io/projected/037bdae4-9edb-4705-b2f2-a394364bb5c8-kube-api-access-8vrrg\") pod \"payload-processing-e2e-ait-1a398fc7-658fb86699-8dzw7\" (UID: \"037bdae4-9edb-4705-b2f2-a394364bb5c8\") " pod="openshift-ingress/payload-processing-e2e-ait-1a398fc7-658fb86699-8dzw7"
Sep 10 21:09:57.086404 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:57.086374    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a398fc7-869656d44b-djr42"
Sep 10 21:09:57.189775 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:57.189746    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1a398fc7-658fb86699-8dzw7"
Sep 10 21:09:57.371947 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:57.371895    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1a398fc7-869656d44b-djr42"]
Sep 10 21:09:57.374603 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:09:57.374570    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f6a0984_3589_4518_ba0c_e6f7a86d35dc.slice/crio-338be21dc51481bae53889b47397d6d43bdf70a60551b60da924df5a510893e9 WatchSource:0}: Error finding container 338be21dc51481bae53889b47397d6d43bdf70a60551b60da924df5a510893e9: Status 404 returned error can't find the container with id 338be21dc51481bae53889b47397d6d43bdf70a60551b60da924df5a510893e9
Sep 10 21:09:57.478764 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:57.477082    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1a398fc7-658fb86699-8dzw7"]
Sep 10 21:09:57.487664 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:09:57.487633    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod037bdae4_9edb_4705_b2f2_a394364bb5c8.slice/crio-312f88dfe3d0230ccebf69b3708c9dfd41ac03fe4cf65150a8aeec9c88e3a5ab WatchSource:0}: Error finding container 312f88dfe3d0230ccebf69b3708c9dfd41ac03fe4cf65150a8aeec9c88e3a5ab: Status 404 returned error can't find the container with id 312f88dfe3d0230ccebf69b3708c9dfd41ac03fe4cf65150a8aeec9c88e3a5ab
Sep 10 21:09:58.042488 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:58.042448    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1a398fc7-658fb86699-8dzw7" event={"ID":"037bdae4-9edb-4705-b2f2-a394364bb5c8","Type":"ContainerStarted","Data":"c581d2c578105cbc821bfb0bb3a912544da399f04ae3f74b1c321716d73c7069"}
Sep 10 21:09:58.042951 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:58.042508    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1a398fc7-658fb86699-8dzw7" event={"ID":"037bdae4-9edb-4705-b2f2-a394364bb5c8","Type":"ContainerStarted","Data":"312f88dfe3d0230ccebf69b3708c9dfd41ac03fe4cf65150a8aeec9c88e3a5ab"}
Sep 10 21:09:58.042951 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:58.042616    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-1a398fc7-658fb86699-8dzw7"
Sep 10 21:09:58.044073 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:58.044048    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a398fc7-869656d44b-djr42" event={"ID":"3f6a0984-3589-4518-ba0c-e6f7a86d35dc","Type":"ContainerStarted","Data":"9279de00acbb8f49870ced6cd8ccf726ffb1ce627ad6450ce8646660dd783fbd"}
Sep 10 21:09:58.044151 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:58.044093    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a398fc7-869656d44b-djr42" event={"ID":"3f6a0984-3589-4518-ba0c-e6f7a86d35dc","Type":"ContainerStarted","Data":"338be21dc51481bae53889b47397d6d43bdf70a60551b60da924df5a510893e9"}
Sep 10 21:09:58.044151 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:58.044138    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a398fc7-869656d44b-djr42"
Sep 10 21:09:58.095009 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:58.094939    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-1a398fc7-658fb86699-8dzw7" podStartSLOduration=2.094915268 podStartE2EDuration="2.094915268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:09:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:09:58.093101532 +0000 UTC m=+715.750010319" watchObservedRunningTime="2026-09-10 21:09:58.094915268 +0000 UTC m=+715.751824043"
Sep 10 21:09:58.248193 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:58.248121    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a398fc7-869656d44b-djr42" podStartSLOduration=2.248096288 podStartE2EDuration="2.248096288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:09:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:09:58.128971406 +0000 UTC m=+715.785880183" watchObservedRunningTime="2026-09-10 21:09:58.248096288 +0000 UTC m=+715.905005063"
Sep 10 21:09:58.248990 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:58.248967    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-63f3d7-69c755474-ld7d2"]
Sep 10 21:09:58.252987 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:58.252966    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-63f3d7-69c755474-ld7d2"
Sep 10 21:09:58.259898 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:58.259870    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-63f3d7\""
Sep 10 21:09:58.260081 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:58.260058    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-63f3d7-dockercfg-9d6rm\""
Sep 10 21:09:58.271269 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:58.271239    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-63f3d7-69c755474-ld7d2"]
Sep 10 21:09:58.360689 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:58.360655    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx2w6\" (UniqueName: \"kubernetes.io/projected/ac06edc0-36b0-4be7-ad07-32edf0982ecb-kube-api-access-wx2w6\") pod \"payload-pre-processing-e2e-autores-63f3d7-69c755474-ld7d2\" (UID: \"ac06edc0-36b0-4be7-ad07-32edf0982ecb\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-63f3d7-69c755474-ld7d2"
Sep 10 21:09:58.360818 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:58.360723    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac06edc0-36b0-4be7-ad07-32edf0982ecb-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-63f3d7-69c755474-ld7d2\" (UID: \"ac06edc0-36b0-4be7-ad07-32edf0982ecb\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-63f3d7-69c755474-ld7d2"
Sep 10 21:09:58.375150 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:58.375123    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-63f3d7-64b49bd9c9-k9chg"]
Sep 10 21:09:58.378975 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:58.378956    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-63f3d7-64b49bd9c9-k9chg"
Sep 10 21:09:58.389644 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:58.389623    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-63f3d7-64b49bd9c9-k9chg"]
Sep 10 21:09:58.461668 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:58.461635    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16ce1986-676d-495a-93f0-6ecf0c6061a7-plugins-config-volume\") pod \"payload-processing-e2e-autores-63f3d7-64b49bd9c9-k9chg\" (UID: \"16ce1986-676d-495a-93f0-6ecf0c6061a7\") " pod="openshift-ingress/payload-processing-e2e-autores-63f3d7-64b49bd9c9-k9chg"
Sep 10 21:09:58.461846 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:58.461701    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx2w6\" (UniqueName: \"kubernetes.io/projected/ac06edc0-36b0-4be7-ad07-32edf0982ecb-kube-api-access-wx2w6\") pod \"payload-pre-processing-e2e-autores-63f3d7-69c755474-ld7d2\" (UID: \"ac06edc0-36b0-4be7-ad07-32edf0982ecb\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-63f3d7-69c755474-ld7d2"
Sep 10 21:09:58.461846 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:58.461755    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj4qv\" (UniqueName: \"kubernetes.io/projected/16ce1986-676d-495a-93f0-6ecf0c6061a7-kube-api-access-nj4qv\") pod \"payload-processing-e2e-autores-63f3d7-64b49bd9c9-k9chg\" (UID: \"16ce1986-676d-495a-93f0-6ecf0c6061a7\") " pod="openshift-ingress/payload-processing-e2e-autores-63f3d7-64b49bd9c9-k9chg"
Sep 10 21:09:58.461846 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:58.461816    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac06edc0-36b0-4be7-ad07-32edf0982ecb-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-63f3d7-69c755474-ld7d2\" (UID: \"ac06edc0-36b0-4be7-ad07-32edf0982ecb\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-63f3d7-69c755474-ld7d2"
Sep 10 21:09:58.462703 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:58.462679    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac06edc0-36b0-4be7-ad07-32edf0982ecb-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-63f3d7-69c755474-ld7d2\" (UID: \"ac06edc0-36b0-4be7-ad07-32edf0982ecb\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-63f3d7-69c755474-ld7d2"
Sep 10 21:09:58.472487 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:58.472459    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx2w6\" (UniqueName: \"kubernetes.io/projected/ac06edc0-36b0-4be7-ad07-32edf0982ecb-kube-api-access-wx2w6\") pod \"payload-pre-processing-e2e-autores-63f3d7-69c755474-ld7d2\" (UID: \"ac06edc0-36b0-4be7-ad07-32edf0982ecb\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-63f3d7-69c755474-ld7d2"
Sep 10 21:09:58.563178 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:58.563137    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16ce1986-676d-495a-93f0-6ecf0c6061a7-plugins-config-volume\") pod \"payload-processing-e2e-autores-63f3d7-64b49bd9c9-k9chg\" (UID: \"16ce1986-676d-495a-93f0-6ecf0c6061a7\") " pod="openshift-ingress/payload-processing-e2e-autores-63f3d7-64b49bd9c9-k9chg"
Sep 10 21:09:58.563330 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:58.563205    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj4qv\" (UniqueName: \"kubernetes.io/projected/16ce1986-676d-495a-93f0-6ecf0c6061a7-kube-api-access-nj4qv\") pod \"payload-processing-e2e-autores-63f3d7-64b49bd9c9-k9chg\" (UID: \"16ce1986-676d-495a-93f0-6ecf0c6061a7\") " pod="openshift-ingress/payload-processing-e2e-autores-63f3d7-64b49bd9c9-k9chg"
Sep 10 21:09:58.564049 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:58.564025    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16ce1986-676d-495a-93f0-6ecf0c6061a7-plugins-config-volume\") pod \"payload-processing-e2e-autores-63f3d7-64b49bd9c9-k9chg\" (UID: \"16ce1986-676d-495a-93f0-6ecf0c6061a7\") " pod="openshift-ingress/payload-processing-e2e-autores-63f3d7-64b49bd9c9-k9chg"
Sep 10 21:09:58.565948 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:58.565921    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-63f3d7-69c755474-ld7d2"
Sep 10 21:09:58.574153 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:58.574127    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj4qv\" (UniqueName: \"kubernetes.io/projected/16ce1986-676d-495a-93f0-6ecf0c6061a7-kube-api-access-nj4qv\") pod \"payload-processing-e2e-autores-63f3d7-64b49bd9c9-k9chg\" (UID: \"16ce1986-676d-495a-93f0-6ecf0c6061a7\") " pod="openshift-ingress/payload-processing-e2e-autores-63f3d7-64b49bd9c9-k9chg"
Sep 10 21:09:58.691840 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:58.691810    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-63f3d7-64b49bd9c9-k9chg"
Sep 10 21:09:58.841933 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:58.841909    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-63f3d7-69c755474-ld7d2"]
Sep 10 21:09:58.844172 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:09:58.844146    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac06edc0_36b0_4be7_ad07_32edf0982ecb.slice/crio-0f93e3a590e9321d6fae93272f14078cfa3cccb62ccadaa3523bb45da1f7f051 WatchSource:0}: Error finding container 0f93e3a590e9321d6fae93272f14078cfa3cccb62ccadaa3523bb45da1f7f051: Status 404 returned error can't find the container with id 0f93e3a590e9321d6fae93272f14078cfa3cccb62ccadaa3523bb45da1f7f051
Sep 10 21:09:58.979207 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:58.979024    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-63f3d7-64b49bd9c9-k9chg"]
Sep 10 21:09:58.981383 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:09:58.981352    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16ce1986_676d_495a_93f0_6ecf0c6061a7.slice/crio-50e212a99b75f166b496bc804b6b5b4361c519dd9a06995b720ccf702c7027d6 WatchSource:0}: Error finding container 50e212a99b75f166b496bc804b6b5b4361c519dd9a06995b720ccf702c7027d6: Status 404 returned error can't find the container with id 50e212a99b75f166b496bc804b6b5b4361c519dd9a06995b720ccf702c7027d6
Sep 10 21:09:59.052608 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:59.052573    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-63f3d7-69c755474-ld7d2" event={"ID":"ac06edc0-36b0-4be7-ad07-32edf0982ecb","Type":"ContainerStarted","Data":"ae2f5228918978bb14340369deb683908d5f3fc6f798935f311085c103556c1d"}
Sep 10 21:09:59.052980 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:59.052641    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-63f3d7-69c755474-ld7d2" event={"ID":"ac06edc0-36b0-4be7-ad07-32edf0982ecb","Type":"ContainerStarted","Data":"0f93e3a590e9321d6fae93272f14078cfa3cccb62ccadaa3523bb45da1f7f051"}
Sep 10 21:09:59.052980 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:59.052672    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-63f3d7-69c755474-ld7d2"
Sep 10 21:09:59.054275 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:09:59.054252    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-63f3d7-64b49bd9c9-k9chg" event={"ID":"16ce1986-676d-495a-93f0-6ecf0c6061a7","Type":"ContainerStarted","Data":"50e212a99b75f166b496bc804b6b5b4361c519dd9a06995b720ccf702c7027d6"}
Sep 10 21:10:00.062147 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:00.062114    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-63f3d7-64b49bd9c9-k9chg" event={"ID":"16ce1986-676d-495a-93f0-6ecf0c6061a7","Type":"ContainerStarted","Data":"1d177ef4ef93b1ee80798e0485e6df45ee1f2a4da1937d0758eb644811bbbb24"}
Sep 10 21:10:00.062627 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:00.062178    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-63f3d7-64b49bd9c9-k9chg"
Sep 10 21:10:00.086141 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:00.086069    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-63f3d7-69c755474-ld7d2" podStartSLOduration=2.086052567 podStartE2EDuration="2.086052567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:09:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:09:59.088613727 +0000 UTC m=+716.745522500" watchObservedRunningTime="2026-09-10 21:10:00.086052567 +0000 UTC m=+717.742961391"
Sep 10 21:10:00.086333 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:00.086157    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-63f3d7-64b49bd9c9-k9chg" podStartSLOduration=2.08615328 podStartE2EDuration="2.08615328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:09:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:10:00.084080031 +0000 UTC m=+717.740988784" watchObservedRunningTime="2026-09-10 21:10:00.08615328 +0000 UTC m=+717.743062051"
Sep 10 21:10:01.010566 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:01.010536    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-47f62f-7656d8ffdb-xht9v"
Sep 10 21:10:01.010817 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:01.010799    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-47f62f-66fbbbb67-hzspx"
Sep 10 21:10:09.056815 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:09.056784    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-1a398fc7-658fb86699-8dzw7"
Sep 10 21:10:09.057160 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:09.056855    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a398fc7-869656d44b-djr42"
Sep 10 21:10:10.064898 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:10.064866    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-63f3d7-69c755474-ld7d2"
Sep 10 21:10:11.070467 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:11.070431    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-63f3d7-64b49bd9c9-k9chg"
Sep 10 21:10:17.561776 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:17.561741    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-000295-7c4d8675bb-98rzs"]
Sep 10 21:10:17.565570 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:17.565552    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-000295-7c4d8675bb-98rzs"
Sep 10 21:10:17.568161 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:17.568137    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-000295\""
Sep 10 21:10:17.577920 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:17.577896    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-000295-7c4d8675bb-98rzs"]
Sep 10 21:10:17.590791 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:17.590771    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-000295-66ccb8699f-7glkj"]
Sep 10 21:10:17.594212 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:17.594195    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-000295-66ccb8699f-7glkj"
Sep 10 21:10:17.596829 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:17.596807    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-000295-dockercfg-lstgn\""
Sep 10 21:10:17.608017 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:17.607995    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-000295-66ccb8699f-7glkj"]
Sep 10 21:10:17.637078 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:17.637054    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e895091-bf03-460b-99ab-242db43dd7d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-000295-7c4d8675bb-98rzs\" (UID: \"2e895091-bf03-460b-99ab-242db43dd7d6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-000295-7c4d8675bb-98rzs"
Sep 10 21:10:17.637228 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:17.637207    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8p58\" (UniqueName: \"kubernetes.io/projected/2e895091-bf03-460b-99ab-242db43dd7d6-kube-api-access-v8p58\") pod \"payload-pre-processing-e2e-inf-b-000295-7c4d8675bb-98rzs\" (UID: \"2e895091-bf03-460b-99ab-242db43dd7d6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-000295-7c4d8675bb-98rzs"
Sep 10 21:10:17.738017 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:17.737977    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8p58\" (UniqueName: \"kubernetes.io/projected/2e895091-bf03-460b-99ab-242db43dd7d6-kube-api-access-v8p58\") pod \"payload-pre-processing-e2e-inf-b-000295-7c4d8675bb-98rzs\" (UID: \"2e895091-bf03-460b-99ab-242db43dd7d6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-000295-7c4d8675bb-98rzs"
Sep 10 21:10:17.738189 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:17.738072    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f603b480-c552-4176-b6e7-779118d030fd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-000295-66ccb8699f-7glkj\" (UID: \"f603b480-c552-4176-b6e7-779118d030fd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-000295-66ccb8699f-7glkj"
Sep 10 21:10:17.738189 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:17.738112    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e895091-bf03-460b-99ab-242db43dd7d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-000295-7c4d8675bb-98rzs\" (UID: \"2e895091-bf03-460b-99ab-242db43dd7d6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-000295-7c4d8675bb-98rzs"
Sep 10 21:10:17.738189 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:17.738159    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2n9g\" (UniqueName: \"kubernetes.io/projected/f603b480-c552-4176-b6e7-779118d030fd-kube-api-access-v2n9g\") pod \"payload-processing-e2e-inf-b-000295-66ccb8699f-7glkj\" (UID: \"f603b480-c552-4176-b6e7-779118d030fd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-000295-66ccb8699f-7glkj"
Sep 10 21:10:17.738914 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:17.738892    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e895091-bf03-460b-99ab-242db43dd7d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-000295-7c4d8675bb-98rzs\" (UID: \"2e895091-bf03-460b-99ab-242db43dd7d6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-000295-7c4d8675bb-98rzs"
Sep 10 21:10:17.746031 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:17.746005    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8p58\" (UniqueName: \"kubernetes.io/projected/2e895091-bf03-460b-99ab-242db43dd7d6-kube-api-access-v8p58\") pod \"payload-pre-processing-e2e-inf-b-000295-7c4d8675bb-98rzs\" (UID: \"2e895091-bf03-460b-99ab-242db43dd7d6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-000295-7c4d8675bb-98rzs"
Sep 10 21:10:17.839329 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:17.839225    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f603b480-c552-4176-b6e7-779118d030fd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-000295-66ccb8699f-7glkj\" (UID: \"f603b480-c552-4176-b6e7-779118d030fd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-000295-66ccb8699f-7glkj"
Sep 10 21:10:17.839329 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:17.839270    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2n9g\" (UniqueName: \"kubernetes.io/projected/f603b480-c552-4176-b6e7-779118d030fd-kube-api-access-v2n9g\") pod \"payload-processing-e2e-inf-b-000295-66ccb8699f-7glkj\" (UID: \"f603b480-c552-4176-b6e7-779118d030fd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-000295-66ccb8699f-7glkj"
Sep 10 21:10:17.839969 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:17.839950    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f603b480-c552-4176-b6e7-779118d030fd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-000295-66ccb8699f-7glkj\" (UID: \"f603b480-c552-4176-b6e7-779118d030fd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-000295-66ccb8699f-7glkj"
Sep 10 21:10:17.848376 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:17.848360    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2n9g\" (UniqueName: \"kubernetes.io/projected/f603b480-c552-4176-b6e7-779118d030fd-kube-api-access-v2n9g\") pod \"payload-processing-e2e-inf-b-000295-66ccb8699f-7glkj\" (UID: \"f603b480-c552-4176-b6e7-779118d030fd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-000295-66ccb8699f-7glkj"
Sep 10 21:10:17.875418 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:17.875388    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-000295-7c4d8675bb-98rzs"
Sep 10 21:10:17.904785 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:17.904755    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-000295-66ccb8699f-7glkj"
Sep 10 21:10:18.166139 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:18.166114    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-000295-7c4d8675bb-98rzs"]
Sep 10 21:10:18.167883 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:10:18.167859    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e895091_bf03_460b_99ab_242db43dd7d6.slice/crio-c9136ed9605c3b08e5e76f84676ba713891924d19cf809b253b34d9ce0483374 WatchSource:0}: Error finding container c9136ed9605c3b08e5e76f84676ba713891924d19cf809b253b34d9ce0483374: Status 404 returned error can't find the container with id c9136ed9605c3b08e5e76f84676ba713891924d19cf809b253b34d9ce0483374
Sep 10 21:10:18.194892 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:18.194872    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-000295-66ccb8699f-7glkj"]
Sep 10 21:10:18.198781 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:10:18.198752    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf603b480_c552_4176_b6e7_779118d030fd.slice/crio-827f07234611ced8349e486431b681f934333e609738c5760d493d15cddb3ba4 WatchSource:0}: Error finding container 827f07234611ced8349e486431b681f934333e609738c5760d493d15cddb3ba4: Status 404 returned error can't find the container with id 827f07234611ced8349e486431b681f934333e609738c5760d493d15cddb3ba4
Sep 10 21:10:19.170679 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:19.170644    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-000295-7c4d8675bb-98rzs" event={"ID":"2e895091-bf03-460b-99ab-242db43dd7d6","Type":"ContainerStarted","Data":"7112e14f5b042afd49dc5840c4b47d5e22009b318cf389c7c59ca4bb6103ef91"}
Sep 10 21:10:19.171106 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:19.170742    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-000295-7c4d8675bb-98rzs"
Sep 10 21:10:19.171106 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:19.170771    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-000295-7c4d8675bb-98rzs" event={"ID":"2e895091-bf03-460b-99ab-242db43dd7d6","Type":"ContainerStarted","Data":"c9136ed9605c3b08e5e76f84676ba713891924d19cf809b253b34d9ce0483374"}
Sep 10 21:10:19.172284 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:19.172260    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-000295-66ccb8699f-7glkj" event={"ID":"f603b480-c552-4176-b6e7-779118d030fd","Type":"ContainerStarted","Data":"063883ecd398a37d007fdaa39083a13d72d79f2bfeefe01519cf095682dcebb9"}
Sep 10 21:10:19.172415 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:19.172311    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-000295-66ccb8699f-7glkj" event={"ID":"f603b480-c552-4176-b6e7-779118d030fd","Type":"ContainerStarted","Data":"827f07234611ced8349e486431b681f934333e609738c5760d493d15cddb3ba4"}
Sep 10 21:10:19.200999 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:19.200952    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-000295-7c4d8675bb-98rzs" podStartSLOduration=2.200937863 podStartE2EDuration="2.200937863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:10:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:10:19.199482891 +0000 UTC m=+736.856391662" watchObservedRunningTime="2026-09-10 21:10:19.200937863 +0000 UTC m=+736.857846634"
Sep 10 21:10:19.228895 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:19.228854    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-000295-66ccb8699f-7glkj" podStartSLOduration=2.228838748 podStartE2EDuration="2.228838748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:10:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:10:19.227426616 +0000 UTC m=+736.884335387" watchObservedRunningTime="2026-09-10 21:10:19.228838748 +0000 UTC m=+736.885747520"
Sep 10 21:10:20.177569 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:20.177537    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-000295-66ccb8699f-7glkj"
Sep 10 21:10:20.257510 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:20.257477    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a398fc7-926lp"]
Sep 10 21:10:20.261313 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:20.261283    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a398fc7-926lp"
Sep 10 21:10:20.267634 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:20.267613    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s4qqn\""
Sep 10 21:10:20.267982 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:20.267966    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:10:20.276031 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:20.276004    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a398fc7-926lp"]
Sep 10 21:10:20.362881 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:20.362852    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f777a6b0-2eab-49bb-99dd-540b2e253b7d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1a398fc7-926lp\" (UID: \"f777a6b0-2eab-49bb-99dd-540b2e253b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a398fc7-926lp"
Sep 10 21:10:20.463800 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:20.463726    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f777a6b0-2eab-49bb-99dd-540b2e253b7d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1a398fc7-926lp\" (UID: \"f777a6b0-2eab-49bb-99dd-540b2e253b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a398fc7-926lp"
Sep 10 21:10:20.464363 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:20.464345    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f777a6b0-2eab-49bb-99dd-540b2e253b7d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1a398fc7-926lp\" (UID: \"f777a6b0-2eab-49bb-99dd-540b2e253b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a398fc7-926lp"
Sep 10 21:10:20.572042 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:20.572000    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a398fc7-926lp"
Sep 10 21:10:20.860130 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:20.860102    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a398fc7-926lp"]
Sep 10 21:10:20.862220 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:10:20.862189    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf777a6b0_2eab_49bb_99dd_540b2e253b7d.slice/crio-fe6fcc05f75951bb3bb0950c27cd898289c58470d9198c8cea7ae593dc83058f WatchSource:0}: Error finding container fe6fcc05f75951bb3bb0950c27cd898289c58470d9198c8cea7ae593dc83058f: Status 404 returned error can't find the container with id fe6fcc05f75951bb3bb0950c27cd898289c58470d9198c8cea7ae593dc83058f
Sep 10 21:10:20.864288 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:20.864269    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 21:10:21.184085 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:21.184004    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a398fc7-926lp" event={"ID":"f777a6b0-2eab-49bb-99dd-540b2e253b7d","Type":"ContainerStarted","Data":"fe6fcc05f75951bb3bb0950c27cd898289c58470d9198c8cea7ae593dc83058f"}
Sep 10 21:10:24.200856 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:24.200826    2581 generic.go:356] "Generic (PLEG): container finished" podID="f777a6b0-2eab-49bb-99dd-540b2e253b7d" containerID="4dac75ad2eb1c0dd4e72806ad776c3aba29941d414007e8bf8ea8d5c726347f2" exitCode=0
Sep 10 21:10:24.201202 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:24.200875    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a398fc7-926lp" event={"ID":"f777a6b0-2eab-49bb-99dd-540b2e253b7d","Type":"ContainerDied","Data":"4dac75ad2eb1c0dd4e72806ad776c3aba29941d414007e8bf8ea8d5c726347f2"}
Sep 10 21:10:25.347867 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:25.347843    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a398fc7-926lp"
Sep 10 21:10:25.509046 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:25.508960    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f777a6b0-2eab-49bb-99dd-540b2e253b7d-maas-api-service-ca\") pod \"f777a6b0-2eab-49bb-99dd-540b2e253b7d\" (UID: \"f777a6b0-2eab-49bb-99dd-540b2e253b7d\") "
Sep 10 21:10:25.509284 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:25.509263    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f777a6b0-2eab-49bb-99dd-540b2e253b7d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f777a6b0-2eab-49bb-99dd-540b2e253b7d" (UID: "f777a6b0-2eab-49bb-99dd-540b2e253b7d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:10:25.610289 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:25.610257    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f777a6b0-2eab-49bb-99dd-540b2e253b7d-maas-api-service-ca\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:10:26.213062 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:26.213024    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a398fc7-926lp" event={"ID":"f777a6b0-2eab-49bb-99dd-540b2e253b7d","Type":"ContainerDied","Data":"fe6fcc05f75951bb3bb0950c27cd898289c58470d9198c8cea7ae593dc83058f"}
Sep 10 21:10:26.213227 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:26.213079    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe6fcc05f75951bb3bb0950c27cd898289c58470d9198c8cea7ae593dc83058f"
Sep 10 21:10:26.213227 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:26.213041    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a398fc7-926lp"
Sep 10 21:10:30.180026 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:30.179995    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-000295-7c4d8675bb-98rzs"
Sep 10 21:10:31.186782 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:31.186749    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-000295-66ccb8699f-7glkj"
Sep 10 21:10:37.051278 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:37.051242    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1a398fc7-658fb86699-8dzw7"]
Sep 10 21:10:37.051799 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:37.051543    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-1a398fc7-658fb86699-8dzw7" podUID="037bdae4-9edb-4705-b2f2-a394364bb5c8" containerName="payload-processing" containerID="cri-o://c581d2c578105cbc821bfb0bb3a912544da399f04ae3f74b1c321716d73c7069" gracePeriod=30
Sep 10 21:10:37.060433 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:37.060406    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1a398fc7-869656d44b-djr42"]
Sep 10 21:10:37.060648 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:37.060601    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a398fc7-869656d44b-djr42" podUID="3f6a0984-3589-4518-ba0c-e6f7a86d35dc" containerName="payload-pre-processing" containerID="cri-o://9279de00acbb8f49870ced6cd8ccf726ffb1ce627ad6450ce8646660dd783fbd" gracePeriod=30
Sep 10 21:10:37.220900 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:37.220857    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a398fc7-926lp"]
Sep 10 21:10:37.222535 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:37.222512    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1a398fc7-926lp"]
Sep 10 21:10:37.270683 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:37.270651    2581 generic.go:356] "Generic (PLEG): container finished" podID="037bdae4-9edb-4705-b2f2-a394364bb5c8" containerID="c581d2c578105cbc821bfb0bb3a912544da399f04ae3f74b1c321716d73c7069" exitCode=0
Sep 10 21:10:37.270878 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:37.270854    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1a398fc7-658fb86699-8dzw7" event={"ID":"037bdae4-9edb-4705-b2f2-a394364bb5c8","Type":"ContainerDied","Data":"c581d2c578105cbc821bfb0bb3a912544da399f04ae3f74b1c321716d73c7069"}
Sep 10 21:10:37.272925 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:37.272901    2581 generic.go:356] "Generic (PLEG): container finished" podID="3f6a0984-3589-4518-ba0c-e6f7a86d35dc" containerID="9279de00acbb8f49870ced6cd8ccf726ffb1ce627ad6450ce8646660dd783fbd" exitCode=0
Sep 10 21:10:37.273038 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:37.272954    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a398fc7-869656d44b-djr42" event={"ID":"3f6a0984-3589-4518-ba0c-e6f7a86d35dc","Type":"ContainerDied","Data":"9279de00acbb8f49870ced6cd8ccf726ffb1ce627ad6450ce8646660dd783fbd"}
Sep 10 21:10:37.336109 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:37.336084    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1a398fc7-658fb86699-8dzw7"
Sep 10 21:10:37.339417 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:37.339399    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a398fc7-869656d44b-djr42"
Sep 10 21:10:37.419387 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:37.419355    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vrrg\" (UniqueName: \"kubernetes.io/projected/037bdae4-9edb-4705-b2f2-a394364bb5c8-kube-api-access-8vrrg\") pod \"037bdae4-9edb-4705-b2f2-a394364bb5c8\" (UID: \"037bdae4-9edb-4705-b2f2-a394364bb5c8\") "
Sep 10 21:10:37.419555 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:37.419407    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f6a0984-3589-4518-ba0c-e6f7a86d35dc-plugins-config-volume\") pod \"3f6a0984-3589-4518-ba0c-e6f7a86d35dc\" (UID: \"3f6a0984-3589-4518-ba0c-e6f7a86d35dc\") "
Sep 10 21:10:37.419555 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:37.419481    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/037bdae4-9edb-4705-b2f2-a394364bb5c8-plugins-config-volume\") pod \"037bdae4-9edb-4705-b2f2-a394364bb5c8\" (UID: \"037bdae4-9edb-4705-b2f2-a394364bb5c8\") "
Sep 10 21:10:37.419555 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:37.419535    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hckh8\" (UniqueName: \"kubernetes.io/projected/3f6a0984-3589-4518-ba0c-e6f7a86d35dc-kube-api-access-hckh8\") pod \"3f6a0984-3589-4518-ba0c-e6f7a86d35dc\" (UID: \"3f6a0984-3589-4518-ba0c-e6f7a86d35dc\") "
Sep 10 21:10:37.419855 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:37.419826    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f6a0984-3589-4518-ba0c-e6f7a86d35dc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3f6a0984-3589-4518-ba0c-e6f7a86d35dc" (UID: "3f6a0984-3589-4518-ba0c-e6f7a86d35dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:10:37.419934 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:37.419869    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/037bdae4-9edb-4705-b2f2-a394364bb5c8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "037bdae4-9edb-4705-b2f2-a394364bb5c8" (UID: "037bdae4-9edb-4705-b2f2-a394364bb5c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:10:37.421627 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:37.421603    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f6a0984-3589-4518-ba0c-e6f7a86d35dc-kube-api-access-hckh8" (OuterVolumeSpecName: "kube-api-access-hckh8") pod "3f6a0984-3589-4518-ba0c-e6f7a86d35dc" (UID: "3f6a0984-3589-4518-ba0c-e6f7a86d35dc"). InnerVolumeSpecName "kube-api-access-hckh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:10:37.421701 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:37.421656    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/037bdae4-9edb-4705-b2f2-a394364bb5c8-kube-api-access-8vrrg" (OuterVolumeSpecName: "kube-api-access-8vrrg") pod "037bdae4-9edb-4705-b2f2-a394364bb5c8" (UID: "037bdae4-9edb-4705-b2f2-a394364bb5c8"). InnerVolumeSpecName "kube-api-access-8vrrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:10:37.520791 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:37.520748    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/037bdae4-9edb-4705-b2f2-a394364bb5c8-plugins-config-volume\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:10:37.520791 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:37.520789    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hckh8\" (UniqueName: \"kubernetes.io/projected/3f6a0984-3589-4518-ba0c-e6f7a86d35dc-kube-api-access-hckh8\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:10:37.520791 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:37.520800    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vrrg\" (UniqueName: \"kubernetes.io/projected/037bdae4-9edb-4705-b2f2-a394364bb5c8-kube-api-access-8vrrg\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:10:37.520791 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:37.520809    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f6a0984-3589-4518-ba0c-e6f7a86d35dc-plugins-config-volume\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:10:38.279599 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:38.279560    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1a398fc7-658fb86699-8dzw7" event={"ID":"037bdae4-9edb-4705-b2f2-a394364bb5c8","Type":"ContainerDied","Data":"312f88dfe3d0230ccebf69b3708c9dfd41ac03fe4cf65150a8aeec9c88e3a5ab"}
Sep 10 21:10:38.280014 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:38.279632    2581 scope.go:117] "RemoveContainer" containerID="c581d2c578105cbc821bfb0bb3a912544da399f04ae3f74b1c321716d73c7069"
Sep 10 21:10:38.280014 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:38.279573    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1a398fc7-658fb86699-8dzw7"
Sep 10 21:10:38.281033 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:38.281011    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a398fc7-869656d44b-djr42" event={"ID":"3f6a0984-3589-4518-ba0c-e6f7a86d35dc","Type":"ContainerDied","Data":"338be21dc51481bae53889b47397d6d43bdf70a60551b60da924df5a510893e9"}
Sep 10 21:10:38.281092 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:38.281030    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1a398fc7-869656d44b-djr42"
Sep 10 21:10:38.289680 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:38.289664    2581 scope.go:117] "RemoveContainer" containerID="9279de00acbb8f49870ced6cd8ccf726ffb1ce627ad6450ce8646660dd783fbd"
Sep 10 21:10:38.345937 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:38.345905    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1a398fc7-658fb86699-8dzw7"]
Sep 10 21:10:38.352636 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:38.352614    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1a398fc7-658fb86699-8dzw7"]
Sep 10 21:10:38.378538 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:38.378505    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1a398fc7-869656d44b-djr42"]
Sep 10 21:10:38.396136 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:38.396113    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1a398fc7-869656d44b-djr42"]
Sep 10 21:10:38.873667 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:38.873622    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="037bdae4-9edb-4705-b2f2-a394364bb5c8" path="/var/lib/kubelet/pods/037bdae4-9edb-4705-b2f2-a394364bb5c8/volumes"
Sep 10 21:10:38.873996 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:38.873981    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f6a0984-3589-4518-ba0c-e6f7a86d35dc" path="/var/lib/kubelet/pods/3f6a0984-3589-4518-ba0c-e6f7a86d35dc/volumes"
Sep 10 21:10:38.874348 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:38.874331    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f777a6b0-2eab-49bb-99dd-540b2e253b7d" path="/var/lib/kubelet/pods/f777a6b0-2eab-49bb-99dd-540b2e253b7d/volumes"
Sep 10 21:10:41.968910 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:41.968876    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a7bbcc15-86df455d-6657d"]
Sep 10 21:10:41.969411 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:41.969243    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f6a0984-3589-4518-ba0c-e6f7a86d35dc" containerName="payload-pre-processing"
Sep 10 21:10:41.969411 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:41.969255    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f6a0984-3589-4518-ba0c-e6f7a86d35dc" containerName="payload-pre-processing"
Sep 10 21:10:41.969411 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:41.969272    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f777a6b0-2eab-49bb-99dd-540b2e253b7d" containerName="revoke-keys"
Sep 10 21:10:41.969411 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:41.969278    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f777a6b0-2eab-49bb-99dd-540b2e253b7d" containerName="revoke-keys"
Sep 10 21:10:41.969411 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:41.969287    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="037bdae4-9edb-4705-b2f2-a394364bb5c8" containerName="payload-processing"
Sep 10 21:10:41.969411 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:41.969292    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="037bdae4-9edb-4705-b2f2-a394364bb5c8" containerName="payload-processing"
Sep 10 21:10:41.969411 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:41.969374    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f6a0984-3589-4518-ba0c-e6f7a86d35dc" containerName="payload-pre-processing"
Sep 10 21:10:41.969411 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:41.969381    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f777a6b0-2eab-49bb-99dd-540b2e253b7d" containerName="revoke-keys"
Sep 10 21:10:41.969411 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:41.969388    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="037bdae4-9edb-4705-b2f2-a394364bb5c8" containerName="payload-processing"
Sep 10 21:10:41.973811 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:41.973789    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a7bbcc15-86df455d-6657d"
Sep 10 21:10:41.976455 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:41.976435    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-a7bbcc15\""
Sep 10 21:10:41.977339 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:41.977319    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-a7bbcc15-dockercfg-kqmhf\""
Sep 10 21:10:41.990093 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:41.990068    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a7bbcc15-86df455d-6657d"]
Sep 10 21:10:42.028206 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:42.028174    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a7bbcc15-75cd9c9dcd-frstp"]
Sep 10 21:10:42.032089 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:42.032064    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a7bbcc15-75cd9c9dcd-frstp"
Sep 10 21:10:42.053161 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:42.053131    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a7bbcc15-75cd9c9dcd-frstp"]
Sep 10 21:10:42.060091 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:42.060062    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22d56ed3-3e81-4a1a-bd48-12b68f1b2a54-plugins-config-volume\") pod \"payload-processing-e2e-migrate-a7bbcc15-75cd9c9dcd-frstp\" (UID: \"22d56ed3-3e81-4a1a-bd48-12b68f1b2a54\") " pod="openshift-ingress/payload-processing-e2e-migrate-a7bbcc15-75cd9c9dcd-frstp"
Sep 10 21:10:42.060210 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:42.060100    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d67v\" (UniqueName: \"kubernetes.io/projected/1ec6b854-e19d-4615-b85b-3c5146f8e527-kube-api-access-4d67v\") pod \"payload-pre-processing-e2e-migrate-a7bbcc15-86df455d-6657d\" (UID: \"1ec6b854-e19d-4615-b85b-3c5146f8e527\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a7bbcc15-86df455d-6657d"
Sep 10 21:10:42.060210 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:42.060177    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ec6b854-e19d-4615-b85b-3c5146f8e527-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-a7bbcc15-86df455d-6657d\" (UID: \"1ec6b854-e19d-4615-b85b-3c5146f8e527\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a7bbcc15-86df455d-6657d"
Sep 10 21:10:42.060210 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:42.060202    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rfbj\" (UniqueName: \"kubernetes.io/projected/22d56ed3-3e81-4a1a-bd48-12b68f1b2a54-kube-api-access-9rfbj\") pod \"payload-processing-e2e-migrate-a7bbcc15-75cd9c9dcd-frstp\" (UID: \"22d56ed3-3e81-4a1a-bd48-12b68f1b2a54\") " pod="openshift-ingress/payload-processing-e2e-migrate-a7bbcc15-75cd9c9dcd-frstp"
Sep 10 21:10:42.160935 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:42.160901    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22d56ed3-3e81-4a1a-bd48-12b68f1b2a54-plugins-config-volume\") pod \"payload-processing-e2e-migrate-a7bbcc15-75cd9c9dcd-frstp\" (UID: \"22d56ed3-3e81-4a1a-bd48-12b68f1b2a54\") " pod="openshift-ingress/payload-processing-e2e-migrate-a7bbcc15-75cd9c9dcd-frstp"
Sep 10 21:10:42.161135 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:42.160945    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d67v\" (UniqueName: \"kubernetes.io/projected/1ec6b854-e19d-4615-b85b-3c5146f8e527-kube-api-access-4d67v\") pod \"payload-pre-processing-e2e-migrate-a7bbcc15-86df455d-6657d\" (UID: \"1ec6b854-e19d-4615-b85b-3c5146f8e527\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a7bbcc15-86df455d-6657d"
Sep 10 21:10:42.161135 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:42.161064    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ec6b854-e19d-4615-b85b-3c5146f8e527-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-a7bbcc15-86df455d-6657d\" (UID: \"1ec6b854-e19d-4615-b85b-3c5146f8e527\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a7bbcc15-86df455d-6657d"
Sep 10 21:10:42.161135 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:42.161100    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rfbj\" (UniqueName: \"kubernetes.io/projected/22d56ed3-3e81-4a1a-bd48-12b68f1b2a54-kube-api-access-9rfbj\") pod \"payload-processing-e2e-migrate-a7bbcc15-75cd9c9dcd-frstp\" (UID: \"22d56ed3-3e81-4a1a-bd48-12b68f1b2a54\") " pod="openshift-ingress/payload-processing-e2e-migrate-a7bbcc15-75cd9c9dcd-frstp"
Sep 10 21:10:42.161768 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:42.161746    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22d56ed3-3e81-4a1a-bd48-12b68f1b2a54-plugins-config-volume\") pod \"payload-processing-e2e-migrate-a7bbcc15-75cd9c9dcd-frstp\" (UID: \"22d56ed3-3e81-4a1a-bd48-12b68f1b2a54\") " pod="openshift-ingress/payload-processing-e2e-migrate-a7bbcc15-75cd9c9dcd-frstp"
Sep 10 21:10:42.161884 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:42.161866    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ec6b854-e19d-4615-b85b-3c5146f8e527-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-a7bbcc15-86df455d-6657d\" (UID: \"1ec6b854-e19d-4615-b85b-3c5146f8e527\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a7bbcc15-86df455d-6657d"
Sep 10 21:10:42.173421 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:42.173393    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rfbj\" (UniqueName: \"kubernetes.io/projected/22d56ed3-3e81-4a1a-bd48-12b68f1b2a54-kube-api-access-9rfbj\") pod \"payload-processing-e2e-migrate-a7bbcc15-75cd9c9dcd-frstp\" (UID: \"22d56ed3-3e81-4a1a-bd48-12b68f1b2a54\") " pod="openshift-ingress/payload-processing-e2e-migrate-a7bbcc15-75cd9c9dcd-frstp"
Sep 10 21:10:42.174081 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:42.174029    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d67v\" (UniqueName: \"kubernetes.io/projected/1ec6b854-e19d-4615-b85b-3c5146f8e527-kube-api-access-4d67v\") pod \"payload-pre-processing-e2e-migrate-a7bbcc15-86df455d-6657d\" (UID: \"1ec6b854-e19d-4615-b85b-3c5146f8e527\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a7bbcc15-86df455d-6657d"
Sep 10 21:10:42.284403 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:42.284316    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a7bbcc15-86df455d-6657d"
Sep 10 21:10:42.344025 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:42.343618    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a7bbcc15-75cd9c9dcd-frstp"
Sep 10 21:10:42.559359 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:42.559334    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a7bbcc15-86df455d-6657d"]
Sep 10 21:10:42.561805 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:10:42.561777    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ec6b854_e19d_4615_b85b_3c5146f8e527.slice/crio-fd31c42448a64c356b6e5b3759feaf330235e97e25749477d44e27cc6f3552ee WatchSource:0}: Error finding container fd31c42448a64c356b6e5b3759feaf330235e97e25749477d44e27cc6f3552ee: Status 404 returned error can't find the container with id fd31c42448a64c356b6e5b3759feaf330235e97e25749477d44e27cc6f3552ee
Sep 10 21:10:42.623921 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:42.623892    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a7bbcc15-75cd9c9dcd-frstp"]
Sep 10 21:10:42.625876 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:10:42.625844    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22d56ed3_3e81_4a1a_bd48_12b68f1b2a54.slice/crio-2d8c69ebeca90c3f86d3d17d579663ccd155a0df9e1a7d6336aa163838cc3598 WatchSource:0}: Error finding container 2d8c69ebeca90c3f86d3d17d579663ccd155a0df9e1a7d6336aa163838cc3598: Status 404 returned error can't find the container with id 2d8c69ebeca90c3f86d3d17d579663ccd155a0df9e1a7d6336aa163838cc3598
Sep 10 21:10:43.311208 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:43.311169    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a7bbcc15-75cd9c9dcd-frstp" event={"ID":"22d56ed3-3e81-4a1a-bd48-12b68f1b2a54","Type":"ContainerStarted","Data":"54652ec8238f89d1e8550c84c0189315463a8a53437609835a3778d58c5658b4"}
Sep 10 21:10:43.311651 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:43.311234    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a7bbcc15-75cd9c9dcd-frstp" event={"ID":"22d56ed3-3e81-4a1a-bd48-12b68f1b2a54","Type":"ContainerStarted","Data":"2d8c69ebeca90c3f86d3d17d579663ccd155a0df9e1a7d6336aa163838cc3598"}
Sep 10 21:10:43.311651 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:43.311265    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-a7bbcc15-75cd9c9dcd-frstp"
Sep 10 21:10:43.313005 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:43.312970    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a7bbcc15-86df455d-6657d" event={"ID":"1ec6b854-e19d-4615-b85b-3c5146f8e527","Type":"ContainerStarted","Data":"63f2282df8f2657eb1f5153da0e69b2108a4ca6a2a50f54cd53b86553b394a68"}
Sep 10 21:10:43.313129 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:43.313030    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a7bbcc15-86df455d-6657d" event={"ID":"1ec6b854-e19d-4615-b85b-3c5146f8e527","Type":"ContainerStarted","Data":"fd31c42448a64c356b6e5b3759feaf330235e97e25749477d44e27cc6f3552ee"}
Sep 10 21:10:43.313129 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:43.313058    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a7bbcc15-86df455d-6657d"
Sep 10 21:10:43.328420 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:43.328367    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-a7bbcc15-75cd9c9dcd-frstp" podStartSLOduration=2.328352664 podStartE2EDuration="2.328352664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:10:43.327437785 +0000 UTC m=+760.984346556" watchObservedRunningTime="2026-09-10 21:10:43.328352664 +0000 UTC m=+760.985261498"
Sep 10 21:10:43.345273 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:43.345220    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a7bbcc15-86df455d-6657d" podStartSLOduration=2.345206181 podStartE2EDuration="2.345206181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:10:43.344631096 +0000 UTC m=+761.001539870" watchObservedRunningTime="2026-09-10 21:10:43.345206181 +0000 UTC m=+761.002114954"
Sep 10 21:10:54.320364 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:54.320335    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-a7bbcc15-75cd9c9dcd-frstp"
Sep 10 21:10:54.320758 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:10:54.320499    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a7bbcc15-86df455d-6657d"
Sep 10 21:11:06.165860 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:06.165826    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a7bbcc15-bzs7t"]
Sep 10 21:11:06.170685 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:06.170662    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a7bbcc15-bzs7t"
Sep 10 21:11:06.172905 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:06.172885    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:11:06.173005 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:06.172973    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s4qqn\""
Sep 10 21:11:06.178623 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:06.178458    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a7bbcc15-bzs7t"]
Sep 10 21:11:06.283225 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:06.283191    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/535f12aa-aba8-463d-973d-788d9ef93963-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a7bbcc15-bzs7t\" (UID: \"535f12aa-aba8-463d-973d-788d9ef93963\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a7bbcc15-bzs7t"
Sep 10 21:11:06.384460 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:06.384428    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/535f12aa-aba8-463d-973d-788d9ef93963-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a7bbcc15-bzs7t\" (UID: \"535f12aa-aba8-463d-973d-788d9ef93963\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a7bbcc15-bzs7t"
Sep 10 21:11:06.385031 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:06.385010    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/535f12aa-aba8-463d-973d-788d9ef93963-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a7bbcc15-bzs7t\" (UID: \"535f12aa-aba8-463d-973d-788d9ef93963\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a7bbcc15-bzs7t"
Sep 10 21:11:06.483066 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:06.482990    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a7bbcc15-bzs7t"
Sep 10 21:11:06.771063 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:06.771024    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a7bbcc15-bzs7t"]
Sep 10 21:11:06.773339 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:11:06.773309    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod535f12aa_aba8_463d_973d_788d9ef93963.slice/crio-1c8c4f84db48c5bf815738889cc11002ee67ff53a3c66ab41e7e9ea4a7718935 WatchSource:0}: Error finding container 1c8c4f84db48c5bf815738889cc11002ee67ff53a3c66ab41e7e9ea4a7718935: Status 404 returned error can't find the container with id 1c8c4f84db48c5bf815738889cc11002ee67ff53a3c66ab41e7e9ea4a7718935
Sep 10 21:11:07.434147 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:07.434108    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a7bbcc15-bzs7t" event={"ID":"535f12aa-aba8-463d-973d-788d9ef93963","Type":"ContainerStarted","Data":"6ea0b28c15e204316ef970323cbb74b51a8b2e29d30093848f98e85a6de9a01b"}
Sep 10 21:11:07.434550 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:07.434179    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a7bbcc15-bzs7t" event={"ID":"535f12aa-aba8-463d-973d-788d9ef93963","Type":"ContainerStarted","Data":"1c8c4f84db48c5bf815738889cc11002ee67ff53a3c66ab41e7e9ea4a7718935"}
Sep 10 21:11:07.451035 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:07.450987    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a7bbcc15-bzs7t" podStartSLOduration=1.450972184 podStartE2EDuration="1.450972184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:11:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:11:07.450341655 +0000 UTC m=+785.107250427" watchObservedRunningTime="2026-09-10 21:11:07.450972184 +0000 UTC m=+785.107880956"
Sep 10 21:11:08.440683 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:08.440644    2581 generic.go:356] "Generic (PLEG): container finished" podID="535f12aa-aba8-463d-973d-788d9ef93963" containerID="6ea0b28c15e204316ef970323cbb74b51a8b2e29d30093848f98e85a6de9a01b" exitCode=0
Sep 10 21:11:08.441054 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:08.440740    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a7bbcc15-bzs7t" event={"ID":"535f12aa-aba8-463d-973d-788d9ef93963","Type":"ContainerDied","Data":"6ea0b28c15e204316ef970323cbb74b51a8b2e29d30093848f98e85a6de9a01b"}
Sep 10 21:11:09.583869 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:09.583840    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a7bbcc15-bzs7t"
Sep 10 21:11:09.716497 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:09.716414    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/535f12aa-aba8-463d-973d-788d9ef93963-maas-api-service-ca\") pod \"535f12aa-aba8-463d-973d-788d9ef93963\" (UID: \"535f12aa-aba8-463d-973d-788d9ef93963\") "
Sep 10 21:11:09.716774 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:09.716750    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/535f12aa-aba8-463d-973d-788d9ef93963-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "535f12aa-aba8-463d-973d-788d9ef93963" (UID: "535f12aa-aba8-463d-973d-788d9ef93963"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:11:09.817428 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:09.817400    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/535f12aa-aba8-463d-973d-788d9ef93963-maas-api-service-ca\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:11:10.452335 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:10.452268    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a7bbcc15-bzs7t" event={"ID":"535f12aa-aba8-463d-973d-788d9ef93963","Type":"ContainerDied","Data":"1c8c4f84db48c5bf815738889cc11002ee67ff53a3c66ab41e7e9ea4a7718935"}
Sep 10 21:11:10.452335 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:10.452325    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a7bbcc15-bzs7t"
Sep 10 21:11:10.452546 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:10.452388    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c8c4f84db48c5bf815738889cc11002ee67ff53a3c66ab41e7e9ea4a7718935"
Sep 10 21:11:12.747076 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:12.747042    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a7bbcc15-75cd9c9dcd-frstp"]
Sep 10 21:11:12.747926 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:12.747850    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-a7bbcc15-75cd9c9dcd-frstp" podUID="22d56ed3-3e81-4a1a-bd48-12b68f1b2a54" containerName="payload-processing" containerID="cri-o://54652ec8238f89d1e8550c84c0189315463a8a53437609835a3778d58c5658b4" gracePeriod=30
Sep 10 21:11:12.750871 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:12.750844    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a7bbcc15-86df455d-6657d"]
Sep 10 21:11:12.751323 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:12.751253    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a7bbcc15-86df455d-6657d" podUID="1ec6b854-e19d-4615-b85b-3c5146f8e527" containerName="payload-pre-processing" containerID="cri-o://63f2282df8f2657eb1f5153da0e69b2108a4ca6a2a50f54cd53b86553b394a68" gracePeriod=30
Sep 10 21:11:12.922628 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:12.922345    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a7bbcc15-bzs7t"]
Sep 10 21:11:12.926592 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:12.926558    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a7bbcc15-bzs7t"]
Sep 10 21:11:13.042448 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:13.042426    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a7bbcc15-75cd9c9dcd-frstp"
Sep 10 21:11:13.045248 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:13.045228    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a7bbcc15-86df455d-6657d"
Sep 10 21:11:13.150257 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:13.150230    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22d56ed3-3e81-4a1a-bd48-12b68f1b2a54-plugins-config-volume\") pod \"22d56ed3-3e81-4a1a-bd48-12b68f1b2a54\" (UID: \"22d56ed3-3e81-4a1a-bd48-12b68f1b2a54\") "
Sep 10 21:11:13.150467 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:13.150266    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4d67v\" (UniqueName: \"kubernetes.io/projected/1ec6b854-e19d-4615-b85b-3c5146f8e527-kube-api-access-4d67v\") pod \"1ec6b854-e19d-4615-b85b-3c5146f8e527\" (UID: \"1ec6b854-e19d-4615-b85b-3c5146f8e527\") "
Sep 10 21:11:13.150467 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:13.150292    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rfbj\" (UniqueName: \"kubernetes.io/projected/22d56ed3-3e81-4a1a-bd48-12b68f1b2a54-kube-api-access-9rfbj\") pod \"22d56ed3-3e81-4a1a-bd48-12b68f1b2a54\" (UID: \"22d56ed3-3e81-4a1a-bd48-12b68f1b2a54\") "
Sep 10 21:11:13.150467 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:13.150402    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ec6b854-e19d-4615-b85b-3c5146f8e527-plugins-config-volume\") pod \"1ec6b854-e19d-4615-b85b-3c5146f8e527\" (UID: \"1ec6b854-e19d-4615-b85b-3c5146f8e527\") "
Sep 10 21:11:13.150681 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:13.150649    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22d56ed3-3e81-4a1a-bd48-12b68f1b2a54-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "22d56ed3-3e81-4a1a-bd48-12b68f1b2a54" (UID: "22d56ed3-3e81-4a1a-bd48-12b68f1b2a54"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:11:13.150742 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:13.150719    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ec6b854-e19d-4615-b85b-3c5146f8e527-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1ec6b854-e19d-4615-b85b-3c5146f8e527" (UID: "1ec6b854-e19d-4615-b85b-3c5146f8e527"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:11:13.152442 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:13.152414    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22d56ed3-3e81-4a1a-bd48-12b68f1b2a54-kube-api-access-9rfbj" (OuterVolumeSpecName: "kube-api-access-9rfbj") pod "22d56ed3-3e81-4a1a-bd48-12b68f1b2a54" (UID: "22d56ed3-3e81-4a1a-bd48-12b68f1b2a54"). InnerVolumeSpecName "kube-api-access-9rfbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:11:13.152442 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:13.152429    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ec6b854-e19d-4615-b85b-3c5146f8e527-kube-api-access-4d67v" (OuterVolumeSpecName: "kube-api-access-4d67v") pod "1ec6b854-e19d-4615-b85b-3c5146f8e527" (UID: "1ec6b854-e19d-4615-b85b-3c5146f8e527"). InnerVolumeSpecName "kube-api-access-4d67v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:11:13.252482 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:13.252284    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22d56ed3-3e81-4a1a-bd48-12b68f1b2a54-plugins-config-volume\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:11:13.252482 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:13.252382    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d67v\" (UniqueName: \"kubernetes.io/projected/1ec6b854-e19d-4615-b85b-3c5146f8e527-kube-api-access-4d67v\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:11:13.252482 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:13.252439    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rfbj\" (UniqueName: \"kubernetes.io/projected/22d56ed3-3e81-4a1a-bd48-12b68f1b2a54-kube-api-access-9rfbj\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:11:13.252482 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:13.252455    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ec6b854-e19d-4615-b85b-3c5146f8e527-plugins-config-volume\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:11:13.468459 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:13.468416    2581 generic.go:356] "Generic (PLEG): container finished" podID="22d56ed3-3e81-4a1a-bd48-12b68f1b2a54" containerID="54652ec8238f89d1e8550c84c0189315463a8a53437609835a3778d58c5658b4" exitCode=0
Sep 10 21:11:13.468667 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:13.468462    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-a7bbcc15-75cd9c9dcd-frstp"
Sep 10 21:11:13.468667 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:13.468499    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a7bbcc15-75cd9c9dcd-frstp" event={"ID":"22d56ed3-3e81-4a1a-bd48-12b68f1b2a54","Type":"ContainerDied","Data":"54652ec8238f89d1e8550c84c0189315463a8a53437609835a3778d58c5658b4"}
Sep 10 21:11:13.468667 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:13.468553    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-a7bbcc15-75cd9c9dcd-frstp" event={"ID":"22d56ed3-3e81-4a1a-bd48-12b68f1b2a54","Type":"ContainerDied","Data":"2d8c69ebeca90c3f86d3d17d579663ccd155a0df9e1a7d6336aa163838cc3598"}
Sep 10 21:11:13.468667 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:13.468580    2581 scope.go:117] "RemoveContainer" containerID="54652ec8238f89d1e8550c84c0189315463a8a53437609835a3778d58c5658b4"
Sep 10 21:11:13.469918 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:13.469899    2581 generic.go:356] "Generic (PLEG): container finished" podID="1ec6b854-e19d-4615-b85b-3c5146f8e527" containerID="63f2282df8f2657eb1f5153da0e69b2108a4ca6a2a50f54cd53b86553b394a68" exitCode=0
Sep 10 21:11:13.470030 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:13.469929    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a7bbcc15-86df455d-6657d" event={"ID":"1ec6b854-e19d-4615-b85b-3c5146f8e527","Type":"ContainerDied","Data":"63f2282df8f2657eb1f5153da0e69b2108a4ca6a2a50f54cd53b86553b394a68"}
Sep 10 21:11:13.470030 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:13.469995    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a7bbcc15-86df455d-6657d" event={"ID":"1ec6b854-e19d-4615-b85b-3c5146f8e527","Type":"ContainerDied","Data":"fd31c42448a64c356b6e5b3759feaf330235e97e25749477d44e27cc6f3552ee"}
Sep 10 21:11:13.470030 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:13.469947    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a7bbcc15-86df455d-6657d"
Sep 10 21:11:13.477467 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:13.477450    2581 scope.go:117] "RemoveContainer" containerID="54652ec8238f89d1e8550c84c0189315463a8a53437609835a3778d58c5658b4"
Sep 10 21:11:13.477803 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:11:13.477779    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54652ec8238f89d1e8550c84c0189315463a8a53437609835a3778d58c5658b4\": container with ID starting with 54652ec8238f89d1e8550c84c0189315463a8a53437609835a3778d58c5658b4 not found: ID does not exist" containerID="54652ec8238f89d1e8550c84c0189315463a8a53437609835a3778d58c5658b4"
Sep 10 21:11:13.477883 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:13.477810    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54652ec8238f89d1e8550c84c0189315463a8a53437609835a3778d58c5658b4"} err="failed to get container status \"54652ec8238f89d1e8550c84c0189315463a8a53437609835a3778d58c5658b4\": rpc error: code = NotFound desc = could not find container \"54652ec8238f89d1e8550c84c0189315463a8a53437609835a3778d58c5658b4\": container with ID starting with 54652ec8238f89d1e8550c84c0189315463a8a53437609835a3778d58c5658b4 not found: ID does not exist"
Sep 10 21:11:13.477883 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:13.477828    2581 scope.go:117] "RemoveContainer" containerID="63f2282df8f2657eb1f5153da0e69b2108a4ca6a2a50f54cd53b86553b394a68"
Sep 10 21:11:13.486047 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:13.486033    2581 scope.go:117] "RemoveContainer" containerID="63f2282df8f2657eb1f5153da0e69b2108a4ca6a2a50f54cd53b86553b394a68"
Sep 10 21:11:13.486279 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:11:13.486260    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63f2282df8f2657eb1f5153da0e69b2108a4ca6a2a50f54cd53b86553b394a68\": container with ID starting with 63f2282df8f2657eb1f5153da0e69b2108a4ca6a2a50f54cd53b86553b394a68 not found: ID does not exist" containerID="63f2282df8f2657eb1f5153da0e69b2108a4ca6a2a50f54cd53b86553b394a68"
Sep 10 21:11:13.486353 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:13.486287    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63f2282df8f2657eb1f5153da0e69b2108a4ca6a2a50f54cd53b86553b394a68"} err="failed to get container status \"63f2282df8f2657eb1f5153da0e69b2108a4ca6a2a50f54cd53b86553b394a68\": rpc error: code = NotFound desc = could not find container \"63f2282df8f2657eb1f5153da0e69b2108a4ca6a2a50f54cd53b86553b394a68\": container with ID starting with 63f2282df8f2657eb1f5153da0e69b2108a4ca6a2a50f54cd53b86553b394a68 not found: ID does not exist"
Sep 10 21:11:13.492960 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:13.492934    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a7bbcc15-75cd9c9dcd-frstp"]
Sep 10 21:11:13.498405 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:13.498385    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-a7bbcc15-75cd9c9dcd-frstp"]
Sep 10 21:11:13.508537 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:13.508514    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a7bbcc15-86df455d-6657d"]
Sep 10 21:11:13.516545 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:13.516526    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a7bbcc15-86df455d-6657d"]
Sep 10 21:11:14.868215 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:14.868182    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ec6b854-e19d-4615-b85b-3c5146f8e527" path="/var/lib/kubelet/pods/1ec6b854-e19d-4615-b85b-3c5146f8e527/volumes"
Sep 10 21:11:14.868589 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:14.868556    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22d56ed3-3e81-4a1a-bd48-12b68f1b2a54" path="/var/lib/kubelet/pods/22d56ed3-3e81-4a1a-bd48-12b68f1b2a54/volumes"
Sep 10 21:11:14.868846 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:14.868833    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="535f12aa-aba8-463d-973d-788d9ef93963" path="/var/lib/kubelet/pods/535f12aa-aba8-463d-973d-788d9ef93963/volumes"
Sep 10 21:11:18.343451 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:18.343420    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b1749e9a-7445ff7946-482rq"]
Sep 10 21:11:18.343842 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:18.343826    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ec6b854-e19d-4615-b85b-3c5146f8e527" containerName="payload-pre-processing"
Sep 10 21:11:18.343899 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:18.343844    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ec6b854-e19d-4615-b85b-3c5146f8e527" containerName="payload-pre-processing"
Sep 10 21:11:18.343899 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:18.343853    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22d56ed3-3e81-4a1a-bd48-12b68f1b2a54" containerName="payload-processing"
Sep 10 21:11:18.343899 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:18.343858    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="22d56ed3-3e81-4a1a-bd48-12b68f1b2a54" containerName="payload-processing"
Sep 10 21:11:18.343899 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:18.343866    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="535f12aa-aba8-463d-973d-788d9ef93963" containerName="revoke-keys"
Sep 10 21:11:18.343899 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:18.343871    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="535f12aa-aba8-463d-973d-788d9ef93963" containerName="revoke-keys"
Sep 10 21:11:18.344051 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:18.343976    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="535f12aa-aba8-463d-973d-788d9ef93963" containerName="revoke-keys"
Sep 10 21:11:18.344051 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:18.343988    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ec6b854-e19d-4615-b85b-3c5146f8e527" containerName="payload-pre-processing"
Sep 10 21:11:18.344051 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:18.343994    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="22d56ed3-3e81-4a1a-bd48-12b68f1b2a54" containerName="payload-processing"
Sep 10 21:11:18.348479 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:18.348454    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b1749e9a-7445ff7946-482rq"
Sep 10 21:11:18.351694 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:18.351674    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b1749e9a-dockercfg-zs6tp\""
Sep 10 21:11:18.352384 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:18.352366    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b1749e9a\""
Sep 10 21:11:18.365939 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:18.365918    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b1749e9a-7445ff7946-482rq"]
Sep 10 21:11:18.487096 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:18.487055    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b1749e9a-5c576dd67-9wvkc"]
Sep 10 21:11:18.490628 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:18.490609    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b1749e9a-5c576dd67-9wvkc"
Sep 10 21:11:18.499220 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:18.499201    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f887b1e-9cd7-45d8-8b80-3cd6f590fe74-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b1749e9a-7445ff7946-482rq\" (UID: \"2f887b1e-9cd7-45d8-8b80-3cd6f590fe74\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b1749e9a-7445ff7946-482rq"
Sep 10 21:11:18.499291 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:18.499237    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs7rn\" (UniqueName: \"kubernetes.io/projected/2f887b1e-9cd7-45d8-8b80-3cd6f590fe74-kube-api-access-gs7rn\") pod \"payload-pre-processing-e2e-ait-b1749e9a-7445ff7946-482rq\" (UID: \"2f887b1e-9cd7-45d8-8b80-3cd6f590fe74\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b1749e9a-7445ff7946-482rq"
Sep 10 21:11:18.512839 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:18.512815    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b1749e9a-5c576dd67-9wvkc"]
Sep 10 21:11:18.600094 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:18.600021    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs7rn\" (UniqueName: \"kubernetes.io/projected/2f887b1e-9cd7-45d8-8b80-3cd6f590fe74-kube-api-access-gs7rn\") pod \"payload-pre-processing-e2e-ait-b1749e9a-7445ff7946-482rq\" (UID: \"2f887b1e-9cd7-45d8-8b80-3cd6f590fe74\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b1749e9a-7445ff7946-482rq"
Sep 10 21:11:18.600094 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:18.600056    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcmrz\" (UniqueName: \"kubernetes.io/projected/d6c95bf6-5ea8-4d8c-a61e-864132be779f-kube-api-access-zcmrz\") pod \"payload-processing-e2e-ait-b1749e9a-5c576dd67-9wvkc\" (UID: \"d6c95bf6-5ea8-4d8c-a61e-864132be779f\") " pod="openshift-ingress/payload-processing-e2e-ait-b1749e9a-5c576dd67-9wvkc"
Sep 10 21:11:18.600346 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:18.600125    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f887b1e-9cd7-45d8-8b80-3cd6f590fe74-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b1749e9a-7445ff7946-482rq\" (UID: \"2f887b1e-9cd7-45d8-8b80-3cd6f590fe74\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b1749e9a-7445ff7946-482rq"
Sep 10 21:11:18.600346 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:18.600155    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6c95bf6-5ea8-4d8c-a61e-864132be779f-plugins-config-volume\") pod \"payload-processing-e2e-ait-b1749e9a-5c576dd67-9wvkc\" (UID: \"d6c95bf6-5ea8-4d8c-a61e-864132be779f\") " pod="openshift-ingress/payload-processing-e2e-ait-b1749e9a-5c576dd67-9wvkc"
Sep 10 21:11:18.600878 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:18.600858    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f887b1e-9cd7-45d8-8b80-3cd6f590fe74-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b1749e9a-7445ff7946-482rq\" (UID: \"2f887b1e-9cd7-45d8-8b80-3cd6f590fe74\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b1749e9a-7445ff7946-482rq"
Sep 10 21:11:18.619667 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:18.619645    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs7rn\" (UniqueName: \"kubernetes.io/projected/2f887b1e-9cd7-45d8-8b80-3cd6f590fe74-kube-api-access-gs7rn\") pod \"payload-pre-processing-e2e-ait-b1749e9a-7445ff7946-482rq\" (UID: \"2f887b1e-9cd7-45d8-8b80-3cd6f590fe74\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b1749e9a-7445ff7946-482rq"
Sep 10 21:11:18.660093 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:18.660042    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b1749e9a-7445ff7946-482rq"
Sep 10 21:11:18.700860 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:18.700825    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6c95bf6-5ea8-4d8c-a61e-864132be779f-plugins-config-volume\") pod \"payload-processing-e2e-ait-b1749e9a-5c576dd67-9wvkc\" (UID: \"d6c95bf6-5ea8-4d8c-a61e-864132be779f\") " pod="openshift-ingress/payload-processing-e2e-ait-b1749e9a-5c576dd67-9wvkc"
Sep 10 21:11:18.701040 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:18.700879    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcmrz\" (UniqueName: \"kubernetes.io/projected/d6c95bf6-5ea8-4d8c-a61e-864132be779f-kube-api-access-zcmrz\") pod \"payload-processing-e2e-ait-b1749e9a-5c576dd67-9wvkc\" (UID: \"d6c95bf6-5ea8-4d8c-a61e-864132be779f\") " pod="openshift-ingress/payload-processing-e2e-ait-b1749e9a-5c576dd67-9wvkc"
Sep 10 21:11:18.701554 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:18.701535    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6c95bf6-5ea8-4d8c-a61e-864132be779f-plugins-config-volume\") pod \"payload-processing-e2e-ait-b1749e9a-5c576dd67-9wvkc\" (UID: \"d6c95bf6-5ea8-4d8c-a61e-864132be779f\") " pod="openshift-ingress/payload-processing-e2e-ait-b1749e9a-5c576dd67-9wvkc"
Sep 10 21:11:18.712617 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:18.712590    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcmrz\" (UniqueName: \"kubernetes.io/projected/d6c95bf6-5ea8-4d8c-a61e-864132be779f-kube-api-access-zcmrz\") pod \"payload-processing-e2e-ait-b1749e9a-5c576dd67-9wvkc\" (UID: \"d6c95bf6-5ea8-4d8c-a61e-864132be779f\") " pod="openshift-ingress/payload-processing-e2e-ait-b1749e9a-5c576dd67-9wvkc"
Sep 10 21:11:18.800417 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:18.800387    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b1749e9a-5c576dd67-9wvkc"
Sep 10 21:11:18.938478 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:18.938452    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b1749e9a-7445ff7946-482rq"]
Sep 10 21:11:18.940089 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:11:18.940067    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f887b1e_9cd7_45d8_8b80_3cd6f590fe74.slice/crio-31979a6a767c8dd13d64a07a8ee83aa74ccbb0c4d2fdabfb4a2282cc60edc19b WatchSource:0}: Error finding container 31979a6a767c8dd13d64a07a8ee83aa74ccbb0c4d2fdabfb4a2282cc60edc19b: Status 404 returned error can't find the container with id 31979a6a767c8dd13d64a07a8ee83aa74ccbb0c4d2fdabfb4a2282cc60edc19b
Sep 10 21:11:19.083715 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:19.079701    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b1749e9a-5c576dd67-9wvkc"]
Sep 10 21:11:19.514621 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:19.514529    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b1749e9a-5c576dd67-9wvkc" event={"ID":"d6c95bf6-5ea8-4d8c-a61e-864132be779f","Type":"ContainerStarted","Data":"008b1fe91075d2712558177d4b9c617e3e24b4d0dca6f376f6444c53a14e306b"}
Sep 10 21:11:19.514621 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:19.514583    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b1749e9a-5c576dd67-9wvkc" event={"ID":"d6c95bf6-5ea8-4d8c-a61e-864132be779f","Type":"ContainerStarted","Data":"b6161bb26705fe2ed670de057052862cc125602c33d0d2bfc713bb3f96b68b51"}
Sep 10 21:11:19.514621 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:19.514614    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b1749e9a-5c576dd67-9wvkc"
Sep 10 21:11:19.516269 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:19.516243    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b1749e9a-7445ff7946-482rq" event={"ID":"2f887b1e-9cd7-45d8-8b80-3cd6f590fe74","Type":"ContainerStarted","Data":"563d1aa8e31bf7cc379af7520bf757ee872f856b45ed5d846b10e0aef3e1cc0a"}
Sep 10 21:11:19.516415 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:19.516289    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b1749e9a-7445ff7946-482rq" event={"ID":"2f887b1e-9cd7-45d8-8b80-3cd6f590fe74","Type":"ContainerStarted","Data":"31979a6a767c8dd13d64a07a8ee83aa74ccbb0c4d2fdabfb4a2282cc60edc19b"}
Sep 10 21:11:19.516415 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:19.516362    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b1749e9a-7445ff7946-482rq"
Sep 10 21:11:19.533401 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:19.533346    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b1749e9a-5c576dd67-9wvkc" podStartSLOduration=1.533331937 podStartE2EDuration="1.533331937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:11:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:11:19.531410314 +0000 UTC m=+797.188319086" watchObservedRunningTime="2026-09-10 21:11:19.533331937 +0000 UTC m=+797.190240710"
Sep 10 21:11:19.548028 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:19.547988    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-b1749e9a-7445ff7946-482rq" podStartSLOduration=1.54797373 podStartE2EDuration="1.54797373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:11:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:11:19.546938173 +0000 UTC m=+797.203846942" watchObservedRunningTime="2026-09-10 21:11:19.54797373 +0000 UTC m=+797.204882502"
Sep 10 21:11:30.523473 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:30.523401    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b1749e9a-5c576dd67-9wvkc"
Sep 10 21:11:30.523859 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:30.523595    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b1749e9a-7445ff7946-482rq"
Sep 10 21:11:42.197404 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:42.197357    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1749e9a-b2wk6"]
Sep 10 21:11:42.201178 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:42.201156    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1749e9a-b2wk6"
Sep 10 21:11:42.203582 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:42.203556    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:11:42.203704 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:42.203596    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s4qqn\""
Sep 10 21:11:42.209728 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:42.209362    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1749e9a-b2wk6"]
Sep 10 21:11:42.308683 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:42.308637    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c39005-7a95-4831-a32f-88314a95fa09-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b1749e9a-b2wk6\" (UID: \"b8c39005-7a95-4831-a32f-88314a95fa09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1749e9a-b2wk6"
Sep 10 21:11:42.409750 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:42.409710    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c39005-7a95-4831-a32f-88314a95fa09-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b1749e9a-b2wk6\" (UID: \"b8c39005-7a95-4831-a32f-88314a95fa09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1749e9a-b2wk6"
Sep 10 21:11:42.410414 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:42.410391    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c39005-7a95-4831-a32f-88314a95fa09-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b1749e9a-b2wk6\" (UID: \"b8c39005-7a95-4831-a32f-88314a95fa09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1749e9a-b2wk6"
Sep 10 21:11:42.514161 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:42.514065    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1749e9a-b2wk6"
Sep 10 21:11:42.786953 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:42.786925    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1749e9a-b2wk6"]
Sep 10 21:11:42.789021 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:11:42.788990    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c39005_7a95_4831_a32f_88314a95fa09.slice/crio-3fdcbb418891f4a757e8ce1d2937915138478b8709aa7fd8b3dfc0a2ebf50053 WatchSource:0}: Error finding container 3fdcbb418891f4a757e8ce1d2937915138478b8709aa7fd8b3dfc0a2ebf50053: Status 404 returned error can't find the container with id 3fdcbb418891f4a757e8ce1d2937915138478b8709aa7fd8b3dfc0a2ebf50053
Sep 10 21:11:43.638858 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:43.638821    2581 generic.go:356] "Generic (PLEG): container finished" podID="b8c39005-7a95-4831-a32f-88314a95fa09" containerID="3fd4a5197e04c6514800c418422a06d4e02be7091bfee0cb1910ca0f89edcee7" exitCode=0
Sep 10 21:11:43.639211 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:43.638901    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1749e9a-b2wk6" event={"ID":"b8c39005-7a95-4831-a32f-88314a95fa09","Type":"ContainerDied","Data":"3fd4a5197e04c6514800c418422a06d4e02be7091bfee0cb1910ca0f89edcee7"}
Sep 10 21:11:43.639211 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:43.638951    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1749e9a-b2wk6" event={"ID":"b8c39005-7a95-4831-a32f-88314a95fa09","Type":"ContainerStarted","Data":"3fdcbb418891f4a757e8ce1d2937915138478b8709aa7fd8b3dfc0a2ebf50053"}
Sep 10 21:11:44.785134 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:44.785111    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1749e9a-b2wk6"
Sep 10 21:11:44.831141 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:44.831112    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c39005-7a95-4831-a32f-88314a95fa09-maas-api-service-ca\") pod \"b8c39005-7a95-4831-a32f-88314a95fa09\" (UID: \"b8c39005-7a95-4831-a32f-88314a95fa09\") "
Sep 10 21:11:44.831505 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:44.831481    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8c39005-7a95-4831-a32f-88314a95fa09-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b8c39005-7a95-4831-a32f-88314a95fa09" (UID: "b8c39005-7a95-4831-a32f-88314a95fa09"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:11:44.931907 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:44.931837    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c39005-7a95-4831-a32f-88314a95fa09-maas-api-service-ca\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:11:45.649736 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:45.649706    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1749e9a-b2wk6"
Sep 10 21:11:45.649908 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:45.649705    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1749e9a-b2wk6" event={"ID":"b8c39005-7a95-4831-a32f-88314a95fa09","Type":"ContainerDied","Data":"3fdcbb418891f4a757e8ce1d2937915138478b8709aa7fd8b3dfc0a2ebf50053"}
Sep 10 21:11:45.649908 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:45.649839    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fdcbb418891f4a757e8ce1d2937915138478b8709aa7fd8b3dfc0a2ebf50053"
Sep 10 21:11:48.221650 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:48.221610    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b1749e9a-5c576dd67-9wvkc"]
Sep 10 21:11:48.222133 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:48.221855    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b1749e9a-5c576dd67-9wvkc" podUID="d6c95bf6-5ea8-4d8c-a61e-864132be779f" containerName="payload-processing" containerID="cri-o://008b1fe91075d2712558177d4b9c617e3e24b4d0dca6f376f6444c53a14e306b" gracePeriod=30
Sep 10 21:11:48.230848 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:48.230807    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b1749e9a-7445ff7946-482rq"]
Sep 10 21:11:48.231377 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:48.231166    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-b1749e9a-7445ff7946-482rq" podUID="2f887b1e-9cd7-45d8-8b80-3cd6f590fe74" containerName="payload-pre-processing" containerID="cri-o://563d1aa8e31bf7cc379af7520bf757ee872f856b45ed5d846b10e0aef3e1cc0a" gracePeriod=30
Sep 10 21:11:48.395438 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:48.395334    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1749e9a-b2wk6"]
Sep 10 21:11:48.399651 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:48.399623    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b1749e9a-b2wk6"]
Sep 10 21:11:48.521318 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:48.521283    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b1749e9a-5c576dd67-9wvkc"
Sep 10 21:11:48.525018 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:48.524993    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b1749e9a-7445ff7946-482rq"
Sep 10 21:11:48.565933 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:48.565904    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6c95bf6-5ea8-4d8c-a61e-864132be779f-plugins-config-volume\") pod \"d6c95bf6-5ea8-4d8c-a61e-864132be779f\" (UID: \"d6c95bf6-5ea8-4d8c-a61e-864132be779f\") "
Sep 10 21:11:48.566111 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:48.566026    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcmrz\" (UniqueName: \"kubernetes.io/projected/d6c95bf6-5ea8-4d8c-a61e-864132be779f-kube-api-access-zcmrz\") pod \"d6c95bf6-5ea8-4d8c-a61e-864132be779f\" (UID: \"d6c95bf6-5ea8-4d8c-a61e-864132be779f\") "
Sep 10 21:11:48.566392 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:48.566362    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6c95bf6-5ea8-4d8c-a61e-864132be779f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d6c95bf6-5ea8-4d8c-a61e-864132be779f" (UID: "d6c95bf6-5ea8-4d8c-a61e-864132be779f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:11:48.568205 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:48.568185    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6c95bf6-5ea8-4d8c-a61e-864132be779f-kube-api-access-zcmrz" (OuterVolumeSpecName: "kube-api-access-zcmrz") pod "d6c95bf6-5ea8-4d8c-a61e-864132be779f" (UID: "d6c95bf6-5ea8-4d8c-a61e-864132be779f"). InnerVolumeSpecName "kube-api-access-zcmrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:11:48.666791 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:48.666760    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f887b1e-9cd7-45d8-8b80-3cd6f590fe74-plugins-config-volume\") pod \"2f887b1e-9cd7-45d8-8b80-3cd6f590fe74\" (UID: \"2f887b1e-9cd7-45d8-8b80-3cd6f590fe74\") "
Sep 10 21:11:48.666973 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:48.666809    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gs7rn\" (UniqueName: \"kubernetes.io/projected/2f887b1e-9cd7-45d8-8b80-3cd6f590fe74-kube-api-access-gs7rn\") pod \"2f887b1e-9cd7-45d8-8b80-3cd6f590fe74\" (UID: \"2f887b1e-9cd7-45d8-8b80-3cd6f590fe74\") "
Sep 10 21:11:48.667054 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:48.667036    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcmrz\" (UniqueName: \"kubernetes.io/projected/d6c95bf6-5ea8-4d8c-a61e-864132be779f-kube-api-access-zcmrz\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:11:48.667110 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:48.667060    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6c95bf6-5ea8-4d8c-a61e-864132be779f-plugins-config-volume\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:11:48.667328 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:48.667106    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f887b1e-9cd7-45d8-8b80-3cd6f590fe74-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2f887b1e-9cd7-45d8-8b80-3cd6f590fe74" (UID: "2f887b1e-9cd7-45d8-8b80-3cd6f590fe74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:11:48.669087 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:48.669061    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f887b1e-9cd7-45d8-8b80-3cd6f590fe74-kube-api-access-gs7rn" (OuterVolumeSpecName: "kube-api-access-gs7rn") pod "2f887b1e-9cd7-45d8-8b80-3cd6f590fe74" (UID: "2f887b1e-9cd7-45d8-8b80-3cd6f590fe74"). InnerVolumeSpecName "kube-api-access-gs7rn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:11:48.669572 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:48.669549    2581 generic.go:356] "Generic (PLEG): container finished" podID="d6c95bf6-5ea8-4d8c-a61e-864132be779f" containerID="008b1fe91075d2712558177d4b9c617e3e24b4d0dca6f376f6444c53a14e306b" exitCode=0
Sep 10 21:11:48.669677 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:48.669591    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b1749e9a-5c576dd67-9wvkc"
Sep 10 21:11:48.669677 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:48.669627    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b1749e9a-5c576dd67-9wvkc" event={"ID":"d6c95bf6-5ea8-4d8c-a61e-864132be779f","Type":"ContainerDied","Data":"008b1fe91075d2712558177d4b9c617e3e24b4d0dca6f376f6444c53a14e306b"}
Sep 10 21:11:48.669762 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:48.669678    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b1749e9a-5c576dd67-9wvkc" event={"ID":"d6c95bf6-5ea8-4d8c-a61e-864132be779f","Type":"ContainerDied","Data":"b6161bb26705fe2ed670de057052862cc125602c33d0d2bfc713bb3f96b68b51"}
Sep 10 21:11:48.669762 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:48.669707    2581 scope.go:117] "RemoveContainer" containerID="008b1fe91075d2712558177d4b9c617e3e24b4d0dca6f376f6444c53a14e306b"
Sep 10 21:11:48.671013 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:48.670993    2581 generic.go:356] "Generic (PLEG): container finished" podID="2f887b1e-9cd7-45d8-8b80-3cd6f590fe74" containerID="563d1aa8e31bf7cc379af7520bf757ee872f856b45ed5d846b10e0aef3e1cc0a" exitCode=0
Sep 10 21:11:48.671099 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:48.671023    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b1749e9a-7445ff7946-482rq"
Sep 10 21:11:48.671099 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:48.671042    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b1749e9a-7445ff7946-482rq" event={"ID":"2f887b1e-9cd7-45d8-8b80-3cd6f590fe74","Type":"ContainerDied","Data":"563d1aa8e31bf7cc379af7520bf757ee872f856b45ed5d846b10e0aef3e1cc0a"}
Sep 10 21:11:48.671099 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:48.671078    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b1749e9a-7445ff7946-482rq" event={"ID":"2f887b1e-9cd7-45d8-8b80-3cd6f590fe74","Type":"ContainerDied","Data":"31979a6a767c8dd13d64a07a8ee83aa74ccbb0c4d2fdabfb4a2282cc60edc19b"}
Sep 10 21:11:48.679017 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:48.678889    2581 scope.go:117] "RemoveContainer" containerID="008b1fe91075d2712558177d4b9c617e3e24b4d0dca6f376f6444c53a14e306b"
Sep 10 21:11:48.679198 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:11:48.679171    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"008b1fe91075d2712558177d4b9c617e3e24b4d0dca6f376f6444c53a14e306b\": container with ID starting with 008b1fe91075d2712558177d4b9c617e3e24b4d0dca6f376f6444c53a14e306b not found: ID does not exist" containerID="008b1fe91075d2712558177d4b9c617e3e24b4d0dca6f376f6444c53a14e306b"
Sep 10 21:11:48.679325 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:48.679202    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"008b1fe91075d2712558177d4b9c617e3e24b4d0dca6f376f6444c53a14e306b"} err="failed to get container status \"008b1fe91075d2712558177d4b9c617e3e24b4d0dca6f376f6444c53a14e306b\": rpc error: code = NotFound desc = could not find container \"008b1fe91075d2712558177d4b9c617e3e24b4d0dca6f376f6444c53a14e306b\": container with ID starting with 008b1fe91075d2712558177d4b9c617e3e24b4d0dca6f376f6444c53a14e306b not found: ID does not exist"
Sep 10 21:11:48.679325 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:48.679216    2581 scope.go:117] "RemoveContainer" containerID="563d1aa8e31bf7cc379af7520bf757ee872f856b45ed5d846b10e0aef3e1cc0a"
Sep 10 21:11:48.686909 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:48.686894    2581 scope.go:117] "RemoveContainer" containerID="563d1aa8e31bf7cc379af7520bf757ee872f856b45ed5d846b10e0aef3e1cc0a"
Sep 10 21:11:48.687162 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:11:48.687145    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"563d1aa8e31bf7cc379af7520bf757ee872f856b45ed5d846b10e0aef3e1cc0a\": container with ID starting with 563d1aa8e31bf7cc379af7520bf757ee872f856b45ed5d846b10e0aef3e1cc0a not found: ID does not exist" containerID="563d1aa8e31bf7cc379af7520bf757ee872f856b45ed5d846b10e0aef3e1cc0a"
Sep 10 21:11:48.687218 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:48.687168    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"563d1aa8e31bf7cc379af7520bf757ee872f856b45ed5d846b10e0aef3e1cc0a"} err="failed to get container status \"563d1aa8e31bf7cc379af7520bf757ee872f856b45ed5d846b10e0aef3e1cc0a\": rpc error: code = NotFound desc = could not find container \"563d1aa8e31bf7cc379af7520bf757ee872f856b45ed5d846b10e0aef3e1cc0a\": container with ID starting with 563d1aa8e31bf7cc379af7520bf757ee872f856b45ed5d846b10e0aef3e1cc0a not found: ID does not exist"
Sep 10 21:11:48.701725 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:48.701706    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b1749e9a-5c576dd67-9wvkc"]
Sep 10 21:11:48.705397 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:48.705376    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b1749e9a-5c576dd67-9wvkc"]
Sep 10 21:11:48.715567 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:48.715539    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b1749e9a-7445ff7946-482rq"]
Sep 10 21:11:48.722268 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:48.722247    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b1749e9a-7445ff7946-482rq"]
Sep 10 21:11:48.768392 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:48.768328    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f887b1e-9cd7-45d8-8b80-3cd6f590fe74-plugins-config-volume\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:11:48.768392 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:48.768350    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs7rn\" (UniqueName: \"kubernetes.io/projected/2f887b1e-9cd7-45d8-8b80-3cd6f590fe74-kube-api-access-gs7rn\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:11:48.868981 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:48.868953    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f887b1e-9cd7-45d8-8b80-3cd6f590fe74" path="/var/lib/kubelet/pods/2f887b1e-9cd7-45d8-8b80-3cd6f590fe74/volumes"
Sep 10 21:11:48.869283 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:48.869269    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8c39005-7a95-4831-a32f-88314a95fa09" path="/var/lib/kubelet/pods/b8c39005-7a95-4831-a32f-88314a95fa09/volumes"
Sep 10 21:11:48.869600 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:48.869587    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6c95bf6-5ea8-4d8c-a61e-864132be779f" path="/var/lib/kubelet/pods/d6c95bf6-5ea8-4d8c-a61e-864132be779f/volumes"
Sep 10 21:11:55.489926 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:55.489889    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7b642ff4-6f57f76bfd-2l4cf"]
Sep 10 21:11:55.490269 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:55.490254    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f887b1e-9cd7-45d8-8b80-3cd6f590fe74" containerName="payload-pre-processing"
Sep 10 21:11:55.490269 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:55.490265    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f887b1e-9cd7-45d8-8b80-3cd6f590fe74" containerName="payload-pre-processing"
Sep 10 21:11:55.490390 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:55.490276    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8c39005-7a95-4831-a32f-88314a95fa09" containerName="revoke-keys"
Sep 10 21:11:55.490390 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:55.490283    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8c39005-7a95-4831-a32f-88314a95fa09" containerName="revoke-keys"
Sep 10 21:11:55.490390 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:55.490315    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6c95bf6-5ea8-4d8c-a61e-864132be779f" containerName="payload-processing"
Sep 10 21:11:55.490390 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:55.490321    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6c95bf6-5ea8-4d8c-a61e-864132be779f" containerName="payload-processing"
Sep 10 21:11:55.490390 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:55.490379    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f887b1e-9cd7-45d8-8b80-3cd6f590fe74" containerName="payload-pre-processing"
Sep 10 21:11:55.490390 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:55.490388    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8c39005-7a95-4831-a32f-88314a95fa09" containerName="revoke-keys"
Sep 10 21:11:55.490629 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:55.490395    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6c95bf6-5ea8-4d8c-a61e-864132be779f" containerName="payload-processing"
Sep 10 21:11:55.495505 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:55.495485    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-7b642ff4-6f57f76bfd-2l4cf"
Sep 10 21:11:55.497528 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:55.497508    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-7b642ff4\""
Sep 10 21:11:55.497648 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:55.497547    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-7b642ff4-dockercfg-96h6k\""
Sep 10 21:11:55.505690 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:55.505656    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7b642ff4-6f57f76bfd-2l4cf"]
Sep 10 21:11:55.529245 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:55.529216    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-7b642ff4-86bd4cd7c-b77km"]
Sep 10 21:11:55.533490 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:55.533466    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-7b642ff4-86bd4cd7c-b77km"
Sep 10 21:11:55.543287 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:55.543264    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-7b642ff4-86bd4cd7c-b77km"]
Sep 10 21:11:55.633350 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:55.633275    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1f8009a-e8a5-431b-ad3c-b9f2780d045f-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-7b642ff4-6f57f76bfd-2l4cf\" (UID: \"d1f8009a-e8a5-431b-ad3c-b9f2780d045f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7b642ff4-6f57f76bfd-2l4cf"
Sep 10 21:11:55.633524 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:55.633384    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428a29ec-b26d-4d13-b25d-94c078dde6f2-plugins-config-volume\") pod \"payload-processing-e2e-derive-7b642ff4-86bd4cd7c-b77km\" (UID: \"428a29ec-b26d-4d13-b25d-94c078dde6f2\") " pod="openshift-ingress/payload-processing-e2e-derive-7b642ff4-86bd4cd7c-b77km"
Sep 10 21:11:55.633524 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:55.633428    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wzbk\" (UniqueName: \"kubernetes.io/projected/d1f8009a-e8a5-431b-ad3c-b9f2780d045f-kube-api-access-8wzbk\") pod \"payload-pre-processing-e2e-derive-7b642ff4-6f57f76bfd-2l4cf\" (UID: \"d1f8009a-e8a5-431b-ad3c-b9f2780d045f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7b642ff4-6f57f76bfd-2l4cf"
Sep 10 21:11:55.633524 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:55.633447    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfdpj\" (UniqueName: \"kubernetes.io/projected/428a29ec-b26d-4d13-b25d-94c078dde6f2-kube-api-access-vfdpj\") pod \"payload-processing-e2e-derive-7b642ff4-86bd4cd7c-b77km\" (UID: \"428a29ec-b26d-4d13-b25d-94c078dde6f2\") " pod="openshift-ingress/payload-processing-e2e-derive-7b642ff4-86bd4cd7c-b77km"
Sep 10 21:11:55.734621 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:55.734578    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wzbk\" (UniqueName: \"kubernetes.io/projected/d1f8009a-e8a5-431b-ad3c-b9f2780d045f-kube-api-access-8wzbk\") pod \"payload-pre-processing-e2e-derive-7b642ff4-6f57f76bfd-2l4cf\" (UID: \"d1f8009a-e8a5-431b-ad3c-b9f2780d045f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7b642ff4-6f57f76bfd-2l4cf"
Sep 10 21:11:55.734809 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:55.734627    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfdpj\" (UniqueName: \"kubernetes.io/projected/428a29ec-b26d-4d13-b25d-94c078dde6f2-kube-api-access-vfdpj\") pod \"payload-processing-e2e-derive-7b642ff4-86bd4cd7c-b77km\" (UID: \"428a29ec-b26d-4d13-b25d-94c078dde6f2\") " pod="openshift-ingress/payload-processing-e2e-derive-7b642ff4-86bd4cd7c-b77km"
Sep 10 21:11:55.734809 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:55.734759    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1f8009a-e8a5-431b-ad3c-b9f2780d045f-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-7b642ff4-6f57f76bfd-2l4cf\" (UID: \"d1f8009a-e8a5-431b-ad3c-b9f2780d045f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7b642ff4-6f57f76bfd-2l4cf"
Sep 10 21:11:55.734809 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:55.734787    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428a29ec-b26d-4d13-b25d-94c078dde6f2-plugins-config-volume\") pod \"payload-processing-e2e-derive-7b642ff4-86bd4cd7c-b77km\" (UID: \"428a29ec-b26d-4d13-b25d-94c078dde6f2\") " pod="openshift-ingress/payload-processing-e2e-derive-7b642ff4-86bd4cd7c-b77km"
Sep 10 21:11:55.735563 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:55.735538    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1f8009a-e8a5-431b-ad3c-b9f2780d045f-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-7b642ff4-6f57f76bfd-2l4cf\" (UID: \"d1f8009a-e8a5-431b-ad3c-b9f2780d045f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7b642ff4-6f57f76bfd-2l4cf"
Sep 10 21:11:55.735670 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:55.735547    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428a29ec-b26d-4d13-b25d-94c078dde6f2-plugins-config-volume\") pod \"payload-processing-e2e-derive-7b642ff4-86bd4cd7c-b77km\" (UID: \"428a29ec-b26d-4d13-b25d-94c078dde6f2\") " pod="openshift-ingress/payload-processing-e2e-derive-7b642ff4-86bd4cd7c-b77km"
Sep 10 21:11:55.744238 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:55.744179    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfdpj\" (UniqueName: \"kubernetes.io/projected/428a29ec-b26d-4d13-b25d-94c078dde6f2-kube-api-access-vfdpj\") pod \"payload-processing-e2e-derive-7b642ff4-86bd4cd7c-b77km\" (UID: \"428a29ec-b26d-4d13-b25d-94c078dde6f2\") " pod="openshift-ingress/payload-processing-e2e-derive-7b642ff4-86bd4cd7c-b77km"
Sep 10 21:11:55.744484 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:55.744462    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wzbk\" (UniqueName: \"kubernetes.io/projected/d1f8009a-e8a5-431b-ad3c-b9f2780d045f-kube-api-access-8wzbk\") pod \"payload-pre-processing-e2e-derive-7b642ff4-6f57f76bfd-2l4cf\" (UID: \"d1f8009a-e8a5-431b-ad3c-b9f2780d045f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-7b642ff4-6f57f76bfd-2l4cf"
Sep 10 21:11:55.806851 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:55.806820    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-7b642ff4-6f57f76bfd-2l4cf"
Sep 10 21:11:55.844028 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:55.843995    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-7b642ff4-86bd4cd7c-b77km"
Sep 10 21:11:56.085797 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:56.085761    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7b642ff4-6f57f76bfd-2l4cf"]
Sep 10 21:11:56.089721 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:11:56.089694    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1f8009a_e8a5_431b_ad3c_b9f2780d045f.slice/crio-46a2a4c6549cdbfa245be6bbb11aac3b4cff5bd7979513a05a3be8d58f81d115 WatchSource:0}: Error finding container 46a2a4c6549cdbfa245be6bbb11aac3b4cff5bd7979513a05a3be8d58f81d115: Status 404 returned error can't find the container with id 46a2a4c6549cdbfa245be6bbb11aac3b4cff5bd7979513a05a3be8d58f81d115
Sep 10 21:11:56.133066 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:56.133038    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-7b642ff4-86bd4cd7c-b77km"]
Sep 10 21:11:56.136781 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:11:56.136757    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod428a29ec_b26d_4d13_b25d_94c078dde6f2.slice/crio-9410ded346edae99dd108adb6189a02674f7ba0c3f197de77527ff9b7a43988c WatchSource:0}: Error finding container 9410ded346edae99dd108adb6189a02674f7ba0c3f197de77527ff9b7a43988c: Status 404 returned error can't find the container with id 9410ded346edae99dd108adb6189a02674f7ba0c3f197de77527ff9b7a43988c
Sep 10 21:11:56.713776 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:56.713735    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-7b642ff4-6f57f76bfd-2l4cf" event={"ID":"d1f8009a-e8a5-431b-ad3c-b9f2780d045f","Type":"ContainerStarted","Data":"47bd77a70b1e3ff331fc3af8a7039c1d0d78a696843d2db0afe5959dc2c951ef"}
Sep 10 21:11:56.714166 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:56.713885    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-7b642ff4-6f57f76bfd-2l4cf"
Sep 10 21:11:56.714166 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:56.713920    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-7b642ff4-6f57f76bfd-2l4cf" event={"ID":"d1f8009a-e8a5-431b-ad3c-b9f2780d045f","Type":"ContainerStarted","Data":"46a2a4c6549cdbfa245be6bbb11aac3b4cff5bd7979513a05a3be8d58f81d115"}
Sep 10 21:11:56.715256 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:56.715232    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-7b642ff4-86bd4cd7c-b77km" event={"ID":"428a29ec-b26d-4d13-b25d-94c078dde6f2","Type":"ContainerStarted","Data":"fb046a20091e26ddb2160b58e8602d549f366fc2d07fd20d29c1f51428682115"}
Sep 10 21:11:56.715409 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:56.715282    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-7b642ff4-86bd4cd7c-b77km" event={"ID":"428a29ec-b26d-4d13-b25d-94c078dde6f2","Type":"ContainerStarted","Data":"9410ded346edae99dd108adb6189a02674f7ba0c3f197de77527ff9b7a43988c"}
Sep 10 21:11:56.715409 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:56.715334    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-7b642ff4-86bd4cd7c-b77km"
Sep 10 21:11:56.732476 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:56.732432    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-7b642ff4-6f57f76bfd-2l4cf" podStartSLOduration=1.7324176260000002 podStartE2EDuration="1.732417626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:11:56.731263396 +0000 UTC m=+834.388172192" watchObservedRunningTime="2026-09-10 21:11:56.732417626 +0000 UTC m=+834.389326398"
Sep 10 21:11:56.750506 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:11:56.750456    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-7b642ff4-86bd4cd7c-b77km" podStartSLOduration=1.75043712 podStartE2EDuration="1.75043712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:11:56.747223456 +0000 UTC m=+834.404132230" watchObservedRunningTime="2026-09-10 21:11:56.75043712 +0000 UTC m=+834.407345894"
Sep 10 21:12:05.191808 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:05.191772    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-47f62f/test-model-47f62f-kserve-6c97f6848c-hqjcq"]
Sep 10 21:12:05.196453 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:05.196437    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-47f62f/test-model-47f62f-kserve-6c97f6848c-hqjcq"
Sep 10 21:12:05.199784 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:05.199764    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-47f62f\"/\"default-dockercfg-rpmdk\""
Sep 10 21:12:05.199934 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:05.199790    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-47f62f\"/\"kube-root-ca.crt\""
Sep 10 21:12:05.199934 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:05.199769    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-47f62f\"/\"openshift-service-ca.crt\""
Sep 10 21:12:05.200056 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:05.200021    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-47f62f\"/\"test-model-47f62f-kserve-self-signed-certs\""
Sep 10 21:12:05.211363 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:05.211344    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-47f62f/test-model-47f62f-kserve-6c97f6848c-hqjcq"]
Sep 10 21:12:05.324279 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:05.324245    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg2jh\" (UniqueName: \"kubernetes.io/projected/e9aca7af-30c4-423a-8cf4-29682d49b618-kube-api-access-kg2jh\") pod \"test-model-47f62f-kserve-6c97f6848c-hqjcq\" (UID: \"e9aca7af-30c4-423a-8cf4-29682d49b618\") " pod="ai-tenant-e2e-inf-a-47f62f/test-model-47f62f-kserve-6c97f6848c-hqjcq"
Sep 10 21:12:05.324476 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:05.324328    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9aca7af-30c4-423a-8cf4-29682d49b618-tls-certs\") pod \"test-model-47f62f-kserve-6c97f6848c-hqjcq\" (UID: \"e9aca7af-30c4-423a-8cf4-29682d49b618\") " pod="ai-tenant-e2e-inf-a-47f62f/test-model-47f62f-kserve-6c97f6848c-hqjcq"
Sep 10 21:12:05.324476 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:05.324364    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9aca7af-30c4-423a-8cf4-29682d49b618-model-cache\") pod \"test-model-47f62f-kserve-6c97f6848c-hqjcq\" (UID: \"e9aca7af-30c4-423a-8cf4-29682d49b618\") " pod="ai-tenant-e2e-inf-a-47f62f/test-model-47f62f-kserve-6c97f6848c-hqjcq"
Sep 10 21:12:05.324476 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:05.324465    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9aca7af-30c4-423a-8cf4-29682d49b618-home\") pod \"test-model-47f62f-kserve-6c97f6848c-hqjcq\" (UID: \"e9aca7af-30c4-423a-8cf4-29682d49b618\") " pod="ai-tenant-e2e-inf-a-47f62f/test-model-47f62f-kserve-6c97f6848c-hqjcq"
Sep 10 21:12:05.324620 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:05.324496    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9aca7af-30c4-423a-8cf4-29682d49b618-tmp-dir\") pod \"test-model-47f62f-kserve-6c97f6848c-hqjcq\" (UID: \"e9aca7af-30c4-423a-8cf4-29682d49b618\") " pod="ai-tenant-e2e-inf-a-47f62f/test-model-47f62f-kserve-6c97f6848c-hqjcq"
Sep 10 21:12:05.324620 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:05.324517    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9aca7af-30c4-423a-8cf4-29682d49b618-dshm\") pod \"test-model-47f62f-kserve-6c97f6848c-hqjcq\" (UID: \"e9aca7af-30c4-423a-8cf4-29682d49b618\") " pod="ai-tenant-e2e-inf-a-47f62f/test-model-47f62f-kserve-6c97f6848c-hqjcq"
Sep 10 21:12:05.425546 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:05.425513    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9aca7af-30c4-423a-8cf4-29682d49b618-tls-certs\") pod \"test-model-47f62f-kserve-6c97f6848c-hqjcq\" (UID: \"e9aca7af-30c4-423a-8cf4-29682d49b618\") " pod="ai-tenant-e2e-inf-a-47f62f/test-model-47f62f-kserve-6c97f6848c-hqjcq"
Sep 10 21:12:05.425700 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:05.425551    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9aca7af-30c4-423a-8cf4-29682d49b618-model-cache\") pod \"test-model-47f62f-kserve-6c97f6848c-hqjcq\" (UID: \"e9aca7af-30c4-423a-8cf4-29682d49b618\") " pod="ai-tenant-e2e-inf-a-47f62f/test-model-47f62f-kserve-6c97f6848c-hqjcq"
Sep 10 21:12:05.425700 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:05.425603    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9aca7af-30c4-423a-8cf4-29682d49b618-home\") pod \"test-model-47f62f-kserve-6c97f6848c-hqjcq\" (UID: \"e9aca7af-30c4-423a-8cf4-29682d49b618\") " pod="ai-tenant-e2e-inf-a-47f62f/test-model-47f62f-kserve-6c97f6848c-hqjcq"
Sep 10 21:12:05.425700 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:05.425624    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9aca7af-30c4-423a-8cf4-29682d49b618-tmp-dir\") pod \"test-model-47f62f-kserve-6c97f6848c-hqjcq\" (UID: \"e9aca7af-30c4-423a-8cf4-29682d49b618\") " pod="ai-tenant-e2e-inf-a-47f62f/test-model-47f62f-kserve-6c97f6848c-hqjcq"
Sep 10 21:12:05.425700 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:05.425642    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9aca7af-30c4-423a-8cf4-29682d49b618-dshm\") pod \"test-model-47f62f-kserve-6c97f6848c-hqjcq\" (UID: \"e9aca7af-30c4-423a-8cf4-29682d49b618\") " pod="ai-tenant-e2e-inf-a-47f62f/test-model-47f62f-kserve-6c97f6848c-hqjcq"
Sep 10 21:12:05.425899 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:05.425702    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg2jh\" (UniqueName: \"kubernetes.io/projected/e9aca7af-30c4-423a-8cf4-29682d49b618-kube-api-access-kg2jh\") pod \"test-model-47f62f-kserve-6c97f6848c-hqjcq\" (UID: \"e9aca7af-30c4-423a-8cf4-29682d49b618\") " pod="ai-tenant-e2e-inf-a-47f62f/test-model-47f62f-kserve-6c97f6848c-hqjcq"
Sep 10 21:12:05.426069 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:05.426038    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9aca7af-30c4-423a-8cf4-29682d49b618-home\") pod \"test-model-47f62f-kserve-6c97f6848c-hqjcq\" (UID: \"e9aca7af-30c4-423a-8cf4-29682d49b618\") " pod="ai-tenant-e2e-inf-a-47f62f/test-model-47f62f-kserve-6c97f6848c-hqjcq"
Sep 10 21:12:05.426149 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:05.426069    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9aca7af-30c4-423a-8cf4-29682d49b618-tmp-dir\") pod \"test-model-47f62f-kserve-6c97f6848c-hqjcq\" (UID: \"e9aca7af-30c4-423a-8cf4-29682d49b618\") " pod="ai-tenant-e2e-inf-a-47f62f/test-model-47f62f-kserve-6c97f6848c-hqjcq"
Sep 10 21:12:05.426149 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:05.426142    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9aca7af-30c4-423a-8cf4-29682d49b618-model-cache\") pod \"test-model-47f62f-kserve-6c97f6848c-hqjcq\" (UID: \"e9aca7af-30c4-423a-8cf4-29682d49b618\") " pod="ai-tenant-e2e-inf-a-47f62f/test-model-47f62f-kserve-6c97f6848c-hqjcq"
Sep 10 21:12:05.427750 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:05.427729    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9aca7af-30c4-423a-8cf4-29682d49b618-dshm\") pod \"test-model-47f62f-kserve-6c97f6848c-hqjcq\" (UID: \"e9aca7af-30c4-423a-8cf4-29682d49b618\") " pod="ai-tenant-e2e-inf-a-47f62f/test-model-47f62f-kserve-6c97f6848c-hqjcq"
Sep 10 21:12:05.428196 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:05.428176    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9aca7af-30c4-423a-8cf4-29682d49b618-tls-certs\") pod \"test-model-47f62f-kserve-6c97f6848c-hqjcq\" (UID: \"e9aca7af-30c4-423a-8cf4-29682d49b618\") " pod="ai-tenant-e2e-inf-a-47f62f/test-model-47f62f-kserve-6c97f6848c-hqjcq"
Sep 10 21:12:05.433574 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:05.433554    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg2jh\" (UniqueName: \"kubernetes.io/projected/e9aca7af-30c4-423a-8cf4-29682d49b618-kube-api-access-kg2jh\") pod \"test-model-47f62f-kserve-6c97f6848c-hqjcq\" (UID: \"e9aca7af-30c4-423a-8cf4-29682d49b618\") " pod="ai-tenant-e2e-inf-a-47f62f/test-model-47f62f-kserve-6c97f6848c-hqjcq"
Sep 10 21:12:05.505555 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:05.505477    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-47f62f/test-model-47f62f-kserve-6c97f6848c-hqjcq"
Sep 10 21:12:05.784670 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:05.784647    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-47f62f/test-model-47f62f-kserve-6c97f6848c-hqjcq"]
Sep 10 21:12:05.787061 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:12:05.787019    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9aca7af_30c4_423a_8cf4_29682d49b618.slice/crio-184a75a216284f4d87aa9d34e2be71b056ece8b1fc57095397b280d8838ebd9c WatchSource:0}: Error finding container 184a75a216284f4d87aa9d34e2be71b056ece8b1fc57095397b280d8838ebd9c: Status 404 returned error can't find the container with id 184a75a216284f4d87aa9d34e2be71b056ece8b1fc57095397b280d8838ebd9c
Sep 10 21:12:06.773513 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:06.773476    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-47f62f/test-model-47f62f-kserve-6c97f6848c-hqjcq" event={"ID":"e9aca7af-30c4-423a-8cf4-29682d49b618","Type":"ContainerStarted","Data":"ba023f87cca87f63f3775eef69af3bcd9ae4f74a959905a00bbbf6cac472aeb4"}
Sep 10 21:12:06.773865 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:06.773521    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-47f62f/test-model-47f62f-kserve-6c97f6848c-hqjcq" event={"ID":"e9aca7af-30c4-423a-8cf4-29682d49b618","Type":"ContainerStarted","Data":"184a75a216284f4d87aa9d34e2be71b056ece8b1fc57095397b280d8838ebd9c"}
Sep 10 21:12:06.794608 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:06.794566    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-47f62f/test-model-47f62f-kserve-6c97f6848c-hqjcq" podStartSLOduration=1.794551859 podStartE2EDuration="1.794551859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:12:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:12:06.793200086 +0000 UTC m=+844.450108862" watchObservedRunningTime="2026-09-10 21:12:06.794551859 +0000 UTC m=+844.451460631"
Sep 10 21:12:07.722863 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:07.722828    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-7b642ff4-6f57f76bfd-2l4cf"
Sep 10 21:12:07.723147 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:07.723126    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-7b642ff4-86bd4cd7c-b77km"
Sep 10 21:12:10.722863 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:10.722827    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-63f3d7-fjkdd"]
Sep 10 21:12:10.726326 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:10.726310    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-63f3d7-fjkdd"
Sep 10 21:12:10.728677 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:10.728644    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s4qqn\""
Sep 10 21:12:10.728677 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:10.728662    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:12:10.733872 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:10.733828    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-63f3d7-fjkdd"]
Sep 10 21:12:10.774560 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:10.774525    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/076e693b-1d23-4685-820c-a472475338ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-63f3d7-fjkdd\" (UID: \"076e693b-1d23-4685-820c-a472475338ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-63f3d7-fjkdd"
Sep 10 21:12:10.875712 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:10.875677    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/076e693b-1d23-4685-820c-a472475338ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-63f3d7-fjkdd\" (UID: \"076e693b-1d23-4685-820c-a472475338ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-63f3d7-fjkdd"
Sep 10 21:12:10.876468 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:10.876444    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/076e693b-1d23-4685-820c-a472475338ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-63f3d7-fjkdd\" (UID: \"076e693b-1d23-4685-820c-a472475338ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-63f3d7-fjkdd"
Sep 10 21:12:11.039820 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:11.039735    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-63f3d7-fjkdd"
Sep 10 21:12:11.309314 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:11.309265    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-63f3d7-fjkdd"]
Sep 10 21:12:11.312401 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:12:11.312366    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod076e693b_1d23_4685_820c_a472475338ea.slice/crio-040f49aff43ed671bb5433b554bd0d63d32eaa9b1780b64fd9cb8dd716796264 WatchSource:0}: Error finding container 040f49aff43ed671bb5433b554bd0d63d32eaa9b1780b64fd9cb8dd716796264: Status 404 returned error can't find the container with id 040f49aff43ed671bb5433b554bd0d63d32eaa9b1780b64fd9cb8dd716796264
Sep 10 21:12:11.805246 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:11.805209    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-63f3d7-fjkdd" event={"ID":"076e693b-1d23-4685-820c-a472475338ea","Type":"ContainerStarted","Data":"8d72ff27baa9695ed0ca69d7e37499453b24f89cd0a73e28aebbde3611dd508f"}
Sep 10 21:12:11.805678 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:11.805274    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-63f3d7-fjkdd" event={"ID":"076e693b-1d23-4685-820c-a472475338ea","Type":"ContainerStarted","Data":"040f49aff43ed671bb5433b554bd0d63d32eaa9b1780b64fd9cb8dd716796264"}
Sep 10 21:12:11.824920 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:11.824832    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-63f3d7-fjkdd" podStartSLOduration=1.824818775 podStartE2EDuration="1.824818775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:12:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:12:11.822659621 +0000 UTC m=+849.479568383" watchObservedRunningTime="2026-09-10 21:12:11.824818775 +0000 UTC m=+849.481727544"
Sep 10 21:12:12.812421 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:12.812383    2581 generic.go:356] "Generic (PLEG): container finished" podID="076e693b-1d23-4685-820c-a472475338ea" containerID="8d72ff27baa9695ed0ca69d7e37499453b24f89cd0a73e28aebbde3611dd508f" exitCode=0
Sep 10 21:12:12.812847 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:12.812467    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-63f3d7-fjkdd" event={"ID":"076e693b-1d23-4685-820c-a472475338ea","Type":"ContainerDied","Data":"8d72ff27baa9695ed0ca69d7e37499453b24f89cd0a73e28aebbde3611dd508f"}
Sep 10 21:12:13.957087 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:13.957057    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-63f3d7-fjkdd"
Sep 10 21:12:14.006485 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:14.006449    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/076e693b-1d23-4685-820c-a472475338ea-maas-api-service-ca\") pod \"076e693b-1d23-4685-820c-a472475338ea\" (UID: \"076e693b-1d23-4685-820c-a472475338ea\") "
Sep 10 21:12:14.006755 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:14.006712    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/076e693b-1d23-4685-820c-a472475338ea-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "076e693b-1d23-4685-820c-a472475338ea" (UID: "076e693b-1d23-4685-820c-a472475338ea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:12:14.006902 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:14.006890    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/076e693b-1d23-4685-820c-a472475338ea-maas-api-service-ca\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:12:14.824467 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:14.824438    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-63f3d7-fjkdd"
Sep 10 21:12:14.824644 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:14.824468    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-63f3d7-fjkdd" event={"ID":"076e693b-1d23-4685-820c-a472475338ea","Type":"ContainerDied","Data":"040f49aff43ed671bb5433b554bd0d63d32eaa9b1780b64fd9cb8dd716796264"}
Sep 10 21:12:14.824644 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:14.824511    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="040f49aff43ed671bb5433b554bd0d63d32eaa9b1780b64fd9cb8dd716796264"
Sep 10 21:12:15.505816 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:15.505783    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-47f62f/test-model-47f62f-kserve-6c97f6848c-hqjcq"
Sep 10 21:12:15.506233 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:15.505835    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-47f62f/test-model-47f62f-kserve-6c97f6848c-hqjcq"
Sep 10 21:12:15.518384 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:15.518358    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-47f62f/test-model-47f62f-kserve-6c97f6848c-hqjcq"
Sep 10 21:12:15.842104 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:15.842065    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-47f62f/test-model-47f62f-kserve-6c97f6848c-hqjcq"
Sep 10 21:12:18.359028 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:18.358994    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7b642ff4-dhsfx"]
Sep 10 21:12:18.359716 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:18.359695    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="076e693b-1d23-4685-820c-a472475338ea" containerName="revoke-keys"
Sep 10 21:12:18.359774 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:18.359720    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="076e693b-1d23-4685-820c-a472475338ea" containerName="revoke-keys"
Sep 10 21:12:18.359844 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:18.359833    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="076e693b-1d23-4685-820c-a472475338ea" containerName="revoke-keys"
Sep 10 21:12:18.363422 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:18.363399    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7b642ff4-dhsfx"
Sep 10 21:12:18.365673 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:18.365650    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s4qqn\""
Sep 10 21:12:18.365806 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:18.365654    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:12:18.371363 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:18.371288    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7b642ff4-dhsfx"]
Sep 10 21:12:18.450148 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:18.450117    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c1b64fb-11b7-491f-b8bd-72d27cf81d10-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-7b642ff4-dhsfx\" (UID: \"9c1b64fb-11b7-491f-b8bd-72d27cf81d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7b642ff4-dhsfx"
Sep 10 21:12:18.551468 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:18.551432    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c1b64fb-11b7-491f-b8bd-72d27cf81d10-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-7b642ff4-dhsfx\" (UID: \"9c1b64fb-11b7-491f-b8bd-72d27cf81d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7b642ff4-dhsfx"
Sep 10 21:12:18.552109 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:18.552088    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c1b64fb-11b7-491f-b8bd-72d27cf81d10-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-7b642ff4-dhsfx\" (UID: \"9c1b64fb-11b7-491f-b8bd-72d27cf81d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7b642ff4-dhsfx"
Sep 10 21:12:18.677618 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:18.677537    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7b642ff4-dhsfx"
Sep 10 21:12:18.950226 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:18.950203    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7b642ff4-dhsfx"]
Sep 10 21:12:18.952318 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:12:18.952270    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c1b64fb_11b7_491f_b8bd_72d27cf81d10.slice/crio-2bd04bbb4042a5c11683c6d11fd894eb8ef1f404249c36872937e4b95ebb8822 WatchSource:0}: Error finding container 2bd04bbb4042a5c11683c6d11fd894eb8ef1f404249c36872937e4b95ebb8822: Status 404 returned error can't find the container with id 2bd04bbb4042a5c11683c6d11fd894eb8ef1f404249c36872937e4b95ebb8822
Sep 10 21:12:19.854719 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:19.854687    2581 generic.go:356] "Generic (PLEG): container finished" podID="9c1b64fb-11b7-491f-b8bd-72d27cf81d10" containerID="fbb2026db22a45ff4c116f34686be930b2d4e8e8a8fdaf984bbbb9886b651948" exitCode=0
Sep 10 21:12:19.855147 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:19.854776    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7b642ff4-dhsfx" event={"ID":"9c1b64fb-11b7-491f-b8bd-72d27cf81d10","Type":"ContainerDied","Data":"fbb2026db22a45ff4c116f34686be930b2d4e8e8a8fdaf984bbbb9886b651948"}
Sep 10 21:12:19.855147 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:19.854842    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7b642ff4-dhsfx" event={"ID":"9c1b64fb-11b7-491f-b8bd-72d27cf81d10","Type":"ContainerStarted","Data":"2bd04bbb4042a5c11683c6d11fd894eb8ef1f404249c36872937e4b95ebb8822"}
Sep 10 21:12:20.993678 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:20.993653    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7b642ff4-dhsfx"
Sep 10 21:12:21.077794 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:21.077764    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c1b64fb-11b7-491f-b8bd-72d27cf81d10-maas-api-service-ca\") pod \"9c1b64fb-11b7-491f-b8bd-72d27cf81d10\" (UID: \"9c1b64fb-11b7-491f-b8bd-72d27cf81d10\") "
Sep 10 21:12:21.078138 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:21.078114    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c1b64fb-11b7-491f-b8bd-72d27cf81d10-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9c1b64fb-11b7-491f-b8bd-72d27cf81d10" (UID: "9c1b64fb-11b7-491f-b8bd-72d27cf81d10"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:12:21.179415 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:21.179334    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c1b64fb-11b7-491f-b8bd-72d27cf81d10-maas-api-service-ca\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:12:21.599156 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:21.599117    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-63f3d7-64b49bd9c9-k9chg"]
Sep 10 21:12:21.599403 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:21.599371    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-63f3d7-64b49bd9c9-k9chg" podUID="16ce1986-676d-495a-93f0-6ecf0c6061a7" containerName="payload-processing" containerID="cri-o://1d177ef4ef93b1ee80798e0485e6df45ee1f2a4da1937d0758eb644811bbbb24" gracePeriod=30
Sep 10 21:12:21.605818 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:21.605597    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-63f3d7-69c755474-ld7d2"]
Sep 10 21:12:21.606077 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:21.606040    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-63f3d7-69c755474-ld7d2" podUID="ac06edc0-36b0-4be7-ad07-32edf0982ecb" containerName="payload-pre-processing" containerID="cri-o://ae2f5228918978bb14340369deb683908d5f3fc6f798935f311085c103556c1d" gracePeriod=30
Sep 10 21:12:21.765329 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:21.765253    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-63f3d7-fjkdd"]
Sep 10 21:12:21.767630 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:21.767585    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-63f3d7-fjkdd"]
Sep 10 21:12:21.868204 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:21.868171    2581 generic.go:356] "Generic (PLEG): container finished" podID="16ce1986-676d-495a-93f0-6ecf0c6061a7" containerID="1d177ef4ef93b1ee80798e0485e6df45ee1f2a4da1937d0758eb644811bbbb24" exitCode=0
Sep 10 21:12:21.868336 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:21.868219    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-63f3d7-64b49bd9c9-k9chg" event={"ID":"16ce1986-676d-495a-93f0-6ecf0c6061a7","Type":"ContainerDied","Data":"1d177ef4ef93b1ee80798e0485e6df45ee1f2a4da1937d0758eb644811bbbb24"}
Sep 10 21:12:21.868336 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:21.868269    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-63f3d7-64b49bd9c9-k9chg" event={"ID":"16ce1986-676d-495a-93f0-6ecf0c6061a7","Type":"ContainerDied","Data":"50e212a99b75f166b496bc804b6b5b4361c519dd9a06995b720ccf702c7027d6"}
Sep 10 21:12:21.868336 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:21.868318    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50e212a99b75f166b496bc804b6b5b4361c519dd9a06995b720ccf702c7027d6"
Sep 10 21:12:21.869840 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:21.869818    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7b642ff4-dhsfx"
Sep 10 21:12:21.869840 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:21.869830    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7b642ff4-dhsfx" event={"ID":"9c1b64fb-11b7-491f-b8bd-72d27cf81d10","Type":"ContainerDied","Data":"2bd04bbb4042a5c11683c6d11fd894eb8ef1f404249c36872937e4b95ebb8822"}
Sep 10 21:12:21.870008 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:21.869869    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bd04bbb4042a5c11683c6d11fd894eb8ef1f404249c36872937e4b95ebb8822"
Sep 10 21:12:21.872137 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:21.871811    2581 generic.go:356] "Generic (PLEG): container finished" podID="ac06edc0-36b0-4be7-ad07-32edf0982ecb" containerID="ae2f5228918978bb14340369deb683908d5f3fc6f798935f311085c103556c1d" exitCode=0
Sep 10 21:12:21.872137 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:21.871859    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-63f3d7-69c755474-ld7d2" event={"ID":"ac06edc0-36b0-4be7-ad07-32edf0982ecb","Type":"ContainerDied","Data":"ae2f5228918978bb14340369deb683908d5f3fc6f798935f311085c103556c1d"}
Sep 10 21:12:21.875829 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:21.875810    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-63f3d7-64b49bd9c9-k9chg"
Sep 10 21:12:21.882595 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:21.882577    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-63f3d7-69c755474-ld7d2"
Sep 10 21:12:21.986246 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:21.986223    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac06edc0-36b0-4be7-ad07-32edf0982ecb-plugins-config-volume\") pod \"ac06edc0-36b0-4be7-ad07-32edf0982ecb\" (UID: \"ac06edc0-36b0-4be7-ad07-32edf0982ecb\") "
Sep 10 21:12:21.986382 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:21.986290    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16ce1986-676d-495a-93f0-6ecf0c6061a7-plugins-config-volume\") pod \"16ce1986-676d-495a-93f0-6ecf0c6061a7\" (UID: \"16ce1986-676d-495a-93f0-6ecf0c6061a7\") "
Sep 10 21:12:21.986382 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:21.986344    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wx2w6\" (UniqueName: \"kubernetes.io/projected/ac06edc0-36b0-4be7-ad07-32edf0982ecb-kube-api-access-wx2w6\") pod \"ac06edc0-36b0-4be7-ad07-32edf0982ecb\" (UID: \"ac06edc0-36b0-4be7-ad07-32edf0982ecb\") "
Sep 10 21:12:21.986382 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:21.986361    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nj4qv\" (UniqueName: \"kubernetes.io/projected/16ce1986-676d-495a-93f0-6ecf0c6061a7-kube-api-access-nj4qv\") pod \"16ce1986-676d-495a-93f0-6ecf0c6061a7\" (UID: \"16ce1986-676d-495a-93f0-6ecf0c6061a7\") "
Sep 10 21:12:21.986651 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:21.986622    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac06edc0-36b0-4be7-ad07-32edf0982ecb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ac06edc0-36b0-4be7-ad07-32edf0982ecb" (UID: "ac06edc0-36b0-4be7-ad07-32edf0982ecb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:12:21.986754 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:21.986715    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16ce1986-676d-495a-93f0-6ecf0c6061a7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "16ce1986-676d-495a-93f0-6ecf0c6061a7" (UID: "16ce1986-676d-495a-93f0-6ecf0c6061a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:12:21.986754 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:21.986727    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac06edc0-36b0-4be7-ad07-32edf0982ecb-plugins-config-volume\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:12:21.988466 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:21.988443    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16ce1986-676d-495a-93f0-6ecf0c6061a7-kube-api-access-nj4qv" (OuterVolumeSpecName: "kube-api-access-nj4qv") pod "16ce1986-676d-495a-93f0-6ecf0c6061a7" (UID: "16ce1986-676d-495a-93f0-6ecf0c6061a7"). InnerVolumeSpecName "kube-api-access-nj4qv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:12:21.988553 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:21.988469    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac06edc0-36b0-4be7-ad07-32edf0982ecb-kube-api-access-wx2w6" (OuterVolumeSpecName: "kube-api-access-wx2w6") pod "ac06edc0-36b0-4be7-ad07-32edf0982ecb" (UID: "ac06edc0-36b0-4be7-ad07-32edf0982ecb"). InnerVolumeSpecName "kube-api-access-wx2w6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:12:22.087501 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:22.087446    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx2w6\" (UniqueName: \"kubernetes.io/projected/ac06edc0-36b0-4be7-ad07-32edf0982ecb-kube-api-access-wx2w6\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:12:22.087865 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:22.087510    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj4qv\" (UniqueName: \"kubernetes.io/projected/16ce1986-676d-495a-93f0-6ecf0c6061a7-kube-api-access-nj4qv\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:12:22.087865 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:22.087524    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16ce1986-676d-495a-93f0-6ecf0c6061a7-plugins-config-volume\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:12:22.868551 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:22.868516    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="076e693b-1d23-4685-820c-a472475338ea" path="/var/lib/kubelet/pods/076e693b-1d23-4685-820c-a472475338ea/volumes"
Sep 10 21:12:22.880029 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:22.880000    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-63f3d7-69c755474-ld7d2" event={"ID":"ac06edc0-36b0-4be7-ad07-32edf0982ecb","Type":"ContainerDied","Data":"0f93e3a590e9321d6fae93272f14078cfa3cccb62ccadaa3523bb45da1f7f051"}
Sep 10 21:12:22.880144 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:22.880039    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-63f3d7-64b49bd9c9-k9chg"
Sep 10 21:12:22.880144 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:22.880070    2581 scope.go:117] "RemoveContainer" containerID="ae2f5228918978bb14340369deb683908d5f3fc6f798935f311085c103556c1d"
Sep 10 21:12:22.880144 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:22.880038    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-63f3d7-69c755474-ld7d2"
Sep 10 21:12:22.899717 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:22.899694    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-63f3d7-64b49bd9c9-k9chg"]
Sep 10 21:12:22.905638 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:22.905615    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-63f3d7-64b49bd9c9-k9chg"]
Sep 10 21:12:22.917900 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:22.917877    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-63f3d7-69c755474-ld7d2"]
Sep 10 21:12:22.922588 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:22.922567    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-63f3d7-69c755474-ld7d2"]
Sep 10 21:12:24.869410 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:24.869369    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16ce1986-676d-495a-93f0-6ecf0c6061a7" path="/var/lib/kubelet/pods/16ce1986-676d-495a-93f0-6ecf0c6061a7/volumes"
Sep 10 21:12:24.869935 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:24.869908    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac06edc0-36b0-4be7-ad07-32edf0982ecb" path="/var/lib/kubelet/pods/ac06edc0-36b0-4be7-ad07-32edf0982ecb/volumes"
Sep 10 21:12:25.665936 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:25.665695    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-7b642ff4-86bd4cd7c-b77km"]
Sep 10 21:12:25.666745 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:25.666693    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-7b642ff4-86bd4cd7c-b77km" podUID="428a29ec-b26d-4d13-b25d-94c078dde6f2" containerName="payload-processing" containerID="cri-o://fb046a20091e26ddb2160b58e8602d549f366fc2d07fd20d29c1f51428682115" gracePeriod=30
Sep 10 21:12:25.675774 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:25.675748    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7b642ff4-6f57f76bfd-2l4cf"]
Sep 10 21:12:25.676012 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:25.675984    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-7b642ff4-6f57f76bfd-2l4cf" podUID="d1f8009a-e8a5-431b-ad3c-b9f2780d045f" containerName="payload-pre-processing" containerID="cri-o://47bd77a70b1e3ff331fc3af8a7039c1d0d78a696843d2db0afe5959dc2c951ef" gracePeriod=30
Sep 10 21:12:25.821966 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:25.821921    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7b642ff4-dhsfx"]
Sep 10 21:12:25.825090 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:25.825062    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-7b642ff4-dhsfx"]
Sep 10 21:12:25.900998 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:25.900953    2581 generic.go:356] "Generic (PLEG): container finished" podID="428a29ec-b26d-4d13-b25d-94c078dde6f2" containerID="fb046a20091e26ddb2160b58e8602d549f366fc2d07fd20d29c1f51428682115" exitCode=0
Sep 10 21:12:25.901464 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:25.901016    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-7b642ff4-86bd4cd7c-b77km" event={"ID":"428a29ec-b26d-4d13-b25d-94c078dde6f2","Type":"ContainerDied","Data":"fb046a20091e26ddb2160b58e8602d549f366fc2d07fd20d29c1f51428682115"}
Sep 10 21:12:25.903168 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:25.903143    2581 generic.go:356] "Generic (PLEG): container finished" podID="d1f8009a-e8a5-431b-ad3c-b9f2780d045f" containerID="47bd77a70b1e3ff331fc3af8a7039c1d0d78a696843d2db0afe5959dc2c951ef" exitCode=0
Sep 10 21:12:25.903315 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:25.903215    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-7b642ff4-6f57f76bfd-2l4cf" event={"ID":"d1f8009a-e8a5-431b-ad3c-b9f2780d045f","Type":"ContainerDied","Data":"47bd77a70b1e3ff331fc3af8a7039c1d0d78a696843d2db0afe5959dc2c951ef"}
Sep 10 21:12:25.970486 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:25.970450    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-7b642ff4-86bd4cd7c-b77km"
Sep 10 21:12:25.973468 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:25.973449    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-7b642ff4-6f57f76bfd-2l4cf"
Sep 10 21:12:26.023286 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:26.023255    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wzbk\" (UniqueName: \"kubernetes.io/projected/d1f8009a-e8a5-431b-ad3c-b9f2780d045f-kube-api-access-8wzbk\") pod \"d1f8009a-e8a5-431b-ad3c-b9f2780d045f\" (UID: \"d1f8009a-e8a5-431b-ad3c-b9f2780d045f\") "
Sep 10 21:12:26.023525 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:26.023346    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfdpj\" (UniqueName: \"kubernetes.io/projected/428a29ec-b26d-4d13-b25d-94c078dde6f2-kube-api-access-vfdpj\") pod \"428a29ec-b26d-4d13-b25d-94c078dde6f2\" (UID: \"428a29ec-b26d-4d13-b25d-94c078dde6f2\") "
Sep 10 21:12:26.023525 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:26.023508    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1f8009a-e8a5-431b-ad3c-b9f2780d045f-plugins-config-volume\") pod \"d1f8009a-e8a5-431b-ad3c-b9f2780d045f\" (UID: \"d1f8009a-e8a5-431b-ad3c-b9f2780d045f\") "
Sep 10 21:12:26.024495 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:26.023823    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428a29ec-b26d-4d13-b25d-94c078dde6f2-plugins-config-volume\") pod \"428a29ec-b26d-4d13-b25d-94c078dde6f2\" (UID: \"428a29ec-b26d-4d13-b25d-94c078dde6f2\") "
Sep 10 21:12:26.024495 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:26.023919    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1f8009a-e8a5-431b-ad3c-b9f2780d045f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d1f8009a-e8a5-431b-ad3c-b9f2780d045f" (UID: "d1f8009a-e8a5-431b-ad3c-b9f2780d045f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:12:26.024495 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:26.024166    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1f8009a-e8a5-431b-ad3c-b9f2780d045f-plugins-config-volume\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:12:26.024817 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:26.024776    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/428a29ec-b26d-4d13-b25d-94c078dde6f2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "428a29ec-b26d-4d13-b25d-94c078dde6f2" (UID: "428a29ec-b26d-4d13-b25d-94c078dde6f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:12:26.026381 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:26.026352    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1f8009a-e8a5-431b-ad3c-b9f2780d045f-kube-api-access-8wzbk" (OuterVolumeSpecName: "kube-api-access-8wzbk") pod "d1f8009a-e8a5-431b-ad3c-b9f2780d045f" (UID: "d1f8009a-e8a5-431b-ad3c-b9f2780d045f"). InnerVolumeSpecName "kube-api-access-8wzbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:12:26.027118 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:26.027093    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/428a29ec-b26d-4d13-b25d-94c078dde6f2-kube-api-access-vfdpj" (OuterVolumeSpecName: "kube-api-access-vfdpj") pod "428a29ec-b26d-4d13-b25d-94c078dde6f2" (UID: "428a29ec-b26d-4d13-b25d-94c078dde6f2"). InnerVolumeSpecName "kube-api-access-vfdpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:12:26.125552 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:26.125518    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfdpj\" (UniqueName: \"kubernetes.io/projected/428a29ec-b26d-4d13-b25d-94c078dde6f2-kube-api-access-vfdpj\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:12:26.125552 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:26.125553    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428a29ec-b26d-4d13-b25d-94c078dde6f2-plugins-config-volume\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:12:26.125758 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:26.125566    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wzbk\" (UniqueName: \"kubernetes.io/projected/d1f8009a-e8a5-431b-ad3c-b9f2780d045f-kube-api-access-8wzbk\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:12:26.869675 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:26.869637    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c1b64fb-11b7-491f-b8bd-72d27cf81d10" path="/var/lib/kubelet/pods/9c1b64fb-11b7-491f-b8bd-72d27cf81d10/volumes"
Sep 10 21:12:26.910015 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:26.909995    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-7b642ff4-6f57f76bfd-2l4cf"
Sep 10 21:12:26.910428 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:26.910033    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-7b642ff4-6f57f76bfd-2l4cf" event={"ID":"d1f8009a-e8a5-431b-ad3c-b9f2780d045f","Type":"ContainerDied","Data":"46a2a4c6549cdbfa245be6bbb11aac3b4cff5bd7979513a05a3be8d58f81d115"}
Sep 10 21:12:26.910428 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:26.910091    2581 scope.go:117] "RemoveContainer" containerID="47bd77a70b1e3ff331fc3af8a7039c1d0d78a696843d2db0afe5959dc2c951ef"
Sep 10 21:12:26.911497 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:26.911476    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-7b642ff4-86bd4cd7c-b77km" event={"ID":"428a29ec-b26d-4d13-b25d-94c078dde6f2","Type":"ContainerDied","Data":"9410ded346edae99dd108adb6189a02674f7ba0c3f197de77527ff9b7a43988c"}
Sep 10 21:12:26.911557 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:26.911502    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-7b642ff4-86bd4cd7c-b77km"
Sep 10 21:12:26.918823 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:26.918803    2581 scope.go:117] "RemoveContainer" containerID="fb046a20091e26ddb2160b58e8602d549f366fc2d07fd20d29c1f51428682115"
Sep 10 21:12:26.930942 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:26.930916    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7b642ff4-6f57f76bfd-2l4cf"]
Sep 10 21:12:26.934670 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:26.934651    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-7b642ff4-6f57f76bfd-2l4cf"]
Sep 10 21:12:26.944958 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:26.944901    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-7b642ff4-86bd4cd7c-b77km"]
Sep 10 21:12:26.947595 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:26.947575    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-7b642ff4-86bd4cd7c-b77km"]
Sep 10 21:12:28.868258 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:28.868223    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="428a29ec-b26d-4d13-b25d-94c078dde6f2" path="/var/lib/kubelet/pods/428a29ec-b26d-4d13-b25d-94c078dde6f2/volumes"
Sep 10 21:12:28.868701 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:28.868617    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1f8009a-e8a5-431b-ad3c-b9f2780d045f" path="/var/lib/kubelet/pods/d1f8009a-e8a5-431b-ad3c-b9f2780d045f/volumes"
Sep 10 21:12:36.234003 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:36.233967    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-cb845d-6d9fcc9cdb-ft9m8"]
Sep 10 21:12:36.234401 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:36.234373    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1f8009a-e8a5-431b-ad3c-b9f2780d045f" containerName="payload-pre-processing"
Sep 10 21:12:36.234401 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:36.234388    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1f8009a-e8a5-431b-ad3c-b9f2780d045f" containerName="payload-pre-processing"
Sep 10 21:12:36.234401 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:36.234398    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac06edc0-36b0-4be7-ad07-32edf0982ecb" containerName="payload-pre-processing"
Sep 10 21:12:36.234401 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:36.234404    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac06edc0-36b0-4be7-ad07-32edf0982ecb" containerName="payload-pre-processing"
Sep 10 21:12:36.234546 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:36.234418    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16ce1986-676d-495a-93f0-6ecf0c6061a7" containerName="payload-processing"
Sep 10 21:12:36.234546 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:36.234424    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="16ce1986-676d-495a-93f0-6ecf0c6061a7" containerName="payload-processing"
Sep 10 21:12:36.234546 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:36.234435    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c1b64fb-11b7-491f-b8bd-72d27cf81d10" containerName="revoke-keys"
Sep 10 21:12:36.234546 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:36.234441    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c1b64fb-11b7-491f-b8bd-72d27cf81d10" containerName="revoke-keys"
Sep 10 21:12:36.234546 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:36.234449    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="428a29ec-b26d-4d13-b25d-94c078dde6f2" containerName="payload-processing"
Sep 10 21:12:36.234546 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:36.234454    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="428a29ec-b26d-4d13-b25d-94c078dde6f2" containerName="payload-processing"
Sep 10 21:12:36.234546 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:36.234524    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1f8009a-e8a5-431b-ad3c-b9f2780d045f" containerName="payload-pre-processing"
Sep 10 21:12:36.234546 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:36.234533    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c1b64fb-11b7-491f-b8bd-72d27cf81d10" containerName="revoke-keys"
Sep 10 21:12:36.234546 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:36.234541    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="16ce1986-676d-495a-93f0-6ecf0c6061a7" containerName="payload-processing"
Sep 10 21:12:36.234546 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:36.234548    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="428a29ec-b26d-4d13-b25d-94c078dde6f2" containerName="payload-processing"
Sep 10 21:12:36.234888 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:36.234556    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac06edc0-36b0-4be7-ad07-32edf0982ecb" containerName="payload-pre-processing"
Sep 10 21:12:36.238917 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:36.238889    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cb845d-6d9fcc9cdb-ft9m8"
Sep 10 21:12:36.241073 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:36.241049    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-cb845d-dockercfg-shk7s\""
Sep 10 21:12:36.241273 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:36.241259    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-cb845d\""
Sep 10 21:12:36.249342 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:36.249313    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-cb845d-6d9fcc9cdb-ft9m8"]
Sep 10 21:12:36.323447 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:36.323413    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0220e8fa-5a34-426d-a6e6-b074a3e98666-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-cb845d-6d9fcc9cdb-ft9m8\" (UID: \"0220e8fa-5a34-426d-a6e6-b074a3e98666\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-cb845d-6d9fcc9cdb-ft9m8"
Sep 10 21:12:36.323638 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:36.323575    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq62h\" (UniqueName: \"kubernetes.io/projected/0220e8fa-5a34-426d-a6e6-b074a3e98666-kube-api-access-dq62h\") pod \"payload-pre-processing-e2e-explicit-cb845d-6d9fcc9cdb-ft9m8\" (UID: \"0220e8fa-5a34-426d-a6e6-b074a3e98666\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-cb845d-6d9fcc9cdb-ft9m8"
Sep 10 21:12:36.424355 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:36.424313    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0220e8fa-5a34-426d-a6e6-b074a3e98666-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-cb845d-6d9fcc9cdb-ft9m8\" (UID: \"0220e8fa-5a34-426d-a6e6-b074a3e98666\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-cb845d-6d9fcc9cdb-ft9m8"
Sep 10 21:12:36.424535 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:36.424472    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq62h\" (UniqueName: \"kubernetes.io/projected/0220e8fa-5a34-426d-a6e6-b074a3e98666-kube-api-access-dq62h\") pod \"payload-pre-processing-e2e-explicit-cb845d-6d9fcc9cdb-ft9m8\" (UID: \"0220e8fa-5a34-426d-a6e6-b074a3e98666\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-cb845d-6d9fcc9cdb-ft9m8"
Sep 10 21:12:36.425075 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:36.425056    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0220e8fa-5a34-426d-a6e6-b074a3e98666-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-cb845d-6d9fcc9cdb-ft9m8\" (UID: \"0220e8fa-5a34-426d-a6e6-b074a3e98666\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-cb845d-6d9fcc9cdb-ft9m8"
Sep 10 21:12:36.434417 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:36.434383    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq62h\" (UniqueName: \"kubernetes.io/projected/0220e8fa-5a34-426d-a6e6-b074a3e98666-kube-api-access-dq62h\") pod \"payload-pre-processing-e2e-explicit-cb845d-6d9fcc9cdb-ft9m8\" (UID: \"0220e8fa-5a34-426d-a6e6-b074a3e98666\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-cb845d-6d9fcc9cdb-ft9m8"
Sep 10 21:12:36.550896 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:36.550858    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cb845d-6d9fcc9cdb-ft9m8"
Sep 10 21:12:36.830152 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:36.830127    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-cb845d-6d9fcc9cdb-ft9m8"]
Sep 10 21:12:36.832444 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:12:36.832410    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0220e8fa_5a34_426d_a6e6_b074a3e98666.slice/crio-7da797d20b78334b789847a9d06de4258a98ff7dc7c1c90d1f7c7b236bd27db4 WatchSource:0}: Error finding container 7da797d20b78334b789847a9d06de4258a98ff7dc7c1c90d1f7c7b236bd27db4: Status 404 returned error can't find the container with id 7da797d20b78334b789847a9d06de4258a98ff7dc7c1c90d1f7c7b236bd27db4
Sep 10 21:12:36.965884 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:36.965845    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cb845d-6d9fcc9cdb-ft9m8" event={"ID":"0220e8fa-5a34-426d-a6e6-b074a3e98666","Type":"ContainerStarted","Data":"60ca4f1000623591abdff0b4a3e237f91aebf296a075c09f6a50de928f83fabd"}
Sep 10 21:12:36.966059 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:36.965988    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cb845d-6d9fcc9cdb-ft9m8"
Sep 10 21:12:36.966059 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:36.966033    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cb845d-6d9fcc9cdb-ft9m8" event={"ID":"0220e8fa-5a34-426d-a6e6-b074a3e98666","Type":"ContainerStarted","Data":"7da797d20b78334b789847a9d06de4258a98ff7dc7c1c90d1f7c7b236bd27db4"}
Sep 10 21:12:36.984444 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:36.984395    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cb845d-6d9fcc9cdb-ft9m8" podStartSLOduration=0.984375308 podStartE2EDuration="984.375308ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:12:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:12:36.981638171 +0000 UTC m=+874.638546943" watchObservedRunningTime="2026-09-10 21:12:36.984375308 +0000 UTC m=+874.641284079"
Sep 10 21:12:47.973711 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:12:47.973677    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cb845d-6d9fcc9cdb-ft9m8"
Sep 10 21:13:02.814440 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:02.814356    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-z6858_c1e06766-b48c-429e-92f1-196bb083360b/console-operator/1.log"
Sep 10 21:13:02.815636 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:02.815609    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-z6858_c1e06766-b48c-429e-92f1-196bb083360b/console-operator/1.log"
Sep 10 21:13:02.818481 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:02.818454    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brrtd_19d8d7ba-45a1-4013-a935-5670abe12b8a/ovn-acl-logging/0.log"
Sep 10 21:13:02.819658 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:02.819641    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brrtd_19d8d7ba-45a1-4013-a935-5670abe12b8a/ovn-acl-logging/0.log"
Sep 10 21:13:19.416998 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:19.416960    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cb845d-4m8gr"]
Sep 10 21:13:19.420626 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:19.420608    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cb845d-4m8gr"
Sep 10 21:13:19.425255 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:19.425234    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:13:19.425372 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:19.425250    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s4qqn\""
Sep 10 21:13:19.436160 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:19.436136    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cb845d-4m8gr"]
Sep 10 21:13:19.524675 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:19.524639    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e758ce1d-1861-49ba-b9df-05afe37a7abe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-cb845d-4m8gr\" (UID: \"e758ce1d-1861-49ba-b9df-05afe37a7abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cb845d-4m8gr"
Sep 10 21:13:19.625678 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:19.625646    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e758ce1d-1861-49ba-b9df-05afe37a7abe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-cb845d-4m8gr\" (UID: \"e758ce1d-1861-49ba-b9df-05afe37a7abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cb845d-4m8gr"
Sep 10 21:13:19.626257 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:19.626235    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e758ce1d-1861-49ba-b9df-05afe37a7abe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-cb845d-4m8gr\" (UID: \"e758ce1d-1861-49ba-b9df-05afe37a7abe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cb845d-4m8gr"
Sep 10 21:13:19.730953 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:19.730870    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cb845d-4m8gr"
Sep 10 21:13:20.010506 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:20.010429    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cb845d-4m8gr"]
Sep 10 21:13:20.014093 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:13:20.014062    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode758ce1d_1861_49ba_b9df_05afe37a7abe.slice/crio-144a641d5578bc24d35717110c591c2e791617ee5d77a7a52176ca36b050d471 WatchSource:0}: Error finding container 144a641d5578bc24d35717110c591c2e791617ee5d77a7a52176ca36b050d471: Status 404 returned error can't find the container with id 144a641d5578bc24d35717110c591c2e791617ee5d77a7a52176ca36b050d471
Sep 10 21:13:20.196697 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:20.196665    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cb845d-4m8gr" event={"ID":"e758ce1d-1861-49ba-b9df-05afe37a7abe","Type":"ContainerStarted","Data":"144a641d5578bc24d35717110c591c2e791617ee5d77a7a52176ca36b050d471"}
Sep 10 21:13:21.203212 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:21.203178    2581 generic.go:356] "Generic (PLEG): container finished" podID="e758ce1d-1861-49ba-b9df-05afe37a7abe" containerID="17b74350ee6d639fb7cf01c31dbee24d6b918e53d79abaa6f962bf0503488267" exitCode=0
Sep 10 21:13:21.203604 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:21.203236    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cb845d-4m8gr" event={"ID":"e758ce1d-1861-49ba-b9df-05afe37a7abe","Type":"ContainerDied","Data":"17b74350ee6d639fb7cf01c31dbee24d6b918e53d79abaa6f962bf0503488267"}
Sep 10 21:13:22.339252 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:22.339225    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cb845d-4m8gr"
Sep 10 21:13:22.453411 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:22.453380    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e758ce1d-1861-49ba-b9df-05afe37a7abe-maas-api-service-ca\") pod \"e758ce1d-1861-49ba-b9df-05afe37a7abe\" (UID: \"e758ce1d-1861-49ba-b9df-05afe37a7abe\") "
Sep 10 21:13:22.453726 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:22.453706    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e758ce1d-1861-49ba-b9df-05afe37a7abe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e758ce1d-1861-49ba-b9df-05afe37a7abe" (UID: "e758ce1d-1861-49ba-b9df-05afe37a7abe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:13:22.554887 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:22.554857    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e758ce1d-1861-49ba-b9df-05afe37a7abe-maas-api-service-ca\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:13:23.216922 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:23.216847    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cb845d-4m8gr"
Sep 10 21:13:23.216922 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:23.216871    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cb845d-4m8gr" event={"ID":"e758ce1d-1861-49ba-b9df-05afe37a7abe","Type":"ContainerDied","Data":"144a641d5578bc24d35717110c591c2e791617ee5d77a7a52176ca36b050d471"}
Sep 10 21:13:23.216922 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:23.216916    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="144a641d5578bc24d35717110c591c2e791617ee5d77a7a52176ca36b050d471"
Sep 10 21:13:29.939874 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:29.939838    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-cb845d-6d9fcc9cdb-ft9m8"]
Sep 10 21:13:29.940369 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:29.940114    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cb845d-6d9fcc9cdb-ft9m8" podUID="0220e8fa-5a34-426d-a6e6-b074a3e98666" containerName="payload-pre-processing" containerID="cri-o://60ca4f1000623591abdff0b4a3e237f91aebf296a075c09f6a50de928f83fabd" gracePeriod=30
Sep 10 21:13:30.150192 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:30.150161    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cb845d-4m8gr"]
Sep 10 21:13:30.165892 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:30.165852    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cb845d-4m8gr"]
Sep 10 21:13:30.201483 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:30.201414    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cb845d-6d9fcc9cdb-ft9m8"
Sep 10 21:13:30.217806 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:30.217778    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0220e8fa-5a34-426d-a6e6-b074a3e98666-plugins-config-volume\") pod \"0220e8fa-5a34-426d-a6e6-b074a3e98666\" (UID: \"0220e8fa-5a34-426d-a6e6-b074a3e98666\") "
Sep 10 21:13:30.217945 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:30.217869    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dq62h\" (UniqueName: \"kubernetes.io/projected/0220e8fa-5a34-426d-a6e6-b074a3e98666-kube-api-access-dq62h\") pod \"0220e8fa-5a34-426d-a6e6-b074a3e98666\" (UID: \"0220e8fa-5a34-426d-a6e6-b074a3e98666\") "
Sep 10 21:13:30.218125 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:30.218102    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0220e8fa-5a34-426d-a6e6-b074a3e98666-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0220e8fa-5a34-426d-a6e6-b074a3e98666" (UID: "0220e8fa-5a34-426d-a6e6-b074a3e98666"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:13:30.218212 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:30.218179    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0220e8fa-5a34-426d-a6e6-b074a3e98666-plugins-config-volume\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:13:30.219885 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:30.219855    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0220e8fa-5a34-426d-a6e6-b074a3e98666-kube-api-access-dq62h" (OuterVolumeSpecName: "kube-api-access-dq62h") pod "0220e8fa-5a34-426d-a6e6-b074a3e98666" (UID: "0220e8fa-5a34-426d-a6e6-b074a3e98666"). InnerVolumeSpecName "kube-api-access-dq62h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:13:30.252387 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:30.252342    2581 generic.go:356] "Generic (PLEG): container finished" podID="0220e8fa-5a34-426d-a6e6-b074a3e98666" containerID="60ca4f1000623591abdff0b4a3e237f91aebf296a075c09f6a50de928f83fabd" exitCode=0
Sep 10 21:13:30.252539 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:30.252402    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cb845d-6d9fcc9cdb-ft9m8"
Sep 10 21:13:30.252539 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:30.252442    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cb845d-6d9fcc9cdb-ft9m8" event={"ID":"0220e8fa-5a34-426d-a6e6-b074a3e98666","Type":"ContainerDied","Data":"60ca4f1000623591abdff0b4a3e237f91aebf296a075c09f6a50de928f83fabd"}
Sep 10 21:13:30.252539 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:30.252502    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-cb845d-6d9fcc9cdb-ft9m8" event={"ID":"0220e8fa-5a34-426d-a6e6-b074a3e98666","Type":"ContainerDied","Data":"7da797d20b78334b789847a9d06de4258a98ff7dc7c1c90d1f7c7b236bd27db4"}
Sep 10 21:13:30.252539 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:30.252530    2581 scope.go:117] "RemoveContainer" containerID="60ca4f1000623591abdff0b4a3e237f91aebf296a075c09f6a50de928f83fabd"
Sep 10 21:13:30.262003 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:30.261983    2581 scope.go:117] "RemoveContainer" containerID="60ca4f1000623591abdff0b4a3e237f91aebf296a075c09f6a50de928f83fabd"
Sep 10 21:13:30.262269 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:13:30.262251    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60ca4f1000623591abdff0b4a3e237f91aebf296a075c09f6a50de928f83fabd\": container with ID starting with 60ca4f1000623591abdff0b4a3e237f91aebf296a075c09f6a50de928f83fabd not found: ID does not exist" containerID="60ca4f1000623591abdff0b4a3e237f91aebf296a075c09f6a50de928f83fabd"
Sep 10 21:13:30.262351 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:30.262278    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60ca4f1000623591abdff0b4a3e237f91aebf296a075c09f6a50de928f83fabd"} err="failed to get container status \"60ca4f1000623591abdff0b4a3e237f91aebf296a075c09f6a50de928f83fabd\": rpc error: code = NotFound desc = could not find container \"60ca4f1000623591abdff0b4a3e237f91aebf296a075c09f6a50de928f83fabd\": container with ID starting with 60ca4f1000623591abdff0b4a3e237f91aebf296a075c09f6a50de928f83fabd not found: ID does not exist"
Sep 10 21:13:30.319244 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:30.319210    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq62h\" (UniqueName: \"kubernetes.io/projected/0220e8fa-5a34-426d-a6e6-b074a3e98666-kube-api-access-dq62h\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:13:30.323337 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:30.323285    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-cb845d-6d9fcc9cdb-ft9m8"]
Sep 10 21:13:30.327869 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:30.327848    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-cb845d-6d9fcc9cdb-ft9m8"]
Sep 10 21:13:30.868361 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:30.868321    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0220e8fa-5a34-426d-a6e6-b074a3e98666" path="/var/lib/kubelet/pods/0220e8fa-5a34-426d-a6e6-b074a3e98666/volumes"
Sep 10 21:13:30.868764 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:30.868744    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e758ce1d-1861-49ba-b9df-05afe37a7abe" path="/var/lib/kubelet/pods/e758ce1d-1861-49ba-b9df-05afe37a7abe/volumes"
Sep 10 21:13:50.548027 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:13:50.547994    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-47f62f/test-model-47f62f-kserve-6c97f6848c-hqjcq"]
Sep 10 21:14:03.407658 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:03.407623    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-47f62f-66fbbbb67-hzspx"]
Sep 10 21:14:03.408461 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:03.408404    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-47f62f-66fbbbb67-hzspx" podUID="aec5e2ad-edad-4725-86e4-85ba96a70cbc" containerName="payload-processing" containerID="cri-o://638f670c664674c43a6c840eac96e92dec76295e9a3af0323928d207bc7c8fa7" gracePeriod=30
Sep 10 21:14:03.422146 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:03.422106    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-47f62f-7656d8ffdb-xht9v"]
Sep 10 21:14:03.422495 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:03.422317    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-47f62f-7656d8ffdb-xht9v" podUID="72088031-9f18-43ae-b436-0940d32a5492" containerName="payload-pre-processing" containerID="cri-o://af71ec2e051504f7902d8921a0e34c7edb14a1b98ff580e863d0b6cadcf1628a" gracePeriod=30
Sep 10 21:14:03.679590 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:03.679566    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-47f62f-66fbbbb67-hzspx"
Sep 10 21:14:03.717588 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:03.717511    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-47f62f-7656d8ffdb-xht9v"
Sep 10 21:14:03.829826 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:03.829799    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aec5e2ad-edad-4725-86e4-85ba96a70cbc-plugins-config-volume\") pod \"aec5e2ad-edad-4725-86e4-85ba96a70cbc\" (UID: \"aec5e2ad-edad-4725-86e4-85ba96a70cbc\") "
Sep 10 21:14:03.830010 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:03.829837    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpqzq\" (UniqueName: \"kubernetes.io/projected/72088031-9f18-43ae-b436-0940d32a5492-kube-api-access-hpqzq\") pod \"72088031-9f18-43ae-b436-0940d32a5492\" (UID: \"72088031-9f18-43ae-b436-0940d32a5492\") "
Sep 10 21:14:03.830010 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:03.829887    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dr9h7\" (UniqueName: \"kubernetes.io/projected/aec5e2ad-edad-4725-86e4-85ba96a70cbc-kube-api-access-dr9h7\") pod \"aec5e2ad-edad-4725-86e4-85ba96a70cbc\" (UID: \"aec5e2ad-edad-4725-86e4-85ba96a70cbc\") "
Sep 10 21:14:03.830145 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:03.830063    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72088031-9f18-43ae-b436-0940d32a5492-plugins-config-volume\") pod \"72088031-9f18-43ae-b436-0940d32a5492\" (UID: \"72088031-9f18-43ae-b436-0940d32a5492\") "
Sep 10 21:14:03.830205 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:03.830175    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aec5e2ad-edad-4725-86e4-85ba96a70cbc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "aec5e2ad-edad-4725-86e4-85ba96a70cbc" (UID: "aec5e2ad-edad-4725-86e4-85ba96a70cbc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:14:03.830473 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:03.830443    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aec5e2ad-edad-4725-86e4-85ba96a70cbc-plugins-config-volume\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:14:03.830473 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:03.830453    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72088031-9f18-43ae-b436-0940d32a5492-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "72088031-9f18-43ae-b436-0940d32a5492" (UID: "72088031-9f18-43ae-b436-0940d32a5492"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:14:03.831975 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:03.831952    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72088031-9f18-43ae-b436-0940d32a5492-kube-api-access-hpqzq" (OuterVolumeSpecName: "kube-api-access-hpqzq") pod "72088031-9f18-43ae-b436-0940d32a5492" (UID: "72088031-9f18-43ae-b436-0940d32a5492"). InnerVolumeSpecName "kube-api-access-hpqzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:14:03.832039 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:03.831988    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aec5e2ad-edad-4725-86e4-85ba96a70cbc-kube-api-access-dr9h7" (OuterVolumeSpecName: "kube-api-access-dr9h7") pod "aec5e2ad-edad-4725-86e4-85ba96a70cbc" (UID: "aec5e2ad-edad-4725-86e4-85ba96a70cbc"). InnerVolumeSpecName "kube-api-access-dr9h7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:14:03.931028 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:03.930957    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpqzq\" (UniqueName: \"kubernetes.io/projected/72088031-9f18-43ae-b436-0940d32a5492-kube-api-access-hpqzq\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:14:03.931028 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:03.930983    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr9h7\" (UniqueName: \"kubernetes.io/projected/aec5e2ad-edad-4725-86e4-85ba96a70cbc-kube-api-access-dr9h7\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:14:03.931028 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:03.930994    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72088031-9f18-43ae-b436-0940d32a5492-plugins-config-volume\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:14:04.436504 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:04.436474    2581 generic.go:356] "Generic (PLEG): container finished" podID="72088031-9f18-43ae-b436-0940d32a5492" containerID="af71ec2e051504f7902d8921a0e34c7edb14a1b98ff580e863d0b6cadcf1628a" exitCode=0
Sep 10 21:14:04.436922 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:04.436520    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-47f62f-7656d8ffdb-xht9v"
Sep 10 21:14:04.436922 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:04.436559    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-47f62f-7656d8ffdb-xht9v" event={"ID":"72088031-9f18-43ae-b436-0940d32a5492","Type":"ContainerDied","Data":"af71ec2e051504f7902d8921a0e34c7edb14a1b98ff580e863d0b6cadcf1628a"}
Sep 10 21:14:04.436922 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:04.436614    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-47f62f-7656d8ffdb-xht9v" event={"ID":"72088031-9f18-43ae-b436-0940d32a5492","Type":"ContainerDied","Data":"0c0e19d0f1f8ab2ed43af1ffbcebfaa8dd3844c06c116fa90f3e1d2bdcd65ba8"}
Sep 10 21:14:04.436922 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:04.436635    2581 scope.go:117] "RemoveContainer" containerID="af71ec2e051504f7902d8921a0e34c7edb14a1b98ff580e863d0b6cadcf1628a"
Sep 10 21:14:04.437951 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:04.437926    2581 generic.go:356] "Generic (PLEG): container finished" podID="aec5e2ad-edad-4725-86e4-85ba96a70cbc" containerID="638f670c664674c43a6c840eac96e92dec76295e9a3af0323928d207bc7c8fa7" exitCode=0
Sep 10 21:14:04.438041 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:04.438026    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-47f62f-66fbbbb67-hzspx" event={"ID":"aec5e2ad-edad-4725-86e4-85ba96a70cbc","Type":"ContainerDied","Data":"638f670c664674c43a6c840eac96e92dec76295e9a3af0323928d207bc7c8fa7"}
Sep 10 21:14:04.438041 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:04.438035    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-47f62f-66fbbbb67-hzspx"
Sep 10 21:14:04.438128 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:04.438060    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-47f62f-66fbbbb67-hzspx" event={"ID":"aec5e2ad-edad-4725-86e4-85ba96a70cbc","Type":"ContainerDied","Data":"21b503b4b31eb257bf3658a5959864ec8e11a2f141c33389eb6be47ea9279da7"}
Sep 10 21:14:04.449524 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:04.449502    2581 scope.go:117] "RemoveContainer" containerID="af71ec2e051504f7902d8921a0e34c7edb14a1b98ff580e863d0b6cadcf1628a"
Sep 10 21:14:04.449915 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:14:04.449828    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af71ec2e051504f7902d8921a0e34c7edb14a1b98ff580e863d0b6cadcf1628a\": container with ID starting with af71ec2e051504f7902d8921a0e34c7edb14a1b98ff580e863d0b6cadcf1628a not found: ID does not exist" containerID="af71ec2e051504f7902d8921a0e34c7edb14a1b98ff580e863d0b6cadcf1628a"
Sep 10 21:14:04.449915 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:04.449859    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af71ec2e051504f7902d8921a0e34c7edb14a1b98ff580e863d0b6cadcf1628a"} err="failed to get container status \"af71ec2e051504f7902d8921a0e34c7edb14a1b98ff580e863d0b6cadcf1628a\": rpc error: code = NotFound desc = could not find container \"af71ec2e051504f7902d8921a0e34c7edb14a1b98ff580e863d0b6cadcf1628a\": container with ID starting with af71ec2e051504f7902d8921a0e34c7edb14a1b98ff580e863d0b6cadcf1628a not found: ID does not exist"
Sep 10 21:14:04.449915 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:04.449885    2581 scope.go:117] "RemoveContainer" containerID="638f670c664674c43a6c840eac96e92dec76295e9a3af0323928d207bc7c8fa7"
Sep 10 21:14:04.458372 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:04.458351    2581 scope.go:117] "RemoveContainer" containerID="638f670c664674c43a6c840eac96e92dec76295e9a3af0323928d207bc7c8fa7"
Sep 10 21:14:04.458623 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:14:04.458604    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"638f670c664674c43a6c840eac96e92dec76295e9a3af0323928d207bc7c8fa7\": container with ID starting with 638f670c664674c43a6c840eac96e92dec76295e9a3af0323928d207bc7c8fa7 not found: ID does not exist" containerID="638f670c664674c43a6c840eac96e92dec76295e9a3af0323928d207bc7c8fa7"
Sep 10 21:14:04.458669 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:04.458630    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"638f670c664674c43a6c840eac96e92dec76295e9a3af0323928d207bc7c8fa7"} err="failed to get container status \"638f670c664674c43a6c840eac96e92dec76295e9a3af0323928d207bc7c8fa7\": rpc error: code = NotFound desc = could not find container \"638f670c664674c43a6c840eac96e92dec76295e9a3af0323928d207bc7c8fa7\": container with ID starting with 638f670c664674c43a6c840eac96e92dec76295e9a3af0323928d207bc7c8fa7 not found: ID does not exist"
Sep 10 21:14:04.463168 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:04.463146    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-47f62f-66fbbbb67-hzspx"]
Sep 10 21:14:04.467606 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:04.467586    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-47f62f-66fbbbb67-hzspx"]
Sep 10 21:14:04.478538 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:04.478505    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-47f62f-7656d8ffdb-xht9v"]
Sep 10 21:14:04.483533 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:04.483512    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-47f62f-7656d8ffdb-xht9v"]
Sep 10 21:14:04.868077 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:04.868044    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72088031-9f18-43ae-b436-0940d32a5492" path="/var/lib/kubelet/pods/72088031-9f18-43ae-b436-0940d32a5492/volumes"
Sep 10 21:14:04.868473 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:04.868458    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aec5e2ad-edad-4725-86e4-85ba96a70cbc" path="/var/lib/kubelet/pods/aec5e2ad-edad-4725-86e4-85ba96a70cbc/volumes"
Sep 10 21:14:05.567571 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:05.567513    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-47f62f/test-model-47f62f-kserve-6c97f6848c-hqjcq" podUID="e9aca7af-30c4-423a-8cf4-29682d49b618" containerName="main" containerID="cri-o://ba023f87cca87f63f3775eef69af3bcd9ae4f74a959905a00bbbf6cac472aeb4" gracePeriod=45
Sep 10 21:14:05.817162 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:05.817137    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-47f62f/test-model-47f62f-kserve-6c97f6848c-hqjcq"
Sep 10 21:14:05.949880 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:05.949793    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kg2jh\" (UniqueName: \"kubernetes.io/projected/e9aca7af-30c4-423a-8cf4-29682d49b618-kube-api-access-kg2jh\") pod \"e9aca7af-30c4-423a-8cf4-29682d49b618\" (UID: \"e9aca7af-30c4-423a-8cf4-29682d49b618\") "
Sep 10 21:14:05.950032 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:05.949921    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9aca7af-30c4-423a-8cf4-29682d49b618-dshm\") pod \"e9aca7af-30c4-423a-8cf4-29682d49b618\" (UID: \"e9aca7af-30c4-423a-8cf4-29682d49b618\") "
Sep 10 21:14:05.950032 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:05.949954    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9aca7af-30c4-423a-8cf4-29682d49b618-model-cache\") pod \"e9aca7af-30c4-423a-8cf4-29682d49b618\" (UID: \"e9aca7af-30c4-423a-8cf4-29682d49b618\") "
Sep 10 21:14:05.950032 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:05.949983    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9aca7af-30c4-423a-8cf4-29682d49b618-tls-certs\") pod \"e9aca7af-30c4-423a-8cf4-29682d49b618\" (UID: \"e9aca7af-30c4-423a-8cf4-29682d49b618\") "
Sep 10 21:14:05.950207 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:05.950060    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9aca7af-30c4-423a-8cf4-29682d49b618-home\") pod \"e9aca7af-30c4-423a-8cf4-29682d49b618\" (UID: \"e9aca7af-30c4-423a-8cf4-29682d49b618\") "
Sep 10 21:14:05.950207 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:05.950113    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9aca7af-30c4-423a-8cf4-29682d49b618-tmp-dir\") pod \"e9aca7af-30c4-423a-8cf4-29682d49b618\" (UID: \"e9aca7af-30c4-423a-8cf4-29682d49b618\") "
Sep 10 21:14:05.950333 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:05.950225    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9aca7af-30c4-423a-8cf4-29682d49b618-model-cache" (OuterVolumeSpecName: "model-cache") pod "e9aca7af-30c4-423a-8cf4-29682d49b618" (UID: "e9aca7af-30c4-423a-8cf4-29682d49b618"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:14:05.950400 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:05.950337    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9aca7af-30c4-423a-8cf4-29682d49b618-home" (OuterVolumeSpecName: "home") pod "e9aca7af-30c4-423a-8cf4-29682d49b618" (UID: "e9aca7af-30c4-423a-8cf4-29682d49b618"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:14:05.950451 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:05.950430    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9aca7af-30c4-423a-8cf4-29682d49b618-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e9aca7af-30c4-423a-8cf4-29682d49b618" (UID: "e9aca7af-30c4-423a-8cf4-29682d49b618"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:14:05.950821 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:05.950650    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9aca7af-30c4-423a-8cf4-29682d49b618-model-cache\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:14:05.950821 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:05.950674    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9aca7af-30c4-423a-8cf4-29682d49b618-home\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:14:05.950821 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:05.950688    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9aca7af-30c4-423a-8cf4-29682d49b618-tmp-dir\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:14:05.952348 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:05.952322    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9aca7af-30c4-423a-8cf4-29682d49b618-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e9aca7af-30c4-423a-8cf4-29682d49b618" (UID: "e9aca7af-30c4-423a-8cf4-29682d49b618"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:14:05.952500 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:05.952482    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9aca7af-30c4-423a-8cf4-29682d49b618-dshm" (OuterVolumeSpecName: "dshm") pod "e9aca7af-30c4-423a-8cf4-29682d49b618" (UID: "e9aca7af-30c4-423a-8cf4-29682d49b618"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:14:05.952500 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:05.952490    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9aca7af-30c4-423a-8cf4-29682d49b618-kube-api-access-kg2jh" (OuterVolumeSpecName: "kube-api-access-kg2jh") pod "e9aca7af-30c4-423a-8cf4-29682d49b618" (UID: "e9aca7af-30c4-423a-8cf4-29682d49b618"). InnerVolumeSpecName "kube-api-access-kg2jh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:14:06.051958 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:06.051908    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9aca7af-30c4-423a-8cf4-29682d49b618-dshm\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:14:06.051958 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:06.051950    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9aca7af-30c4-423a-8cf4-29682d49b618-tls-certs\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:14:06.051958 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:06.051960    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg2jh\" (UniqueName: \"kubernetes.io/projected/e9aca7af-30c4-423a-8cf4-29682d49b618-kube-api-access-kg2jh\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:14:06.451915 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:06.451874    2581 generic.go:356] "Generic (PLEG): container finished" podID="e9aca7af-30c4-423a-8cf4-29682d49b618" containerID="ba023f87cca87f63f3775eef69af3bcd9ae4f74a959905a00bbbf6cac472aeb4" exitCode=0
Sep 10 21:14:06.452093 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:06.451924    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-47f62f/test-model-47f62f-kserve-6c97f6848c-hqjcq"
Sep 10 21:14:06.452093 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:06.451968    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-47f62f/test-model-47f62f-kserve-6c97f6848c-hqjcq" event={"ID":"e9aca7af-30c4-423a-8cf4-29682d49b618","Type":"ContainerDied","Data":"ba023f87cca87f63f3775eef69af3bcd9ae4f74a959905a00bbbf6cac472aeb4"}
Sep 10 21:14:06.452093 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:06.451998    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-47f62f/test-model-47f62f-kserve-6c97f6848c-hqjcq" event={"ID":"e9aca7af-30c4-423a-8cf4-29682d49b618","Type":"ContainerDied","Data":"184a75a216284f4d87aa9d34e2be71b056ece8b1fc57095397b280d8838ebd9c"}
Sep 10 21:14:06.452093 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:06.452015    2581 scope.go:117] "RemoveContainer" containerID="ba023f87cca87f63f3775eef69af3bcd9ae4f74a959905a00bbbf6cac472aeb4"
Sep 10 21:14:06.461276 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:06.461256    2581 scope.go:117] "RemoveContainer" containerID="ba023f87cca87f63f3775eef69af3bcd9ae4f74a959905a00bbbf6cac472aeb4"
Sep 10 21:14:06.461587 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:14:06.461566    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba023f87cca87f63f3775eef69af3bcd9ae4f74a959905a00bbbf6cac472aeb4\": container with ID starting with ba023f87cca87f63f3775eef69af3bcd9ae4f74a959905a00bbbf6cac472aeb4 not found: ID does not exist" containerID="ba023f87cca87f63f3775eef69af3bcd9ae4f74a959905a00bbbf6cac472aeb4"
Sep 10 21:14:06.461666 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:06.461599    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba023f87cca87f63f3775eef69af3bcd9ae4f74a959905a00bbbf6cac472aeb4"} err="failed to get container status \"ba023f87cca87f63f3775eef69af3bcd9ae4f74a959905a00bbbf6cac472aeb4\": rpc error: code = NotFound desc = could not find container \"ba023f87cca87f63f3775eef69af3bcd9ae4f74a959905a00bbbf6cac472aeb4\": container with ID starting with ba023f87cca87f63f3775eef69af3bcd9ae4f74a959905a00bbbf6cac472aeb4 not found: ID does not exist"
Sep 10 21:14:06.473505 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:06.473482    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-47f62f/test-model-47f62f-kserve-6c97f6848c-hqjcq"]
Sep 10 21:14:06.477077 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:06.477057    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-47f62f/test-model-47f62f-kserve-6c97f6848c-hqjcq"]
Sep 10 21:14:06.871173 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:06.871131    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9aca7af-30c4-423a-8cf4-29682d49b618" path="/var/lib/kubelet/pods/e9aca7af-30c4-423a-8cf4-29682d49b618/volumes"
Sep 10 21:14:06.971092 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:06.971061    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"]
Sep 10 21:14:06.971447 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:06.971435    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e758ce1d-1861-49ba-b9df-05afe37a7abe" containerName="revoke-keys"
Sep 10 21:14:06.971447 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:06.971449    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e758ce1d-1861-49ba-b9df-05afe37a7abe" containerName="revoke-keys"
Sep 10 21:14:06.971540 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:06.971467    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9aca7af-30c4-423a-8cf4-29682d49b618" containerName="main"
Sep 10 21:14:06.971540 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:06.971473    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9aca7af-30c4-423a-8cf4-29682d49b618" containerName="main"
Sep 10 21:14:06.971540 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:06.971480    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0220e8fa-5a34-426d-a6e6-b074a3e98666" containerName="payload-pre-processing"
Sep 10 21:14:06.971540 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:06.971487    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0220e8fa-5a34-426d-a6e6-b074a3e98666" containerName="payload-pre-processing"
Sep 10 21:14:06.971540 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:06.971495    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72088031-9f18-43ae-b436-0940d32a5492" containerName="payload-pre-processing"
Sep 10 21:14:06.971540 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:06.971500    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="72088031-9f18-43ae-b436-0940d32a5492" containerName="payload-pre-processing"
Sep 10 21:14:06.971540 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:06.971513    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aec5e2ad-edad-4725-86e4-85ba96a70cbc" containerName="payload-processing"
Sep 10 21:14:06.971540 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:06.971518    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="aec5e2ad-edad-4725-86e4-85ba96a70cbc" containerName="payload-processing"
Sep 10 21:14:06.971782 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:06.971566    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="72088031-9f18-43ae-b436-0940d32a5492" containerName="payload-pre-processing"
Sep 10 21:14:06.971782 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:06.971575    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e758ce1d-1861-49ba-b9df-05afe37a7abe" containerName="revoke-keys"
Sep 10 21:14:06.971782 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:06.971582    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9aca7af-30c4-423a-8cf4-29682d49b618" containerName="main"
Sep 10 21:14:06.971782 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:06.971590    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0220e8fa-5a34-426d-a6e6-b074a3e98666" containerName="payload-pre-processing"
Sep 10 21:14:06.971782 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:06.971596    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="aec5e2ad-edad-4725-86e4-85ba96a70cbc" containerName="payload-processing"
Sep 10 21:14:06.975846 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:06.975820    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"
Sep 10 21:14:06.978354 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:06.978329    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-db4a68ea-openshift-default-dockercfg-hh6k2\""
Sep 10 21:14:06.990665 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:06.990642    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"]
Sep 10 21:14:07.060769 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:07.060731    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8c2e3e4-83b3-44f2-857b-331077d03271-istio-podinfo\") pod \"e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj\" (UID: \"f8c2e3e4-83b3-44f2-857b-331077d03271\") " pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"
Sep 10 21:14:07.060769 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:07.060770    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbql4\" (UniqueName: \"kubernetes.io/projected/f8c2e3e4-83b3-44f2-857b-331077d03271-kube-api-access-nbql4\") pod \"e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj\" (UID: \"f8c2e3e4-83b3-44f2-857b-331077d03271\") " pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"
Sep 10 21:14:07.060971 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:07.060792    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c2e3e4-83b3-44f2-857b-331077d03271-istio-data\") pod \"e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj\" (UID: \"f8c2e3e4-83b3-44f2-857b-331077d03271\") " pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"
Sep 10 21:14:07.060971 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:07.060897    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c2e3e4-83b3-44f2-857b-331077d03271-credential-socket\") pod \"e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj\" (UID: \"f8c2e3e4-83b3-44f2-857b-331077d03271\") " pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"
Sep 10 21:14:07.060971 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:07.060949    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c2e3e4-83b3-44f2-857b-331077d03271-workload-socket\") pod \"e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj\" (UID: \"f8c2e3e4-83b3-44f2-857b-331077d03271\") " pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"
Sep 10 21:14:07.060971 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:07.060965    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8c2e3e4-83b3-44f2-857b-331077d03271-istiod-ca-cert\") pod \"e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj\" (UID: \"f8c2e3e4-83b3-44f2-857b-331077d03271\") " pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"
Sep 10 21:14:07.061100 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:07.060982    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8c2e3e4-83b3-44f2-857b-331077d03271-istio-token\") pod \"e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj\" (UID: \"f8c2e3e4-83b3-44f2-857b-331077d03271\") " pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"
Sep 10 21:14:07.061100 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:07.061001    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8c2e3e4-83b3-44f2-857b-331077d03271-workload-certs\") pod \"e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj\" (UID: \"f8c2e3e4-83b3-44f2-857b-331077d03271\") " pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"
Sep 10 21:14:07.061100 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:07.061074    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8c2e3e4-83b3-44f2-857b-331077d03271-istio-envoy\") pod \"e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj\" (UID: \"f8c2e3e4-83b3-44f2-857b-331077d03271\") " pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"
Sep 10 21:14:07.162665 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:07.162583    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8c2e3e4-83b3-44f2-857b-331077d03271-istio-envoy\") pod \"e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj\" (UID: \"f8c2e3e4-83b3-44f2-857b-331077d03271\") " pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"
Sep 10 21:14:07.162665 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:07.162645    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8c2e3e4-83b3-44f2-857b-331077d03271-istio-podinfo\") pod \"e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj\" (UID: \"f8c2e3e4-83b3-44f2-857b-331077d03271\") " pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"
Sep 10 21:14:07.162665 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:07.162667    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbql4\" (UniqueName: \"kubernetes.io/projected/f8c2e3e4-83b3-44f2-857b-331077d03271-kube-api-access-nbql4\") pod \"e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj\" (UID: \"f8c2e3e4-83b3-44f2-857b-331077d03271\") " pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"
Sep 10 21:14:07.162946 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:07.162685    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c2e3e4-83b3-44f2-857b-331077d03271-istio-data\") pod \"e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj\" (UID: \"f8c2e3e4-83b3-44f2-857b-331077d03271\") " pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"
Sep 10 21:14:07.162946 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:07.162744    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c2e3e4-83b3-44f2-857b-331077d03271-credential-socket\") pod \"e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj\" (UID: \"f8c2e3e4-83b3-44f2-857b-331077d03271\") " pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"
Sep 10 21:14:07.162946 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:07.162794    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c2e3e4-83b3-44f2-857b-331077d03271-workload-socket\") pod \"e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj\" (UID: \"f8c2e3e4-83b3-44f2-857b-331077d03271\") " pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"
Sep 10 21:14:07.162946 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:07.162817    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8c2e3e4-83b3-44f2-857b-331077d03271-istiod-ca-cert\") pod \"e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj\" (UID: \"f8c2e3e4-83b3-44f2-857b-331077d03271\") " pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"
Sep 10 21:14:07.162946 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:07.162843    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8c2e3e4-83b3-44f2-857b-331077d03271-istio-token\") pod \"e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj\" (UID: \"f8c2e3e4-83b3-44f2-857b-331077d03271\") " pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"
Sep 10 21:14:07.162946 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:07.162869    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8c2e3e4-83b3-44f2-857b-331077d03271-workload-certs\") pod \"e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj\" (UID: \"f8c2e3e4-83b3-44f2-857b-331077d03271\") " pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"
Sep 10 21:14:07.163242 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:07.163123    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c2e3e4-83b3-44f2-857b-331077d03271-istio-data\") pod \"e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj\" (UID: \"f8c2e3e4-83b3-44f2-857b-331077d03271\") " pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"
Sep 10 21:14:07.163242 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:07.163205    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c2e3e4-83b3-44f2-857b-331077d03271-workload-socket\") pod \"e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj\" (UID: \"f8c2e3e4-83b3-44f2-857b-331077d03271\") " pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"
Sep 10 21:14:07.163363 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:07.163275    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8c2e3e4-83b3-44f2-857b-331077d03271-workload-certs\") pod \"e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj\" (UID: \"f8c2e3e4-83b3-44f2-857b-331077d03271\") " pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"
Sep 10 21:14:07.163363 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:07.163345    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c2e3e4-83b3-44f2-857b-331077d03271-credential-socket\") pod \"e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj\" (UID: \"f8c2e3e4-83b3-44f2-857b-331077d03271\") " pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"
Sep 10 21:14:07.163620 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:07.163603    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8c2e3e4-83b3-44f2-857b-331077d03271-istiod-ca-cert\") pod \"e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj\" (UID: \"f8c2e3e4-83b3-44f2-857b-331077d03271\") " pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"
Sep 10 21:14:07.164920 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:07.164901    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8c2e3e4-83b3-44f2-857b-331077d03271-istio-envoy\") pod \"e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj\" (UID: \"f8c2e3e4-83b3-44f2-857b-331077d03271\") " pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"
Sep 10 21:14:07.165287 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:07.165254    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8c2e3e4-83b3-44f2-857b-331077d03271-istio-podinfo\") pod \"e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj\" (UID: \"f8c2e3e4-83b3-44f2-857b-331077d03271\") " pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"
Sep 10 21:14:07.171433 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:07.171409    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8c2e3e4-83b3-44f2-857b-331077d03271-istio-token\") pod \"e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj\" (UID: \"f8c2e3e4-83b3-44f2-857b-331077d03271\") " pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"
Sep 10 21:14:07.171533 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:07.171518    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbql4\" (UniqueName: \"kubernetes.io/projected/f8c2e3e4-83b3-44f2-857b-331077d03271-kube-api-access-nbql4\") pod \"e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj\" (UID: \"f8c2e3e4-83b3-44f2-857b-331077d03271\") " pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"
Sep 10 21:14:07.287092 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:07.287043    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"
Sep 10 21:14:07.566952 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:07.566924    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"]
Sep 10 21:14:07.568548 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:14:07.568518    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8c2e3e4_83b3_44f2_857b_331077d03271.slice/crio-a9cb718df7f8c40e7a858f100c4ba035901fa774475c8ed8c55e89b1e091ae48 WatchSource:0}: Error finding container a9cb718df7f8c40e7a858f100c4ba035901fa774475c8ed8c55e89b1e091ae48: Status 404 returned error can't find the container with id a9cb718df7f8c40e7a858f100c4ba035901fa774475c8ed8c55e89b1e091ae48
Sep 10 21:14:07.570742 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:07.570712    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 21:14:07.570833 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:07.570780    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 21:14:07.570833 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:07.570814    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 21:14:08.464889 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:08.464856    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj" event={"ID":"f8c2e3e4-83b3-44f2-857b-331077d03271","Type":"ContainerStarted","Data":"cd8d57dee9bf2cb98b85b3dcaa952db8969156e34000240afc7ef30865650434"}
Sep 10 21:14:08.465279 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:08.464905    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj" event={"ID":"f8c2e3e4-83b3-44f2-857b-331077d03271","Type":"ContainerStarted","Data":"a9cb718df7f8c40e7a858f100c4ba035901fa774475c8ed8c55e89b1e091ae48"}
Sep 10 21:14:08.487480 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:08.487438    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj" podStartSLOduration=2.487422872 podStartE2EDuration="2.487422872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:14:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:14:08.485958277 +0000 UTC m=+966.142867062" watchObservedRunningTime="2026-09-10 21:14:08.487422872 +0000 UTC m=+966.144331644"
Sep 10 21:14:09.288185 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:09.288146    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"
Sep 10 21:14:09.293203 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:09.293178    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"
Sep 10 21:14:09.470087 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:09.470052    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"
Sep 10 21:14:09.470896 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:09.470879    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"
Sep 10 21:14:12.422252 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:12.422212    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-db4a68ea-84597fc849-4nrtm"]
Sep 10 21:14:12.425772 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:12.425748    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-db4a68ea-84597fc849-4nrtm"
Sep 10 21:14:12.427914 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:12.427862    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-db4a68ea\""
Sep 10 21:14:12.428051 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:12.427962    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-db4a68ea-dockercfg-lpf2s\""
Sep 10 21:14:12.436923 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:12.436901    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-db4a68ea-84597fc849-4nrtm"]
Sep 10 21:14:12.611510 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:12.611474    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phq45\" (UniqueName: \"kubernetes.io/projected/480bae13-46c1-4671-a53b-ce4cbb4811a6-kube-api-access-phq45\") pod \"payload-processing-e2e-mt-db4a68ea-84597fc849-4nrtm\" (UID: \"480bae13-46c1-4671-a53b-ce4cbb4811a6\") " pod="openshift-ingress/payload-processing-e2e-mt-db4a68ea-84597fc849-4nrtm"
Sep 10 21:14:12.611679 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:12.611534    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/480bae13-46c1-4671-a53b-ce4cbb4811a6-plugins-config-volume\") pod \"payload-processing-e2e-mt-db4a68ea-84597fc849-4nrtm\" (UID: \"480bae13-46c1-4671-a53b-ce4cbb4811a6\") " pod="openshift-ingress/payload-processing-e2e-mt-db4a68ea-84597fc849-4nrtm"
Sep 10 21:14:12.712260 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:12.712179    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phq45\" (UniqueName: \"kubernetes.io/projected/480bae13-46c1-4671-a53b-ce4cbb4811a6-kube-api-access-phq45\") pod \"payload-processing-e2e-mt-db4a68ea-84597fc849-4nrtm\" (UID: \"480bae13-46c1-4671-a53b-ce4cbb4811a6\") " pod="openshift-ingress/payload-processing-e2e-mt-db4a68ea-84597fc849-4nrtm"
Sep 10 21:14:12.712260 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:12.712250    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/480bae13-46c1-4671-a53b-ce4cbb4811a6-plugins-config-volume\") pod \"payload-processing-e2e-mt-db4a68ea-84597fc849-4nrtm\" (UID: \"480bae13-46c1-4671-a53b-ce4cbb4811a6\") " pod="openshift-ingress/payload-processing-e2e-mt-db4a68ea-84597fc849-4nrtm"
Sep 10 21:14:12.713060 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:12.713033    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/480bae13-46c1-4671-a53b-ce4cbb4811a6-plugins-config-volume\") pod \"payload-processing-e2e-mt-db4a68ea-84597fc849-4nrtm\" (UID: \"480bae13-46c1-4671-a53b-ce4cbb4811a6\") " pod="openshift-ingress/payload-processing-e2e-mt-db4a68ea-84597fc849-4nrtm"
Sep 10 21:14:12.719846 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:12.719823    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phq45\" (UniqueName: \"kubernetes.io/projected/480bae13-46c1-4671-a53b-ce4cbb4811a6-kube-api-access-phq45\") pod \"payload-processing-e2e-mt-db4a68ea-84597fc849-4nrtm\" (UID: \"480bae13-46c1-4671-a53b-ce4cbb4811a6\") " pod="openshift-ingress/payload-processing-e2e-mt-db4a68ea-84597fc849-4nrtm"
Sep 10 21:14:12.737742 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:12.737705    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-db4a68ea-84597fc849-4nrtm"
Sep 10 21:14:13.013989 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:13.012943    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-db4a68ea-84597fc849-4nrtm"]
Sep 10 21:14:13.017341 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:14:13.014604    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod480bae13_46c1_4671_a53b_ce4cbb4811a6.slice/crio-0c3ecc5ac6d56984589c2c2d3ffeb5f7fd4bc4ad748c2e0eae1cf4e9313b15b1 WatchSource:0}: Error finding container 0c3ecc5ac6d56984589c2c2d3ffeb5f7fd4bc4ad748c2e0eae1cf4e9313b15b1: Status 404 returned error can't find the container with id 0c3ecc5ac6d56984589c2c2d3ffeb5f7fd4bc4ad748c2e0eae1cf4e9313b15b1
Sep 10 21:14:13.494658 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:13.494621    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-db4a68ea-84597fc849-4nrtm" event={"ID":"480bae13-46c1-4671-a53b-ce4cbb4811a6","Type":"ContainerStarted","Data":"283916db85373df2bceeab250a99226d719e0dbf5c47d2073674ae5c61d503cb"}
Sep 10 21:14:13.495128 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:13.494675    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-db4a68ea-84597fc849-4nrtm" event={"ID":"480bae13-46c1-4671-a53b-ce4cbb4811a6","Type":"ContainerStarted","Data":"0c3ecc5ac6d56984589c2c2d3ffeb5f7fd4bc4ad748c2e0eae1cf4e9313b15b1"}
Sep 10 21:14:13.495128 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:13.494714    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-db4a68ea-84597fc849-4nrtm"
Sep 10 21:14:13.521628 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:13.521576    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-db4a68ea-84597fc849-4nrtm" podStartSLOduration=1.52156084 podStartE2EDuration="1.52156084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:14:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:14:13.518274749 +0000 UTC m=+971.175183521" watchObservedRunningTime="2026-09-10 21:14:13.52156084 +0000 UTC m=+971.178469612"
Sep 10 21:14:21.841276 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:21.841246    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-000295-66ccb8699f-7glkj"]
Sep 10 21:14:21.841754 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:21.841561    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-000295-66ccb8699f-7glkj" podUID="f603b480-c552-4176-b6e7-779118d030fd" containerName="payload-processing" containerID="cri-o://063883ecd398a37d007fdaa39083a13d72d79f2bfeefe01519cf095682dcebb9" gracePeriod=30
Sep 10 21:14:21.849367 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:21.849342    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-000295-7c4d8675bb-98rzs"]
Sep 10 21:14:21.849756 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:21.849672    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-000295-7c4d8675bb-98rzs" podUID="2e895091-bf03-460b-99ab-242db43dd7d6" containerName="payload-pre-processing" containerID="cri-o://7112e14f5b042afd49dc5840c4b47d5e22009b318cf389c7c59ca4bb6103ef91" gracePeriod=30
Sep 10 21:14:22.195018 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:22.194945    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-000295-66ccb8699f-7glkj"
Sep 10 21:14:22.199173 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:22.199146    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-000295-7c4d8675bb-98rzs"
Sep 10 21:14:22.205220 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:22.205198    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2n9g\" (UniqueName: \"kubernetes.io/projected/f603b480-c552-4176-b6e7-779118d030fd-kube-api-access-v2n9g\") pod \"f603b480-c552-4176-b6e7-779118d030fd\" (UID: \"f603b480-c552-4176-b6e7-779118d030fd\") "
Sep 10 21:14:22.205377 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:22.205277    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f603b480-c552-4176-b6e7-779118d030fd-plugins-config-volume\") pod \"f603b480-c552-4176-b6e7-779118d030fd\" (UID: \"f603b480-c552-4176-b6e7-779118d030fd\") "
Sep 10 21:14:22.205668 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:22.205642    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f603b480-c552-4176-b6e7-779118d030fd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f603b480-c552-4176-b6e7-779118d030fd" (UID: "f603b480-c552-4176-b6e7-779118d030fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:14:22.207316 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:22.207284    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f603b480-c552-4176-b6e7-779118d030fd-kube-api-access-v2n9g" (OuterVolumeSpecName: "kube-api-access-v2n9g") pod "f603b480-c552-4176-b6e7-779118d030fd" (UID: "f603b480-c552-4176-b6e7-779118d030fd"). InnerVolumeSpecName "kube-api-access-v2n9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:14:22.306679 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:22.306645    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8p58\" (UniqueName: \"kubernetes.io/projected/2e895091-bf03-460b-99ab-242db43dd7d6-kube-api-access-v8p58\") pod \"2e895091-bf03-460b-99ab-242db43dd7d6\" (UID: \"2e895091-bf03-460b-99ab-242db43dd7d6\") "
Sep 10 21:14:22.306679 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:22.306682    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e895091-bf03-460b-99ab-242db43dd7d6-plugins-config-volume\") pod \"2e895091-bf03-460b-99ab-242db43dd7d6\" (UID: \"2e895091-bf03-460b-99ab-242db43dd7d6\") "
Sep 10 21:14:22.306930 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:22.306879    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2n9g\" (UniqueName: \"kubernetes.io/projected/f603b480-c552-4176-b6e7-779118d030fd-kube-api-access-v2n9g\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:14:22.306930 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:22.306891    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f603b480-c552-4176-b6e7-779118d030fd-plugins-config-volume\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:14:22.307060 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:22.307034    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e895091-bf03-460b-99ab-242db43dd7d6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2e895091-bf03-460b-99ab-242db43dd7d6" (UID: "2e895091-bf03-460b-99ab-242db43dd7d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:14:22.308728 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:22.308706    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e895091-bf03-460b-99ab-242db43dd7d6-kube-api-access-v8p58" (OuterVolumeSpecName: "kube-api-access-v8p58") pod "2e895091-bf03-460b-99ab-242db43dd7d6" (UID: "2e895091-bf03-460b-99ab-242db43dd7d6"). InnerVolumeSpecName "kube-api-access-v8p58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:14:22.407710 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:22.407613    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8p58\" (UniqueName: \"kubernetes.io/projected/2e895091-bf03-460b-99ab-242db43dd7d6-kube-api-access-v8p58\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:14:22.407710 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:22.407645    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e895091-bf03-460b-99ab-242db43dd7d6-plugins-config-volume\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:14:22.545671 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:22.545637    2581 generic.go:356] "Generic (PLEG): container finished" podID="2e895091-bf03-460b-99ab-242db43dd7d6" containerID="7112e14f5b042afd49dc5840c4b47d5e22009b318cf389c7c59ca4bb6103ef91" exitCode=0
Sep 10 21:14:22.545859 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:22.545682    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-000295-7c4d8675bb-98rzs"
Sep 10 21:14:22.545859 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:22.545714    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-000295-7c4d8675bb-98rzs" event={"ID":"2e895091-bf03-460b-99ab-242db43dd7d6","Type":"ContainerDied","Data":"7112e14f5b042afd49dc5840c4b47d5e22009b318cf389c7c59ca4bb6103ef91"}
Sep 10 21:14:22.545859 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:22.545765    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-000295-7c4d8675bb-98rzs" event={"ID":"2e895091-bf03-460b-99ab-242db43dd7d6","Type":"ContainerDied","Data":"c9136ed9605c3b08e5e76f84676ba713891924d19cf809b253b34d9ce0483374"}
Sep 10 21:14:22.545859 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:22.545786    2581 scope.go:117] "RemoveContainer" containerID="7112e14f5b042afd49dc5840c4b47d5e22009b318cf389c7c59ca4bb6103ef91"
Sep 10 21:14:22.546908 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:22.546887    2581 generic.go:356] "Generic (PLEG): container finished" podID="f603b480-c552-4176-b6e7-779118d030fd" containerID="063883ecd398a37d007fdaa39083a13d72d79f2bfeefe01519cf095682dcebb9" exitCode=0
Sep 10 21:14:22.547005 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:22.546947    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-000295-66ccb8699f-7glkj"
Sep 10 21:14:22.547005 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:22.546964    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-000295-66ccb8699f-7glkj" event={"ID":"f603b480-c552-4176-b6e7-779118d030fd","Type":"ContainerDied","Data":"063883ecd398a37d007fdaa39083a13d72d79f2bfeefe01519cf095682dcebb9"}
Sep 10 21:14:22.547005 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:22.546997    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-000295-66ccb8699f-7glkj" event={"ID":"f603b480-c552-4176-b6e7-779118d030fd","Type":"ContainerDied","Data":"827f07234611ced8349e486431b681f934333e609738c5760d493d15cddb3ba4"}
Sep 10 21:14:22.555157 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:22.555136    2581 scope.go:117] "RemoveContainer" containerID="7112e14f5b042afd49dc5840c4b47d5e22009b318cf389c7c59ca4bb6103ef91"
Sep 10 21:14:22.555470 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:14:22.555448    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7112e14f5b042afd49dc5840c4b47d5e22009b318cf389c7c59ca4bb6103ef91\": container with ID starting with 7112e14f5b042afd49dc5840c4b47d5e22009b318cf389c7c59ca4bb6103ef91 not found: ID does not exist" containerID="7112e14f5b042afd49dc5840c4b47d5e22009b318cf389c7c59ca4bb6103ef91"
Sep 10 21:14:22.555550 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:22.555476    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7112e14f5b042afd49dc5840c4b47d5e22009b318cf389c7c59ca4bb6103ef91"} err="failed to get container status \"7112e14f5b042afd49dc5840c4b47d5e22009b318cf389c7c59ca4bb6103ef91\": rpc error: code = NotFound desc = could not find container \"7112e14f5b042afd49dc5840c4b47d5e22009b318cf389c7c59ca4bb6103ef91\": container with ID starting with 7112e14f5b042afd49dc5840c4b47d5e22009b318cf389c7c59ca4bb6103ef91 not found: ID does not exist"
Sep 10 21:14:22.555550 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:22.555493    2581 scope.go:117] "RemoveContainer" containerID="063883ecd398a37d007fdaa39083a13d72d79f2bfeefe01519cf095682dcebb9"
Sep 10 21:14:22.581072 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:22.581039    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-000295-7c4d8675bb-98rzs"]
Sep 10 21:14:22.584133 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:22.584106    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-000295-7c4d8675bb-98rzs"]
Sep 10 21:14:22.587633 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:22.587617    2581 scope.go:117] "RemoveContainer" containerID="063883ecd398a37d007fdaa39083a13d72d79f2bfeefe01519cf095682dcebb9"
Sep 10 21:14:22.587982 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:14:22.587958    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"063883ecd398a37d007fdaa39083a13d72d79f2bfeefe01519cf095682dcebb9\": container with ID starting with 063883ecd398a37d007fdaa39083a13d72d79f2bfeefe01519cf095682dcebb9 not found: ID does not exist" containerID="063883ecd398a37d007fdaa39083a13d72d79f2bfeefe01519cf095682dcebb9"
Sep 10 21:14:22.588061 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:22.588010    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"063883ecd398a37d007fdaa39083a13d72d79f2bfeefe01519cf095682dcebb9"} err="failed to get container status \"063883ecd398a37d007fdaa39083a13d72d79f2bfeefe01519cf095682dcebb9\": rpc error: code = NotFound desc = could not find container \"063883ecd398a37d007fdaa39083a13d72d79f2bfeefe01519cf095682dcebb9\": container with ID starting with 063883ecd398a37d007fdaa39083a13d72d79f2bfeefe01519cf095682dcebb9 not found: ID does not exist"
Sep 10 21:14:22.591224 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:22.591189    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-000295-66ccb8699f-7glkj"]
Sep 10 21:14:22.592935 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:22.592895    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-000295-66ccb8699f-7glkj"]
Sep 10 21:14:22.868175 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:22.868144    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e895091-bf03-460b-99ab-242db43dd7d6" path="/var/lib/kubelet/pods/2e895091-bf03-460b-99ab-242db43dd7d6/volumes"
Sep 10 21:14:22.868546 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:22.868533    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f603b480-c552-4176-b6e7-779118d030fd" path="/var/lib/kubelet/pods/f603b480-c552-4176-b6e7-779118d030fd/volumes"
Sep 10 21:14:24.502496 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:24.502465    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-db4a68ea-84597fc849-4nrtm"
Sep 10 21:14:25.511510 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:25.511428    2581 patch_prober.go:28] interesting pod/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 21:14:25.511510 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:25.511496    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj" podUID="f8c2e3e4-83b3-44f2-857b-331077d03271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 21:14:26.579480 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:26.579450    2581 generic.go:356] "Generic (PLEG): container finished" podID="f8c2e3e4-83b3-44f2-857b-331077d03271" containerID="cd8d57dee9bf2cb98b85b3dcaa952db8969156e34000240afc7ef30865650434" exitCode=0
Sep 10 21:14:26.579898 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:26.579528    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj" event={"ID":"f8c2e3e4-83b3-44f2-857b-331077d03271","Type":"ContainerDied","Data":"cd8d57dee9bf2cb98b85b3dcaa952db8969156e34000240afc7ef30865650434"}
Sep 10 21:14:26.579966 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:26.579949    2581 scope.go:117] "RemoveContainer" containerID="cd8d57dee9bf2cb98b85b3dcaa952db8969156e34000240afc7ef30865650434"
Sep 10 21:14:26.580960 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:26.580931    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 21:14:26.581059 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:26.580997    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 21:14:26.581059 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:26.581031    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 21:14:27.588637 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:27.588602    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj" event={"ID":"f8c2e3e4-83b3-44f2-857b-331077d03271","Type":"ContainerStarted","Data":"7ef87c73871f1252feffb323c425464cdfb0367ed03129b6c926ec28c9e8bae7"}
Sep 10 21:14:28.287892 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:28.287849    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"
Sep 10 21:14:28.288122 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:28.288100    2581 patch_prober.go:28] interesting pod/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 10 21:14:28.288169 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:28.288145    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj" podUID="f8c2e3e4-83b3-44f2-857b-331077d03271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 10 21:14:29.287693 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:29.287653    2581 patch_prober.go:28] interesting pod/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 10 21:14:29.288143 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:29.287723    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj" podUID="f8c2e3e4-83b3-44f2-857b-331077d03271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 10 21:14:30.287902 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:30.287856    2581 patch_prober.go:28] interesting pod/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 10 21:14:30.288281 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:30.287920    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj" podUID="f8c2e3e4-83b3-44f2-857b-331077d03271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 10 21:14:31.288226 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:31.288187    2581 patch_prober.go:28] interesting pod/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 10 21:14:31.288632 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:31.288240    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj" podUID="f8c2e3e4-83b3-44f2-857b-331077d03271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 10 21:14:32.287889 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:32.287833    2581 patch_prober.go:28] interesting pod/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 10 21:14:32.288072 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:32.287917    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj" podUID="f8c2e3e4-83b3-44f2-857b-331077d03271" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 10 21:14:33.291581 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:33.291550    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"
Sep 10 21:14:33.292036 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:33.291825    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"
Sep 10 21:14:33.292652 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:33.292627    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"
Sep 10 21:14:35.397500 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:35.397464    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-828c7c-79f4cf7bfd-5f4t2"]
Sep 10 21:14:35.397862 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:35.397821    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f603b480-c552-4176-b6e7-779118d030fd" containerName="payload-processing"
Sep 10 21:14:35.397862 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:35.397831    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f603b480-c552-4176-b6e7-779118d030fd" containerName="payload-processing"
Sep 10 21:14:35.397862 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:35.397841    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e895091-bf03-460b-99ab-242db43dd7d6" containerName="payload-pre-processing"
Sep 10 21:14:35.397862 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:35.397847    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e895091-bf03-460b-99ab-242db43dd7d6" containerName="payload-pre-processing"
Sep 10 21:14:35.397997 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:35.397908    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f603b480-c552-4176-b6e7-779118d030fd" containerName="payload-processing"
Sep 10 21:14:35.397997 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:35.397915    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e895091-bf03-460b-99ab-242db43dd7d6" containerName="payload-pre-processing"
Sep 10 21:14:35.399857 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:35.399840    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-828c7c-79f4cf7bfd-5f4t2"
Sep 10 21:14:35.403403 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:35.403374    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-828c7c\""
Sep 10 21:14:35.403403 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:35.403396    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-828c7c-dockercfg-lkwzv\""
Sep 10 21:14:35.412010 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:35.411985    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-828c7c-79f4cf7bfd-5f4t2"]
Sep 10 21:14:35.538863 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:35.538812    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c29d5c7c-8766-490a-b1e5-e453245d8b42-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-828c7c-79f4cf7bfd-5f4t2\" (UID: \"c29d5c7c-8766-490a-b1e5-e453245d8b42\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-828c7c-79f4cf7bfd-5f4t2"
Sep 10 21:14:35.538863 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:35.538852    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7b6r\" (UniqueName: \"kubernetes.io/projected/c29d5c7c-8766-490a-b1e5-e453245d8b42-kube-api-access-n7b6r\") pod \"payload-pre-processing-e2e-shared-a-w3-828c7c-79f4cf7bfd-5f4t2\" (UID: \"c29d5c7c-8766-490a-b1e5-e453245d8b42\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-828c7c-79f4cf7bfd-5f4t2"
Sep 10 21:14:35.639525 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:35.639490    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c29d5c7c-8766-490a-b1e5-e453245d8b42-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-828c7c-79f4cf7bfd-5f4t2\" (UID: \"c29d5c7c-8766-490a-b1e5-e453245d8b42\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-828c7c-79f4cf7bfd-5f4t2"
Sep 10 21:14:35.639525 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:35.639525    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7b6r\" (UniqueName: \"kubernetes.io/projected/c29d5c7c-8766-490a-b1e5-e453245d8b42-kube-api-access-n7b6r\") pod \"payload-pre-processing-e2e-shared-a-w3-828c7c-79f4cf7bfd-5f4t2\" (UID: \"c29d5c7c-8766-490a-b1e5-e453245d8b42\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-828c7c-79f4cf7bfd-5f4t2"
Sep 10 21:14:35.640358 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:35.640330    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c29d5c7c-8766-490a-b1e5-e453245d8b42-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-828c7c-79f4cf7bfd-5f4t2\" (UID: \"c29d5c7c-8766-490a-b1e5-e453245d8b42\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-828c7c-79f4cf7bfd-5f4t2"
Sep 10 21:14:35.650838 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:35.650777    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7b6r\" (UniqueName: \"kubernetes.io/projected/c29d5c7c-8766-490a-b1e5-e453245d8b42-kube-api-access-n7b6r\") pod \"payload-pre-processing-e2e-shared-a-w3-828c7c-79f4cf7bfd-5f4t2\" (UID: \"c29d5c7c-8766-490a-b1e5-e453245d8b42\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-828c7c-79f4cf7bfd-5f4t2"
Sep 10 21:14:35.710940 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:35.710898    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-828c7c-79f4cf7bfd-5f4t2"
Sep 10 21:14:35.982581 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:35.982503    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-828c7c-79f4cf7bfd-5f4t2"]
Sep 10 21:14:35.985047 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:14:35.985018    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc29d5c7c_8766_490a_b1e5_e453245d8b42.slice/crio-f640be2ac09695a880d60bcec0538e26974022d567641df91d8415ac7a3926f4 WatchSource:0}: Error finding container f640be2ac09695a880d60bcec0538e26974022d567641df91d8415ac7a3926f4: Status 404 returned error can't find the container with id f640be2ac09695a880d60bcec0538e26974022d567641df91d8415ac7a3926f4
Sep 10 21:14:36.637754 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:36.637714    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-828c7c-79f4cf7bfd-5f4t2" event={"ID":"c29d5c7c-8766-490a-b1e5-e453245d8b42","Type":"ContainerStarted","Data":"2d785e9ce21d017468cd1e9b973ca88e3e6c301fd76e66d9602d09bb8e50a001"}
Sep 10 21:14:36.638238 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:36.637783    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-828c7c-79f4cf7bfd-5f4t2" event={"ID":"c29d5c7c-8766-490a-b1e5-e453245d8b42","Type":"ContainerStarted","Data":"f640be2ac09695a880d60bcec0538e26974022d567641df91d8415ac7a3926f4"}
Sep 10 21:14:36.638238 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:36.637806    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-828c7c-79f4cf7bfd-5f4t2"
Sep 10 21:14:36.655173 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:36.655124    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-828c7c-79f4cf7bfd-5f4t2" podStartSLOduration=1.655108056 podStartE2EDuration="1.655108056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:14:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:14:36.654164636 +0000 UTC m=+994.311073422" watchObservedRunningTime="2026-09-10 21:14:36.655108056 +0000 UTC m=+994.312016828"
Sep 10 21:14:42.942078 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:42.942041    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db4a68ea-6f59j"]
Sep 10 21:14:42.945353 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:42.945332    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db4a68ea-6f59j"
Sep 10 21:14:42.947397 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:42.947375    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:14:42.947502 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:42.947470    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s4qqn\""
Sep 10 21:14:42.953462 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:42.953427    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db4a68ea-6f59j"]
Sep 10 21:14:43.113180 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:43.113147    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f411bd1a-8c4a-4c96-ab5e-fa6c57bd9470-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-db4a68ea-6f59j\" (UID: \"f411bd1a-8c4a-4c96-ab5e-fa6c57bd9470\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db4a68ea-6f59j"
Sep 10 21:14:43.214311 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:43.214215    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f411bd1a-8c4a-4c96-ab5e-fa6c57bd9470-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-db4a68ea-6f59j\" (UID: \"f411bd1a-8c4a-4c96-ab5e-fa6c57bd9470\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db4a68ea-6f59j"
Sep 10 21:14:43.214831 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:43.214813    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f411bd1a-8c4a-4c96-ab5e-fa6c57bd9470-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-db4a68ea-6f59j\" (UID: \"f411bd1a-8c4a-4c96-ab5e-fa6c57bd9470\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db4a68ea-6f59j"
Sep 10 21:14:43.257179 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:43.257149    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db4a68ea-6f59j"
Sep 10 21:14:43.527216 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:43.527140    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db4a68ea-6f59j"]
Sep 10 21:14:43.530652 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:14:43.530626    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf411bd1a_8c4a_4c96_ab5e_fa6c57bd9470.slice/crio-f68c97ede4a3c63dbd283ad6d326f63ccd223adab9aca071320bb93f2c9fb17b WatchSource:0}: Error finding container f68c97ede4a3c63dbd283ad6d326f63ccd223adab9aca071320bb93f2c9fb17b: Status 404 returned error can't find the container with id f68c97ede4a3c63dbd283ad6d326f63ccd223adab9aca071320bb93f2c9fb17b
Sep 10 21:14:43.674839 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:43.674810    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db4a68ea-6f59j" event={"ID":"f411bd1a-8c4a-4c96-ab5e-fa6c57bd9470","Type":"ContainerStarted","Data":"f68c97ede4a3c63dbd283ad6d326f63ccd223adab9aca071320bb93f2c9fb17b"}
Sep 10 21:14:44.681966 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:44.681930    2581 generic.go:356] "Generic (PLEG): container finished" podID="f411bd1a-8c4a-4c96-ab5e-fa6c57bd9470" containerID="d48cdfe611d4d7b9ab08fd6bd184d57208580b894c2632787d21a0bd04fa4226" exitCode=0
Sep 10 21:14:44.682349 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:44.681995    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db4a68ea-6f59j" event={"ID":"f411bd1a-8c4a-4c96-ab5e-fa6c57bd9470","Type":"ContainerDied","Data":"d48cdfe611d4d7b9ab08fd6bd184d57208580b894c2632787d21a0bd04fa4226"}
Sep 10 21:14:45.813689 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:45.813659    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db4a68ea-6f59j"
Sep 10 21:14:45.939941 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:45.939859    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f411bd1a-8c4a-4c96-ab5e-fa6c57bd9470-maas-api-service-ca\") pod \"f411bd1a-8c4a-4c96-ab5e-fa6c57bd9470\" (UID: \"f411bd1a-8c4a-4c96-ab5e-fa6c57bd9470\") "
Sep 10 21:14:45.940246 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:45.940226    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f411bd1a-8c4a-4c96-ab5e-fa6c57bd9470-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f411bd1a-8c4a-4c96-ab5e-fa6c57bd9470" (UID: "f411bd1a-8c4a-4c96-ab5e-fa6c57bd9470"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:14:46.041519 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:46.041480    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f411bd1a-8c4a-4c96-ab5e-fa6c57bd9470-maas-api-service-ca\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:14:46.692659 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:46.692630    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db4a68ea-6f59j"
Sep 10 21:14:46.692830 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:46.692629    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db4a68ea-6f59j" event={"ID":"f411bd1a-8c4a-4c96-ab5e-fa6c57bd9470","Type":"ContainerDied","Data":"f68c97ede4a3c63dbd283ad6d326f63ccd223adab9aca071320bb93f2c9fb17b"}
Sep 10 21:14:46.692830 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:46.692764    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f68c97ede4a3c63dbd283ad6d326f63ccd223adab9aca071320bb93f2c9fb17b"
Sep 10 21:14:47.645104 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:47.645076    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-828c7c-79f4cf7bfd-5f4t2"
Sep 10 21:14:53.480485 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:53.480448    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-db4a68ea-84597fc849-4nrtm"]
Sep 10 21:14:53.480974 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:53.480695    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-db4a68ea-84597fc849-4nrtm" podUID="480bae13-46c1-4671-a53b-ce4cbb4811a6" containerName="payload-processing" containerID="cri-o://283916db85373df2bceeab250a99226d719e0dbf5c47d2073674ae5c61d503cb" gracePeriod=30
Sep 10 21:14:53.636109 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:53.636076    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db4a68ea-6f59j"]
Sep 10 21:14:53.638335 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:53.638280    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db4a68ea-6f59j"]
Sep 10 21:14:53.725467 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:53.725440    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-db4a68ea-84597fc849-4nrtm"
Sep 10 21:14:53.731185 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:53.731117    2581 generic.go:356] "Generic (PLEG): container finished" podID="480bae13-46c1-4671-a53b-ce4cbb4811a6" containerID="283916db85373df2bceeab250a99226d719e0dbf5c47d2073674ae5c61d503cb" exitCode=0
Sep 10 21:14:53.731185 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:53.731157    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-db4a68ea-84597fc849-4nrtm"
Sep 10 21:14:53.731409 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:53.731215    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-db4a68ea-84597fc849-4nrtm" event={"ID":"480bae13-46c1-4671-a53b-ce4cbb4811a6","Type":"ContainerDied","Data":"283916db85373df2bceeab250a99226d719e0dbf5c47d2073674ae5c61d503cb"}
Sep 10 21:14:53.731409 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:53.731282    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-db4a68ea-84597fc849-4nrtm" event={"ID":"480bae13-46c1-4671-a53b-ce4cbb4811a6","Type":"ContainerDied","Data":"0c3ecc5ac6d56984589c2c2d3ffeb5f7fd4bc4ad748c2e0eae1cf4e9313b15b1"}
Sep 10 21:14:53.731409 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:53.731332    2581 scope.go:117] "RemoveContainer" containerID="283916db85373df2bceeab250a99226d719e0dbf5c47d2073674ae5c61d503cb"
Sep 10 21:14:53.740953 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:53.740923    2581 scope.go:117] "RemoveContainer" containerID="283916db85373df2bceeab250a99226d719e0dbf5c47d2073674ae5c61d503cb"
Sep 10 21:14:53.741399 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:14:53.741336    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"283916db85373df2bceeab250a99226d719e0dbf5c47d2073674ae5c61d503cb\": container with ID starting with 283916db85373df2bceeab250a99226d719e0dbf5c47d2073674ae5c61d503cb not found: ID does not exist" containerID="283916db85373df2bceeab250a99226d719e0dbf5c47d2073674ae5c61d503cb"
Sep 10 21:14:53.741399 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:53.741371    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"283916db85373df2bceeab250a99226d719e0dbf5c47d2073674ae5c61d503cb"} err="failed to get container status \"283916db85373df2bceeab250a99226d719e0dbf5c47d2073674ae5c61d503cb\": rpc error: code = NotFound desc = could not find container \"283916db85373df2bceeab250a99226d719e0dbf5c47d2073674ae5c61d503cb\": container with ID starting with 283916db85373df2bceeab250a99226d719e0dbf5c47d2073674ae5c61d503cb not found: ID does not exist"
Sep 10 21:14:53.909229 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:53.909196    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/480bae13-46c1-4671-a53b-ce4cbb4811a6-plugins-config-volume\") pod \"480bae13-46c1-4671-a53b-ce4cbb4811a6\" (UID: \"480bae13-46c1-4671-a53b-ce4cbb4811a6\") "
Sep 10 21:14:53.909432 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:53.909250    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phq45\" (UniqueName: \"kubernetes.io/projected/480bae13-46c1-4671-a53b-ce4cbb4811a6-kube-api-access-phq45\") pod \"480bae13-46c1-4671-a53b-ce4cbb4811a6\" (UID: \"480bae13-46c1-4671-a53b-ce4cbb4811a6\") "
Sep 10 21:14:53.909611 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:53.909580    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/480bae13-46c1-4671-a53b-ce4cbb4811a6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "480bae13-46c1-4671-a53b-ce4cbb4811a6" (UID: "480bae13-46c1-4671-a53b-ce4cbb4811a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:14:53.911371 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:53.911353    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/480bae13-46c1-4671-a53b-ce4cbb4811a6-kube-api-access-phq45" (OuterVolumeSpecName: "kube-api-access-phq45") pod "480bae13-46c1-4671-a53b-ce4cbb4811a6" (UID: "480bae13-46c1-4671-a53b-ce4cbb4811a6"). InnerVolumeSpecName "kube-api-access-phq45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:14:54.009907 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:54.009841    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/480bae13-46c1-4671-a53b-ce4cbb4811a6-plugins-config-volume\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:14:54.009907 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:54.009863    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phq45\" (UniqueName: \"kubernetes.io/projected/480bae13-46c1-4671-a53b-ce4cbb4811a6-kube-api-access-phq45\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:14:54.060605 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:54.060576    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-db4a68ea-84597fc849-4nrtm"]
Sep 10 21:14:54.066291 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:54.066266    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-db4a68ea-84597fc849-4nrtm"]
Sep 10 21:14:54.867977 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:54.867942    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="480bae13-46c1-4671-a53b-ce4cbb4811a6" path="/var/lib/kubelet/pods/480bae13-46c1-4671-a53b-ce4cbb4811a6/volumes"
Sep 10 21:14:54.868346 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:14:54.868264    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f411bd1a-8c4a-4c96-ab5e-fa6c57bd9470" path="/var/lib/kubelet/pods/f411bd1a-8c4a-4c96-ab5e-fa6c57bd9470/volumes"
Sep 10 21:15:00.163954 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:00.163915    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817915-ngwj8"]
Sep 10 21:15:00.164471 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:00.164454    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="480bae13-46c1-4671-a53b-ce4cbb4811a6" containerName="payload-processing"
Sep 10 21:15:00.164530 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:00.164473    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="480bae13-46c1-4671-a53b-ce4cbb4811a6" containerName="payload-processing"
Sep 10 21:15:00.164530 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:00.164493    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f411bd1a-8c4a-4c96-ab5e-fa6c57bd9470" containerName="revoke-keys"
Sep 10 21:15:00.164530 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:00.164503    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f411bd1a-8c4a-4c96-ab5e-fa6c57bd9470" containerName="revoke-keys"
Sep 10 21:15:00.164651 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:00.164619    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="480bae13-46c1-4671-a53b-ce4cbb4811a6" containerName="payload-processing"
Sep 10 21:15:00.164651 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:00.164630    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f411bd1a-8c4a-4c96-ab5e-fa6c57bd9470" containerName="revoke-keys"
Sep 10 21:15:00.167860 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:00.167839    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-828c7c-29817915-hzkvj"]
Sep 10 21:15:00.168016 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:00.167951    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817915-ngwj8"
Sep 10 21:15:00.170373 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:00.170351    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s4qqn\""
Sep 10 21:15:00.171189 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:00.171170    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-828c7c-29817915-hzkvj"
Sep 10 21:15:00.178854 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:00.178827    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817915-ngwj8"
Sep 10 21:15:00.187047 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:00.187012    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817915-ngwj8"]
Sep 10 21:15:00.187785 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:00.187765    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-828c7c-29817915-hzkvj"
Sep 10 21:15:00.189515 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:00.189494    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-828c7c-29817915-hzkvj"]
Sep 10 21:15:00.460821 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:00.460770    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817915-ngwj8"]
Sep 10 21:15:00.463703 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:15:00.463666    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod616f2812_d98d_4e01_8197_405863e91fe1.slice/crio-d97816a55baf6318983fd4f6eda7c298b9d2e4a9ca62a5fa0127277bde3921e4 WatchSource:0}: Error finding container d97816a55baf6318983fd4f6eda7c298b9d2e4a9ca62a5fa0127277bde3921e4: Status 404 returned error can't find the container with id d97816a55baf6318983fd4f6eda7c298b9d2e4a9ca62a5fa0127277bde3921e4
Sep 10 21:15:00.472842 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:00.472814    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-828c7c-29817915-hzkvj"]
Sep 10 21:15:00.477100 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:15:00.477076    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3ee754b_56e0_4483_85ba_02e2bd77769a.slice/crio-af1c01f85af9c17a6e1c82e192966f9bfddebe3561d73143460b857617803d81 WatchSource:0}: Error finding container af1c01f85af9c17a6e1c82e192966f9bfddebe3561d73143460b857617803d81: Status 404 returned error can't find the container with id af1c01f85af9c17a6e1c82e192966f9bfddebe3561d73143460b857617803d81
Sep 10 21:15:00.519166 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:00.519138    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"]
Sep 10 21:15:00.519433 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:00.519406    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj" podUID="f8c2e3e4-83b3-44f2-857b-331077d03271" containerName="istio-proxy" containerID="cri-o://7ef87c73871f1252feffb323c425464cdfb0367ed03129b6c926ec28c9e8bae7" gracePeriod=30
Sep 10 21:15:00.775364 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:00.775336    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-828c7c-29817915-hzkvj" event={"ID":"e3ee754b-56e0-4483-85ba-02e2bd77769a","Type":"ContainerStarted","Data":"af1c01f85af9c17a6e1c82e192966f9bfddebe3561d73143460b857617803d81"}
Sep 10 21:15:00.776804 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:00.776781    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817915-ngwj8" event={"ID":"616f2812-d98d-4e01-8197-405863e91fe1","Type":"ContainerStarted","Data":"b818f43432670c5daeed5aabf3aa9de2228f6a9cd6ddbb4be4dd68fdf461e743"}
Sep 10 21:15:00.776900 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:00.776822    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817915-ngwj8" event={"ID":"616f2812-d98d-4e01-8197-405863e91fe1","Type":"ContainerStarted","Data":"d97816a55baf6318983fd4f6eda7c298b9d2e4a9ca62a5fa0127277bde3921e4"}
Sep 10 21:15:00.804223 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:00.804182    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817915-ngwj8" podStartSLOduration=0.804166517 podStartE2EDuration="804.166517ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:15:00.803172116 +0000 UTC m=+1018.460080890" watchObservedRunningTime="2026-09-10 21:15:00.804166517 +0000 UTC m=+1018.461075289"
Sep 10 21:15:01.783400 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:01.783291    2581 generic.go:356] "Generic (PLEG): container finished" podID="616f2812-d98d-4e01-8197-405863e91fe1" containerID="b818f43432670c5daeed5aabf3aa9de2228f6a9cd6ddbb4be4dd68fdf461e743" exitCode=0
Sep 10 21:15:01.783784 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:01.783396    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817915-ngwj8" event={"ID":"616f2812-d98d-4e01-8197-405863e91fe1","Type":"ContainerDied","Data":"b818f43432670c5daeed5aabf3aa9de2228f6a9cd6ddbb4be4dd68fdf461e743"}
Sep 10 21:15:01.784947 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:01.784922    2581 generic.go:356] "Generic (PLEG): container finished" podID="e3ee754b-56e0-4483-85ba-02e2bd77769a" containerID="5f6380fc890cad81f0a43f59fda1bcffbf326eb4cef1ce7615cbff0a4b829a8e" exitCode=0
Sep 10 21:15:01.785061 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:01.784985    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-828c7c-29817915-hzkvj" event={"ID":"e3ee754b-56e0-4483-85ba-02e2bd77769a","Type":"ContainerDied","Data":"5f6380fc890cad81f0a43f59fda1bcffbf326eb4cef1ce7615cbff0a4b829a8e"}
Sep 10 21:15:02.939945 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:02.939918    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817915-ngwj8"
Sep 10 21:15:02.942006 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:02.941990    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-828c7c-29817915-hzkvj"
Sep 10 21:15:03.294095 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:03.294057    2581 patch_prober.go:28] interesting pod/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 21:15:03.294283 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:03.294124    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj" podUID="f8c2e3e4-83b3-44f2-857b-331077d03271" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 21:15:03.796487 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:03.796460    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817915-ngwj8"
Sep 10 21:15:03.796635 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:03.796495    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817915-ngwj8" event={"ID":"616f2812-d98d-4e01-8197-405863e91fe1","Type":"ContainerDied","Data":"d97816a55baf6318983fd4f6eda7c298b9d2e4a9ca62a5fa0127277bde3921e4"}
Sep 10 21:15:03.796635 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:03.796541    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d97816a55baf6318983fd4f6eda7c298b9d2e4a9ca62a5fa0127277bde3921e4"
Sep 10 21:15:03.797856 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:03.797836    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-828c7c-29817915-hzkvj"
Sep 10 21:15:03.797971 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:03.797838    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-828c7c-29817915-hzkvj" event={"ID":"e3ee754b-56e0-4483-85ba-02e2bd77769a","Type":"ContainerDied","Data":"af1c01f85af9c17a6e1c82e192966f9bfddebe3561d73143460b857617803d81"}
Sep 10 21:15:03.797971 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:03.797964    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af1c01f85af9c17a6e1c82e192966f9bfddebe3561d73143460b857617803d81"
Sep 10 21:15:04.198901 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:04.198830    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9f6475-85fb54bc6b-m6ktj"]
Sep 10 21:15:04.199240 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:04.199174    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="616f2812-d98d-4e01-8197-405863e91fe1" containerName="cleanup"
Sep 10 21:15:04.199240 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:04.199186    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="616f2812-d98d-4e01-8197-405863e91fe1" containerName="cleanup"
Sep 10 21:15:04.199240 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:04.199198    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3ee754b-56e0-4483-85ba-02e2bd77769a" containerName="cleanup"
Sep 10 21:15:04.199240 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:04.199204    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3ee754b-56e0-4483-85ba-02e2bd77769a" containerName="cleanup"
Sep 10 21:15:04.199404 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:04.199271    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="616f2812-d98d-4e01-8197-405863e91fe1" containerName="cleanup"
Sep 10 21:15:04.199404 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:04.199282    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3ee754b-56e0-4483-85ba-02e2bd77769a" containerName="cleanup"
Sep 10 21:15:04.202210 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:04.202192    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9f6475-85fb54bc6b-m6ktj"
Sep 10 21:15:04.204267 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:04.204245    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-9f6475\""
Sep 10 21:15:04.213345 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:04.213324    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9f6475-85fb54bc6b-m6ktj"]
Sep 10 21:15:04.235377 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:04.235345    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-9f6475-5d7d84f979-nkxnj"]
Sep 10 21:15:04.238743 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:04.238721    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9f6475-5d7d84f979-nkxnj"
Sep 10 21:15:04.240840 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:04.240819    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-9f6475-dockercfg-j6jz8\""
Sep 10 21:15:04.249661 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:04.249642    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-9f6475-5d7d84f979-nkxnj"]
Sep 10 21:15:04.303343 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:04.303319    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbphr\" (UniqueName: \"kubernetes.io/projected/b6895de4-48ac-4813-98ce-52b92ebfa04f-kube-api-access-wbphr\") pod \"payload-pre-processing-e2e-shared-b-w3-9f6475-85fb54bc6b-m6ktj\" (UID: \"b6895de4-48ac-4813-98ce-52b92ebfa04f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9f6475-85fb54bc6b-m6ktj"
Sep 10 21:15:04.303481 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:04.303394    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6895de4-48ac-4813-98ce-52b92ebfa04f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-9f6475-85fb54bc6b-m6ktj\" (UID: \"b6895de4-48ac-4813-98ce-52b92ebfa04f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9f6475-85fb54bc6b-m6ktj"
Sep 10 21:15:04.303481 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:04.303423    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a65e733-908c-4846-97b6-d6b6b4f1c094-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-9f6475-5d7d84f979-nkxnj\" (UID: \"8a65e733-908c-4846-97b6-d6b6b4f1c094\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9f6475-5d7d84f979-nkxnj"
Sep 10 21:15:04.303584 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:04.303500    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfvwq\" (UniqueName: \"kubernetes.io/projected/8a65e733-908c-4846-97b6-d6b6b4f1c094-kube-api-access-hfvwq\") pod \"payload-processing-e2e-shared-b-w3-9f6475-5d7d84f979-nkxnj\" (UID: \"8a65e733-908c-4846-97b6-d6b6b4f1c094\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9f6475-5d7d84f979-nkxnj"
Sep 10 21:15:04.404348 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:04.404287    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbphr\" (UniqueName: \"kubernetes.io/projected/b6895de4-48ac-4813-98ce-52b92ebfa04f-kube-api-access-wbphr\") pod \"payload-pre-processing-e2e-shared-b-w3-9f6475-85fb54bc6b-m6ktj\" (UID: \"b6895de4-48ac-4813-98ce-52b92ebfa04f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9f6475-85fb54bc6b-m6ktj"
Sep 10 21:15:04.404490 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:04.404400    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6895de4-48ac-4813-98ce-52b92ebfa04f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-9f6475-85fb54bc6b-m6ktj\" (UID: \"b6895de4-48ac-4813-98ce-52b92ebfa04f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9f6475-85fb54bc6b-m6ktj"
Sep 10 21:15:04.404490 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:04.404437    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a65e733-908c-4846-97b6-d6b6b4f1c094-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-9f6475-5d7d84f979-nkxnj\" (UID: \"8a65e733-908c-4846-97b6-d6b6b4f1c094\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9f6475-5d7d84f979-nkxnj"
Sep 10 21:15:04.404490 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:04.404482    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfvwq\" (UniqueName: \"kubernetes.io/projected/8a65e733-908c-4846-97b6-d6b6b4f1c094-kube-api-access-hfvwq\") pod \"payload-processing-e2e-shared-b-w3-9f6475-5d7d84f979-nkxnj\" (UID: \"8a65e733-908c-4846-97b6-d6b6b4f1c094\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9f6475-5d7d84f979-nkxnj"
Sep 10 21:15:04.405199 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:04.405175    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6895de4-48ac-4813-98ce-52b92ebfa04f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-9f6475-85fb54bc6b-m6ktj\" (UID: \"b6895de4-48ac-4813-98ce-52b92ebfa04f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9f6475-85fb54bc6b-m6ktj"
Sep 10 21:15:04.405323 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:04.405275    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a65e733-908c-4846-97b6-d6b6b4f1c094-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-9f6475-5d7d84f979-nkxnj\" (UID: \"8a65e733-908c-4846-97b6-d6b6b4f1c094\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9f6475-5d7d84f979-nkxnj"
Sep 10 21:15:04.412893 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:04.412870    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfvwq\" (UniqueName: \"kubernetes.io/projected/8a65e733-908c-4846-97b6-d6b6b4f1c094-kube-api-access-hfvwq\") pod \"payload-processing-e2e-shared-b-w3-9f6475-5d7d84f979-nkxnj\" (UID: \"8a65e733-908c-4846-97b6-d6b6b4f1c094\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9f6475-5d7d84f979-nkxnj"
Sep 10 21:15:04.412893 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:04.412892    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbphr\" (UniqueName: \"kubernetes.io/projected/b6895de4-48ac-4813-98ce-52b92ebfa04f-kube-api-access-wbphr\") pod \"payload-pre-processing-e2e-shared-b-w3-9f6475-85fb54bc6b-m6ktj\" (UID: \"b6895de4-48ac-4813-98ce-52b92ebfa04f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9f6475-85fb54bc6b-m6ktj"
Sep 10 21:15:04.513537 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:04.513464    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9f6475-85fb54bc6b-m6ktj"
Sep 10 21:15:04.548571 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:04.548547    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9f6475-5d7d84f979-nkxnj"
Sep 10 21:15:04.790242 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:04.790159    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9f6475-85fb54bc6b-m6ktj"]
Sep 10 21:15:04.793597 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:15:04.793567    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6895de4_48ac_4813_98ce_52b92ebfa04f.slice/crio-9a6f78b52cb1b55ce2295796edb1635408c3acec08d2529222ec6ad7c9b34353 WatchSource:0}: Error finding container 9a6f78b52cb1b55ce2295796edb1635408c3acec08d2529222ec6ad7c9b34353: Status 404 returned error can't find the container with id 9a6f78b52cb1b55ce2295796edb1635408c3acec08d2529222ec6ad7c9b34353
Sep 10 21:15:04.803907 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:04.803877    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9f6475-85fb54bc6b-m6ktj" event={"ID":"b6895de4-48ac-4813-98ce-52b92ebfa04f","Type":"ContainerStarted","Data":"9a6f78b52cb1b55ce2295796edb1635408c3acec08d2529222ec6ad7c9b34353"}
Sep 10 21:15:04.834907 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:04.834829    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-9f6475-5d7d84f979-nkxnj"]
Sep 10 21:15:04.836149 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:15:04.836127    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a65e733_908c_4846_97b6_d6b6b4f1c094.slice/crio-127de3d3db57bf7fc23abe0cd625be15b8dae2f1f723d08b200cfc789d74d1af WatchSource:0}: Error finding container 127de3d3db57bf7fc23abe0cd625be15b8dae2f1f723d08b200cfc789d74d1af: Status 404 returned error can't find the container with id 127de3d3db57bf7fc23abe0cd625be15b8dae2f1f723d08b200cfc789d74d1af
Sep 10 21:15:05.776711 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.776680    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"
Sep 10 21:15:05.812039 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.811998    2581 generic.go:356] "Generic (PLEG): container finished" podID="f8c2e3e4-83b3-44f2-857b-331077d03271" containerID="7ef87c73871f1252feffb323c425464cdfb0367ed03129b6c926ec28c9e8bae7" exitCode=0
Sep 10 21:15:05.812216 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.812050    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"
Sep 10 21:15:05.812216 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.812079    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj" event={"ID":"f8c2e3e4-83b3-44f2-857b-331077d03271","Type":"ContainerDied","Data":"7ef87c73871f1252feffb323c425464cdfb0367ed03129b6c926ec28c9e8bae7"}
Sep 10 21:15:05.812216 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.812138    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj" event={"ID":"f8c2e3e4-83b3-44f2-857b-331077d03271","Type":"ContainerDied","Data":"a9cb718df7f8c40e7a858f100c4ba035901fa774475c8ed8c55e89b1e091ae48"}
Sep 10 21:15:05.812216 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.812164    2581 scope.go:117] "RemoveContainer" containerID="7ef87c73871f1252feffb323c425464cdfb0367ed03129b6c926ec28c9e8bae7"
Sep 10 21:15:05.813892 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.813865    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9f6475-85fb54bc6b-m6ktj" event={"ID":"b6895de4-48ac-4813-98ce-52b92ebfa04f","Type":"ContainerStarted","Data":"2238f0b824aacd7b29ba56649daa877ae700e22d0e2d11ed68ed8df2ef959c25"}
Sep 10 21:15:05.814013 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.813952    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9f6475-85fb54bc6b-m6ktj"
Sep 10 21:15:05.815648 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.815618    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9f6475-5d7d84f979-nkxnj" event={"ID":"8a65e733-908c-4846-97b6-d6b6b4f1c094","Type":"ContainerStarted","Data":"411a8bc53f90f1072857e5284e1a9ce33dd287c039e63dfc1f93f1775291dfd1"}
Sep 10 21:15:05.815764 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.815670    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9f6475-5d7d84f979-nkxnj" event={"ID":"8a65e733-908c-4846-97b6-d6b6b4f1c094","Type":"ContainerStarted","Data":"127de3d3db57bf7fc23abe0cd625be15b8dae2f1f723d08b200cfc789d74d1af"}
Sep 10 21:15:05.815764 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.815720    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9f6475-5d7d84f979-nkxnj"
Sep 10 21:15:05.818044 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.818023    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8c2e3e4-83b3-44f2-857b-331077d03271-istiod-ca-cert\") pod \"f8c2e3e4-83b3-44f2-857b-331077d03271\" (UID: \"f8c2e3e4-83b3-44f2-857b-331077d03271\") "
Sep 10 21:15:05.818119 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.818053    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c2e3e4-83b3-44f2-857b-331077d03271-credential-socket\") pod \"f8c2e3e4-83b3-44f2-857b-331077d03271\" (UID: \"f8c2e3e4-83b3-44f2-857b-331077d03271\") "
Sep 10 21:15:05.818119 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.818073    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c2e3e4-83b3-44f2-857b-331077d03271-istio-data\") pod \"f8c2e3e4-83b3-44f2-857b-331077d03271\" (UID: \"f8c2e3e4-83b3-44f2-857b-331077d03271\") "
Sep 10 21:15:05.818119 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.818108    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8c2e3e4-83b3-44f2-857b-331077d03271-istio-envoy\") pod \"f8c2e3e4-83b3-44f2-857b-331077d03271\" (UID: \"f8c2e3e4-83b3-44f2-857b-331077d03271\") "
Sep 10 21:15:05.818251 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.818157    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8c2e3e4-83b3-44f2-857b-331077d03271-workload-certs\") pod \"f8c2e3e4-83b3-44f2-857b-331077d03271\" (UID: \"f8c2e3e4-83b3-44f2-857b-331077d03271\") "
Sep 10 21:15:05.818251 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.818198    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8c2e3e4-83b3-44f2-857b-331077d03271-istio-podinfo\") pod \"f8c2e3e4-83b3-44f2-857b-331077d03271\" (UID: \"f8c2e3e4-83b3-44f2-857b-331077d03271\") "
Sep 10 21:15:05.818251 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.818225    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8c2e3e4-83b3-44f2-857b-331077d03271-istio-token\") pod \"f8c2e3e4-83b3-44f2-857b-331077d03271\" (UID: \"f8c2e3e4-83b3-44f2-857b-331077d03271\") "
Sep 10 21:15:05.818379 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.818251    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c2e3e4-83b3-44f2-857b-331077d03271-workload-socket\") pod \"f8c2e3e4-83b3-44f2-857b-331077d03271\" (UID: \"f8c2e3e4-83b3-44f2-857b-331077d03271\") "
Sep 10 21:15:05.818379 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.818318    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbql4\" (UniqueName: \"kubernetes.io/projected/f8c2e3e4-83b3-44f2-857b-331077d03271-kube-api-access-nbql4\") pod \"f8c2e3e4-83b3-44f2-857b-331077d03271\" (UID: \"f8c2e3e4-83b3-44f2-857b-331077d03271\") "
Sep 10 21:15:05.818457 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.818426    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8c2e3e4-83b3-44f2-857b-331077d03271-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f8c2e3e4-83b3-44f2-857b-331077d03271" (UID: "f8c2e3e4-83b3-44f2-857b-331077d03271"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:15:05.818508 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.818453    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c2e3e4-83b3-44f2-857b-331077d03271-istio-data" (OuterVolumeSpecName: "istio-data") pod "f8c2e3e4-83b3-44f2-857b-331077d03271" (UID: "f8c2e3e4-83b3-44f2-857b-331077d03271"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:15:05.818694 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.818674    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8c2e3e4-83b3-44f2-857b-331077d03271-istiod-ca-cert\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:15:05.818788 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.818695    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c2e3e4-83b3-44f2-857b-331077d03271-istio-data\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:15:05.819187 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.819104    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c2e3e4-83b3-44f2-857b-331077d03271-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f8c2e3e4-83b3-44f2-857b-331077d03271" (UID: "f8c2e3e4-83b3-44f2-857b-331077d03271"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:15:05.819187 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.819154    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c2e3e4-83b3-44f2-857b-331077d03271-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f8c2e3e4-83b3-44f2-857b-331077d03271" (UID: "f8c2e3e4-83b3-44f2-857b-331077d03271"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:15:05.819517 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.819492    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c2e3e4-83b3-44f2-857b-331077d03271-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f8c2e3e4-83b3-44f2-857b-331077d03271" (UID: "f8c2e3e4-83b3-44f2-857b-331077d03271"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:15:05.821171 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.821144    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c2e3e4-83b3-44f2-857b-331077d03271-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f8c2e3e4-83b3-44f2-857b-331077d03271" (UID: "f8c2e3e4-83b3-44f2-857b-331077d03271"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:15:05.821272 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.821235    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8c2e3e4-83b3-44f2-857b-331077d03271-kube-api-access-nbql4" (OuterVolumeSpecName: "kube-api-access-nbql4") pod "f8c2e3e4-83b3-44f2-857b-331077d03271" (UID: "f8c2e3e4-83b3-44f2-857b-331077d03271"). InnerVolumeSpecName "kube-api-access-nbql4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:15:05.821471 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.821443    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8c2e3e4-83b3-44f2-857b-331077d03271-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f8c2e3e4-83b3-44f2-857b-331077d03271" (UID: "f8c2e3e4-83b3-44f2-857b-331077d03271"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 21:15:05.821471 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.821458    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8c2e3e4-83b3-44f2-857b-331077d03271-istio-token" (OuterVolumeSpecName: "istio-token") pod "f8c2e3e4-83b3-44f2-857b-331077d03271" (UID: "f8c2e3e4-83b3-44f2-857b-331077d03271"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:15:05.823482 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.823462    2581 scope.go:117] "RemoveContainer" containerID="cd8d57dee9bf2cb98b85b3dcaa952db8969156e34000240afc7ef30865650434"
Sep 10 21:15:05.833436 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.833392    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9f6475-85fb54bc6b-m6ktj" podStartSLOduration=1.8333781839999999 podStartE2EDuration="1.833378184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:15:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:15:05.831790477 +0000 UTC m=+1023.488699251" watchObservedRunningTime="2026-09-10 21:15:05.833378184 +0000 UTC m=+1023.490286955"
Sep 10 21:15:05.835452 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.835433    2581 scope.go:117] "RemoveContainer" containerID="7ef87c73871f1252feffb323c425464cdfb0367ed03129b6c926ec28c9e8bae7"
Sep 10 21:15:05.835748 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:15:05.835723    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ef87c73871f1252feffb323c425464cdfb0367ed03129b6c926ec28c9e8bae7\": container with ID starting with 7ef87c73871f1252feffb323c425464cdfb0367ed03129b6c926ec28c9e8bae7 not found: ID does not exist" containerID="7ef87c73871f1252feffb323c425464cdfb0367ed03129b6c926ec28c9e8bae7"
Sep 10 21:15:05.835815 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.835753    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ef87c73871f1252feffb323c425464cdfb0367ed03129b6c926ec28c9e8bae7"} err="failed to get container status \"7ef87c73871f1252feffb323c425464cdfb0367ed03129b6c926ec28c9e8bae7\": rpc error: code = NotFound desc = could not find container \"7ef87c73871f1252feffb323c425464cdfb0367ed03129b6c926ec28c9e8bae7\": container with ID starting with 7ef87c73871f1252feffb323c425464cdfb0367ed03129b6c926ec28c9e8bae7 not found: ID does not exist"
Sep 10 21:15:05.835815 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.835771    2581 scope.go:117] "RemoveContainer" containerID="cd8d57dee9bf2cb98b85b3dcaa952db8969156e34000240afc7ef30865650434"
Sep 10 21:15:05.836050 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:15:05.836033    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd8d57dee9bf2cb98b85b3dcaa952db8969156e34000240afc7ef30865650434\": container with ID starting with cd8d57dee9bf2cb98b85b3dcaa952db8969156e34000240afc7ef30865650434 not found: ID does not exist" containerID="cd8d57dee9bf2cb98b85b3dcaa952db8969156e34000240afc7ef30865650434"
Sep 10 21:15:05.836112 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.836057    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd8d57dee9bf2cb98b85b3dcaa952db8969156e34000240afc7ef30865650434"} err="failed to get container status \"cd8d57dee9bf2cb98b85b3dcaa952db8969156e34000240afc7ef30865650434\": rpc error: code = NotFound desc = could not find container \"cd8d57dee9bf2cb98b85b3dcaa952db8969156e34000240afc7ef30865650434\": container with ID starting with cd8d57dee9bf2cb98b85b3dcaa952db8969156e34000240afc7ef30865650434 not found: ID does not exist"
Sep 10 21:15:05.849186 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.849148    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9f6475-5d7d84f979-nkxnj" podStartSLOduration=1.8491388720000002 podStartE2EDuration="1.849138872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:15:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:15:05.84758206 +0000 UTC m=+1023.504490835" watchObservedRunningTime="2026-09-10 21:15:05.849138872 +0000 UTC m=+1023.506047644"
Sep 10 21:15:05.919722 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.919694    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8c2e3e4-83b3-44f2-857b-331077d03271-workload-certs\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:15:05.919722 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.919722    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8c2e3e4-83b3-44f2-857b-331077d03271-istio-podinfo\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:15:05.919860 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.919734    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8c2e3e4-83b3-44f2-857b-331077d03271-istio-token\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:15:05.919860 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.919749    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c2e3e4-83b3-44f2-857b-331077d03271-workload-socket\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:15:05.919860 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.919762    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbql4\" (UniqueName: \"kubernetes.io/projected/f8c2e3e4-83b3-44f2-857b-331077d03271-kube-api-access-nbql4\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:15:05.919860 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.919775    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c2e3e4-83b3-44f2-857b-331077d03271-credential-socket\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:15:05.919860 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:05.919787    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8c2e3e4-83b3-44f2-857b-331077d03271-istio-envoy\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:15:06.133670 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:06.133633    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"]
Sep 10 21:15:06.137887 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:06.137863    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-db4a68ea-openshift-default-b8bbbd9bd-ljzkj"]
Sep 10 21:15:06.868123 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:06.868087    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8c2e3e4-83b3-44f2-857b-331077d03271" path="/var/lib/kubelet/pods/f8c2e3e4-83b3-44f2-857b-331077d03271/volumes"
Sep 10 21:15:14.712191 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:14.712159    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"]
Sep 10 21:15:14.712725 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:14.712666    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8c2e3e4-83b3-44f2-857b-331077d03271" containerName="istio-proxy"
Sep 10 21:15:14.712725 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:14.712687    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8c2e3e4-83b3-44f2-857b-331077d03271" containerName="istio-proxy"
Sep 10 21:15:14.712725 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:14.712713    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8c2e3e4-83b3-44f2-857b-331077d03271" containerName="istio-proxy"
Sep 10 21:15:14.712725 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:14.712721    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8c2e3e4-83b3-44f2-857b-331077d03271" containerName="istio-proxy"
Sep 10 21:15:14.712930 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:14.712798    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8c2e3e4-83b3-44f2-857b-331077d03271" containerName="istio-proxy"
Sep 10 21:15:14.712930 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:14.712813    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8c2e3e4-83b3-44f2-857b-331077d03271" containerName="istio-proxy"
Sep 10 21:15:14.717819 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:14.717797    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"
Sep 10 21:15:14.719924 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:14.719902    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-14cee3-openshift-default-dockercfg-pqh9l\""
Sep 10 21:15:14.726511 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:14.726489    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"]
Sep 10 21:15:14.802903 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:14.802847    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07844b88-223f-4b79-983d-f33403a57ec0-istio-podinfo\") pod \"e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2\" (UID: \"07844b88-223f-4b79-983d-f33403a57ec0\") " pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"
Sep 10 21:15:14.803136 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:14.802909    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07844b88-223f-4b79-983d-f33403a57ec0-istio-token\") pod \"e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2\" (UID: \"07844b88-223f-4b79-983d-f33403a57ec0\") " pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"
Sep 10 21:15:14.803136 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:14.803024    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07844b88-223f-4b79-983d-f33403a57ec0-workload-socket\") pod \"e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2\" (UID: \"07844b88-223f-4b79-983d-f33403a57ec0\") " pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"
Sep 10 21:15:14.803136 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:14.803077    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07844b88-223f-4b79-983d-f33403a57ec0-istio-envoy\") pod \"e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2\" (UID: \"07844b88-223f-4b79-983d-f33403a57ec0\") " pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"
Sep 10 21:15:14.803136 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:14.803114    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp2sh\" (UniqueName: \"kubernetes.io/projected/07844b88-223f-4b79-983d-f33403a57ec0-kube-api-access-pp2sh\") pod \"e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2\" (UID: \"07844b88-223f-4b79-983d-f33403a57ec0\") " pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"
Sep 10 21:15:14.803712 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:14.803152    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07844b88-223f-4b79-983d-f33403a57ec0-workload-certs\") pod \"e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2\" (UID: \"07844b88-223f-4b79-983d-f33403a57ec0\") " pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"
Sep 10 21:15:14.803712 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:14.803180    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07844b88-223f-4b79-983d-f33403a57ec0-istio-data\") pod \"e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2\" (UID: \"07844b88-223f-4b79-983d-f33403a57ec0\") " pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"
Sep 10 21:15:14.803712 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:14.803210    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07844b88-223f-4b79-983d-f33403a57ec0-credential-socket\") pod \"e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2\" (UID: \"07844b88-223f-4b79-983d-f33403a57ec0\") " pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"
Sep 10 21:15:14.803712 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:14.803348    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07844b88-223f-4b79-983d-f33403a57ec0-istiod-ca-cert\") pod \"e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2\" (UID: \"07844b88-223f-4b79-983d-f33403a57ec0\") " pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"
Sep 10 21:15:14.904031 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:14.903995    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07844b88-223f-4b79-983d-f33403a57ec0-istio-podinfo\") pod \"e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2\" (UID: \"07844b88-223f-4b79-983d-f33403a57ec0\") " pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"
Sep 10 21:15:14.904031 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:14.904035    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07844b88-223f-4b79-983d-f33403a57ec0-istio-token\") pod \"e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2\" (UID: \"07844b88-223f-4b79-983d-f33403a57ec0\") " pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"
Sep 10 21:15:14.904244 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:14.904074    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07844b88-223f-4b79-983d-f33403a57ec0-workload-socket\") pod \"e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2\" (UID: \"07844b88-223f-4b79-983d-f33403a57ec0\") " pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"
Sep 10 21:15:14.904244 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:14.904105    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07844b88-223f-4b79-983d-f33403a57ec0-istio-envoy\") pod \"e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2\" (UID: \"07844b88-223f-4b79-983d-f33403a57ec0\") " pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"
Sep 10 21:15:14.904244 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:14.904131    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp2sh\" (UniqueName: \"kubernetes.io/projected/07844b88-223f-4b79-983d-f33403a57ec0-kube-api-access-pp2sh\") pod \"e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2\" (UID: \"07844b88-223f-4b79-983d-f33403a57ec0\") " pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"
Sep 10 21:15:14.904244 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:14.904165    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07844b88-223f-4b79-983d-f33403a57ec0-workload-certs\") pod \"e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2\" (UID: \"07844b88-223f-4b79-983d-f33403a57ec0\") " pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"
Sep 10 21:15:14.904577 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:14.904271    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07844b88-223f-4b79-983d-f33403a57ec0-istio-data\") pod \"e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2\" (UID: \"07844b88-223f-4b79-983d-f33403a57ec0\") " pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"
Sep 10 21:15:14.904577 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:14.904374    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07844b88-223f-4b79-983d-f33403a57ec0-credential-socket\") pod \"e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2\" (UID: \"07844b88-223f-4b79-983d-f33403a57ec0\") " pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"
Sep 10 21:15:14.904577 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:14.904458    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07844b88-223f-4b79-983d-f33403a57ec0-istiod-ca-cert\") pod \"e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2\" (UID: \"07844b88-223f-4b79-983d-f33403a57ec0\") " pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"
Sep 10 21:15:14.904741 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:14.904589    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07844b88-223f-4b79-983d-f33403a57ec0-workload-socket\") pod \"e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2\" (UID: \"07844b88-223f-4b79-983d-f33403a57ec0\") " pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"
Sep 10 21:15:14.904741 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:14.904629    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07844b88-223f-4b79-983d-f33403a57ec0-workload-certs\") pod \"e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2\" (UID: \"07844b88-223f-4b79-983d-f33403a57ec0\") " pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"
Sep 10 21:15:14.904854 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:14.904751    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07844b88-223f-4b79-983d-f33403a57ec0-istio-data\") pod \"e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2\" (UID: \"07844b88-223f-4b79-983d-f33403a57ec0\") " pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"
Sep 10 21:15:14.904971 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:14.904849    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07844b88-223f-4b79-983d-f33403a57ec0-credential-socket\") pod \"e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2\" (UID: \"07844b88-223f-4b79-983d-f33403a57ec0\") " pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"
Sep 10 21:15:14.905210 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:14.905189    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07844b88-223f-4b79-983d-f33403a57ec0-istiod-ca-cert\") pod \"e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2\" (UID: \"07844b88-223f-4b79-983d-f33403a57ec0\") " pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"
Sep 10 21:15:14.906779 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:14.906764    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07844b88-223f-4b79-983d-f33403a57ec0-istio-podinfo\") pod \"e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2\" (UID: \"07844b88-223f-4b79-983d-f33403a57ec0\") " pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"
Sep 10 21:15:14.906845 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:14.906780    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07844b88-223f-4b79-983d-f33403a57ec0-istio-envoy\") pod \"e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2\" (UID: \"07844b88-223f-4b79-983d-f33403a57ec0\") " pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"
Sep 10 21:15:14.912510 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:14.912482    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07844b88-223f-4b79-983d-f33403a57ec0-istio-token\") pod \"e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2\" (UID: \"07844b88-223f-4b79-983d-f33403a57ec0\") " pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"
Sep 10 21:15:14.912957 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:14.912937    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp2sh\" (UniqueName: \"kubernetes.io/projected/07844b88-223f-4b79-983d-f33403a57ec0-kube-api-access-pp2sh\") pod \"e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2\" (UID: \"07844b88-223f-4b79-983d-f33403a57ec0\") " pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"
Sep 10 21:15:15.030647 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:15.030566    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"
Sep 10 21:15:15.307289 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:15.307261    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"]
Sep 10 21:15:15.309346 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:15:15.309292    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07844b88_223f_4b79_983d_f33403a57ec0.slice/crio-153c5aab0aa399dda3643b1ae67e2b164bacf4526833665cc56199405b312455 WatchSource:0}: Error finding container 153c5aab0aa399dda3643b1ae67e2b164bacf4526833665cc56199405b312455: Status 404 returned error can't find the container with id 153c5aab0aa399dda3643b1ae67e2b164bacf4526833665cc56199405b312455
Sep 10 21:15:15.311544 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:15.311510    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 21:15:15.311638 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:15.311576    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 21:15:15.311638 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:15.311604    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 21:15:15.870468 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:15.870420    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2" event={"ID":"07844b88-223f-4b79-983d-f33403a57ec0","Type":"ContainerStarted","Data":"a135dbd85f6573b53b0c129e3b40551744a389d48f275ee823ae9bc6bc11c067"}
Sep 10 21:15:15.870836 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:15.870497    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2" event={"ID":"07844b88-223f-4b79-983d-f33403a57ec0","Type":"ContainerStarted","Data":"153c5aab0aa399dda3643b1ae67e2b164bacf4526833665cc56199405b312455"}
Sep 10 21:15:15.893203 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:15.893149    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2" podStartSLOduration=1.893134627 podStartE2EDuration="1.893134627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:15:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:15:15.891068213 +0000 UTC m=+1033.547976986" watchObservedRunningTime="2026-09-10 21:15:15.893134627 +0000 UTC m=+1033.550043399"
Sep 10 21:15:16.031371 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:16.031337    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"
Sep 10 21:15:16.036219 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:16.036189    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"
Sep 10 21:15:16.823449 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:16.823419    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9f6475-85fb54bc6b-m6ktj"
Sep 10 21:15:16.823644 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:16.823589    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9f6475-5d7d84f979-nkxnj"
Sep 10 21:15:16.876268 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:16.876125    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"
Sep 10 21:15:16.877411 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:16.877390    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"
Sep 10 21:15:19.646560 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:19.646524    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-14cee3-7b557b4c78-sjrs7"]
Sep 10 21:15:19.650560 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:19.650539    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-14cee3-7b557b4c78-sjrs7"
Sep 10 21:15:19.652807 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:19.652783    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-14cee3\""
Sep 10 21:15:19.661805 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:19.661784    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-14cee3-7b557b4c78-sjrs7"]
Sep 10 21:15:19.692587 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:19.692551    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-14cee3-74676b8c7f-nppg7"]
Sep 10 21:15:19.696243 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:19.696226    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-14cee3-74676b8c7f-nppg7"
Sep 10 21:15:19.698337 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:19.698309    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-14cee3-dockercfg-cd46l\""
Sep 10 21:15:19.706154 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:19.706129    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-14cee3-74676b8c7f-nppg7"]
Sep 10 21:15:19.753698 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:19.753649    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81f0bb0c-1901-4432-a48c-f97b087bfecd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-14cee3-7b557b4c78-sjrs7\" (UID: \"81f0bb0c-1901-4432-a48c-f97b087bfecd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-14cee3-7b557b4c78-sjrs7"
Sep 10 21:15:19.753950 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:19.753747    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt56k\" (UniqueName: \"kubernetes.io/projected/81f0bb0c-1901-4432-a48c-f97b087bfecd-kube-api-access-vt56k\") pod \"payload-pre-processing-e2e-shared-a-w2-14cee3-7b557b4c78-sjrs7\" (UID: \"81f0bb0c-1901-4432-a48c-f97b087bfecd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-14cee3-7b557b4c78-sjrs7"
Sep 10 21:15:19.854628 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:19.854591    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81f0bb0c-1901-4432-a48c-f97b087bfecd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-14cee3-7b557b4c78-sjrs7\" (UID: \"81f0bb0c-1901-4432-a48c-f97b087bfecd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-14cee3-7b557b4c78-sjrs7"
Sep 10 21:15:19.854818 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:19.854640    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzhlz\" (UniqueName: \"kubernetes.io/projected/e04ce7a3-c789-4c0e-835b-0421e6f12186-kube-api-access-nzhlz\") pod \"payload-processing-e2e-shared-a-w2-14cee3-74676b8c7f-nppg7\" (UID: \"e04ce7a3-c789-4c0e-835b-0421e6f12186\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-14cee3-74676b8c7f-nppg7"
Sep 10 21:15:19.854818 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:19.854669    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt56k\" (UniqueName: \"kubernetes.io/projected/81f0bb0c-1901-4432-a48c-f97b087bfecd-kube-api-access-vt56k\") pod \"payload-pre-processing-e2e-shared-a-w2-14cee3-7b557b4c78-sjrs7\" (UID: \"81f0bb0c-1901-4432-a48c-f97b087bfecd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-14cee3-7b557b4c78-sjrs7"
Sep 10 21:15:19.854818 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:19.854685    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e04ce7a3-c789-4c0e-835b-0421e6f12186-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-14cee3-74676b8c7f-nppg7\" (UID: \"e04ce7a3-c789-4c0e-835b-0421e6f12186\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-14cee3-74676b8c7f-nppg7"
Sep 10 21:15:19.855411 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:19.855390    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81f0bb0c-1901-4432-a48c-f97b087bfecd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-14cee3-7b557b4c78-sjrs7\" (UID: \"81f0bb0c-1901-4432-a48c-f97b087bfecd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-14cee3-7b557b4c78-sjrs7"
Sep 10 21:15:19.870764 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:19.870733    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt56k\" (UniqueName: \"kubernetes.io/projected/81f0bb0c-1901-4432-a48c-f97b087bfecd-kube-api-access-vt56k\") pod \"payload-pre-processing-e2e-shared-a-w2-14cee3-7b557b4c78-sjrs7\" (UID: \"81f0bb0c-1901-4432-a48c-f97b087bfecd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-14cee3-7b557b4c78-sjrs7"
Sep 10 21:15:19.956101 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:19.956001    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzhlz\" (UniqueName: \"kubernetes.io/projected/e04ce7a3-c789-4c0e-835b-0421e6f12186-kube-api-access-nzhlz\") pod \"payload-processing-e2e-shared-a-w2-14cee3-74676b8c7f-nppg7\" (UID: \"e04ce7a3-c789-4c0e-835b-0421e6f12186\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-14cee3-74676b8c7f-nppg7"
Sep 10 21:15:19.956101 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:19.956054    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e04ce7a3-c789-4c0e-835b-0421e6f12186-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-14cee3-74676b8c7f-nppg7\" (UID: \"e04ce7a3-c789-4c0e-835b-0421e6f12186\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-14cee3-74676b8c7f-nppg7"
Sep 10 21:15:19.956850 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:19.956825    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e04ce7a3-c789-4c0e-835b-0421e6f12186-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-14cee3-74676b8c7f-nppg7\" (UID: \"e04ce7a3-c789-4c0e-835b-0421e6f12186\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-14cee3-74676b8c7f-nppg7"
Sep 10 21:15:19.962040 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:19.962018    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-14cee3-7b557b4c78-sjrs7"
Sep 10 21:15:19.964077 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:19.964050    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzhlz\" (UniqueName: \"kubernetes.io/projected/e04ce7a3-c789-4c0e-835b-0421e6f12186-kube-api-access-nzhlz\") pod \"payload-processing-e2e-shared-a-w2-14cee3-74676b8c7f-nppg7\" (UID: \"e04ce7a3-c789-4c0e-835b-0421e6f12186\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-14cee3-74676b8c7f-nppg7"
Sep 10 21:15:20.007548 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:20.007513    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-14cee3-74676b8c7f-nppg7"
Sep 10 21:15:20.235974 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:20.235879    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-14cee3-7b557b4c78-sjrs7"]
Sep 10 21:15:20.239432 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:15:20.239403    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f0bb0c_1901_4432_a48c_f97b087bfecd.slice/crio-1690e86c9d49737955c29d319bfa0c7b8f24b4cefaef4fd3f3d23032b493b161 WatchSource:0}: Error finding container 1690e86c9d49737955c29d319bfa0c7b8f24b4cefaef4fd3f3d23032b493b161: Status 404 returned error can't find the container with id 1690e86c9d49737955c29d319bfa0c7b8f24b4cefaef4fd3f3d23032b493b161
Sep 10 21:15:20.297928 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:20.297902    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-14cee3-74676b8c7f-nppg7"]
Sep 10 21:15:20.300199 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:15:20.300168    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode04ce7a3_c789_4c0e_835b_0421e6f12186.slice/crio-e2f3c138aef7fadccf317f1312597b0998fe7b32777d402865aa1a31dcf692fe WatchSource:0}: Error finding container e2f3c138aef7fadccf317f1312597b0998fe7b32777d402865aa1a31dcf692fe: Status 404 returned error can't find the container with id e2f3c138aef7fadccf317f1312597b0998fe7b32777d402865aa1a31dcf692fe
Sep 10 21:15:20.907357 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:20.907317    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-14cee3-7b557b4c78-sjrs7" event={"ID":"81f0bb0c-1901-4432-a48c-f97b087bfecd","Type":"ContainerStarted","Data":"778def8e21f324aafdc6e4f6b49418694cca57073994c2fd88a1e7cce7a4de07"}
Sep 10 21:15:20.907824 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:20.907375    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-14cee3-7b557b4c78-sjrs7" event={"ID":"81f0bb0c-1901-4432-a48c-f97b087bfecd","Type":"ContainerStarted","Data":"1690e86c9d49737955c29d319bfa0c7b8f24b4cefaef4fd3f3d23032b493b161"}
Sep 10 21:15:20.907824 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:20.907478    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-14cee3-7b557b4c78-sjrs7"
Sep 10 21:15:20.908936 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:20.908903    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-14cee3-74676b8c7f-nppg7" event={"ID":"e04ce7a3-c789-4c0e-835b-0421e6f12186","Type":"ContainerStarted","Data":"880ef82277914daa113ac3fbe733fd9fb820593a3b054b77e5295ccced7e1441"}
Sep 10 21:15:20.909073 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:20.908945    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-14cee3-74676b8c7f-nppg7" event={"ID":"e04ce7a3-c789-4c0e-835b-0421e6f12186","Type":"ContainerStarted","Data":"e2f3c138aef7fadccf317f1312597b0998fe7b32777d402865aa1a31dcf692fe"}
Sep 10 21:15:20.909073 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:20.908973    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-14cee3-74676b8c7f-nppg7"
Sep 10 21:15:20.926917 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:20.926867    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-14cee3-7b557b4c78-sjrs7" podStartSLOduration=1.926853748 podStartE2EDuration="1.926853748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:15:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:15:20.924477496 +0000 UTC m=+1038.581386267" watchObservedRunningTime="2026-09-10 21:15:20.926853748 +0000 UTC m=+1038.583762519"
Sep 10 21:15:20.949892 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:20.949854    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-14cee3-74676b8c7f-nppg7" podStartSLOduration=1.949841175 podStartE2EDuration="1.949841175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:15:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:15:20.947475216 +0000 UTC m=+1038.604383987" watchObservedRunningTime="2026-09-10 21:15:20.949841175 +0000 UTC m=+1038.606749946"
Sep 10 21:15:31.916821 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:31.916788    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-14cee3-74676b8c7f-nppg7"
Sep 10 21:15:31.917221 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:31.917020    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-14cee3-7b557b4c78-sjrs7"
Sep 10 21:15:32.917475 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:32.917440    2581 patch_prober.go:28] interesting pod/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 21:15:32.917835 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:32.917509    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2" podUID="07844b88-223f-4b79-983d-f33403a57ec0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 21:15:35.011319 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:35.011267    2581 generic.go:356] "Generic (PLEG): container finished" podID="07844b88-223f-4b79-983d-f33403a57ec0" containerID="a135dbd85f6573b53b0c129e3b40551744a389d48f275ee823ae9bc6bc11c067" exitCode=0
Sep 10 21:15:35.011753 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:35.011340    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2" event={"ID":"07844b88-223f-4b79-983d-f33403a57ec0","Type":"ContainerDied","Data":"a135dbd85f6573b53b0c129e3b40551744a389d48f275ee823ae9bc6bc11c067"}
Sep 10 21:15:35.011753 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:35.011722    2581 scope.go:117] "RemoveContainer" containerID="a135dbd85f6573b53b0c129e3b40551744a389d48f275ee823ae9bc6bc11c067"
Sep 10 21:15:35.012845 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:35.012817    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 21:15:35.012903 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:35.012882    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 21:15:35.012940 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:35.012914    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 21:15:36.019968 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:36.019930    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2" event={"ID":"07844b88-223f-4b79-983d-f33403a57ec0","Type":"ContainerStarted","Data":"d1c63ce0dbe5aa172c6c50d7f1a0b483d9233e1c4255c3d33c2aa9fa008cfb3e"}
Sep 10 21:15:36.030984 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:36.030955    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"
Sep 10 21:15:36.031146 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:36.031124    2581 patch_prober.go:28] interesting pod/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Sep 10 21:15:36.031217 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:36.031180    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2" podUID="07844b88-223f-4b79-983d-f33403a57ec0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 10 21:15:37.031158 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:37.031128    2581 patch_prober.go:28] interesting pod/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Sep 10 21:15:37.031538 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:37.031186    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2" podUID="07844b88-223f-4b79-983d-f33403a57ec0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 10 21:15:38.031738 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:38.031700    2581 patch_prober.go:28] interesting pod/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Sep 10 21:15:38.032111 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:38.031755    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2" podUID="07844b88-223f-4b79-983d-f33403a57ec0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 10 21:15:39.031037 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:39.030998    2581 patch_prober.go:28] interesting pod/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Sep 10 21:15:39.031239 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:39.031071    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2" podUID="07844b88-223f-4b79-983d-f33403a57ec0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 10 21:15:40.031148 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:40.031114    2581 patch_prober.go:28] interesting pod/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Sep 10 21:15:40.031585 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:40.031172    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2" podUID="07844b88-223f-4b79-983d-f33403a57ec0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 10 21:15:41.031437 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:41.031398    2581 patch_prober.go:28] interesting pod/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Sep 10 21:15:41.031822 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:41.031461    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2" podUID="07844b88-223f-4b79-983d-f33403a57ec0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 10 21:15:42.034506 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:42.034475    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"
Sep 10 21:15:42.034929 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:42.034828    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"
Sep 10 21:15:42.035596 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:42.035578    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"
Sep 10 21:15:48.621830 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:48.621793    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-aca18f-7cd949bc79-8w99q"]
Sep 10 21:15:48.626329 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:48.626291    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-aca18f-7cd949bc79-8w99q"
Sep 10 21:15:48.629050 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:48.629031    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-aca18f-dockercfg-fdht9\""
Sep 10 21:15:48.629990 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:48.629974    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-aca18f\""
Sep 10 21:15:48.650099 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:48.650077    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-aca18f-7cd949bc79-8w99q"]
Sep 10 21:15:48.730056 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:48.730025    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c50519ad-bd6d-4d75-b6fd-2d47b5691747-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-aca18f-7cd949bc79-8w99q\" (UID: \"c50519ad-bd6d-4d75-b6fd-2d47b5691747\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-aca18f-7cd949bc79-8w99q"
Sep 10 21:15:48.730056 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:48.730061    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbtnq\" (UniqueName: \"kubernetes.io/projected/c50519ad-bd6d-4d75-b6fd-2d47b5691747-kube-api-access-zbtnq\") pod \"payload-processing-e2e-shared-b-w2-aca18f-7cd949bc79-8w99q\" (UID: \"c50519ad-bd6d-4d75-b6fd-2d47b5691747\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-aca18f-7cd949bc79-8w99q"
Sep 10 21:15:48.830765 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:48.830730    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c50519ad-bd6d-4d75-b6fd-2d47b5691747-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-aca18f-7cd949bc79-8w99q\" (UID: \"c50519ad-bd6d-4d75-b6fd-2d47b5691747\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-aca18f-7cd949bc79-8w99q"
Sep 10 21:15:48.830919 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:48.830819    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbtnq\" (UniqueName: \"kubernetes.io/projected/c50519ad-bd6d-4d75-b6fd-2d47b5691747-kube-api-access-zbtnq\") pod \"payload-processing-e2e-shared-b-w2-aca18f-7cd949bc79-8w99q\" (UID: \"c50519ad-bd6d-4d75-b6fd-2d47b5691747\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-aca18f-7cd949bc79-8w99q"
Sep 10 21:15:48.831502 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:48.831483    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c50519ad-bd6d-4d75-b6fd-2d47b5691747-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-aca18f-7cd949bc79-8w99q\" (UID: \"c50519ad-bd6d-4d75-b6fd-2d47b5691747\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-aca18f-7cd949bc79-8w99q"
Sep 10 21:15:48.840429 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:48.840400    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbtnq\" (UniqueName: \"kubernetes.io/projected/c50519ad-bd6d-4d75-b6fd-2d47b5691747-kube-api-access-zbtnq\") pod \"payload-processing-e2e-shared-b-w2-aca18f-7cd949bc79-8w99q\" (UID: \"c50519ad-bd6d-4d75-b6fd-2d47b5691747\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-aca18f-7cd949bc79-8w99q"
Sep 10 21:15:48.936264 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:48.936188    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-aca18f-7cd949bc79-8w99q"
Sep 10 21:15:49.207803 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:49.207782    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-aca18f-7cd949bc79-8w99q"]
Sep 10 21:15:49.209914 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:15:49.209885    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc50519ad_bd6d_4d75_b6fd_2d47b5691747.slice/crio-dcb5a1269ab29eaa205d70f9a8374d2287f5115e092147e282e59a9450bccd06 WatchSource:0}: Error finding container dcb5a1269ab29eaa205d70f9a8374d2287f5115e092147e282e59a9450bccd06: Status 404 returned error can't find the container with id dcb5a1269ab29eaa205d70f9a8374d2287f5115e092147e282e59a9450bccd06
Sep 10 21:15:50.098876 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:50.098837    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-aca18f-7cd949bc79-8w99q" event={"ID":"c50519ad-bd6d-4d75-b6fd-2d47b5691747","Type":"ContainerStarted","Data":"ae2321303feedee9e32ffa47768d20ed19ff8f68f2cf2ed2a8e3545262275ab9"}
Sep 10 21:15:50.099249 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:50.098892    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-aca18f-7cd949bc79-8w99q" event={"ID":"c50519ad-bd6d-4d75-b6fd-2d47b5691747","Type":"ContainerStarted","Data":"dcb5a1269ab29eaa205d70f9a8374d2287f5115e092147e282e59a9450bccd06"}
Sep 10 21:15:50.099249 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:15:50.098966    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-aca18f-7cd949bc79-8w99q"
Sep 10 21:16:01.107174 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:16:01.107098    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-aca18f-7cd949bc79-8w99q"
Sep 10 21:16:01.124199 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:16:01.124139    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-aca18f-7cd949bc79-8w99q" podStartSLOduration=13.124119206 podStartE2EDuration="13.124119206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:15:50.132212497 +0000 UTC m=+1067.789121271" watchObservedRunningTime="2026-09-10 21:16:01.124119206 +0000 UTC m=+1078.781027979"
Sep 10 21:16:02.923557 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:16:02.923522    2581 scope.go:117] "RemoveContainer" containerID="1d177ef4ef93b1ee80798e0485e6df45ee1f2a4da1937d0758eb644811bbbb24"
Sep 10 21:16:26.964619 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:16:26.964580    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-669d5f22-6469f49c86-mnv8x"]
Sep 10 21:16:26.968193 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:16:26.968172    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-669d5f22-6469f49c86-mnv8x"
Sep 10 21:16:26.970654 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:16:26.970614    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-669d5f22\""
Sep 10 21:16:26.970785 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:16:26.970674    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-669d5f22-dockercfg-wg6bl\""
Sep 10 21:16:26.981323 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:16:26.981268    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-669d5f22-6469f49c86-mnv8x"]
Sep 10 21:16:26.991766 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:16:26.991742    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-669d5f22-64447dbd57-9rqsf"]
Sep 10 21:16:26.995167 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:16:26.995152    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-669d5f22-64447dbd57-9rqsf"
Sep 10 21:16:27.005656 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:16:27.005630    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-669d5f22-64447dbd57-9rqsf"]
Sep 10 21:16:27.081363 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:16:27.081291    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3608c89-c069-44f2-b286-99b92fb4d05e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-669d5f22-6469f49c86-mnv8x\" (UID: \"a3608c89-c069-44f2-b286-99b92fb4d05e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-669d5f22-6469f49c86-mnv8x"
Sep 10 21:16:27.081563 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:16:27.081393    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s85j2\" (UniqueName: \"kubernetes.io/projected/d22ccaff-06ee-4c9d-b9e3-0fd812880238-kube-api-access-s85j2\") pod \"payload-processing-e2e-mt-669d5f22-64447dbd57-9rqsf\" (UID: \"d22ccaff-06ee-4c9d-b9e3-0fd812880238\") " pod="openshift-ingress/payload-processing-e2e-mt-669d5f22-64447dbd57-9rqsf"
Sep 10 21:16:27.081563 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:16:27.081437    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d22ccaff-06ee-4c9d-b9e3-0fd812880238-plugins-config-volume\") pod \"payload-processing-e2e-mt-669d5f22-64447dbd57-9rqsf\" (UID: \"d22ccaff-06ee-4c9d-b9e3-0fd812880238\") " pod="openshift-ingress/payload-processing-e2e-mt-669d5f22-64447dbd57-9rqsf"
Sep 10 21:16:27.081563 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:16:27.081482    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4zbj\" (UniqueName: \"kubernetes.io/projected/a3608c89-c069-44f2-b286-99b92fb4d05e-kube-api-access-c4zbj\") pod \"payload-pre-processing-e2e-mt-669d5f22-6469f49c86-mnv8x\" (UID: \"a3608c89-c069-44f2-b286-99b92fb4d05e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-669d5f22-6469f49c86-mnv8x"
Sep 10 21:16:27.182528 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:16:27.182492    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3608c89-c069-44f2-b286-99b92fb4d05e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-669d5f22-6469f49c86-mnv8x\" (UID: \"a3608c89-c069-44f2-b286-99b92fb4d05e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-669d5f22-6469f49c86-mnv8x"
Sep 10 21:16:27.182717 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:16:27.182546    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s85j2\" (UniqueName: \"kubernetes.io/projected/d22ccaff-06ee-4c9d-b9e3-0fd812880238-kube-api-access-s85j2\") pod \"payload-processing-e2e-mt-669d5f22-64447dbd57-9rqsf\" (UID: \"d22ccaff-06ee-4c9d-b9e3-0fd812880238\") " pod="openshift-ingress/payload-processing-e2e-mt-669d5f22-64447dbd57-9rqsf"
Sep 10 21:16:27.182717 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:16:27.182581    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d22ccaff-06ee-4c9d-b9e3-0fd812880238-plugins-config-volume\") pod \"payload-processing-e2e-mt-669d5f22-64447dbd57-9rqsf\" (UID: \"d22ccaff-06ee-4c9d-b9e3-0fd812880238\") " pod="openshift-ingress/payload-processing-e2e-mt-669d5f22-64447dbd57-9rqsf"
Sep 10 21:16:27.182907 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:16:27.182880    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4zbj\" (UniqueName: \"kubernetes.io/projected/a3608c89-c069-44f2-b286-99b92fb4d05e-kube-api-access-c4zbj\") pod \"payload-pre-processing-e2e-mt-669d5f22-6469f49c86-mnv8x\" (UID: \"a3608c89-c069-44f2-b286-99b92fb4d05e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-669d5f22-6469f49c86-mnv8x"
Sep 10 21:16:27.183338 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:16:27.183319    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3608c89-c069-44f2-b286-99b92fb4d05e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-669d5f22-6469f49c86-mnv8x\" (UID: \"a3608c89-c069-44f2-b286-99b92fb4d05e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-669d5f22-6469f49c86-mnv8x"
Sep 10 21:16:27.183520 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:16:27.183501    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d22ccaff-06ee-4c9d-b9e3-0fd812880238-plugins-config-volume\") pod \"payload-processing-e2e-mt-669d5f22-64447dbd57-9rqsf\" (UID: \"d22ccaff-06ee-4c9d-b9e3-0fd812880238\") " pod="openshift-ingress/payload-processing-e2e-mt-669d5f22-64447dbd57-9rqsf"
Sep 10 21:16:27.190289 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:16:27.190267    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s85j2\" (UniqueName: \"kubernetes.io/projected/d22ccaff-06ee-4c9d-b9e3-0fd812880238-kube-api-access-s85j2\") pod \"payload-processing-e2e-mt-669d5f22-64447dbd57-9rqsf\" (UID: \"d22ccaff-06ee-4c9d-b9e3-0fd812880238\") " pod="openshift-ingress/payload-processing-e2e-mt-669d5f22-64447dbd57-9rqsf"
Sep 10 21:16:27.190646 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:16:27.190418    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4zbj\" (UniqueName: \"kubernetes.io/projected/a3608c89-c069-44f2-b286-99b92fb4d05e-kube-api-access-c4zbj\") pod \"payload-pre-processing-e2e-mt-669d5f22-6469f49c86-mnv8x\" (UID: \"a3608c89-c069-44f2-b286-99b92fb4d05e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-669d5f22-6469f49c86-mnv8x"
Sep 10 21:16:27.278249 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:16:27.278159    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-669d5f22-6469f49c86-mnv8x"
Sep 10 21:16:27.308720 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:16:27.308690    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-669d5f22-64447dbd57-9rqsf"
Sep 10 21:16:27.555140 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:16:27.555112    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-669d5f22-6469f49c86-mnv8x"]
Sep 10 21:16:27.557165 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:16:27.557140    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3608c89_c069_44f2_b286_99b92fb4d05e.slice/crio-2173b47d09e1d553535e2af258e18b2f61a296e636a97e0b02c6d303b835faa2 WatchSource:0}: Error finding container 2173b47d09e1d553535e2af258e18b2f61a296e636a97e0b02c6d303b835faa2: Status 404 returned error can't find the container with id 2173b47d09e1d553535e2af258e18b2f61a296e636a97e0b02c6d303b835faa2
Sep 10 21:16:27.599548 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:16:27.599517    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-669d5f22-64447dbd57-9rqsf"]
Sep 10 21:16:27.602076 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:16:27.602046    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd22ccaff_06ee_4c9d_b9e3_0fd812880238.slice/crio-10b36042f604ef79faa4646b3534e792307a1236305efd441f8d67d40c025a25 WatchSource:0}: Error finding container 10b36042f604ef79faa4646b3534e792307a1236305efd441f8d67d40c025a25: Status 404 returned error can't find the container with id 10b36042f604ef79faa4646b3534e792307a1236305efd441f8d67d40c025a25
Sep 10 21:16:28.304592 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:16:28.304537    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-669d5f22-6469f49c86-mnv8x" event={"ID":"a3608c89-c069-44f2-b286-99b92fb4d05e","Type":"ContainerStarted","Data":"77061e99fc9f04db054b30c29c2a4fe6902dd9968c72818ab16e74c679902e94"}
Sep 10 21:16:28.305054 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:16:28.304641    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-669d5f22-6469f49c86-mnv8x" event={"ID":"a3608c89-c069-44f2-b286-99b92fb4d05e","Type":"ContainerStarted","Data":"2173b47d09e1d553535e2af258e18b2f61a296e636a97e0b02c6d303b835faa2"}
Sep 10 21:16:28.305054 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:16:28.304676    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-669d5f22-6469f49c86-mnv8x"
Sep 10 21:16:28.306484 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:16:28.306457    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-669d5f22-64447dbd57-9rqsf" event={"ID":"d22ccaff-06ee-4c9d-b9e3-0fd812880238","Type":"ContainerStarted","Data":"a29d6eb89fb1c14c70ece4c26797dfcfa93fcbf08e6e1d8ae5342bc17fd65eb3"}
Sep 10 21:16:28.306599 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:16:28.306499    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-669d5f22-64447dbd57-9rqsf" event={"ID":"d22ccaff-06ee-4c9d-b9e3-0fd812880238","Type":"ContainerStarted","Data":"10b36042f604ef79faa4646b3534e792307a1236305efd441f8d67d40c025a25"}
Sep 10 21:16:28.306599 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:16:28.306535    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-669d5f22-64447dbd57-9rqsf"
Sep 10 21:16:28.322213 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:16:28.322168    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-669d5f22-6469f49c86-mnv8x" podStartSLOduration=2.322156254 podStartE2EDuration="2.322156254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:16:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:16:28.320873234 +0000 UTC m=+1105.977782007" watchObservedRunningTime="2026-09-10 21:16:28.322156254 +0000 UTC m=+1105.979065084"
Sep 10 21:16:28.337336 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:16:28.337275    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-669d5f22-64447dbd57-9rqsf" podStartSLOduration=2.33726423 podStartE2EDuration="2.33726423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:16:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:16:28.335821624 +0000 UTC m=+1105.992730392" watchObservedRunningTime="2026-09-10 21:16:28.33726423 +0000 UTC m=+1105.994173062"
Sep 10 21:16:39.314009 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:16:39.313975    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-669d5f22-64447dbd57-9rqsf"
Sep 10 21:16:39.314500 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:16:39.314140    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-669d5f22-6469f49c86-mnv8x"
Sep 10 21:17:02.947193 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:02.947162    2581 scope.go:117] "RemoveContainer" containerID="4dac75ad2eb1c0dd4e72806ad776c3aba29941d414007e8bf8ea8d5c726347f2"
Sep 10 21:17:39.671337 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:39.671230    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-669d5f22-7h7p4"]
Sep 10 21:17:39.674962 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:39.674939    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-669d5f22-7h7p4"
Sep 10 21:17:39.677634 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:39.677604    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:17:39.677761 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:39.677667    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s4qqn\""
Sep 10 21:17:39.685998 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:39.685966    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-669d5f22-7h7p4"]
Sep 10 21:17:39.830938 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:39.830907    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/873d35f7-c728-4a94-94a4-6063f62be3d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-669d5f22-7h7p4\" (UID: \"873d35f7-c728-4a94-94a4-6063f62be3d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-669d5f22-7h7p4"
Sep 10 21:17:39.932178 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:39.932083    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/873d35f7-c728-4a94-94a4-6063f62be3d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-669d5f22-7h7p4\" (UID: \"873d35f7-c728-4a94-94a4-6063f62be3d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-669d5f22-7h7p4"
Sep 10 21:17:39.932759 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:39.932737    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/873d35f7-c728-4a94-94a4-6063f62be3d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-669d5f22-7h7p4\" (UID: \"873d35f7-c728-4a94-94a4-6063f62be3d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-669d5f22-7h7p4"
Sep 10 21:17:39.986869 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:39.986830    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-669d5f22-7h7p4"
Sep 10 21:17:40.262710 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:40.262680    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-669d5f22-7h7p4"]
Sep 10 21:17:40.265022 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:17:40.264981    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod873d35f7_c728_4a94_94a4_6063f62be3d1.slice/crio-98b5b0326dace731f3bbdc0075bb2e903dc8dba8f36ec985cf10608bdd832beb WatchSource:0}: Error finding container 98b5b0326dace731f3bbdc0075bb2e903dc8dba8f36ec985cf10608bdd832beb: Status 404 returned error can't find the container with id 98b5b0326dace731f3bbdc0075bb2e903dc8dba8f36ec985cf10608bdd832beb
Sep 10 21:17:40.699456 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:40.699419    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-669d5f22-7h7p4" event={"ID":"873d35f7-c728-4a94-94a4-6063f62be3d1","Type":"ContainerStarted","Data":"af36511b646c6aaab911f476cf7b2bc5e3a387320744f08e83e6a7436a846ba6"}
Sep 10 21:17:40.699825 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:40.699477    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-669d5f22-7h7p4" event={"ID":"873d35f7-c728-4a94-94a4-6063f62be3d1","Type":"ContainerStarted","Data":"98b5b0326dace731f3bbdc0075bb2e903dc8dba8f36ec985cf10608bdd832beb"}
Sep 10 21:17:40.721105 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:40.721057    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-669d5f22-7h7p4" podStartSLOduration=1.721043215 podStartE2EDuration="1.721043215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:17:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:17:40.720113735 +0000 UTC m=+1178.377022507" watchObservedRunningTime="2026-09-10 21:17:40.721043215 +0000 UTC m=+1178.377952033"
Sep 10 21:17:41.707354 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:41.707320    2581 generic.go:356] "Generic (PLEG): container finished" podID="873d35f7-c728-4a94-94a4-6063f62be3d1" containerID="af36511b646c6aaab911f476cf7b2bc5e3a387320744f08e83e6a7436a846ba6" exitCode=0
Sep 10 21:17:41.707857 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:41.707396    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-669d5f22-7h7p4" event={"ID":"873d35f7-c728-4a94-94a4-6063f62be3d1","Type":"ContainerDied","Data":"af36511b646c6aaab911f476cf7b2bc5e3a387320744f08e83e6a7436a846ba6"}
Sep 10 21:17:42.849284 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:42.849255    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-669d5f22-7h7p4"
Sep 10 21:17:42.961401 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:42.961371    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/873d35f7-c728-4a94-94a4-6063f62be3d1-maas-api-service-ca\") pod \"873d35f7-c728-4a94-94a4-6063f62be3d1\" (UID: \"873d35f7-c728-4a94-94a4-6063f62be3d1\") "
Sep 10 21:17:42.961787 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:42.961758    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/873d35f7-c728-4a94-94a4-6063f62be3d1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "873d35f7-c728-4a94-94a4-6063f62be3d1" (UID: "873d35f7-c728-4a94-94a4-6063f62be3d1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:17:43.062279 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:43.062257    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/873d35f7-c728-4a94-94a4-6063f62be3d1-maas-api-service-ca\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:17:43.719885 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:43.719842    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-669d5f22-7h7p4" event={"ID":"873d35f7-c728-4a94-94a4-6063f62be3d1","Type":"ContainerDied","Data":"98b5b0326dace731f3bbdc0075bb2e903dc8dba8f36ec985cf10608bdd832beb"}
Sep 10 21:17:43.720060 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:43.719905    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98b5b0326dace731f3bbdc0075bb2e903dc8dba8f36ec985cf10608bdd832beb"
Sep 10 21:17:43.720060 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:43.719839    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-669d5f22-7h7p4"
Sep 10 21:17:55.277131 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:55.277097    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-669d5f22-64447dbd57-9rqsf"]
Sep 10 21:17:55.277716 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:55.277373    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-669d5f22-64447dbd57-9rqsf" podUID="d22ccaff-06ee-4c9d-b9e3-0fd812880238" containerName="payload-processing" containerID="cri-o://a29d6eb89fb1c14c70ece4c26797dfcfa93fcbf08e6e1d8ae5342bc17fd65eb3" gracePeriod=30
Sep 10 21:17:55.287234 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:55.287204    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-669d5f22-6469f49c86-mnv8x"]
Sep 10 21:17:55.287558 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:55.287521    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-669d5f22-6469f49c86-mnv8x" podUID="a3608c89-c069-44f2-b286-99b92fb4d05e" containerName="payload-pre-processing" containerID="cri-o://77061e99fc9f04db054b30c29c2a4fe6902dd9968c72818ab16e74c679902e94" gracePeriod=30
Sep 10 21:17:55.464784 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:55.464706    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-669d5f22-7h7p4"]
Sep 10 21:17:55.467825 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:55.467794    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-669d5f22-7h7p4"]
Sep 10 21:17:55.537525 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:55.537461    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-669d5f22-64447dbd57-9rqsf"
Sep 10 21:17:55.571710 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:55.571687    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d22ccaff-06ee-4c9d-b9e3-0fd812880238-plugins-config-volume\") pod \"d22ccaff-06ee-4c9d-b9e3-0fd812880238\" (UID: \"d22ccaff-06ee-4c9d-b9e3-0fd812880238\") "
Sep 10 21:17:55.571843 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:55.571821    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s85j2\" (UniqueName: \"kubernetes.io/projected/d22ccaff-06ee-4c9d-b9e3-0fd812880238-kube-api-access-s85j2\") pod \"d22ccaff-06ee-4c9d-b9e3-0fd812880238\" (UID: \"d22ccaff-06ee-4c9d-b9e3-0fd812880238\") "
Sep 10 21:17:55.572108 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:55.572080    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d22ccaff-06ee-4c9d-b9e3-0fd812880238-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d22ccaff-06ee-4c9d-b9e3-0fd812880238" (UID: "d22ccaff-06ee-4c9d-b9e3-0fd812880238"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:17:55.573810 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:55.573791    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d22ccaff-06ee-4c9d-b9e3-0fd812880238-kube-api-access-s85j2" (OuterVolumeSpecName: "kube-api-access-s85j2") pod "d22ccaff-06ee-4c9d-b9e3-0fd812880238" (UID: "d22ccaff-06ee-4c9d-b9e3-0fd812880238"). InnerVolumeSpecName "kube-api-access-s85j2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:17:55.575047 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:55.575029    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-669d5f22-6469f49c86-mnv8x"
Sep 10 21:17:55.672407 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:55.672365    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4zbj\" (UniqueName: \"kubernetes.io/projected/a3608c89-c069-44f2-b286-99b92fb4d05e-kube-api-access-c4zbj\") pod \"a3608c89-c069-44f2-b286-99b92fb4d05e\" (UID: \"a3608c89-c069-44f2-b286-99b92fb4d05e\") "
Sep 10 21:17:55.672407 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:55.672412    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3608c89-c069-44f2-b286-99b92fb4d05e-plugins-config-volume\") pod \"a3608c89-c069-44f2-b286-99b92fb4d05e\" (UID: \"a3608c89-c069-44f2-b286-99b92fb4d05e\") "
Sep 10 21:17:55.672699 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:55.672677    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s85j2\" (UniqueName: \"kubernetes.io/projected/d22ccaff-06ee-4c9d-b9e3-0fd812880238-kube-api-access-s85j2\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:17:55.672784 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:55.672704    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d22ccaff-06ee-4c9d-b9e3-0fd812880238-plugins-config-volume\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:17:55.672852 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:55.672805    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3608c89-c069-44f2-b286-99b92fb4d05e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a3608c89-c069-44f2-b286-99b92fb4d05e" (UID: "a3608c89-c069-44f2-b286-99b92fb4d05e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:17:55.674452 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:55.674422    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3608c89-c069-44f2-b286-99b92fb4d05e-kube-api-access-c4zbj" (OuterVolumeSpecName: "kube-api-access-c4zbj") pod "a3608c89-c069-44f2-b286-99b92fb4d05e" (UID: "a3608c89-c069-44f2-b286-99b92fb4d05e"). InnerVolumeSpecName "kube-api-access-c4zbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:17:55.773586 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:55.773552    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4zbj\" (UniqueName: \"kubernetes.io/projected/a3608c89-c069-44f2-b286-99b92fb4d05e-kube-api-access-c4zbj\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:17:55.773586 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:55.773583    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3608c89-c069-44f2-b286-99b92fb4d05e-plugins-config-volume\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:17:55.793615 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:55.793587    2581 generic.go:356] "Generic (PLEG): container finished" podID="d22ccaff-06ee-4c9d-b9e3-0fd812880238" containerID="a29d6eb89fb1c14c70ece4c26797dfcfa93fcbf08e6e1d8ae5342bc17fd65eb3" exitCode=0
Sep 10 21:17:55.793763 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:55.793630    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-669d5f22-64447dbd57-9rqsf"
Sep 10 21:17:55.793763 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:55.793681    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-669d5f22-64447dbd57-9rqsf" event={"ID":"d22ccaff-06ee-4c9d-b9e3-0fd812880238","Type":"ContainerDied","Data":"a29d6eb89fb1c14c70ece4c26797dfcfa93fcbf08e6e1d8ae5342bc17fd65eb3"}
Sep 10 21:17:55.793763 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:55.793739    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-669d5f22-64447dbd57-9rqsf" event={"ID":"d22ccaff-06ee-4c9d-b9e3-0fd812880238","Type":"ContainerDied","Data":"10b36042f604ef79faa4646b3534e792307a1236305efd441f8d67d40c025a25"}
Sep 10 21:17:55.793933 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:55.793773    2581 scope.go:117] "RemoveContainer" containerID="a29d6eb89fb1c14c70ece4c26797dfcfa93fcbf08e6e1d8ae5342bc17fd65eb3"
Sep 10 21:17:55.795052 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:55.795034    2581 generic.go:356] "Generic (PLEG): container finished" podID="a3608c89-c069-44f2-b286-99b92fb4d05e" containerID="77061e99fc9f04db054b30c29c2a4fe6902dd9968c72818ab16e74c679902e94" exitCode=0
Sep 10 21:17:55.795150 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:55.795056    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-669d5f22-6469f49c86-mnv8x" event={"ID":"a3608c89-c069-44f2-b286-99b92fb4d05e","Type":"ContainerDied","Data":"77061e99fc9f04db054b30c29c2a4fe6902dd9968c72818ab16e74c679902e94"}
Sep 10 21:17:55.795150 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:55.795076    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-669d5f22-6469f49c86-mnv8x"
Sep 10 21:17:55.795150 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:55.795102    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-669d5f22-6469f49c86-mnv8x" event={"ID":"a3608c89-c069-44f2-b286-99b92fb4d05e","Type":"ContainerDied","Data":"2173b47d09e1d553535e2af258e18b2f61a296e636a97e0b02c6d303b835faa2"}
Sep 10 21:17:55.803915 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:55.803896    2581 scope.go:117] "RemoveContainer" containerID="a29d6eb89fb1c14c70ece4c26797dfcfa93fcbf08e6e1d8ae5342bc17fd65eb3"
Sep 10 21:17:55.804206 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:17:55.804182    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a29d6eb89fb1c14c70ece4c26797dfcfa93fcbf08e6e1d8ae5342bc17fd65eb3\": container with ID starting with a29d6eb89fb1c14c70ece4c26797dfcfa93fcbf08e6e1d8ae5342bc17fd65eb3 not found: ID does not exist" containerID="a29d6eb89fb1c14c70ece4c26797dfcfa93fcbf08e6e1d8ae5342bc17fd65eb3"
Sep 10 21:17:55.804331 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:55.804211    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a29d6eb89fb1c14c70ece4c26797dfcfa93fcbf08e6e1d8ae5342bc17fd65eb3"} err="failed to get container status \"a29d6eb89fb1c14c70ece4c26797dfcfa93fcbf08e6e1d8ae5342bc17fd65eb3\": rpc error: code = NotFound desc = could not find container \"a29d6eb89fb1c14c70ece4c26797dfcfa93fcbf08e6e1d8ae5342bc17fd65eb3\": container with ID starting with a29d6eb89fb1c14c70ece4c26797dfcfa93fcbf08e6e1d8ae5342bc17fd65eb3 not found: ID does not exist"
Sep 10 21:17:55.804331 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:55.804228    2581 scope.go:117] "RemoveContainer" containerID="77061e99fc9f04db054b30c29c2a4fe6902dd9968c72818ab16e74c679902e94"
Sep 10 21:17:55.812217 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:55.812200    2581 scope.go:117] "RemoveContainer" containerID="77061e99fc9f04db054b30c29c2a4fe6902dd9968c72818ab16e74c679902e94"
Sep 10 21:17:55.812475 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:17:55.812458    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77061e99fc9f04db054b30c29c2a4fe6902dd9968c72818ab16e74c679902e94\": container with ID starting with 77061e99fc9f04db054b30c29c2a4fe6902dd9968c72818ab16e74c679902e94 not found: ID does not exist" containerID="77061e99fc9f04db054b30c29c2a4fe6902dd9968c72818ab16e74c679902e94"
Sep 10 21:17:55.812549 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:55.812480    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77061e99fc9f04db054b30c29c2a4fe6902dd9968c72818ab16e74c679902e94"} err="failed to get container status \"77061e99fc9f04db054b30c29c2a4fe6902dd9968c72818ab16e74c679902e94\": rpc error: code = NotFound desc = could not find container \"77061e99fc9f04db054b30c29c2a4fe6902dd9968c72818ab16e74c679902e94\": container with ID starting with 77061e99fc9f04db054b30c29c2a4fe6902dd9968c72818ab16e74c679902e94 not found: ID does not exist"
Sep 10 21:17:55.819657 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:55.819636    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-669d5f22-6469f49c86-mnv8x"]
Sep 10 21:17:55.823424 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:55.823403    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-669d5f22-6469f49c86-mnv8x"]
Sep 10 21:17:55.833582 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:55.833561    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-669d5f22-64447dbd57-9rqsf"]
Sep 10 21:17:55.839476 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:55.839455    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-669d5f22-64447dbd57-9rqsf"]
Sep 10 21:17:56.868723 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:56.868693    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="873d35f7-c728-4a94-94a4-6063f62be3d1" path="/var/lib/kubelet/pods/873d35f7-c728-4a94-94a4-6063f62be3d1/volumes"
Sep 10 21:17:56.869105 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:56.869052    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3608c89-c069-44f2-b286-99b92fb4d05e" path="/var/lib/kubelet/pods/a3608c89-c069-44f2-b286-99b92fb4d05e/volumes"
Sep 10 21:17:56.869424 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:17:56.869409    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d22ccaff-06ee-4c9d-b9e3-0fd812880238" path="/var/lib/kubelet/pods/d22ccaff-06ee-4c9d-b9e3-0fd812880238/volumes"
Sep 10 21:18:02.852109 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:02.852080    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-z6858_c1e06766-b48c-429e-92f1-196bb083360b/console-operator/1.log"
Sep 10 21:18:02.852819 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:02.852802    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-z6858_c1e06766-b48c-429e-92f1-196bb083360b/console-operator/1.log"
Sep 10 21:18:02.855765 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:02.855746    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brrtd_19d8d7ba-45a1-4013-a935-5670abe12b8a/ovn-acl-logging/0.log"
Sep 10 21:18:02.856506 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:02.856492    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brrtd_19d8d7ba-45a1-4013-a935-5670abe12b8a/ovn-acl-logging/0.log"
Sep 10 21:18:02.967642 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:02.967612    2581 scope.go:117] "RemoveContainer" containerID="3fd4a5197e04c6514800c418422a06d4e02be7091bfee0cb1910ca0f89edcee7"
Sep 10 21:18:02.976187 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:02.976168    2581 scope.go:117] "RemoveContainer" containerID="6ea0b28c15e204316ef970323cbb74b51a8b2e29d30093848f98e85a6de9a01b"
Sep 10 21:18:45.504775 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:45.504742    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-828c7c/sub-test-model-828c7c-kserve-79f77576b5-tfpjm"]
Sep 10 21:18:45.505370 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:45.505351    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3608c89-c069-44f2-b286-99b92fb4d05e" containerName="payload-pre-processing"
Sep 10 21:18:45.505370 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:45.505372    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3608c89-c069-44f2-b286-99b92fb4d05e" containerName="payload-pre-processing"
Sep 10 21:18:45.505539 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:45.505417    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="873d35f7-c728-4a94-94a4-6063f62be3d1" containerName="revoke-keys"
Sep 10 21:18:45.505539 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:45.505426    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="873d35f7-c728-4a94-94a4-6063f62be3d1" containerName="revoke-keys"
Sep 10 21:18:45.505539 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:45.505437    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d22ccaff-06ee-4c9d-b9e3-0fd812880238" containerName="payload-processing"
Sep 10 21:18:45.505539 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:45.505446    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d22ccaff-06ee-4c9d-b9e3-0fd812880238" containerName="payload-processing"
Sep 10 21:18:45.505735 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:45.505553    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d22ccaff-06ee-4c9d-b9e3-0fd812880238" containerName="payload-processing"
Sep 10 21:18:45.505735 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:45.505567    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3608c89-c069-44f2-b286-99b92fb4d05e" containerName="payload-pre-processing"
Sep 10 21:18:45.505735 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:45.505577    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="873d35f7-c728-4a94-94a4-6063f62be3d1" containerName="revoke-keys"
Sep 10 21:18:45.508544 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:45.508524    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-828c7c/sub-test-model-828c7c-kserve-79f77576b5-tfpjm"
Sep 10 21:18:45.511436 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:45.511411    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-828c7c\"/\"default-dockercfg-5bcln\""
Sep 10 21:18:45.511583 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:45.511435    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-828c7c\"/\"kube-root-ca.crt\""
Sep 10 21:18:45.511583 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:45.511435    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-828c7c\"/\"openshift-service-ca.crt\""
Sep 10 21:18:45.511583 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:45.511411    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-828c7c\"/\"sub-test-model-828c7c-kserve-self-signed-certs\""
Sep 10 21:18:45.518711 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:45.518690    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-828c7c/sub-test-model-828c7c-kserve-79f77576b5-tfpjm"]
Sep 10 21:18:45.531900 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:45.531865    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f2c2996-38b4-4834-8bc2-7839b2c246cc-model-cache\") pod \"sub-test-model-828c7c-kserve-79f77576b5-tfpjm\" (UID: \"2f2c2996-38b4-4834-8bc2-7839b2c246cc\") " pod="ai-tenant-e2e-shared-a-w3-828c7c/sub-test-model-828c7c-kserve-79f77576b5-tfpjm"
Sep 10 21:18:45.532024 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:45.531927    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f2c2996-38b4-4834-8bc2-7839b2c246cc-dshm\") pod \"sub-test-model-828c7c-kserve-79f77576b5-tfpjm\" (UID: \"2f2c2996-38b4-4834-8bc2-7839b2c246cc\") " pod="ai-tenant-e2e-shared-a-w3-828c7c/sub-test-model-828c7c-kserve-79f77576b5-tfpjm"
Sep 10 21:18:45.532024 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:45.531967    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f2c2996-38b4-4834-8bc2-7839b2c246cc-home\") pod \"sub-test-model-828c7c-kserve-79f77576b5-tfpjm\" (UID: \"2f2c2996-38b4-4834-8bc2-7839b2c246cc\") " pod="ai-tenant-e2e-shared-a-w3-828c7c/sub-test-model-828c7c-kserve-79f77576b5-tfpjm"
Sep 10 21:18:45.532118 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:45.532017    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmz4p\" (UniqueName: \"kubernetes.io/projected/2f2c2996-38b4-4834-8bc2-7839b2c246cc-kube-api-access-tmz4p\") pod \"sub-test-model-828c7c-kserve-79f77576b5-tfpjm\" (UID: \"2f2c2996-38b4-4834-8bc2-7839b2c246cc\") " pod="ai-tenant-e2e-shared-a-w3-828c7c/sub-test-model-828c7c-kserve-79f77576b5-tfpjm"
Sep 10 21:18:45.532118 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:45.532065    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f2c2996-38b4-4834-8bc2-7839b2c246cc-tls-certs\") pod \"sub-test-model-828c7c-kserve-79f77576b5-tfpjm\" (UID: \"2f2c2996-38b4-4834-8bc2-7839b2c246cc\") " pod="ai-tenant-e2e-shared-a-w3-828c7c/sub-test-model-828c7c-kserve-79f77576b5-tfpjm"
Sep 10 21:18:45.532118 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:45.532114    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f2c2996-38b4-4834-8bc2-7839b2c246cc-tmp-dir\") pod \"sub-test-model-828c7c-kserve-79f77576b5-tfpjm\" (UID: \"2f2c2996-38b4-4834-8bc2-7839b2c246cc\") " pod="ai-tenant-e2e-shared-a-w3-828c7c/sub-test-model-828c7c-kserve-79f77576b5-tfpjm"
Sep 10 21:18:45.633334 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:45.633277    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f2c2996-38b4-4834-8bc2-7839b2c246cc-model-cache\") pod \"sub-test-model-828c7c-kserve-79f77576b5-tfpjm\" (UID: \"2f2c2996-38b4-4834-8bc2-7839b2c246cc\") " pod="ai-tenant-e2e-shared-a-w3-828c7c/sub-test-model-828c7c-kserve-79f77576b5-tfpjm"
Sep 10 21:18:45.633518 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:45.633342    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f2c2996-38b4-4834-8bc2-7839b2c246cc-dshm\") pod \"sub-test-model-828c7c-kserve-79f77576b5-tfpjm\" (UID: \"2f2c2996-38b4-4834-8bc2-7839b2c246cc\") " pod="ai-tenant-e2e-shared-a-w3-828c7c/sub-test-model-828c7c-kserve-79f77576b5-tfpjm"
Sep 10 21:18:45.633518 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:45.633362    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f2c2996-38b4-4834-8bc2-7839b2c246cc-home\") pod \"sub-test-model-828c7c-kserve-79f77576b5-tfpjm\" (UID: \"2f2c2996-38b4-4834-8bc2-7839b2c246cc\") " pod="ai-tenant-e2e-shared-a-w3-828c7c/sub-test-model-828c7c-kserve-79f77576b5-tfpjm"
Sep 10 21:18:45.633518 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:45.633383    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmz4p\" (UniqueName: \"kubernetes.io/projected/2f2c2996-38b4-4834-8bc2-7839b2c246cc-kube-api-access-tmz4p\") pod \"sub-test-model-828c7c-kserve-79f77576b5-tfpjm\" (UID: \"2f2c2996-38b4-4834-8bc2-7839b2c246cc\") " pod="ai-tenant-e2e-shared-a-w3-828c7c/sub-test-model-828c7c-kserve-79f77576b5-tfpjm"
Sep 10 21:18:45.633518 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:45.633410    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f2c2996-38b4-4834-8bc2-7839b2c246cc-tls-certs\") pod \"sub-test-model-828c7c-kserve-79f77576b5-tfpjm\" (UID: \"2f2c2996-38b4-4834-8bc2-7839b2c246cc\") " pod="ai-tenant-e2e-shared-a-w3-828c7c/sub-test-model-828c7c-kserve-79f77576b5-tfpjm"
Sep 10 21:18:45.633518 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:45.633441    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f2c2996-38b4-4834-8bc2-7839b2c246cc-tmp-dir\") pod \"sub-test-model-828c7c-kserve-79f77576b5-tfpjm\" (UID: \"2f2c2996-38b4-4834-8bc2-7839b2c246cc\") " pod="ai-tenant-e2e-shared-a-w3-828c7c/sub-test-model-828c7c-kserve-79f77576b5-tfpjm"
Sep 10 21:18:45.633793 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:45.633730    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f2c2996-38b4-4834-8bc2-7839b2c246cc-model-cache\") pod \"sub-test-model-828c7c-kserve-79f77576b5-tfpjm\" (UID: \"2f2c2996-38b4-4834-8bc2-7839b2c246cc\") " pod="ai-tenant-e2e-shared-a-w3-828c7c/sub-test-model-828c7c-kserve-79f77576b5-tfpjm"
Sep 10 21:18:45.633793 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:45.633773    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f2c2996-38b4-4834-8bc2-7839b2c246cc-home\") pod \"sub-test-model-828c7c-kserve-79f77576b5-tfpjm\" (UID: \"2f2c2996-38b4-4834-8bc2-7839b2c246cc\") " pod="ai-tenant-e2e-shared-a-w3-828c7c/sub-test-model-828c7c-kserve-79f77576b5-tfpjm"
Sep 10 21:18:45.633875 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:45.633857    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f2c2996-38b4-4834-8bc2-7839b2c246cc-tmp-dir\") pod \"sub-test-model-828c7c-kserve-79f77576b5-tfpjm\" (UID: \"2f2c2996-38b4-4834-8bc2-7839b2c246cc\") " pod="ai-tenant-e2e-shared-a-w3-828c7c/sub-test-model-828c7c-kserve-79f77576b5-tfpjm"
Sep 10 21:18:45.635704 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:45.635684    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f2c2996-38b4-4834-8bc2-7839b2c246cc-dshm\") pod \"sub-test-model-828c7c-kserve-79f77576b5-tfpjm\" (UID: \"2f2c2996-38b4-4834-8bc2-7839b2c246cc\") " pod="ai-tenant-e2e-shared-a-w3-828c7c/sub-test-model-828c7c-kserve-79f77576b5-tfpjm"
Sep 10 21:18:45.636021 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:45.635998    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f2c2996-38b4-4834-8bc2-7839b2c246cc-tls-certs\") pod \"sub-test-model-828c7c-kserve-79f77576b5-tfpjm\" (UID: \"2f2c2996-38b4-4834-8bc2-7839b2c246cc\") " pod="ai-tenant-e2e-shared-a-w3-828c7c/sub-test-model-828c7c-kserve-79f77576b5-tfpjm"
Sep 10 21:18:45.643769 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:45.643740    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmz4p\" (UniqueName: \"kubernetes.io/projected/2f2c2996-38b4-4834-8bc2-7839b2c246cc-kube-api-access-tmz4p\") pod \"sub-test-model-828c7c-kserve-79f77576b5-tfpjm\" (UID: \"2f2c2996-38b4-4834-8bc2-7839b2c246cc\") " pod="ai-tenant-e2e-shared-a-w3-828c7c/sub-test-model-828c7c-kserve-79f77576b5-tfpjm"
Sep 10 21:18:45.819119 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:45.819081    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-828c7c/sub-test-model-828c7c-kserve-79f77576b5-tfpjm"
Sep 10 21:18:46.092349 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:46.092324    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-828c7c/sub-test-model-828c7c-kserve-79f77576b5-tfpjm"]
Sep 10 21:18:46.094291 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:18:46.094263    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f2c2996_38b4_4834_8bc2_7839b2c246cc.slice/crio-8763f6702c4d49f5b4ba71a284eeba4a761542d3bbd93a4b6ef25d3efe81db7a WatchSource:0}: Error finding container 8763f6702c4d49f5b4ba71a284eeba4a761542d3bbd93a4b6ef25d3efe81db7a: Status 404 returned error can't find the container with id 8763f6702c4d49f5b4ba71a284eeba4a761542d3bbd93a4b6ef25d3efe81db7a
Sep 10 21:18:47.078410 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:47.078370    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-828c7c/sub-test-model-828c7c-kserve-79f77576b5-tfpjm" event={"ID":"2f2c2996-38b4-4834-8bc2-7839b2c246cc","Type":"ContainerStarted","Data":"3e45d767f602e55d4025470907f6f085f0e668604a0f647b4aca12fc07607e75"}
Sep 10 21:18:47.078410 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:47.078416    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-828c7c/sub-test-model-828c7c-kserve-79f77576b5-tfpjm" event={"ID":"2f2c2996-38b4-4834-8bc2-7839b2c246cc","Type":"ContainerStarted","Data":"8763f6702c4d49f5b4ba71a284eeba4a761542d3bbd93a4b6ef25d3efe81db7a"}
Sep 10 21:18:47.108094 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:47.108027    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-828c7c/sub-test-model-828c7c-kserve-79f77576b5-tfpjm" podStartSLOduration=2.108012737 podStartE2EDuration="2.108012737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:18:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:18:47.105051166 +0000 UTC m=+1244.761959937" watchObservedRunningTime="2026-09-10 21:18:47.108012737 +0000 UTC m=+1244.764921508"
Sep 10 21:18:55.820014 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:55.819924    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-828c7c/sub-test-model-828c7c-kserve-79f77576b5-tfpjm"
Sep 10 21:18:55.820421 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:55.820011    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-828c7c/sub-test-model-828c7c-kserve-79f77576b5-tfpjm"
Sep 10 21:18:55.832351 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:55.832327    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-828c7c/sub-test-model-828c7c-kserve-79f77576b5-tfpjm"
Sep 10 21:18:56.140684 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:18:56.140605    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-828c7c/sub-test-model-828c7c-kserve-79f77576b5-tfpjm"
Sep 10 21:19:03.005894 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:19:03.005857    2581 scope.go:117] "RemoveContainer" containerID="fbb2026db22a45ff4c116f34686be930b2d4e8e8a8fdaf984bbbb9886b651948"
Sep 10 21:19:03.014180 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:19:03.014162    2581 scope.go:117] "RemoveContainer" containerID="8d72ff27baa9695ed0ca69d7e37499453b24f89cd0a73e28aebbde3611dd508f"
Sep 10 21:19:53.990029 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:19:53.989992    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0778a4-68b598b54b-crlzd"]
Sep 10 21:19:53.993416 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:19:53.993395    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0778a4-68b598b54b-crlzd"
Sep 10 21:19:53.995595 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:19:53.995576    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-0778a4-dockercfg-4g7j6\""
Sep 10 21:19:53.995706 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:19:53.995608    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-0778a4\""
Sep 10 21:19:54.008417 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:19:54.008387    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0778a4-68b598b54b-crlzd"]
Sep 10 21:19:54.081241 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:19:54.081207    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzjzv\" (UniqueName: \"kubernetes.io/projected/7c30bfcc-4817-427d-b4f0-0f72ed878b55-kube-api-access-qzjzv\") pod \"payload-pre-processing-e2e-api-a-0778a4-68b598b54b-crlzd\" (UID: \"7c30bfcc-4817-427d-b4f0-0f72ed878b55\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0778a4-68b598b54b-crlzd"
Sep 10 21:19:54.081465 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:19:54.081263    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c30bfcc-4817-427d-b4f0-0f72ed878b55-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0778a4-68b598b54b-crlzd\" (UID: \"7c30bfcc-4817-427d-b4f0-0f72ed878b55\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0778a4-68b598b54b-crlzd"
Sep 10 21:19:54.182272 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:19:54.182228    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c30bfcc-4817-427d-b4f0-0f72ed878b55-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0778a4-68b598b54b-crlzd\" (UID: \"7c30bfcc-4817-427d-b4f0-0f72ed878b55\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0778a4-68b598b54b-crlzd"
Sep 10 21:19:54.182499 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:19:54.182378    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzjzv\" (UniqueName: \"kubernetes.io/projected/7c30bfcc-4817-427d-b4f0-0f72ed878b55-kube-api-access-qzjzv\") pod \"payload-pre-processing-e2e-api-a-0778a4-68b598b54b-crlzd\" (UID: \"7c30bfcc-4817-427d-b4f0-0f72ed878b55\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0778a4-68b598b54b-crlzd"
Sep 10 21:19:54.183014 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:19:54.182992    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c30bfcc-4817-427d-b4f0-0f72ed878b55-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0778a4-68b598b54b-crlzd\" (UID: \"7c30bfcc-4817-427d-b4f0-0f72ed878b55\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0778a4-68b598b54b-crlzd"
Sep 10 21:19:54.190679 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:19:54.190656    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzjzv\" (UniqueName: \"kubernetes.io/projected/7c30bfcc-4817-427d-b4f0-0f72ed878b55-kube-api-access-qzjzv\") pod \"payload-pre-processing-e2e-api-a-0778a4-68b598b54b-crlzd\" (UID: \"7c30bfcc-4817-427d-b4f0-0f72ed878b55\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0778a4-68b598b54b-crlzd"
Sep 10 21:19:54.303965 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:19:54.303930    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0778a4-68b598b54b-crlzd"
Sep 10 21:19:54.581157 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:19:54.581125    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0778a4-68b598b54b-crlzd"]
Sep 10 21:19:54.583006 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:19:54.582976    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c30bfcc_4817_427d_b4f0_0f72ed878b55.slice/crio-1a44fbd2a4be39df151c7c1fdda1e5c5140fe315c70925752feb80bb79a98590 WatchSource:0}: Error finding container 1a44fbd2a4be39df151c7c1fdda1e5c5140fe315c70925752feb80bb79a98590: Status 404 returned error can't find the container with id 1a44fbd2a4be39df151c7c1fdda1e5c5140fe315c70925752feb80bb79a98590
Sep 10 21:19:55.438857 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:19:55.438822    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0778a4-68b598b54b-crlzd" event={"ID":"7c30bfcc-4817-427d-b4f0-0f72ed878b55","Type":"ContainerStarted","Data":"4bd879d03a7aea42c1b01d9b6081077591c2b3c13a8a782508b132d8ae9d10a4"}
Sep 10 21:19:55.439234 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:19:55.438874    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0778a4-68b598b54b-crlzd" event={"ID":"7c30bfcc-4817-427d-b4f0-0f72ed878b55","Type":"ContainerStarted","Data":"1a44fbd2a4be39df151c7c1fdda1e5c5140fe315c70925752feb80bb79a98590"}
Sep 10 21:19:55.439234 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:19:55.438937    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0778a4-68b598b54b-crlzd"
Sep 10 21:19:55.458245 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:19:55.458197    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0778a4-68b598b54b-crlzd" podStartSLOduration=2.458181502 podStartE2EDuration="2.458181502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:19:55.455816498 +0000 UTC m=+1313.112725271" watchObservedRunningTime="2026-09-10 21:19:55.458181502 +0000 UTC m=+1313.115090271"
Sep 10 21:19:55.486312 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:19:55.486257    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-828c7c/sub-test-model-828c7c-kserve-79f77576b5-tfpjm"]
Sep 10 21:20:03.036033 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:03.036000    2581 scope.go:117] "RemoveContainer" containerID="17b74350ee6d639fb7cf01c31dbee24d6b918e53d79abaa6f962bf0503488267"
Sep 10 21:20:06.446687 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:06.446656    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0778a4-68b598b54b-crlzd"
Sep 10 21:20:10.506815 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:10.506765    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-828c7c/sub-test-model-828c7c-kserve-79f77576b5-tfpjm" podUID="2f2c2996-38b4-4834-8bc2-7839b2c246cc" containerName="main" containerID="cri-o://3e45d767f602e55d4025470907f6f085f0e668604a0f647b4aca12fc07607e75" gracePeriod=45
Sep 10 21:20:10.769334 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:10.769263    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-828c7c/sub-test-model-828c7c-kserve-79f77576b5-tfpjm"
Sep 10 21:20:10.840560 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:10.840520    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f2c2996-38b4-4834-8bc2-7839b2c246cc-home\") pod \"2f2c2996-38b4-4834-8bc2-7839b2c246cc\" (UID: \"2f2c2996-38b4-4834-8bc2-7839b2c246cc\") "
Sep 10 21:20:10.840730 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:10.840637    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmz4p\" (UniqueName: \"kubernetes.io/projected/2f2c2996-38b4-4834-8bc2-7839b2c246cc-kube-api-access-tmz4p\") pod \"2f2c2996-38b4-4834-8bc2-7839b2c246cc\" (UID: \"2f2c2996-38b4-4834-8bc2-7839b2c246cc\") "
Sep 10 21:20:10.840730 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:10.840658    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f2c2996-38b4-4834-8bc2-7839b2c246cc-tls-certs\") pod \"2f2c2996-38b4-4834-8bc2-7839b2c246cc\" (UID: \"2f2c2996-38b4-4834-8bc2-7839b2c246cc\") "
Sep 10 21:20:10.840730 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:10.840686    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f2c2996-38b4-4834-8bc2-7839b2c246cc-tmp-dir\") pod \"2f2c2996-38b4-4834-8bc2-7839b2c246cc\" (UID: \"2f2c2996-38b4-4834-8bc2-7839b2c246cc\") "
Sep 10 21:20:10.840922 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:10.840736    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f2c2996-38b4-4834-8bc2-7839b2c246cc-dshm\") pod \"2f2c2996-38b4-4834-8bc2-7839b2c246cc\" (UID: \"2f2c2996-38b4-4834-8bc2-7839b2c246cc\") "
Sep 10 21:20:10.840922 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:10.840766    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f2c2996-38b4-4834-8bc2-7839b2c246cc-model-cache\") pod \"2f2c2996-38b4-4834-8bc2-7839b2c246cc\" (UID: \"2f2c2996-38b4-4834-8bc2-7839b2c246cc\") "
Sep 10 21:20:10.840922 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:10.840830    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f2c2996-38b4-4834-8bc2-7839b2c246cc-home" (OuterVolumeSpecName: "home") pod "2f2c2996-38b4-4834-8bc2-7839b2c246cc" (UID: "2f2c2996-38b4-4834-8bc2-7839b2c246cc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:20:10.841088 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:10.841002    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f2c2996-38b4-4834-8bc2-7839b2c246cc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2f2c2996-38b4-4834-8bc2-7839b2c246cc" (UID: "2f2c2996-38b4-4834-8bc2-7839b2c246cc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:20:10.841088 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:10.841058    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f2c2996-38b4-4834-8bc2-7839b2c246cc-home\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:20:10.841193 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:10.841153    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f2c2996-38b4-4834-8bc2-7839b2c246cc-model-cache" (OuterVolumeSpecName: "model-cache") pod "2f2c2996-38b4-4834-8bc2-7839b2c246cc" (UID: "2f2c2996-38b4-4834-8bc2-7839b2c246cc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:20:10.842880 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:10.842848    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f2c2996-38b4-4834-8bc2-7839b2c246cc-dshm" (OuterVolumeSpecName: "dshm") pod "2f2c2996-38b4-4834-8bc2-7839b2c246cc" (UID: "2f2c2996-38b4-4834-8bc2-7839b2c246cc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:20:10.842880 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:10.842873    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f2c2996-38b4-4834-8bc2-7839b2c246cc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2f2c2996-38b4-4834-8bc2-7839b2c246cc" (UID: "2f2c2996-38b4-4834-8bc2-7839b2c246cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:20:10.843024 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:10.842943    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f2c2996-38b4-4834-8bc2-7839b2c246cc-kube-api-access-tmz4p" (OuterVolumeSpecName: "kube-api-access-tmz4p") pod "2f2c2996-38b4-4834-8bc2-7839b2c246cc" (UID: "2f2c2996-38b4-4834-8bc2-7839b2c246cc"). InnerVolumeSpecName "kube-api-access-tmz4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:20:10.942496 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:10.942457    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmz4p\" (UniqueName: \"kubernetes.io/projected/2f2c2996-38b4-4834-8bc2-7839b2c246cc-kube-api-access-tmz4p\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:20:10.942496 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:10.942492    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f2c2996-38b4-4834-8bc2-7839b2c246cc-tls-certs\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:20:10.942496 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:10.942501    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f2c2996-38b4-4834-8bc2-7839b2c246cc-tmp-dir\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:20:10.942735 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:10.942509    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f2c2996-38b4-4834-8bc2-7839b2c246cc-dshm\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:20:10.942735 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:10.942518    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f2c2996-38b4-4834-8bc2-7839b2c246cc-model-cache\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:20:11.526408 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:11.526373    2581 generic.go:356] "Generic (PLEG): container finished" podID="2f2c2996-38b4-4834-8bc2-7839b2c246cc" containerID="3e45d767f602e55d4025470907f6f085f0e668604a0f647b4aca12fc07607e75" exitCode=0
Sep 10 21:20:11.526408 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:11.526413    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-828c7c/sub-test-model-828c7c-kserve-79f77576b5-tfpjm"
Sep 10 21:20:11.526921 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:11.526460    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-828c7c/sub-test-model-828c7c-kserve-79f77576b5-tfpjm" event={"ID":"2f2c2996-38b4-4834-8bc2-7839b2c246cc","Type":"ContainerDied","Data":"3e45d767f602e55d4025470907f6f085f0e668604a0f647b4aca12fc07607e75"}
Sep 10 21:20:11.526921 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:11.526502    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-828c7c/sub-test-model-828c7c-kserve-79f77576b5-tfpjm" event={"ID":"2f2c2996-38b4-4834-8bc2-7839b2c246cc","Type":"ContainerDied","Data":"8763f6702c4d49f5b4ba71a284eeba4a761542d3bbd93a4b6ef25d3efe81db7a"}
Sep 10 21:20:11.526921 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:11.526529    2581 scope.go:117] "RemoveContainer" containerID="3e45d767f602e55d4025470907f6f085f0e668604a0f647b4aca12fc07607e75"
Sep 10 21:20:11.535165 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:11.535145    2581 scope.go:117] "RemoveContainer" containerID="3e45d767f602e55d4025470907f6f085f0e668604a0f647b4aca12fc07607e75"
Sep 10 21:20:11.535528 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:20:11.535502    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e45d767f602e55d4025470907f6f085f0e668604a0f647b4aca12fc07607e75\": container with ID starting with 3e45d767f602e55d4025470907f6f085f0e668604a0f647b4aca12fc07607e75 not found: ID does not exist" containerID="3e45d767f602e55d4025470907f6f085f0e668604a0f647b4aca12fc07607e75"
Sep 10 21:20:11.535633 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:11.535532    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e45d767f602e55d4025470907f6f085f0e668604a0f647b4aca12fc07607e75"} err="failed to get container status \"3e45d767f602e55d4025470907f6f085f0e668604a0f647b4aca12fc07607e75\": rpc error: code = NotFound desc = could not find container \"3e45d767f602e55d4025470907f6f085f0e668604a0f647b4aca12fc07607e75\": container with ID starting with 3e45d767f602e55d4025470907f6f085f0e668604a0f647b4aca12fc07607e75 not found: ID does not exist"
Sep 10 21:20:11.543426 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:11.543397    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-828c7c/sub-test-model-828c7c-kserve-79f77576b5-tfpjm"]
Sep 10 21:20:11.546804 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:11.546780    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-828c7c/sub-test-model-828c7c-kserve-79f77576b5-tfpjm"]
Sep 10 21:20:12.874599 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:12.874560    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f2c2996-38b4-4834-8bc2-7839b2c246cc" path="/var/lib/kubelet/pods/2f2c2996-38b4-4834-8bc2-7839b2c246cc/volumes"
Sep 10 21:20:18.027518 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:18.027487    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"]
Sep 10 21:20:18.027911 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:18.027897    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f2c2996-38b4-4834-8bc2-7839b2c246cc" containerName="main"
Sep 10 21:20:18.027911 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:18.027909    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f2c2996-38b4-4834-8bc2-7839b2c246cc" containerName="main"
Sep 10 21:20:18.027999 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:18.027990    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f2c2996-38b4-4834-8bc2-7839b2c246cc" containerName="main"
Sep 10 21:20:18.037779 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:18.037750    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"
Sep 10 21:20:18.040755 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:18.040717    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-806a33-openshift-default-dockercfg-4bfqv\""
Sep 10 21:20:18.044033 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:18.044006    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"]
Sep 10 21:20:18.111985 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:18.111946    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a2904f3-27e3-4ad1-8007-5bc4156ac287-credential-socket\") pod \"e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx\" (UID: \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\") " pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"
Sep 10 21:20:18.112161 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:18.112003    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a2904f3-27e3-4ad1-8007-5bc4156ac287-istio-envoy\") pod \"e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx\" (UID: \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\") " pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"
Sep 10 21:20:18.112161 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:18.112031    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a2904f3-27e3-4ad1-8007-5bc4156ac287-istio-podinfo\") pod \"e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx\" (UID: \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\") " pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"
Sep 10 21:20:18.112161 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:18.112077    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a2904f3-27e3-4ad1-8007-5bc4156ac287-istiod-ca-cert\") pod \"e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx\" (UID: \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\") " pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"
Sep 10 21:20:18.112161 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:18.112144    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a2904f3-27e3-4ad1-8007-5bc4156ac287-workload-socket\") pod \"e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx\" (UID: \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\") " pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"
Sep 10 21:20:18.112349 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:18.112179    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a2904f3-27e3-4ad1-8007-5bc4156ac287-istio-data\") pod \"e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx\" (UID: \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\") " pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"
Sep 10 21:20:18.112349 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:18.112224    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdgbc\" (UniqueName: \"kubernetes.io/projected/5a2904f3-27e3-4ad1-8007-5bc4156ac287-kube-api-access-zdgbc\") pod \"e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx\" (UID: \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\") " pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"
Sep 10 21:20:18.112349 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:18.112243    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a2904f3-27e3-4ad1-8007-5bc4156ac287-istio-token\") pod \"e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx\" (UID: \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\") " pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"
Sep 10 21:20:18.112349 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:18.112284    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a2904f3-27e3-4ad1-8007-5bc4156ac287-workload-certs\") pod \"e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx\" (UID: \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\") " pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"
Sep 10 21:20:18.213664 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:18.213624    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a2904f3-27e3-4ad1-8007-5bc4156ac287-istio-token\") pod \"e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx\" (UID: \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\") " pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"
Sep 10 21:20:18.213853 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:18.213675    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a2904f3-27e3-4ad1-8007-5bc4156ac287-workload-certs\") pod \"e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx\" (UID: \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\") " pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"
Sep 10 21:20:18.213853 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:18.213705    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a2904f3-27e3-4ad1-8007-5bc4156ac287-credential-socket\") pod \"e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx\" (UID: \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\") " pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"
Sep 10 21:20:18.213853 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:18.213724    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a2904f3-27e3-4ad1-8007-5bc4156ac287-istio-envoy\") pod \"e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx\" (UID: \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\") " pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"
Sep 10 21:20:18.213853 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:18.213746    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a2904f3-27e3-4ad1-8007-5bc4156ac287-istio-podinfo\") pod \"e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx\" (UID: \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\") " pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"
Sep 10 21:20:18.213853 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:18.213794    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a2904f3-27e3-4ad1-8007-5bc4156ac287-istiod-ca-cert\") pod \"e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx\" (UID: \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\") " pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"
Sep 10 21:20:18.213853 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:18.213816    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a2904f3-27e3-4ad1-8007-5bc4156ac287-workload-socket\") pod \"e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx\" (UID: \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\") " pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"
Sep 10 21:20:18.213853 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:18.213845    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a2904f3-27e3-4ad1-8007-5bc4156ac287-istio-data\") pod \"e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx\" (UID: \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\") " pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"
Sep 10 21:20:18.214262 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:18.213897    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdgbc\" (UniqueName: \"kubernetes.io/projected/5a2904f3-27e3-4ad1-8007-5bc4156ac287-kube-api-access-zdgbc\") pod \"e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx\" (UID: \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\") " pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"
Sep 10 21:20:18.214262 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:18.214170    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a2904f3-27e3-4ad1-8007-5bc4156ac287-credential-socket\") pod \"e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx\" (UID: \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\") " pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"
Sep 10 21:20:18.214262 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:18.214201    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a2904f3-27e3-4ad1-8007-5bc4156ac287-workload-certs\") pod \"e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx\" (UID: \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\") " pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"
Sep 10 21:20:18.214414 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:18.214344    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a2904f3-27e3-4ad1-8007-5bc4156ac287-workload-socket\") pod \"e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx\" (UID: \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\") " pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"
Sep 10 21:20:18.214469 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:18.214440    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a2904f3-27e3-4ad1-8007-5bc4156ac287-istio-data\") pod \"e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx\" (UID: \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\") " pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"
Sep 10 21:20:18.214606 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:18.214589    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a2904f3-27e3-4ad1-8007-5bc4156ac287-istiod-ca-cert\") pod \"e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx\" (UID: \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\") " pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"
Sep 10 21:20:18.216154 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:18.216134    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a2904f3-27e3-4ad1-8007-5bc4156ac287-istio-envoy\") pod \"e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx\" (UID: \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\") " pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"
Sep 10 21:20:18.216396 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:18.216380    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a2904f3-27e3-4ad1-8007-5bc4156ac287-istio-podinfo\") pod \"e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx\" (UID: \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\") " pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"
Sep 10 21:20:18.221562 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:18.221532    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a2904f3-27e3-4ad1-8007-5bc4156ac287-istio-token\") pod \"e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx\" (UID: \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\") " pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"
Sep 10 21:20:18.221655 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:18.221573    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdgbc\" (UniqueName: \"kubernetes.io/projected/5a2904f3-27e3-4ad1-8007-5bc4156ac287-kube-api-access-zdgbc\") pod \"e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx\" (UID: \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\") " pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"
Sep 10 21:20:18.351355 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:18.351322    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"
Sep 10 21:20:18.631488 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:18.631465    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"]
Sep 10 21:20:18.633152 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:20:18.633124    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a2904f3_27e3_4ad1_8007_5bc4156ac287.slice/crio-652aa9532b51502c3cede53c394a23e983fff590e24a2389c41d34f4a8b24be4 WatchSource:0}: Error finding container 652aa9532b51502c3cede53c394a23e983fff590e24a2389c41d34f4a8b24be4: Status 404 returned error can't find the container with id 652aa9532b51502c3cede53c394a23e983fff590e24a2389c41d34f4a8b24be4
Sep 10 21:20:18.635736 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:18.635693    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 21:20:18.635812 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:18.635786    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 21:20:18.635851 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:18.635831    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 21:20:19.573227 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:19.573191    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx" event={"ID":"5a2904f3-27e3-4ad1-8007-5bc4156ac287","Type":"ContainerStarted","Data":"f5cb8988a434a1605a5dee09586eaccc63cefa43ffde7901a5fac9fced6a5bc6"}
Sep 10 21:20:19.573712 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:19.573248    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx" event={"ID":"5a2904f3-27e3-4ad1-8007-5bc4156ac287","Type":"ContainerStarted","Data":"652aa9532b51502c3cede53c394a23e983fff590e24a2389c41d34f4a8b24be4"}
Sep 10 21:20:19.593431 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:19.593376    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx" podStartSLOduration=1.593358031 podStartE2EDuration="1.593358031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:20:19.591868184 +0000 UTC m=+1337.248776955" watchObservedRunningTime="2026-09-10 21:20:19.593358031 +0000 UTC m=+1337.250266803"
Sep 10 21:20:20.351725 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:20.351690    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"
Sep 10 21:20:20.356907 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:20.356881    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"
Sep 10 21:20:20.578624 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:20.578597    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"
Sep 10 21:20:20.579485 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:20.579466    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"
Sep 10 21:20:23.604919 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:23.604884    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-806a33-b6659b6f5-zlvsx"]
Sep 10 21:20:23.608556 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:23.608532    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-806a33-b6659b6f5-zlvsx"
Sep 10 21:20:23.610632 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:23.610604    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-806a33\""
Sep 10 21:20:23.610821 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:23.610808    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-806a33-dockercfg-z4vsc\""
Sep 10 21:20:23.621944 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:23.621919    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-806a33-b6659b6f5-zlvsx"]
Sep 10 21:20:23.671584 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:23.671552    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8181c395-bfce-4521-9b2c-812cb2ebc22d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-806a33-b6659b6f5-zlvsx\" (UID: \"8181c395-bfce-4521-9b2c-812cb2ebc22d\") " pod="openshift-ingress/payload-processing-e2e-api-b-806a33-b6659b6f5-zlvsx"
Sep 10 21:20:23.671744 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:23.671619    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6k4p\" (UniqueName: \"kubernetes.io/projected/8181c395-bfce-4521-9b2c-812cb2ebc22d-kube-api-access-c6k4p\") pod \"payload-processing-e2e-api-b-806a33-b6659b6f5-zlvsx\" (UID: \"8181c395-bfce-4521-9b2c-812cb2ebc22d\") " pod="openshift-ingress/payload-processing-e2e-api-b-806a33-b6659b6f5-zlvsx"
Sep 10 21:20:23.772699 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:23.772663    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8181c395-bfce-4521-9b2c-812cb2ebc22d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-806a33-b6659b6f5-zlvsx\" (UID: \"8181c395-bfce-4521-9b2c-812cb2ebc22d\") " pod="openshift-ingress/payload-processing-e2e-api-b-806a33-b6659b6f5-zlvsx"
Sep 10 21:20:23.772870 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:23.772730    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6k4p\" (UniqueName: \"kubernetes.io/projected/8181c395-bfce-4521-9b2c-812cb2ebc22d-kube-api-access-c6k4p\") pod \"payload-processing-e2e-api-b-806a33-b6659b6f5-zlvsx\" (UID: \"8181c395-bfce-4521-9b2c-812cb2ebc22d\") " pod="openshift-ingress/payload-processing-e2e-api-b-806a33-b6659b6f5-zlvsx"
Sep 10 21:20:23.773379 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:23.773355    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8181c395-bfce-4521-9b2c-812cb2ebc22d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-806a33-b6659b6f5-zlvsx\" (UID: \"8181c395-bfce-4521-9b2c-812cb2ebc22d\") " pod="openshift-ingress/payload-processing-e2e-api-b-806a33-b6659b6f5-zlvsx"
Sep 10 21:20:23.780703 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:23.780676    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6k4p\" (UniqueName: \"kubernetes.io/projected/8181c395-bfce-4521-9b2c-812cb2ebc22d-kube-api-access-c6k4p\") pod \"payload-processing-e2e-api-b-806a33-b6659b6f5-zlvsx\" (UID: \"8181c395-bfce-4521-9b2c-812cb2ebc22d\") " pod="openshift-ingress/payload-processing-e2e-api-b-806a33-b6659b6f5-zlvsx"
Sep 10 21:20:23.920183 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:23.920099    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-806a33-b6659b6f5-zlvsx"
Sep 10 21:20:24.194651 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:24.194624    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-806a33-b6659b6f5-zlvsx"]
Sep 10 21:20:24.196587 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:20:24.196560    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8181c395_bfce_4521_9b2c_812cb2ebc22d.slice/crio-92d39977697fd55538906f1a60911d5fee04e434a811e985e7dd4e4fe8e0c88d WatchSource:0}: Error finding container 92d39977697fd55538906f1a60911d5fee04e434a811e985e7dd4e4fe8e0c88d: Status 404 returned error can't find the container with id 92d39977697fd55538906f1a60911d5fee04e434a811e985e7dd4e4fe8e0c88d
Sep 10 21:20:24.601473 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:24.601431    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-806a33-b6659b6f5-zlvsx" event={"ID":"8181c395-bfce-4521-9b2c-812cb2ebc22d","Type":"ContainerStarted","Data":"f9502361630a4696eec6cf37df934b901153ce7897a95d0de875151ac6e999bd"}
Sep 10 21:20:24.601635 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:24.601502    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-806a33-b6659b6f5-zlvsx" event={"ID":"8181c395-bfce-4521-9b2c-812cb2ebc22d","Type":"ContainerStarted","Data":"92d39977697fd55538906f1a60911d5fee04e434a811e985e7dd4e4fe8e0c88d"}
Sep 10 21:20:24.601635 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:24.601533    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-806a33-b6659b6f5-zlvsx"
Sep 10 21:20:24.621278 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:24.621200    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-806a33-b6659b6f5-zlvsx" podStartSLOduration=1.621183365 podStartE2EDuration="1.621183365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:20:24.620541297 +0000 UTC m=+1342.277450072" watchObservedRunningTime="2026-09-10 21:20:24.621183365 +0000 UTC m=+1342.278092136"
Sep 10 21:20:35.609160 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:35.609125    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-806a33-b6659b6f5-zlvsx"
Sep 10 21:20:43.711085 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:43.711053    2581 generic.go:356] "Generic (PLEG): container finished" podID="5a2904f3-27e3-4ad1-8007-5bc4156ac287" containerID="f5cb8988a434a1605a5dee09586eaccc63cefa43ffde7901a5fac9fced6a5bc6" exitCode=0
Sep 10 21:20:43.711503 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:43.711125    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx" event={"ID":"5a2904f3-27e3-4ad1-8007-5bc4156ac287","Type":"ContainerDied","Data":"f5cb8988a434a1605a5dee09586eaccc63cefa43ffde7901a5fac9fced6a5bc6"}
Sep 10 21:20:43.711566 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:43.711549    2581 scope.go:117] "RemoveContainer" containerID="f5cb8988a434a1605a5dee09586eaccc63cefa43ffde7901a5fac9fced6a5bc6"
Sep 10 21:20:43.712700 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:43.712670    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 21:20:43.712778 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:43.712735    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 21:20:43.712778 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:43.712768    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 21:20:44.718744 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:44.718710    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx" event={"ID":"5a2904f3-27e3-4ad1-8007-5bc4156ac287","Type":"ContainerStarted","Data":"60eac9d5bd071901fc1d8bdc06526d026ef103bf247a836247de681d8251f5aa"}
Sep 10 21:20:45.351930 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:45.351895    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"
Sep 10 21:20:45.352123 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:45.352093    2581 patch_prober.go:28] interesting pod/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 10 21:20:45.352212 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:45.352169    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx" podUID="5a2904f3-27e3-4ad1-8007-5bc4156ac287" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 10 21:20:46.352254 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:46.352216    2581 patch_prober.go:28] interesting pod/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 10 21:20:46.352698 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:46.352290    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx" podUID="5a2904f3-27e3-4ad1-8007-5bc4156ac287" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 10 21:20:47.351732 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:47.351696    2581 patch_prober.go:28] interesting pod/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 10 21:20:47.351942 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:47.351764    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx" podUID="5a2904f3-27e3-4ad1-8007-5bc4156ac287" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 10 21:20:48.352629 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:48.352595    2581 patch_prober.go:28] interesting pod/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 10 21:20:48.353082 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:48.352655    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx" podUID="5a2904f3-27e3-4ad1-8007-5bc4156ac287" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 10 21:20:49.352400 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:49.352359    2581 patch_prober.go:28] interesting pod/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 10 21:20:49.352582 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:49.352414    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx" podUID="5a2904f3-27e3-4ad1-8007-5bc4156ac287" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 10 21:20:49.656874 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:49.656790    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0778a4-slwqk"]
Sep 10 21:20:49.659546 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:49.659522    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0778a4-slwqk"
Sep 10 21:20:49.664106 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:49.664074    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s4qqn\""
Sep 10 21:20:49.664362 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:49.664342    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:20:49.673537 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:49.673509    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0778a4-slwqk"]
Sep 10 21:20:49.730673 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:49.730637    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98a5899-3284-4c80-8dba-0dd72a2d535c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0778a4-slwqk\" (UID: \"a98a5899-3284-4c80-8dba-0dd72a2d535c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0778a4-slwqk"
Sep 10 21:20:49.831858 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:49.831819    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98a5899-3284-4c80-8dba-0dd72a2d535c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0778a4-slwqk\" (UID: \"a98a5899-3284-4c80-8dba-0dd72a2d535c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0778a4-slwqk"
Sep 10 21:20:49.832809 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:49.832788    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98a5899-3284-4c80-8dba-0dd72a2d535c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0778a4-slwqk\" (UID: \"a98a5899-3284-4c80-8dba-0dd72a2d535c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0778a4-slwqk"
Sep 10 21:20:49.976217 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:49.976138    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0778a4-slwqk"
Sep 10 21:20:50.256408 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:50.256382    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0778a4-slwqk"]
Sep 10 21:20:50.257866 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:20:50.257840    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda98a5899_3284_4c80_8dba_0dd72a2d535c.slice/crio-54e22a778df09eac4ed4344dc9db07a4f4e33f40d4f7337fde96621b00922907 WatchSource:0}: Error finding container 54e22a778df09eac4ed4344dc9db07a4f4e33f40d4f7337fde96621b00922907: Status 404 returned error can't find the container with id 54e22a778df09eac4ed4344dc9db07a4f4e33f40d4f7337fde96621b00922907
Sep 10 21:20:50.356604 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:50.356577    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"
Sep 10 21:20:50.356865 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:50.356844    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"
Sep 10 21:20:50.357598 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:50.357580    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"
Sep 10 21:20:50.750268 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:50.750233    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0778a4-slwqk" event={"ID":"a98a5899-3284-4c80-8dba-0dd72a2d535c","Type":"ContainerStarted","Data":"0b3baeafb2ca126ad0b82c79b2469f9bc94189f5c3a5ac3b12738eaf4669f2f0"}
Sep 10 21:20:50.750776 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:50.750290    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0778a4-slwqk" event={"ID":"a98a5899-3284-4c80-8dba-0dd72a2d535c","Type":"ContainerStarted","Data":"54e22a778df09eac4ed4344dc9db07a4f4e33f40d4f7337fde96621b00922907"}
Sep 10 21:20:50.766617 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:50.766568    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0778a4-slwqk" podStartSLOduration=1.766550578 podStartE2EDuration="1.766550578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:20:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:20:50.766145565 +0000 UTC m=+1368.423054337" watchObservedRunningTime="2026-09-10 21:20:50.766550578 +0000 UTC m=+1368.423459352"
Sep 10 21:20:51.756565 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:51.756531    2581 generic.go:356] "Generic (PLEG): container finished" podID="a98a5899-3284-4c80-8dba-0dd72a2d535c" containerID="0b3baeafb2ca126ad0b82c79b2469f9bc94189f5c3a5ac3b12738eaf4669f2f0" exitCode=0
Sep 10 21:20:51.756966 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:51.756623    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0778a4-slwqk" event={"ID":"a98a5899-3284-4c80-8dba-0dd72a2d535c","Type":"ContainerDied","Data":"0b3baeafb2ca126ad0b82c79b2469f9bc94189f5c3a5ac3b12738eaf4669f2f0"}
Sep 10 21:20:52.903559 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:52.903534    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0778a4-slwqk"
Sep 10 21:20:52.960797 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:52.960767    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98a5899-3284-4c80-8dba-0dd72a2d535c-maas-api-service-ca\") pod \"a98a5899-3284-4c80-8dba-0dd72a2d535c\" (UID: \"a98a5899-3284-4c80-8dba-0dd72a2d535c\") "
Sep 10 21:20:52.961096 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:52.961076    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a98a5899-3284-4c80-8dba-0dd72a2d535c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a98a5899-3284-4c80-8dba-0dd72a2d535c" (UID: "a98a5899-3284-4c80-8dba-0dd72a2d535c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:20:52.961197 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:52.961186    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98a5899-3284-4c80-8dba-0dd72a2d535c-maas-api-service-ca\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:20:53.769180 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:53.769150    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0778a4-slwqk"
Sep 10 21:20:53.769409 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:53.769202    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0778a4-slwqk" event={"ID":"a98a5899-3284-4c80-8dba-0dd72a2d535c","Type":"ContainerDied","Data":"54e22a778df09eac4ed4344dc9db07a4f4e33f40d4f7337fde96621b00922907"}
Sep 10 21:20:53.769409 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:20:53.769245    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54e22a778df09eac4ed4344dc9db07a4f4e33f40d4f7337fde96621b00922907"
Sep 10 21:21:03.056030 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:03.056001    2581 scope.go:117] "RemoveContainer" containerID="d48cdfe611d4d7b9ab08fd6bd184d57208580b894c2632787d21a0bd04fa4226"
Sep 10 21:21:05.463723 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:05.463689    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0778a4-68b598b54b-crlzd"]
Sep 10 21:21:05.464214 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:05.463896    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0778a4-68b598b54b-crlzd" podUID="7c30bfcc-4817-427d-b4f0-0f72ed878b55" containerName="payload-pre-processing" containerID="cri-o://4bd879d03a7aea42c1b01d9b6081077591c2b3c13a8a782508b132d8ae9d10a4" gracePeriod=30
Sep 10 21:21:05.611091 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:05.611056    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0778a4-slwqk"]
Sep 10 21:21:05.613127 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:05.613105    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0778a4-slwqk"]
Sep 10 21:21:05.754812 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:05.754781    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0778a4-68b598b54b-crlzd"
Sep 10 21:21:05.837147 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:05.836835    2581 generic.go:356] "Generic (PLEG): container finished" podID="7c30bfcc-4817-427d-b4f0-0f72ed878b55" containerID="4bd879d03a7aea42c1b01d9b6081077591c2b3c13a8a782508b132d8ae9d10a4" exitCode=0
Sep 10 21:21:05.837147 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:05.836926    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0778a4-68b598b54b-crlzd" event={"ID":"7c30bfcc-4817-427d-b4f0-0f72ed878b55","Type":"ContainerDied","Data":"4bd879d03a7aea42c1b01d9b6081077591c2b3c13a8a782508b132d8ae9d10a4"}
Sep 10 21:21:05.837147 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:05.837006    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0778a4-68b598b54b-crlzd" event={"ID":"7c30bfcc-4817-427d-b4f0-0f72ed878b55","Type":"ContainerDied","Data":"1a44fbd2a4be39df151c7c1fdda1e5c5140fe315c70925752feb80bb79a98590"}
Sep 10 21:21:05.837147 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:05.837040    2581 scope.go:117] "RemoveContainer" containerID="4bd879d03a7aea42c1b01d9b6081077591c2b3c13a8a782508b132d8ae9d10a4"
Sep 10 21:21:05.838140 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:05.838110    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0778a4-68b598b54b-crlzd"
Sep 10 21:21:05.847184 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:05.847165    2581 scope.go:117] "RemoveContainer" containerID="4bd879d03a7aea42c1b01d9b6081077591c2b3c13a8a782508b132d8ae9d10a4"
Sep 10 21:21:05.847531 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:21:05.847510    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bd879d03a7aea42c1b01d9b6081077591c2b3c13a8a782508b132d8ae9d10a4\": container with ID starting with 4bd879d03a7aea42c1b01d9b6081077591c2b3c13a8a782508b132d8ae9d10a4 not found: ID does not exist" containerID="4bd879d03a7aea42c1b01d9b6081077591c2b3c13a8a782508b132d8ae9d10a4"
Sep 10 21:21:05.847604 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:05.847544    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bd879d03a7aea42c1b01d9b6081077591c2b3c13a8a782508b132d8ae9d10a4"} err="failed to get container status \"4bd879d03a7aea42c1b01d9b6081077591c2b3c13a8a782508b132d8ae9d10a4\": rpc error: code = NotFound desc = could not find container \"4bd879d03a7aea42c1b01d9b6081077591c2b3c13a8a782508b132d8ae9d10a4\": container with ID starting with 4bd879d03a7aea42c1b01d9b6081077591c2b3c13a8a782508b132d8ae9d10a4 not found: ID does not exist"
Sep 10 21:21:05.884787 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:05.884756    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qzjzv\" (UniqueName: \"kubernetes.io/projected/7c30bfcc-4817-427d-b4f0-0f72ed878b55-kube-api-access-qzjzv\") pod \"7c30bfcc-4817-427d-b4f0-0f72ed878b55\" (UID: \"7c30bfcc-4817-427d-b4f0-0f72ed878b55\") "
Sep 10 21:21:05.884925 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:05.884826    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c30bfcc-4817-427d-b4f0-0f72ed878b55-plugins-config-volume\") pod \"7c30bfcc-4817-427d-b4f0-0f72ed878b55\" (UID: \"7c30bfcc-4817-427d-b4f0-0f72ed878b55\") "
Sep 10 21:21:05.885183 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:05.885154    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c30bfcc-4817-427d-b4f0-0f72ed878b55-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7c30bfcc-4817-427d-b4f0-0f72ed878b55" (UID: "7c30bfcc-4817-427d-b4f0-0f72ed878b55"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:21:05.886829 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:05.886804    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c30bfcc-4817-427d-b4f0-0f72ed878b55-kube-api-access-qzjzv" (OuterVolumeSpecName: "kube-api-access-qzjzv") pod "7c30bfcc-4817-427d-b4f0-0f72ed878b55" (UID: "7c30bfcc-4817-427d-b4f0-0f72ed878b55"). InnerVolumeSpecName "kube-api-access-qzjzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:21:05.986117 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:05.986027    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzjzv\" (UniqueName: \"kubernetes.io/projected/7c30bfcc-4817-427d-b4f0-0f72ed878b55-kube-api-access-qzjzv\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:21:05.986117 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:05.986055    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c30bfcc-4817-427d-b4f0-0f72ed878b55-plugins-config-volume\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:21:06.162791 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:06.162763    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0778a4-68b598b54b-crlzd"]
Sep 10 21:21:06.167671 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:06.167648    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0778a4-68b598b54b-crlzd"]
Sep 10 21:21:06.868465 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:06.868435    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c30bfcc-4817-427d-b4f0-0f72ed878b55" path="/var/lib/kubelet/pods/7c30bfcc-4817-427d-b4f0-0f72ed878b55/volumes"
Sep 10 21:21:06.868853 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:06.868783    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a98a5899-3284-4c80-8dba-0dd72a2d535c" path="/var/lib/kubelet/pods/a98a5899-3284-4c80-8dba-0dd72a2d535c/volumes"
Sep 10 21:21:13.206106 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:13.206071    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-806a33-7wcjk"]
Sep 10 21:21:13.206720 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:13.206701    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a98a5899-3284-4c80-8dba-0dd72a2d535c" containerName="revoke-keys"
Sep 10 21:21:13.206765 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:13.206725    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a98a5899-3284-4c80-8dba-0dd72a2d535c" containerName="revoke-keys"
Sep 10 21:21:13.206765 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:13.206741    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c30bfcc-4817-427d-b4f0-0f72ed878b55" containerName="payload-pre-processing"
Sep 10 21:21:13.206765 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:13.206749    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c30bfcc-4817-427d-b4f0-0f72ed878b55" containerName="payload-pre-processing"
Sep 10 21:21:13.206861 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:13.206844    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c30bfcc-4817-427d-b4f0-0f72ed878b55" containerName="payload-pre-processing"
Sep 10 21:21:13.206895 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:13.206861    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a98a5899-3284-4c80-8dba-0dd72a2d535c" containerName="revoke-keys"
Sep 10 21:21:13.209977 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:13.209957    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-806a33-7wcjk"
Sep 10 21:21:13.212460 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:13.212428    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:21:13.212608 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:13.212482    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s4qqn\""
Sep 10 21:21:13.216218 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:13.216184    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-806a33-7wcjk"]
Sep 10 21:21:13.370583 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:13.370540    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58e8f2a7-e33b-4ed4-9840-c64766a41818-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-806a33-7wcjk\" (UID: \"58e8f2a7-e33b-4ed4-9840-c64766a41818\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-806a33-7wcjk"
Sep 10 21:21:13.471502 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:13.471411    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58e8f2a7-e33b-4ed4-9840-c64766a41818-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-806a33-7wcjk\" (UID: \"58e8f2a7-e33b-4ed4-9840-c64766a41818\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-806a33-7wcjk"
Sep 10 21:21:13.472406 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:13.472381    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58e8f2a7-e33b-4ed4-9840-c64766a41818-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-806a33-7wcjk\" (UID: \"58e8f2a7-e33b-4ed4-9840-c64766a41818\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-806a33-7wcjk"
Sep 10 21:21:13.522103 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:13.522069    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-806a33-7wcjk"
Sep 10 21:21:13.803507 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:13.803484    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-806a33-7wcjk"]
Sep 10 21:21:13.805792 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:21:13.805766    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58e8f2a7_e33b_4ed4_9840_c64766a41818.slice/crio-2733096a29fa9b8f556177086b192e18ef63b7507739238765e8d4992f157c43 WatchSource:0}: Error finding container 2733096a29fa9b8f556177086b192e18ef63b7507739238765e8d4992f157c43: Status 404 returned error can't find the container with id 2733096a29fa9b8f556177086b192e18ef63b7507739238765e8d4992f157c43
Sep 10 21:21:13.888983 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:13.888946    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-806a33-7wcjk" event={"ID":"58e8f2a7-e33b-4ed4-9840-c64766a41818","Type":"ContainerStarted","Data":"2733096a29fa9b8f556177086b192e18ef63b7507739238765e8d4992f157c43"}
Sep 10 21:21:14.895375 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:14.895338    2581 generic.go:356] "Generic (PLEG): container finished" podID="58e8f2a7-e33b-4ed4-9840-c64766a41818" containerID="261e614a11c1ab65284dc9c83e5086e3908e8a44a57f86ab1d4e39b126139aa8" exitCode=0
Sep 10 21:21:14.895764 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:14.895401    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-806a33-7wcjk" event={"ID":"58e8f2a7-e33b-4ed4-9840-c64766a41818","Type":"ContainerDied","Data":"261e614a11c1ab65284dc9c83e5086e3908e8a44a57f86ab1d4e39b126139aa8"}
Sep 10 21:21:16.042836 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:16.042810    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-806a33-7wcjk"
Sep 10 21:21:16.107318 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:16.107255    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58e8f2a7-e33b-4ed4-9840-c64766a41818-maas-api-service-ca\") pod \"58e8f2a7-e33b-4ed4-9840-c64766a41818\" (UID: \"58e8f2a7-e33b-4ed4-9840-c64766a41818\") "
Sep 10 21:21:16.107609 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:16.107583    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58e8f2a7-e33b-4ed4-9840-c64766a41818-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "58e8f2a7-e33b-4ed4-9840-c64766a41818" (UID: "58e8f2a7-e33b-4ed4-9840-c64766a41818"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:21:16.208833 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:16.208756    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/58e8f2a7-e33b-4ed4-9840-c64766a41818-maas-api-service-ca\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:21:16.915192 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:16.915161    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-806a33-7wcjk"
Sep 10 21:21:16.915396 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:16.915162    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-806a33-7wcjk" event={"ID":"58e8f2a7-e33b-4ed4-9840-c64766a41818","Type":"ContainerDied","Data":"2733096a29fa9b8f556177086b192e18ef63b7507739238765e8d4992f157c43"}
Sep 10 21:21:16.915396 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:16.915292    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2733096a29fa9b8f556177086b192e18ef63b7507739238765e8d4992f157c43"
Sep 10 21:21:28.739445 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:28.739410    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-806a33-b6659b6f5-zlvsx"]
Sep 10 21:21:28.739985 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:28.739633    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-806a33-b6659b6f5-zlvsx" podUID="8181c395-bfce-4521-9b2c-812cb2ebc22d" containerName="payload-processing" containerID="cri-o://f9502361630a4696eec6cf37df934b901153ce7897a95d0de875151ac6e999bd" gracePeriod=30
Sep 10 21:21:28.921492 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:28.921458    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-806a33-7wcjk"]
Sep 10 21:21:28.923458 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:28.923435    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-806a33-7wcjk"]
Sep 10 21:21:28.992542 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:28.991992    2581 generic.go:356] "Generic (PLEG): container finished" podID="8181c395-bfce-4521-9b2c-812cb2ebc22d" containerID="f9502361630a4696eec6cf37df934b901153ce7897a95d0de875151ac6e999bd" exitCode=0
Sep 10 21:21:28.992542 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:28.992156    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-806a33-b6659b6f5-zlvsx" event={"ID":"8181c395-bfce-4521-9b2c-812cb2ebc22d","Type":"ContainerDied","Data":"f9502361630a4696eec6cf37df934b901153ce7897a95d0de875151ac6e999bd"}
Sep 10 21:21:28.992542 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:28.992214    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-806a33-b6659b6f5-zlvsx" event={"ID":"8181c395-bfce-4521-9b2c-812cb2ebc22d","Type":"ContainerDied","Data":"92d39977697fd55538906f1a60911d5fee04e434a811e985e7dd4e4fe8e0c88d"}
Sep 10 21:21:28.992542 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:28.992241    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92d39977697fd55538906f1a60911d5fee04e434a811e985e7dd4e4fe8e0c88d"
Sep 10 21:21:29.014524 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:29.014226    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-806a33-b6659b6f5-zlvsx"
Sep 10 21:21:29.128288 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:29.128253    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6k4p\" (UniqueName: \"kubernetes.io/projected/8181c395-bfce-4521-9b2c-812cb2ebc22d-kube-api-access-c6k4p\") pod \"8181c395-bfce-4521-9b2c-812cb2ebc22d\" (UID: \"8181c395-bfce-4521-9b2c-812cb2ebc22d\") "
Sep 10 21:21:29.128478 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:29.128326    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8181c395-bfce-4521-9b2c-812cb2ebc22d-plugins-config-volume\") pod \"8181c395-bfce-4521-9b2c-812cb2ebc22d\" (UID: \"8181c395-bfce-4521-9b2c-812cb2ebc22d\") "
Sep 10 21:21:29.128767 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:29.128741    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8181c395-bfce-4521-9b2c-812cb2ebc22d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8181c395-bfce-4521-9b2c-812cb2ebc22d" (UID: "8181c395-bfce-4521-9b2c-812cb2ebc22d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:21:29.130437 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:29.130417    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8181c395-bfce-4521-9b2c-812cb2ebc22d-kube-api-access-c6k4p" (OuterVolumeSpecName: "kube-api-access-c6k4p") pod "8181c395-bfce-4521-9b2c-812cb2ebc22d" (UID: "8181c395-bfce-4521-9b2c-812cb2ebc22d"). InnerVolumeSpecName "kube-api-access-c6k4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:21:29.229054 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:29.229026    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6k4p\" (UniqueName: \"kubernetes.io/projected/8181c395-bfce-4521-9b2c-812cb2ebc22d-kube-api-access-c6k4p\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:21:29.229054 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:29.229051    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8181c395-bfce-4521-9b2c-812cb2ebc22d-plugins-config-volume\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:21:29.997781 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:29.997747    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-806a33-b6659b6f5-zlvsx"
Sep 10 21:21:30.020438 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:30.020407    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-806a33-b6659b6f5-zlvsx"]
Sep 10 21:21:30.026004 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:30.025968    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-806a33-b6659b6f5-zlvsx"]
Sep 10 21:21:30.873615 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:30.873580    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58e8f2a7-e33b-4ed4-9840-c64766a41818" path="/var/lib/kubelet/pods/58e8f2a7-e33b-4ed4-9840-c64766a41818/volumes"
Sep 10 21:21:30.873969 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:30.873956    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8181c395-bfce-4521-9b2c-812cb2ebc22d" path="/var/lib/kubelet/pods/8181c395-bfce-4521-9b2c-812cb2ebc22d/volumes"
Sep 10 21:21:35.968129 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:35.968094    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"]
Sep 10 21:21:35.968577 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:35.968329    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx" podUID="5a2904f3-27e3-4ad1-8007-5bc4156ac287" containerName="istio-proxy" containerID="cri-o://60eac9d5bd071901fc1d8bdc06526d026ef103bf247a836247de681d8251f5aa" gracePeriod=30
Sep 10 21:21:36.343772 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:36.343735    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-14cee3-llxxp"]
Sep 10 21:21:36.344206 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:36.344189    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8181c395-bfce-4521-9b2c-812cb2ebc22d" containerName="payload-processing"
Sep 10 21:21:36.344274 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:36.344210    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8181c395-bfce-4521-9b2c-812cb2ebc22d" containerName="payload-processing"
Sep 10 21:21:36.344274 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:36.344230    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58e8f2a7-e33b-4ed4-9840-c64766a41818" containerName="revoke-keys"
Sep 10 21:21:36.344274 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:36.344235    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="58e8f2a7-e33b-4ed4-9840-c64766a41818" containerName="revoke-keys"
Sep 10 21:21:36.344416 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:36.344327    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="58e8f2a7-e33b-4ed4-9840-c64766a41818" containerName="revoke-keys"
Sep 10 21:21:36.344416 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:36.344337    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8181c395-bfce-4521-9b2c-812cb2ebc22d" containerName="payload-processing"
Sep 10 21:21:36.347489 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:36.347472    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-14cee3-llxxp"
Sep 10 21:21:36.349928 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:36.349910    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s4qqn\""
Sep 10 21:21:36.349995 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:36.349910    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:21:36.355566 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:36.355541    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-14cee3-llxxp"]
Sep 10 21:21:36.397781 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:36.397749    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81673b14-17da-4701-b7b7-4ca5d05a536f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-14cee3-llxxp\" (UID: \"81673b14-17da-4701-b7b7-4ca5d05a536f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-14cee3-llxxp"
Sep 10 21:21:36.499175 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:36.499140    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81673b14-17da-4701-b7b7-4ca5d05a536f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-14cee3-llxxp\" (UID: \"81673b14-17da-4701-b7b7-4ca5d05a536f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-14cee3-llxxp"
Sep 10 21:21:36.499773 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:36.499753    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81673b14-17da-4701-b7b7-4ca5d05a536f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-14cee3-llxxp\" (UID: \"81673b14-17da-4701-b7b7-4ca5d05a536f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-14cee3-llxxp"
Sep 10 21:21:36.658626 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:36.658542    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-14cee3-llxxp"
Sep 10 21:21:36.931922 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:36.931898    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-14cee3-llxxp"]
Sep 10 21:21:36.934114 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:21:36.934084    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81673b14_17da_4701_b7b7_4ca5d05a536f.slice/crio-55945915aa22e46978d806762c2027fbe205daa87add1a1216e0dfc34a48c732 WatchSource:0}: Error finding container 55945915aa22e46978d806762c2027fbe205daa87add1a1216e0dfc34a48c732: Status 404 returned error can't find the container with id 55945915aa22e46978d806762c2027fbe205daa87add1a1216e0dfc34a48c732
Sep 10 21:21:37.034040 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:37.034002    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-14cee3-llxxp" event={"ID":"81673b14-17da-4701-b7b7-4ca5d05a536f","Type":"ContainerStarted","Data":"55945915aa22e46978d806762c2027fbe205daa87add1a1216e0dfc34a48c732"}
Sep 10 21:21:38.004223 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:38.004139    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f7eb29-56d4bb5998-bzd2r"]
Sep 10 21:21:38.007584 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:38.007564    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7eb29-56d4bb5998-bzd2r"
Sep 10 21:21:38.009698 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:38.009670    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f7eb29\""
Sep 10 21:21:38.019586 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:38.019563    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f7eb29-56d4bb5998-bzd2r"]
Sep 10 21:21:38.042200 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:38.042170    2581 generic.go:356] "Generic (PLEG): container finished" podID="81673b14-17da-4701-b7b7-4ca5d05a536f" containerID="dd973375df5663608a410d1914c690888f9239aaa014c88ee76aaafbedf26c14" exitCode=0
Sep 10 21:21:38.042681 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:38.042249    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-14cee3-llxxp" event={"ID":"81673b14-17da-4701-b7b7-4ca5d05a536f","Type":"ContainerDied","Data":"dd973375df5663608a410d1914c690888f9239aaa014c88ee76aaafbedf26c14"}
Sep 10 21:21:38.114145 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:38.114104    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfbf31da-aee8-46bf-8c52-4e8b15be7743-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f7eb29-56d4bb5998-bzd2r\" (UID: \"cfbf31da-aee8-46bf-8c52-4e8b15be7743\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7eb29-56d4bb5998-bzd2r"
Sep 10 21:21:38.114343 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:38.114177    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lphj\" (UniqueName: \"kubernetes.io/projected/cfbf31da-aee8-46bf-8c52-4e8b15be7743-kube-api-access-8lphj\") pod \"payload-pre-processing-e2e-ipp-a-f7eb29-56d4bb5998-bzd2r\" (UID: \"cfbf31da-aee8-46bf-8c52-4e8b15be7743\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7eb29-56d4bb5998-bzd2r"
Sep 10 21:21:38.214913 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:38.214873    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfbf31da-aee8-46bf-8c52-4e8b15be7743-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f7eb29-56d4bb5998-bzd2r\" (UID: \"cfbf31da-aee8-46bf-8c52-4e8b15be7743\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7eb29-56d4bb5998-bzd2r"
Sep 10 21:21:38.215085 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:38.215045    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lphj\" (UniqueName: \"kubernetes.io/projected/cfbf31da-aee8-46bf-8c52-4e8b15be7743-kube-api-access-8lphj\") pod \"payload-pre-processing-e2e-ipp-a-f7eb29-56d4bb5998-bzd2r\" (UID: \"cfbf31da-aee8-46bf-8c52-4e8b15be7743\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7eb29-56d4bb5998-bzd2r"
Sep 10 21:21:38.215694 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:38.215668    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfbf31da-aee8-46bf-8c52-4e8b15be7743-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f7eb29-56d4bb5998-bzd2r\" (UID: \"cfbf31da-aee8-46bf-8c52-4e8b15be7743\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7eb29-56d4bb5998-bzd2r"
Sep 10 21:21:38.223389 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:38.223363    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lphj\" (UniqueName: \"kubernetes.io/projected/cfbf31da-aee8-46bf-8c52-4e8b15be7743-kube-api-access-8lphj\") pod \"payload-pre-processing-e2e-ipp-a-f7eb29-56d4bb5998-bzd2r\" (UID: \"cfbf31da-aee8-46bf-8c52-4e8b15be7743\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7eb29-56d4bb5998-bzd2r"
Sep 10 21:21:38.319040 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:38.319008    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7eb29-56d4bb5998-bzd2r"
Sep 10 21:21:38.591928 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:38.591902    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f7eb29-56d4bb5998-bzd2r"]
Sep 10 21:21:38.593673 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:21:38.593645    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfbf31da_aee8_46bf_8c52_4e8b15be7743.slice/crio-fcf2d5883c657a39150e571b2fd3eacfd618f9ab7b7cd85134b8cf055a2cbd81 WatchSource:0}: Error finding container fcf2d5883c657a39150e571b2fd3eacfd618f9ab7b7cd85134b8cf055a2cbd81: Status 404 returned error can't find the container with id fcf2d5883c657a39150e571b2fd3eacfd618f9ab7b7cd85134b8cf055a2cbd81
Sep 10 21:21:39.050371 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:39.050334    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7eb29-56d4bb5998-bzd2r" event={"ID":"cfbf31da-aee8-46bf-8c52-4e8b15be7743","Type":"ContainerStarted","Data":"a86142f81b31aa8bf8748247d5f5e1f42bc5002d95bcb663348f58fe02ddceaf"}
Sep 10 21:21:39.050788 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:39.050397    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7eb29-56d4bb5998-bzd2r" event={"ID":"cfbf31da-aee8-46bf-8c52-4e8b15be7743","Type":"ContainerStarted","Data":"fcf2d5883c657a39150e571b2fd3eacfd618f9ab7b7cd85134b8cf055a2cbd81"}
Sep 10 21:21:39.050788 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:39.050427    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7eb29-56d4bb5998-bzd2r"
Sep 10 21:21:39.067662 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:39.067615    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7eb29-56d4bb5998-bzd2r" podStartSLOduration=2.067596373 podStartE2EDuration="2.067596373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:21:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:21:39.065869935 +0000 UTC m=+1416.722778708" watchObservedRunningTime="2026-09-10 21:21:39.067596373 +0000 UTC m=+1416.724505145"
Sep 10 21:21:39.188369 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:39.188341    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-14cee3-llxxp"
Sep 10 21:21:39.225082 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:39.225048    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81673b14-17da-4701-b7b7-4ca5d05a536f-maas-api-service-ca\") pod \"81673b14-17da-4701-b7b7-4ca5d05a536f\" (UID: \"81673b14-17da-4701-b7b7-4ca5d05a536f\") "
Sep 10 21:21:39.225490 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:39.225459    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81673b14-17da-4701-b7b7-4ca5d05a536f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "81673b14-17da-4701-b7b7-4ca5d05a536f" (UID: "81673b14-17da-4701-b7b7-4ca5d05a536f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:21:39.326394 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:39.326287    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81673b14-17da-4701-b7b7-4ca5d05a536f-maas-api-service-ca\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:21:40.057061 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:40.057031    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-14cee3-llxxp"
Sep 10 21:21:40.057548 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:40.057066    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-14cee3-llxxp" event={"ID":"81673b14-17da-4701-b7b7-4ca5d05a536f","Type":"ContainerDied","Data":"55945915aa22e46978d806762c2027fbe205daa87add1a1216e0dfc34a48c732"}
Sep 10 21:21:40.057548 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:40.057115    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55945915aa22e46978d806762c2027fbe205daa87add1a1216e0dfc34a48c732"
Sep 10 21:21:41.229110 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:41.229058    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"
Sep 10 21:21:41.349549 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:41.349517    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a2904f3-27e3-4ad1-8007-5bc4156ac287-credential-socket\") pod \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\" (UID: \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\") "
Sep 10 21:21:41.349727 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:41.349558    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a2904f3-27e3-4ad1-8007-5bc4156ac287-istio-podinfo\") pod \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\" (UID: \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\") "
Sep 10 21:21:41.349727 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:41.349576    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a2904f3-27e3-4ad1-8007-5bc4156ac287-workload-certs\") pod \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\" (UID: \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\") "
Sep 10 21:21:41.349727 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:41.349616    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a2904f3-27e3-4ad1-8007-5bc4156ac287-istio-envoy\") pod \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\" (UID: \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\") "
Sep 10 21:21:41.349727 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:41.349654    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a2904f3-27e3-4ad1-8007-5bc4156ac287-istiod-ca-cert\") pod \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\" (UID: \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\") "
Sep 10 21:21:41.349727 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:41.349681    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zdgbc\" (UniqueName: \"kubernetes.io/projected/5a2904f3-27e3-4ad1-8007-5bc4156ac287-kube-api-access-zdgbc\") pod \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\" (UID: \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\") "
Sep 10 21:21:41.349727 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:41.349724    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a2904f3-27e3-4ad1-8007-5bc4156ac287-istio-token\") pod \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\" (UID: \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\") "
Sep 10 21:21:41.350056 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:41.349766    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a2904f3-27e3-4ad1-8007-5bc4156ac287-workload-socket\") pod \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\" (UID: \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\") "
Sep 10 21:21:41.350056 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:41.349808    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a2904f3-27e3-4ad1-8007-5bc4156ac287-istio-data\") pod \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\" (UID: \"5a2904f3-27e3-4ad1-8007-5bc4156ac287\") "
Sep 10 21:21:41.350056 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:41.349841    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a2904f3-27e3-4ad1-8007-5bc4156ac287-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5a2904f3-27e3-4ad1-8007-5bc4156ac287" (UID: "5a2904f3-27e3-4ad1-8007-5bc4156ac287"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:21:41.350223 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:41.350078    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a2904f3-27e3-4ad1-8007-5bc4156ac287-credential-socket\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:21:41.350223 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:41.350094    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a2904f3-27e3-4ad1-8007-5bc4156ac287-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5a2904f3-27e3-4ad1-8007-5bc4156ac287" (UID: "5a2904f3-27e3-4ad1-8007-5bc4156ac287"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:21:41.350223 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:41.350139    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a2904f3-27e3-4ad1-8007-5bc4156ac287-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5a2904f3-27e3-4ad1-8007-5bc4156ac287" (UID: "5a2904f3-27e3-4ad1-8007-5bc4156ac287"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:21:41.350420 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:41.350270    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a2904f3-27e3-4ad1-8007-5bc4156ac287-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5a2904f3-27e3-4ad1-8007-5bc4156ac287" (UID: "5a2904f3-27e3-4ad1-8007-5bc4156ac287"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:21:41.350420 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:41.350286    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a2904f3-27e3-4ad1-8007-5bc4156ac287-istio-data" (OuterVolumeSpecName: "istio-data") pod "5a2904f3-27e3-4ad1-8007-5bc4156ac287" (UID: "5a2904f3-27e3-4ad1-8007-5bc4156ac287"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:21:41.351923 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:41.351897    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a2904f3-27e3-4ad1-8007-5bc4156ac287-kube-api-access-zdgbc" (OuterVolumeSpecName: "kube-api-access-zdgbc") pod "5a2904f3-27e3-4ad1-8007-5bc4156ac287" (UID: "5a2904f3-27e3-4ad1-8007-5bc4156ac287"). InnerVolumeSpecName "kube-api-access-zdgbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:21:41.352034 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:41.351936    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a2904f3-27e3-4ad1-8007-5bc4156ac287-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5a2904f3-27e3-4ad1-8007-5bc4156ac287" (UID: "5a2904f3-27e3-4ad1-8007-5bc4156ac287"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 21:21:41.352034 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:41.351956    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a2904f3-27e3-4ad1-8007-5bc4156ac287-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5a2904f3-27e3-4ad1-8007-5bc4156ac287" (UID: "5a2904f3-27e3-4ad1-8007-5bc4156ac287"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:21:41.352034 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:41.351951    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a2904f3-27e3-4ad1-8007-5bc4156ac287-istio-token" (OuterVolumeSpecName: "istio-token") pod "5a2904f3-27e3-4ad1-8007-5bc4156ac287" (UID: "5a2904f3-27e3-4ad1-8007-5bc4156ac287"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:21:41.451389 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:41.451359    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a2904f3-27e3-4ad1-8007-5bc4156ac287-istio-podinfo\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:21:41.451389 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:41.451386    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a2904f3-27e3-4ad1-8007-5bc4156ac287-workload-certs\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:21:41.451389 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:41.451395    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a2904f3-27e3-4ad1-8007-5bc4156ac287-istio-envoy\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:21:41.451594 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:41.451403    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a2904f3-27e3-4ad1-8007-5bc4156ac287-istiod-ca-cert\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:21:41.451594 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:41.451412    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdgbc\" (UniqueName: \"kubernetes.io/projected/5a2904f3-27e3-4ad1-8007-5bc4156ac287-kube-api-access-zdgbc\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:21:41.451594 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:41.451421    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a2904f3-27e3-4ad1-8007-5bc4156ac287-istio-token\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:21:41.451594 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:41.451429    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a2904f3-27e3-4ad1-8007-5bc4156ac287-workload-socket\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:21:41.451594 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:41.451437    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a2904f3-27e3-4ad1-8007-5bc4156ac287-istio-data\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:21:42.069432 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:42.069401    2581 generic.go:356] "Generic (PLEG): container finished" podID="5a2904f3-27e3-4ad1-8007-5bc4156ac287" containerID="60eac9d5bd071901fc1d8bdc06526d026ef103bf247a836247de681d8251f5aa" exitCode=0
Sep 10 21:21:42.069611 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:42.069448    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"
Sep 10 21:21:42.069611 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:42.069487    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx" event={"ID":"5a2904f3-27e3-4ad1-8007-5bc4156ac287","Type":"ContainerDied","Data":"60eac9d5bd071901fc1d8bdc06526d026ef103bf247a836247de681d8251f5aa"}
Sep 10 21:21:42.069611 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:42.069549    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx" event={"ID":"5a2904f3-27e3-4ad1-8007-5bc4156ac287","Type":"ContainerDied","Data":"652aa9532b51502c3cede53c394a23e983fff590e24a2389c41d34f4a8b24be4"}
Sep 10 21:21:42.069611 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:42.069578    2581 scope.go:117] "RemoveContainer" containerID="60eac9d5bd071901fc1d8bdc06526d026ef103bf247a836247de681d8251f5aa"
Sep 10 21:21:42.079141 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:42.079124    2581 scope.go:117] "RemoveContainer" containerID="f5cb8988a434a1605a5dee09586eaccc63cefa43ffde7901a5fac9fced6a5bc6"
Sep 10 21:21:42.087916 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:42.087747    2581 scope.go:117] "RemoveContainer" containerID="60eac9d5bd071901fc1d8bdc06526d026ef103bf247a836247de681d8251f5aa"
Sep 10 21:21:42.088083 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:21:42.088062    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60eac9d5bd071901fc1d8bdc06526d026ef103bf247a836247de681d8251f5aa\": container with ID starting with 60eac9d5bd071901fc1d8bdc06526d026ef103bf247a836247de681d8251f5aa not found: ID does not exist" containerID="60eac9d5bd071901fc1d8bdc06526d026ef103bf247a836247de681d8251f5aa"
Sep 10 21:21:42.088132 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:42.088091    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60eac9d5bd071901fc1d8bdc06526d026ef103bf247a836247de681d8251f5aa"} err="failed to get container status \"60eac9d5bd071901fc1d8bdc06526d026ef103bf247a836247de681d8251f5aa\": rpc error: code = NotFound desc = could not find container \"60eac9d5bd071901fc1d8bdc06526d026ef103bf247a836247de681d8251f5aa\": container with ID starting with 60eac9d5bd071901fc1d8bdc06526d026ef103bf247a836247de681d8251f5aa not found: ID does not exist"
Sep 10 21:21:42.088132 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:42.088109    2581 scope.go:117] "RemoveContainer" containerID="f5cb8988a434a1605a5dee09586eaccc63cefa43ffde7901a5fac9fced6a5bc6"
Sep 10 21:21:42.088395 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:21:42.088374    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5cb8988a434a1605a5dee09586eaccc63cefa43ffde7901a5fac9fced6a5bc6\": container with ID starting with f5cb8988a434a1605a5dee09586eaccc63cefa43ffde7901a5fac9fced6a5bc6 not found: ID does not exist" containerID="f5cb8988a434a1605a5dee09586eaccc63cefa43ffde7901a5fac9fced6a5bc6"
Sep 10 21:21:42.088456 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:42.088417    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5cb8988a434a1605a5dee09586eaccc63cefa43ffde7901a5fac9fced6a5bc6"} err="failed to get container status \"f5cb8988a434a1605a5dee09586eaccc63cefa43ffde7901a5fac9fced6a5bc6\": rpc error: code = NotFound desc = could not find container \"f5cb8988a434a1605a5dee09586eaccc63cefa43ffde7901a5fac9fced6a5bc6\": container with ID starting with f5cb8988a434a1605a5dee09586eaccc63cefa43ffde7901a5fac9fced6a5bc6 not found: ID does not exist"
Sep 10 21:21:42.091619 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:42.091599    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"]
Sep 10 21:21:42.095044 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:42.095021    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-806a33-openshift-default-57cc8c59c6-j9vdx"]
Sep 10 21:21:42.868621 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:42.868587    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a2904f3-27e3-4ad1-8007-5bc4156ac287" path="/var/lib/kubelet/pods/5a2904f3-27e3-4ad1-8007-5bc4156ac287/volumes"
Sep 10 21:21:47.095956 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:47.095923    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-14cee3-74676b8c7f-nppg7"]
Sep 10 21:21:47.096481 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:47.096186    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-14cee3-74676b8c7f-nppg7" podUID="e04ce7a3-c789-4c0e-835b-0421e6f12186" containerName="payload-processing" containerID="cri-o://880ef82277914daa113ac3fbe733fd9fb820593a3b054b77e5295ccced7e1441" gracePeriod=30
Sep 10 21:21:47.099983 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:47.099956    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-14cee3-7b557b4c78-sjrs7"]
Sep 10 21:21:47.100337 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:47.100244    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-14cee3-7b557b4c78-sjrs7" podUID="81f0bb0c-1901-4432-a48c-f97b087bfecd" containerName="payload-pre-processing" containerID="cri-o://778def8e21f324aafdc6e4f6b49418694cca57073994c2fd88a1e7cce7a4de07" gracePeriod=30
Sep 10 21:21:47.332490 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:47.332458    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-14cee3-llxxp"]
Sep 10 21:21:47.334703 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:47.334669    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-14cee3-llxxp"]
Sep 10 21:21:47.422081 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:47.422053    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-14cee3-7b557b4c78-sjrs7"
Sep 10 21:21:47.424899 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:47.424876    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-14cee3-74676b8c7f-nppg7"
Sep 10 21:21:47.509427 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:47.509393    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzhlz\" (UniqueName: \"kubernetes.io/projected/e04ce7a3-c789-4c0e-835b-0421e6f12186-kube-api-access-nzhlz\") pod \"e04ce7a3-c789-4c0e-835b-0421e6f12186\" (UID: \"e04ce7a3-c789-4c0e-835b-0421e6f12186\") "
Sep 10 21:21:47.509595 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:47.509473    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81f0bb0c-1901-4432-a48c-f97b087bfecd-plugins-config-volume\") pod \"81f0bb0c-1901-4432-a48c-f97b087bfecd\" (UID: \"81f0bb0c-1901-4432-a48c-f97b087bfecd\") "
Sep 10 21:21:47.509595 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:47.509508    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vt56k\" (UniqueName: \"kubernetes.io/projected/81f0bb0c-1901-4432-a48c-f97b087bfecd-kube-api-access-vt56k\") pod \"81f0bb0c-1901-4432-a48c-f97b087bfecd\" (UID: \"81f0bb0c-1901-4432-a48c-f97b087bfecd\") "
Sep 10 21:21:47.509595 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:47.509545    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e04ce7a3-c789-4c0e-835b-0421e6f12186-plugins-config-volume\") pod \"e04ce7a3-c789-4c0e-835b-0421e6f12186\" (UID: \"e04ce7a3-c789-4c0e-835b-0421e6f12186\") "
Sep 10 21:21:47.509934 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:47.509908    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81f0bb0c-1901-4432-a48c-f97b087bfecd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "81f0bb0c-1901-4432-a48c-f97b087bfecd" (UID: "81f0bb0c-1901-4432-a48c-f97b087bfecd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:21:47.509934 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:47.509917    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e04ce7a3-c789-4c0e-835b-0421e6f12186-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e04ce7a3-c789-4c0e-835b-0421e6f12186" (UID: "e04ce7a3-c789-4c0e-835b-0421e6f12186"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:21:47.511551 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:47.511530    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e04ce7a3-c789-4c0e-835b-0421e6f12186-kube-api-access-nzhlz" (OuterVolumeSpecName: "kube-api-access-nzhlz") pod "e04ce7a3-c789-4c0e-835b-0421e6f12186" (UID: "e04ce7a3-c789-4c0e-835b-0421e6f12186"). InnerVolumeSpecName "kube-api-access-nzhlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:21:47.511713 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:47.511689    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81f0bb0c-1901-4432-a48c-f97b087bfecd-kube-api-access-vt56k" (OuterVolumeSpecName: "kube-api-access-vt56k") pod "81f0bb0c-1901-4432-a48c-f97b087bfecd" (UID: "81f0bb0c-1901-4432-a48c-f97b087bfecd"). InnerVolumeSpecName "kube-api-access-vt56k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:21:47.610631 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:47.610596    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81f0bb0c-1901-4432-a48c-f97b087bfecd-plugins-config-volume\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:21:47.610631 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:47.610627    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt56k\" (UniqueName: \"kubernetes.io/projected/81f0bb0c-1901-4432-a48c-f97b087bfecd-kube-api-access-vt56k\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:21:47.610819 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:47.610642    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e04ce7a3-c789-4c0e-835b-0421e6f12186-plugins-config-volume\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:21:47.610819 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:47.610653    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzhlz\" (UniqueName: \"kubernetes.io/projected/e04ce7a3-c789-4c0e-835b-0421e6f12186-kube-api-access-nzhlz\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:21:48.105836 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:48.105805    2581 generic.go:356] "Generic (PLEG): container finished" podID="81f0bb0c-1901-4432-a48c-f97b087bfecd" containerID="778def8e21f324aafdc6e4f6b49418694cca57073994c2fd88a1e7cce7a4de07" exitCode=0
Sep 10 21:21:48.106314 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:48.105860    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-14cee3-7b557b4c78-sjrs7"
Sep 10 21:21:48.106314 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:48.105890    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-14cee3-7b557b4c78-sjrs7" event={"ID":"81f0bb0c-1901-4432-a48c-f97b087bfecd","Type":"ContainerDied","Data":"778def8e21f324aafdc6e4f6b49418694cca57073994c2fd88a1e7cce7a4de07"}
Sep 10 21:21:48.106314 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:48.105946    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-14cee3-7b557b4c78-sjrs7" event={"ID":"81f0bb0c-1901-4432-a48c-f97b087bfecd","Type":"ContainerDied","Data":"1690e86c9d49737955c29d319bfa0c7b8f24b4cefaef4fd3f3d23032b493b161"}
Sep 10 21:21:48.106314 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:48.105977    2581 scope.go:117] "RemoveContainer" containerID="778def8e21f324aafdc6e4f6b49418694cca57073994c2fd88a1e7cce7a4de07"
Sep 10 21:21:48.107319 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:48.107219    2581 generic.go:356] "Generic (PLEG): container finished" podID="e04ce7a3-c789-4c0e-835b-0421e6f12186" containerID="880ef82277914daa113ac3fbe733fd9fb820593a3b054b77e5295ccced7e1441" exitCode=0
Sep 10 21:21:48.107319 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:48.107258    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-14cee3-74676b8c7f-nppg7"
Sep 10 21:21:48.107319 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:48.107277    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-14cee3-74676b8c7f-nppg7" event={"ID":"e04ce7a3-c789-4c0e-835b-0421e6f12186","Type":"ContainerDied","Data":"880ef82277914daa113ac3fbe733fd9fb820593a3b054b77e5295ccced7e1441"}
Sep 10 21:21:48.107455 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:48.107337    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-14cee3-74676b8c7f-nppg7" event={"ID":"e04ce7a3-c789-4c0e-835b-0421e6f12186","Type":"ContainerDied","Data":"e2f3c138aef7fadccf317f1312597b0998fe7b32777d402865aa1a31dcf692fe"}
Sep 10 21:21:48.122028 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:48.122008    2581 scope.go:117] "RemoveContainer" containerID="778def8e21f324aafdc6e4f6b49418694cca57073994c2fd88a1e7cce7a4de07"
Sep 10 21:21:48.122290 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:21:48.122273    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"778def8e21f324aafdc6e4f6b49418694cca57073994c2fd88a1e7cce7a4de07\": container with ID starting with 778def8e21f324aafdc6e4f6b49418694cca57073994c2fd88a1e7cce7a4de07 not found: ID does not exist" containerID="778def8e21f324aafdc6e4f6b49418694cca57073994c2fd88a1e7cce7a4de07"
Sep 10 21:21:48.122359 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:48.122312    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"778def8e21f324aafdc6e4f6b49418694cca57073994c2fd88a1e7cce7a4de07"} err="failed to get container status \"778def8e21f324aafdc6e4f6b49418694cca57073994c2fd88a1e7cce7a4de07\": rpc error: code = NotFound desc = could not find container \"778def8e21f324aafdc6e4f6b49418694cca57073994c2fd88a1e7cce7a4de07\": container with ID starting with 778def8e21f324aafdc6e4f6b49418694cca57073994c2fd88a1e7cce7a4de07 not found: ID does not exist"
Sep 10 21:21:48.122359 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:48.122332    2581 scope.go:117] "RemoveContainer" containerID="880ef82277914daa113ac3fbe733fd9fb820593a3b054b77e5295ccced7e1441"
Sep 10 21:21:48.129944 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:48.129926    2581 scope.go:117] "RemoveContainer" containerID="880ef82277914daa113ac3fbe733fd9fb820593a3b054b77e5295ccced7e1441"
Sep 10 21:21:48.130207 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:21:48.130190    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"880ef82277914daa113ac3fbe733fd9fb820593a3b054b77e5295ccced7e1441\": container with ID starting with 880ef82277914daa113ac3fbe733fd9fb820593a3b054b77e5295ccced7e1441 not found: ID does not exist" containerID="880ef82277914daa113ac3fbe733fd9fb820593a3b054b77e5295ccced7e1441"
Sep 10 21:21:48.130272 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:48.130214    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"880ef82277914daa113ac3fbe733fd9fb820593a3b054b77e5295ccced7e1441"} err="failed to get container status \"880ef82277914daa113ac3fbe733fd9fb820593a3b054b77e5295ccced7e1441\": rpc error: code = NotFound desc = could not find container \"880ef82277914daa113ac3fbe733fd9fb820593a3b054b77e5295ccced7e1441\": container with ID starting with 880ef82277914daa113ac3fbe733fd9fb820593a3b054b77e5295ccced7e1441 not found: ID does not exist"
Sep 10 21:21:48.134532 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:48.134511    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-14cee3-7b557b4c78-sjrs7"]
Sep 10 21:21:48.138145 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:48.138127    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-14cee3-7b557b4c78-sjrs7"]
Sep 10 21:21:48.148350 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:48.148331    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-14cee3-74676b8c7f-nppg7"]
Sep 10 21:21:48.152458 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:48.152441    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-14cee3-74676b8c7f-nppg7"]
Sep 10 21:21:48.869028 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:48.868993    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81673b14-17da-4701-b7b7-4ca5d05a536f" path="/var/lib/kubelet/pods/81673b14-17da-4701-b7b7-4ca5d05a536f/volumes"
Sep 10 21:21:48.869395 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:48.869380    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81f0bb0c-1901-4432-a48c-f97b087bfecd" path="/var/lib/kubelet/pods/81f0bb0c-1901-4432-a48c-f97b087bfecd/volumes"
Sep 10 21:21:48.869717 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:48.869705    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e04ce7a3-c789-4c0e-835b-0421e6f12186" path="/var/lib/kubelet/pods/e04ce7a3-c789-4c0e-835b-0421e6f12186/volumes"
Sep 10 21:21:50.059646 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:50.059615    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7eb29-56d4bb5998-bzd2r"
Sep 10 21:21:54.221708 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:54.221673    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"]
Sep 10 21:21:54.222143 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:54.221933    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2" podUID="07844b88-223f-4b79-983d-f33403a57ec0" containerName="istio-proxy" containerID="cri-o://d1c63ce0dbe5aa172c6c50d7f1a0b483d9233e1c4255c3d33c2aa9fa008cfb3e" gracePeriod=30
Sep 10 21:21:54.588651 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:54.588619    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-aca18f-wtmnw"]
Sep 10 21:21:54.589057 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:54.589044    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a2904f3-27e3-4ad1-8007-5bc4156ac287" containerName="istio-proxy"
Sep 10 21:21:54.589114 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:54.589059    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a2904f3-27e3-4ad1-8007-5bc4156ac287" containerName="istio-proxy"
Sep 10 21:21:54.589114 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:54.589073    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e04ce7a3-c789-4c0e-835b-0421e6f12186" containerName="payload-processing"
Sep 10 21:21:54.589114 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:54.589082    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e04ce7a3-c789-4c0e-835b-0421e6f12186" containerName="payload-processing"
Sep 10 21:21:54.589114 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:54.589097    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a2904f3-27e3-4ad1-8007-5bc4156ac287" containerName="istio-proxy"
Sep 10 21:21:54.589114 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:54.589104    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a2904f3-27e3-4ad1-8007-5bc4156ac287" containerName="istio-proxy"
Sep 10 21:21:54.589114 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:54.589113    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81673b14-17da-4701-b7b7-4ca5d05a536f" containerName="revoke-keys"
Sep 10 21:21:54.589291 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:54.589119    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="81673b14-17da-4701-b7b7-4ca5d05a536f" containerName="revoke-keys"
Sep 10 21:21:54.589291 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:54.589127    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81f0bb0c-1901-4432-a48c-f97b087bfecd" containerName="payload-pre-processing"
Sep 10 21:21:54.589291 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:54.589132    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="81f0bb0c-1901-4432-a48c-f97b087bfecd" containerName="payload-pre-processing"
Sep 10 21:21:54.589291 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:54.589194    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a2904f3-27e3-4ad1-8007-5bc4156ac287" containerName="istio-proxy"
Sep 10 21:21:54.589291 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:54.589205    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="81f0bb0c-1901-4432-a48c-f97b087bfecd" containerName="payload-pre-processing"
Sep 10 21:21:54.589291 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:54.589212    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a2904f3-27e3-4ad1-8007-5bc4156ac287" containerName="istio-proxy"
Sep 10 21:21:54.589291 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:54.589218    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e04ce7a3-c789-4c0e-835b-0421e6f12186" containerName="payload-processing"
Sep 10 21:21:54.589291 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:54.589223    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="81673b14-17da-4701-b7b7-4ca5d05a536f" containerName="revoke-keys"
Sep 10 21:21:54.593922 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:54.593903    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-aca18f-wtmnw"
Sep 10 21:21:54.596086 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:54.596057    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:21:54.596408 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:54.596383    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s4qqn\""
Sep 10 21:21:54.599335 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:54.599289    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-aca18f-wtmnw"]
Sep 10 21:21:54.778215 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:54.778179    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cc004e7-7523-42b2-be51-8d5a1918b371-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-aca18f-wtmnw\" (UID: \"9cc004e7-7523-42b2-be51-8d5a1918b371\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-aca18f-wtmnw"
Sep 10 21:21:54.879785 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:54.879702    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cc004e7-7523-42b2-be51-8d5a1918b371-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-aca18f-wtmnw\" (UID: \"9cc004e7-7523-42b2-be51-8d5a1918b371\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-aca18f-wtmnw"
Sep 10 21:21:54.880346 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:54.880293    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cc004e7-7523-42b2-be51-8d5a1918b371-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-aca18f-wtmnw\" (UID: \"9cc004e7-7523-42b2-be51-8d5a1918b371\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-aca18f-wtmnw"
Sep 10 21:21:54.906670 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:54.906643    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-aca18f-wtmnw"
Sep 10 21:21:55.184356 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:55.184325    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-aca18f-wtmnw"]
Sep 10 21:21:55.185257 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:21:55.185226    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cc004e7_7523_42b2_be51_8d5a1918b371.slice/crio-ff8d995aa6755655d6e69deddfc16dfe09048ebac41f183c034a303ecd282d19 WatchSource:0}: Error finding container ff8d995aa6755655d6e69deddfc16dfe09048ebac41f183c034a303ecd282d19: Status 404 returned error can't find the container with id ff8d995aa6755655d6e69deddfc16dfe09048ebac41f183c034a303ecd282d19
Sep 10 21:21:56.155724 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:56.155635    2581 generic.go:356] "Generic (PLEG): container finished" podID="9cc004e7-7523-42b2-be51-8d5a1918b371" containerID="1fd6bc98dca3500b5829939771ff0f5a9eb1460736cdbf86f73227e3d8a54b92" exitCode=0
Sep 10 21:21:56.156085 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:56.155727    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-aca18f-wtmnw" event={"ID":"9cc004e7-7523-42b2-be51-8d5a1918b371","Type":"ContainerDied","Data":"1fd6bc98dca3500b5829939771ff0f5a9eb1460736cdbf86f73227e3d8a54b92"}
Sep 10 21:21:56.156085 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:56.155778    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-aca18f-wtmnw" event={"ID":"9cc004e7-7523-42b2-be51-8d5a1918b371","Type":"ContainerStarted","Data":"ff8d995aa6755655d6e69deddfc16dfe09048ebac41f183c034a303ecd282d19"}
Sep 10 21:21:57.036859 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:57.036813    2581 patch_prober.go:28] interesting pod/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 21:21:57.037028 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:57.036877    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2" podUID="07844b88-223f-4b79-983d-f33403a57ec0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 21:21:57.293817 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:57.293794    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-aca18f-wtmnw"
Sep 10 21:21:57.402169 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:57.402134    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cc004e7-7523-42b2-be51-8d5a1918b371-maas-api-service-ca\") pod \"9cc004e7-7523-42b2-be51-8d5a1918b371\" (UID: \"9cc004e7-7523-42b2-be51-8d5a1918b371\") "
Sep 10 21:21:57.402541 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:57.402519    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cc004e7-7523-42b2-be51-8d5a1918b371-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9cc004e7-7523-42b2-be51-8d5a1918b371" (UID: "9cc004e7-7523-42b2-be51-8d5a1918b371"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:21:57.503752 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:57.503725    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cc004e7-7523-42b2-be51-8d5a1918b371-maas-api-service-ca\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:21:58.167648 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:58.167616    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-aca18f-wtmnw"
Sep 10 21:21:58.167648 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:58.167641    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-aca18f-wtmnw" event={"ID":"9cc004e7-7523-42b2-be51-8d5a1918b371","Type":"ContainerDied","Data":"ff8d995aa6755655d6e69deddfc16dfe09048ebac41f183c034a303ecd282d19"}
Sep 10 21:21:58.167913 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:58.167685    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff8d995aa6755655d6e69deddfc16dfe09048ebac41f183c034a303ecd282d19"
Sep 10 21:21:59.470407 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:59.470380    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"
Sep 10 21:21:59.623607 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:59.623578    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07844b88-223f-4b79-983d-f33403a57ec0-istio-token\") pod \"07844b88-223f-4b79-983d-f33403a57ec0\" (UID: \"07844b88-223f-4b79-983d-f33403a57ec0\") "
Sep 10 21:21:59.623806 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:59.623624    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07844b88-223f-4b79-983d-f33403a57ec0-workload-certs\") pod \"07844b88-223f-4b79-983d-f33403a57ec0\" (UID: \"07844b88-223f-4b79-983d-f33403a57ec0\") "
Sep 10 21:21:59.623806 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:59.623678    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pp2sh\" (UniqueName: \"kubernetes.io/projected/07844b88-223f-4b79-983d-f33403a57ec0-kube-api-access-pp2sh\") pod \"07844b88-223f-4b79-983d-f33403a57ec0\" (UID: \"07844b88-223f-4b79-983d-f33403a57ec0\") "
Sep 10 21:21:59.623806 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:59.623714    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07844b88-223f-4b79-983d-f33403a57ec0-istiod-ca-cert\") pod \"07844b88-223f-4b79-983d-f33403a57ec0\" (UID: \"07844b88-223f-4b79-983d-f33403a57ec0\") "
Sep 10 21:21:59.623806 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:59.623762    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07844b88-223f-4b79-983d-f33403a57ec0-workload-socket\") pod \"07844b88-223f-4b79-983d-f33403a57ec0\" (UID: \"07844b88-223f-4b79-983d-f33403a57ec0\") "
Sep 10 21:21:59.624038 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:59.623814    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07844b88-223f-4b79-983d-f33403a57ec0-istio-podinfo\") pod \"07844b88-223f-4b79-983d-f33403a57ec0\" (UID: \"07844b88-223f-4b79-983d-f33403a57ec0\") "
Sep 10 21:21:59.624038 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:59.623966    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07844b88-223f-4b79-983d-f33403a57ec0-credential-socket\") pod \"07844b88-223f-4b79-983d-f33403a57ec0\" (UID: \"07844b88-223f-4b79-983d-f33403a57ec0\") "
Sep 10 21:21:59.624038 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:59.624034    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07844b88-223f-4b79-983d-f33403a57ec0-istio-data\") pod \"07844b88-223f-4b79-983d-f33403a57ec0\" (UID: \"07844b88-223f-4b79-983d-f33403a57ec0\") "
Sep 10 21:21:59.624191 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:59.624018    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07844b88-223f-4b79-983d-f33403a57ec0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "07844b88-223f-4b79-983d-f33403a57ec0" (UID: "07844b88-223f-4b79-983d-f33403a57ec0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:21:59.624191 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:59.624067    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07844b88-223f-4b79-983d-f33403a57ec0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "07844b88-223f-4b79-983d-f33403a57ec0" (UID: "07844b88-223f-4b79-983d-f33403a57ec0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:21:59.624191 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:59.624117    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07844b88-223f-4b79-983d-f33403a57ec0-istio-envoy\") pod \"07844b88-223f-4b79-983d-f33403a57ec0\" (UID: \"07844b88-223f-4b79-983d-f33403a57ec0\") "
Sep 10 21:21:59.624382 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:59.624243    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07844b88-223f-4b79-983d-f33403a57ec0-istio-data" (OuterVolumeSpecName: "istio-data") pod "07844b88-223f-4b79-983d-f33403a57ec0" (UID: "07844b88-223f-4b79-983d-f33403a57ec0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:21:59.624382 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:59.624267    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07844b88-223f-4b79-983d-f33403a57ec0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "07844b88-223f-4b79-983d-f33403a57ec0" (UID: "07844b88-223f-4b79-983d-f33403a57ec0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:21:59.624382 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:59.624280    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07844b88-223f-4b79-983d-f33403a57ec0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "07844b88-223f-4b79-983d-f33403a57ec0" (UID: "07844b88-223f-4b79-983d-f33403a57ec0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:21:59.624558 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:59.624533    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07844b88-223f-4b79-983d-f33403a57ec0-istio-data\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:21:59.624618 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:59.624560    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07844b88-223f-4b79-983d-f33403a57ec0-workload-certs\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:21:59.624618 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:59.624573    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07844b88-223f-4b79-983d-f33403a57ec0-istiod-ca-cert\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:21:59.624618 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:59.624585    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07844b88-223f-4b79-983d-f33403a57ec0-workload-socket\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:21:59.624618 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:59.624607    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07844b88-223f-4b79-983d-f33403a57ec0-credential-socket\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:21:59.626168 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:59.626142    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07844b88-223f-4b79-983d-f33403a57ec0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "07844b88-223f-4b79-983d-f33403a57ec0" (UID: "07844b88-223f-4b79-983d-f33403a57ec0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:21:59.626265 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:59.626176    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07844b88-223f-4b79-983d-f33403a57ec0-istio-token" (OuterVolumeSpecName: "istio-token") pod "07844b88-223f-4b79-983d-f33403a57ec0" (UID: "07844b88-223f-4b79-983d-f33403a57ec0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:21:59.626265 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:59.626211    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07844b88-223f-4b79-983d-f33403a57ec0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "07844b88-223f-4b79-983d-f33403a57ec0" (UID: "07844b88-223f-4b79-983d-f33403a57ec0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 21:21:59.626265 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:59.626207    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07844b88-223f-4b79-983d-f33403a57ec0-kube-api-access-pp2sh" (OuterVolumeSpecName: "kube-api-access-pp2sh") pod "07844b88-223f-4b79-983d-f33403a57ec0" (UID: "07844b88-223f-4b79-983d-f33403a57ec0"). InnerVolumeSpecName "kube-api-access-pp2sh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:21:59.725110 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:59.725078    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07844b88-223f-4b79-983d-f33403a57ec0-istio-podinfo\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:21:59.725110 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:59.725106    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07844b88-223f-4b79-983d-f33403a57ec0-istio-envoy\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:21:59.725110 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:59.725117    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07844b88-223f-4b79-983d-f33403a57ec0-istio-token\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:21:59.725428 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:21:59.725126    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp2sh\" (UniqueName: \"kubernetes.io/projected/07844b88-223f-4b79-983d-f33403a57ec0-kube-api-access-pp2sh\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:22:00.180051 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:00.180019    2581 generic.go:356] "Generic (PLEG): container finished" podID="07844b88-223f-4b79-983d-f33403a57ec0" containerID="d1c63ce0dbe5aa172c6c50d7f1a0b483d9233e1c4255c3d33c2aa9fa008cfb3e" exitCode=0
Sep 10 21:22:00.180247 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:00.180063    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2" event={"ID":"07844b88-223f-4b79-983d-f33403a57ec0","Type":"ContainerDied","Data":"d1c63ce0dbe5aa172c6c50d7f1a0b483d9233e1c4255c3d33c2aa9fa008cfb3e"}
Sep 10 21:22:00.180247 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:00.180110    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2" event={"ID":"07844b88-223f-4b79-983d-f33403a57ec0","Type":"ContainerDied","Data":"153c5aab0aa399dda3643b1ae67e2b164bacf4526833665cc56199405b312455"}
Sep 10 21:22:00.180247 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:00.180068    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"
Sep 10 21:22:00.180247 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:00.180140    2581 scope.go:117] "RemoveContainer" containerID="d1c63ce0dbe5aa172c6c50d7f1a0b483d9233e1c4255c3d33c2aa9fa008cfb3e"
Sep 10 21:22:00.190393 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:00.190374    2581 scope.go:117] "RemoveContainer" containerID="a135dbd85f6573b53b0c129e3b40551744a389d48f275ee823ae9bc6bc11c067"
Sep 10 21:22:00.198699 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:00.198680    2581 scope.go:117] "RemoveContainer" containerID="d1c63ce0dbe5aa172c6c50d7f1a0b483d9233e1c4255c3d33c2aa9fa008cfb3e"
Sep 10 21:22:00.198997 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:22:00.198974    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1c63ce0dbe5aa172c6c50d7f1a0b483d9233e1c4255c3d33c2aa9fa008cfb3e\": container with ID starting with d1c63ce0dbe5aa172c6c50d7f1a0b483d9233e1c4255c3d33c2aa9fa008cfb3e not found: ID does not exist" containerID="d1c63ce0dbe5aa172c6c50d7f1a0b483d9233e1c4255c3d33c2aa9fa008cfb3e"
Sep 10 21:22:00.199049 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:00.199005    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1c63ce0dbe5aa172c6c50d7f1a0b483d9233e1c4255c3d33c2aa9fa008cfb3e"} err="failed to get container status \"d1c63ce0dbe5aa172c6c50d7f1a0b483d9233e1c4255c3d33c2aa9fa008cfb3e\": rpc error: code = NotFound desc = could not find container \"d1c63ce0dbe5aa172c6c50d7f1a0b483d9233e1c4255c3d33c2aa9fa008cfb3e\": container with ID starting with d1c63ce0dbe5aa172c6c50d7f1a0b483d9233e1c4255c3d33c2aa9fa008cfb3e not found: ID does not exist"
Sep 10 21:22:00.199049 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:00.199022    2581 scope.go:117] "RemoveContainer" containerID="a135dbd85f6573b53b0c129e3b40551744a389d48f275ee823ae9bc6bc11c067"
Sep 10 21:22:00.199328 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:22:00.199309    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a135dbd85f6573b53b0c129e3b40551744a389d48f275ee823ae9bc6bc11c067\": container with ID starting with a135dbd85f6573b53b0c129e3b40551744a389d48f275ee823ae9bc6bc11c067 not found: ID does not exist" containerID="a135dbd85f6573b53b0c129e3b40551744a389d48f275ee823ae9bc6bc11c067"
Sep 10 21:22:00.199398 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:00.199334    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a135dbd85f6573b53b0c129e3b40551744a389d48f275ee823ae9bc6bc11c067"} err="failed to get container status \"a135dbd85f6573b53b0c129e3b40551744a389d48f275ee823ae9bc6bc11c067\": rpc error: code = NotFound desc = could not find container \"a135dbd85f6573b53b0c129e3b40551744a389d48f275ee823ae9bc6bc11c067\": container with ID starting with a135dbd85f6573b53b0c129e3b40551744a389d48f275ee823ae9bc6bc11c067 not found: ID does not exist"
Sep 10 21:22:00.207747 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:00.207724    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"]
Sep 10 21:22:00.214536 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:00.214516    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-14cee3-openshift-default-6985794d4-4hpm2"]
Sep 10 21:22:00.868336 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:00.868284    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07844b88-223f-4b79-983d-f33403a57ec0" path="/var/lib/kubelet/pods/07844b88-223f-4b79-983d-f33403a57ec0/volumes"
Sep 10 21:22:01.918709 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:01.918670    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"]
Sep 10 21:22:01.919167 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:01.919151    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07844b88-223f-4b79-983d-f33403a57ec0" containerName="istio-proxy"
Sep 10 21:22:01.919226 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:01.919168    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="07844b88-223f-4b79-983d-f33403a57ec0" containerName="istio-proxy"
Sep 10 21:22:01.919226 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:01.919176    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cc004e7-7523-42b2-be51-8d5a1918b371" containerName="revoke-keys"
Sep 10 21:22:01.919226 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:01.919182    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cc004e7-7523-42b2-be51-8d5a1918b371" containerName="revoke-keys"
Sep 10 21:22:01.919368 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:01.919250    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="07844b88-223f-4b79-983d-f33403a57ec0" containerName="istio-proxy"
Sep 10 21:22:01.919368 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:01.919260    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cc004e7-7523-42b2-be51-8d5a1918b371" containerName="revoke-keys"
Sep 10 21:22:01.919368 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:01.919271    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="07844b88-223f-4b79-983d-f33403a57ec0" containerName="istio-proxy"
Sep 10 21:22:01.924089 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:01.924071    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"
Sep 10 21:22:01.928757 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:01.928737    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-1427b1-openshift-default-dockercfg-d5lw5\""
Sep 10 21:22:01.937695 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:01.937672    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"]
Sep 10 21:22:01.943432 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:01.943401    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba1f825-34bb-41e6-a692-35d2de203a14-credential-socket\") pod \"e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml\" (UID: \"1ba1f825-34bb-41e6-a692-35d2de203a14\") " pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"
Sep 10 21:22:01.943541 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:01.943446    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba1f825-34bb-41e6-a692-35d2de203a14-workload-socket\") pod \"e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml\" (UID: \"1ba1f825-34bb-41e6-a692-35d2de203a14\") " pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"
Sep 10 21:22:01.943541 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:01.943518    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ba1f825-34bb-41e6-a692-35d2de203a14-workload-certs\") pod \"e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml\" (UID: \"1ba1f825-34bb-41e6-a692-35d2de203a14\") " pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"
Sep 10 21:22:01.943657 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:01.943577    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b794\" (UniqueName: \"kubernetes.io/projected/1ba1f825-34bb-41e6-a692-35d2de203a14-kube-api-access-4b794\") pod \"e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml\" (UID: \"1ba1f825-34bb-41e6-a692-35d2de203a14\") " pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"
Sep 10 21:22:01.943657 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:01.943608    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ba1f825-34bb-41e6-a692-35d2de203a14-istio-podinfo\") pod \"e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml\" (UID: \"1ba1f825-34bb-41e6-a692-35d2de203a14\") " pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"
Sep 10 21:22:01.943737 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:01.943675    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ba1f825-34bb-41e6-a692-35d2de203a14-istiod-ca-cert\") pod \"e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml\" (UID: \"1ba1f825-34bb-41e6-a692-35d2de203a14\") " pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"
Sep 10 21:22:01.943737 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:01.943720    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ba1f825-34bb-41e6-a692-35d2de203a14-istio-envoy\") pod \"e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml\" (UID: \"1ba1f825-34bb-41e6-a692-35d2de203a14\") " pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"
Sep 10 21:22:01.943868 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:01.943848    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ba1f825-34bb-41e6-a692-35d2de203a14-istio-token\") pod \"e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml\" (UID: \"1ba1f825-34bb-41e6-a692-35d2de203a14\") " pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"
Sep 10 21:22:01.943942 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:01.943901    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ba1f825-34bb-41e6-a692-35d2de203a14-istio-data\") pod \"e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml\" (UID: \"1ba1f825-34bb-41e6-a692-35d2de203a14\") " pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"
Sep 10 21:22:02.044630 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:02.044598    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ba1f825-34bb-41e6-a692-35d2de203a14-istio-token\") pod \"e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml\" (UID: \"1ba1f825-34bb-41e6-a692-35d2de203a14\") " pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"
Sep 10 21:22:02.044811 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:02.044646    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ba1f825-34bb-41e6-a692-35d2de203a14-istio-data\") pod \"e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml\" (UID: \"1ba1f825-34bb-41e6-a692-35d2de203a14\") " pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"
Sep 10 21:22:02.044811 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:02.044674    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba1f825-34bb-41e6-a692-35d2de203a14-credential-socket\") pod \"e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml\" (UID: \"1ba1f825-34bb-41e6-a692-35d2de203a14\") " pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"
Sep 10 21:22:02.044811 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:02.044700    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba1f825-34bb-41e6-a692-35d2de203a14-workload-socket\") pod \"e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml\" (UID: \"1ba1f825-34bb-41e6-a692-35d2de203a14\") " pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"
Sep 10 21:22:02.044811 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:02.044747    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ba1f825-34bb-41e6-a692-35d2de203a14-workload-certs\") pod \"e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml\" (UID: \"1ba1f825-34bb-41e6-a692-35d2de203a14\") " pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"
Sep 10 21:22:02.044811 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:02.044802    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b794\" (UniqueName: \"kubernetes.io/projected/1ba1f825-34bb-41e6-a692-35d2de203a14-kube-api-access-4b794\") pod \"e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml\" (UID: \"1ba1f825-34bb-41e6-a692-35d2de203a14\") " pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"
Sep 10 21:22:02.045082 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:02.044829    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ba1f825-34bb-41e6-a692-35d2de203a14-istio-podinfo\") pod \"e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml\" (UID: \"1ba1f825-34bb-41e6-a692-35d2de203a14\") " pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"
Sep 10 21:22:02.045082 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:02.044871    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ba1f825-34bb-41e6-a692-35d2de203a14-istiod-ca-cert\") pod \"e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml\" (UID: \"1ba1f825-34bb-41e6-a692-35d2de203a14\") " pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"
Sep 10 21:22:02.045082 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:02.044900    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ba1f825-34bb-41e6-a692-35d2de203a14-istio-envoy\") pod \"e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml\" (UID: \"1ba1f825-34bb-41e6-a692-35d2de203a14\") " pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"
Sep 10 21:22:02.045082 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:02.045042    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ba1f825-34bb-41e6-a692-35d2de203a14-istio-data\") pod \"e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml\" (UID: \"1ba1f825-34bb-41e6-a692-35d2de203a14\") " pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"
Sep 10 21:22:02.045242 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:02.045083    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba1f825-34bb-41e6-a692-35d2de203a14-credential-socket\") pod \"e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml\" (UID: \"1ba1f825-34bb-41e6-a692-35d2de203a14\") " pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"
Sep 10 21:22:02.045508 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:02.045488    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ba1f825-34bb-41e6-a692-35d2de203a14-workload-certs\") pod \"e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml\" (UID: \"1ba1f825-34bb-41e6-a692-35d2de203a14\") " pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"
Sep 10 21:22:02.045609 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:02.045546    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba1f825-34bb-41e6-a692-35d2de203a14-workload-socket\") pod \"e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml\" (UID: \"1ba1f825-34bb-41e6-a692-35d2de203a14\") " pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"
Sep 10 21:22:02.045609 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:02.045601    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ba1f825-34bb-41e6-a692-35d2de203a14-istiod-ca-cert\") pod \"e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml\" (UID: \"1ba1f825-34bb-41e6-a692-35d2de203a14\") " pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"
Sep 10 21:22:02.047269 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:02.047247    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ba1f825-34bb-41e6-a692-35d2de203a14-istio-podinfo\") pod \"e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml\" (UID: \"1ba1f825-34bb-41e6-a692-35d2de203a14\") " pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"
Sep 10 21:22:02.047393 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:02.047269    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ba1f825-34bb-41e6-a692-35d2de203a14-istio-envoy\") pod \"e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml\" (UID: \"1ba1f825-34bb-41e6-a692-35d2de203a14\") " pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"
Sep 10 21:22:02.053877 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:02.053852    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ba1f825-34bb-41e6-a692-35d2de203a14-istio-token\") pod \"e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml\" (UID: \"1ba1f825-34bb-41e6-a692-35d2de203a14\") " pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"
Sep 10 21:22:02.054040 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:02.054025    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b794\" (UniqueName: \"kubernetes.io/projected/1ba1f825-34bb-41e6-a692-35d2de203a14-kube-api-access-4b794\") pod \"e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml\" (UID: \"1ba1f825-34bb-41e6-a692-35d2de203a14\") " pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"
Sep 10 21:22:02.237549 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:02.237463    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"
Sep 10 21:22:02.514713 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:02.514685    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"]
Sep 10 21:22:02.516619 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:22:02.516593    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ba1f825_34bb_41e6_a692_35d2de203a14.slice/crio-2850a1fc49e0279adc00e264d57e39985b2d25041dc5645c521ef07d20613e62 WatchSource:0}: Error finding container 2850a1fc49e0279adc00e264d57e39985b2d25041dc5645c521ef07d20613e62: Status 404 returned error can't find the container with id 2850a1fc49e0279adc00e264d57e39985b2d25041dc5645c521ef07d20613e62
Sep 10 21:22:02.518745 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:02.518713    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 21:22:02.518830 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:02.518779    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 21:22:02.518830 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:02.518806    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 21:22:03.197374 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:03.197335    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml" event={"ID":"1ba1f825-34bb-41e6-a692-35d2de203a14","Type":"ContainerStarted","Data":"37e2b7e01776ae1ab6694ea54035222aea50625bc1c4bc9e20467e5bd225924c"}
Sep 10 21:22:03.197717 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:03.197403    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml" event={"ID":"1ba1f825-34bb-41e6-a692-35d2de203a14","Type":"ContainerStarted","Data":"2850a1fc49e0279adc00e264d57e39985b2d25041dc5645c521ef07d20613e62"}
Sep 10 21:22:03.217619 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:03.217562    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml" podStartSLOduration=2.217545759 podStartE2EDuration="2.217545759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:22:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:22:03.216426699 +0000 UTC m=+1440.873335472" watchObservedRunningTime="2026-09-10 21:22:03.217545759 +0000 UTC m=+1440.874454531"
Sep 10 21:22:03.238281 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:03.238254    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"
Sep 10 21:22:03.243037 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:03.243009    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"
Sep 10 21:22:04.202741 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:04.202712    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"
Sep 10 21:22:04.203577 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:04.203560    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"
Sep 10 21:22:04.827034 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:04.827002    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-aca18f-7cd949bc79-8w99q"]
Sep 10 21:22:04.827270 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:04.827244    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-aca18f-7cd949bc79-8w99q" podUID="c50519ad-bd6d-4d75-b6fd-2d47b5691747" containerName="payload-processing" containerID="cri-o://ae2321303feedee9e32ffa47768d20ed19ff8f68f2cf2ed2a8e3545262275ab9" gracePeriod=30
Sep 10 21:22:04.979275 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:04.979241    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-aca18f-wtmnw"]
Sep 10 21:22:04.983553 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:04.983527    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-aca18f-wtmnw"]
Sep 10 21:22:05.083116 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:05.083056    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-aca18f-7cd949bc79-8w99q"
Sep 10 21:22:05.171042 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:05.171010    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbtnq\" (UniqueName: \"kubernetes.io/projected/c50519ad-bd6d-4d75-b6fd-2d47b5691747-kube-api-access-zbtnq\") pod \"c50519ad-bd6d-4d75-b6fd-2d47b5691747\" (UID: \"c50519ad-bd6d-4d75-b6fd-2d47b5691747\") "
Sep 10 21:22:05.171197 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:05.171071    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c50519ad-bd6d-4d75-b6fd-2d47b5691747-plugins-config-volume\") pod \"c50519ad-bd6d-4d75-b6fd-2d47b5691747\" (UID: \"c50519ad-bd6d-4d75-b6fd-2d47b5691747\") "
Sep 10 21:22:05.171474 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:05.171452    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c50519ad-bd6d-4d75-b6fd-2d47b5691747-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c50519ad-bd6d-4d75-b6fd-2d47b5691747" (UID: "c50519ad-bd6d-4d75-b6fd-2d47b5691747"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:22:05.173101 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:05.173078    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c50519ad-bd6d-4d75-b6fd-2d47b5691747-kube-api-access-zbtnq" (OuterVolumeSpecName: "kube-api-access-zbtnq") pod "c50519ad-bd6d-4d75-b6fd-2d47b5691747" (UID: "c50519ad-bd6d-4d75-b6fd-2d47b5691747"). InnerVolumeSpecName "kube-api-access-zbtnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:22:05.209666 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:05.209636    2581 generic.go:356] "Generic (PLEG): container finished" podID="c50519ad-bd6d-4d75-b6fd-2d47b5691747" containerID="ae2321303feedee9e32ffa47768d20ed19ff8f68f2cf2ed2a8e3545262275ab9" exitCode=0
Sep 10 21:22:05.209960 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:05.209684    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-aca18f-7cd949bc79-8w99q"
Sep 10 21:22:05.209960 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:05.209723    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-aca18f-7cd949bc79-8w99q" event={"ID":"c50519ad-bd6d-4d75-b6fd-2d47b5691747","Type":"ContainerDied","Data":"ae2321303feedee9e32ffa47768d20ed19ff8f68f2cf2ed2a8e3545262275ab9"}
Sep 10 21:22:05.209960 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:05.209777    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-aca18f-7cd949bc79-8w99q" event={"ID":"c50519ad-bd6d-4d75-b6fd-2d47b5691747","Type":"ContainerDied","Data":"dcb5a1269ab29eaa205d70f9a8374d2287f5115e092147e282e59a9450bccd06"}
Sep 10 21:22:05.209960 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:05.209805    2581 scope.go:117] "RemoveContainer" containerID="ae2321303feedee9e32ffa47768d20ed19ff8f68f2cf2ed2a8e3545262275ab9"
Sep 10 21:22:05.226488 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:05.226470    2581 scope.go:117] "RemoveContainer" containerID="ae2321303feedee9e32ffa47768d20ed19ff8f68f2cf2ed2a8e3545262275ab9"
Sep 10 21:22:05.226735 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:22:05.226719    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae2321303feedee9e32ffa47768d20ed19ff8f68f2cf2ed2a8e3545262275ab9\": container with ID starting with ae2321303feedee9e32ffa47768d20ed19ff8f68f2cf2ed2a8e3545262275ab9 not found: ID does not exist" containerID="ae2321303feedee9e32ffa47768d20ed19ff8f68f2cf2ed2a8e3545262275ab9"
Sep 10 21:22:05.226782 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:05.226741    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae2321303feedee9e32ffa47768d20ed19ff8f68f2cf2ed2a8e3545262275ab9"} err="failed to get container status \"ae2321303feedee9e32ffa47768d20ed19ff8f68f2cf2ed2a8e3545262275ab9\": rpc error: code = NotFound desc = could not find container \"ae2321303feedee9e32ffa47768d20ed19ff8f68f2cf2ed2a8e3545262275ab9\": container with ID starting with ae2321303feedee9e32ffa47768d20ed19ff8f68f2cf2ed2a8e3545262275ab9 not found: ID does not exist"
Sep 10 21:22:05.236567 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:05.236545    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-aca18f-7cd949bc79-8w99q"]
Sep 10 21:22:05.240130 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:05.240111    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-aca18f-7cd949bc79-8w99q"]
Sep 10 21:22:05.272810 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:05.272775    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c50519ad-bd6d-4d75-b6fd-2d47b5691747-plugins-config-volume\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:22:05.272810 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:05.272812    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbtnq\" (UniqueName: \"kubernetes.io/projected/c50519ad-bd6d-4d75-b6fd-2d47b5691747-kube-api-access-zbtnq\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:22:06.868501 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:06.868467    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cc004e7-7523-42b2-be51-8d5a1918b371" path="/var/lib/kubelet/pods/9cc004e7-7523-42b2-be51-8d5a1918b371/volumes"
Sep 10 21:22:06.868875 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:06.868859    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c50519ad-bd6d-4d75-b6fd-2d47b5691747" path="/var/lib/kubelet/pods/c50519ad-bd6d-4d75-b6fd-2d47b5691747/volumes"
Sep 10 21:22:07.119026 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:07.118937    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1427b1-787494c786-hs22s"]
Sep 10 21:22:07.119423 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:07.119404    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07844b88-223f-4b79-983d-f33403a57ec0" containerName="istio-proxy"
Sep 10 21:22:07.119531 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:07.119424    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="07844b88-223f-4b79-983d-f33403a57ec0" containerName="istio-proxy"
Sep 10 21:22:07.119531 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:07.119447    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c50519ad-bd6d-4d75-b6fd-2d47b5691747" containerName="payload-processing"
Sep 10 21:22:07.119531 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:07.119456    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c50519ad-bd6d-4d75-b6fd-2d47b5691747" containerName="payload-processing"
Sep 10 21:22:07.119681 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:07.119563    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c50519ad-bd6d-4d75-b6fd-2d47b5691747" containerName="payload-processing"
Sep 10 21:22:07.123629 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:07.123608    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1427b1-787494c786-hs22s"
Sep 10 21:22:07.125784 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:07.125760    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-1427b1-dockercfg-5khc9\""
Sep 10 21:22:07.125784 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:07.125776    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-1427b1\""
Sep 10 21:22:07.131628 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:07.131610    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1427b1-787494c786-hs22s"]
Sep 10 21:22:07.190685 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:07.190657    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62zsk\" (UniqueName: \"kubernetes.io/projected/caba7506-b396-4302-b039-45c432c2b44a-kube-api-access-62zsk\") pod \"payload-pre-processing-e2e-ipp-b-1427b1-787494c786-hs22s\" (UID: \"caba7506-b396-4302-b039-45c432c2b44a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1427b1-787494c786-hs22s"
Sep 10 21:22:07.190800 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:07.190711    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caba7506-b396-4302-b039-45c432c2b44a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1427b1-787494c786-hs22s\" (UID: \"caba7506-b396-4302-b039-45c432c2b44a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1427b1-787494c786-hs22s"
Sep 10 21:22:07.253583 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:07.253554    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1427b1-76558d5ddf-bh527"]
Sep 10 21:22:07.256938 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:07.256916    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1427b1-76558d5ddf-bh527"
Sep 10 21:22:07.266484 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:07.266461    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1427b1-76558d5ddf-bh527"]
Sep 10 21:22:07.291306 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:07.291265    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/626c8e14-8583-4d7c-9ded-e3048e1018e3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1427b1-76558d5ddf-bh527\" (UID: \"626c8e14-8583-4d7c-9ded-e3048e1018e3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1427b1-76558d5ddf-bh527"
Sep 10 21:22:07.291444 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:07.291388    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caba7506-b396-4302-b039-45c432c2b44a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1427b1-787494c786-hs22s\" (UID: \"caba7506-b396-4302-b039-45c432c2b44a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1427b1-787494c786-hs22s"
Sep 10 21:22:07.291535 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:07.291517    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62zsk\" (UniqueName: \"kubernetes.io/projected/caba7506-b396-4302-b039-45c432c2b44a-kube-api-access-62zsk\") pod \"payload-pre-processing-e2e-ipp-b-1427b1-787494c786-hs22s\" (UID: \"caba7506-b396-4302-b039-45c432c2b44a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1427b1-787494c786-hs22s"
Sep 10 21:22:07.291594 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:07.291563    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg2jd\" (UniqueName: \"kubernetes.io/projected/626c8e14-8583-4d7c-9ded-e3048e1018e3-kube-api-access-cg2jd\") pod \"payload-processing-e2e-ipp-b-1427b1-76558d5ddf-bh527\" (UID: \"626c8e14-8583-4d7c-9ded-e3048e1018e3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1427b1-76558d5ddf-bh527"
Sep 10 21:22:07.292199 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:07.292175    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caba7506-b396-4302-b039-45c432c2b44a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1427b1-787494c786-hs22s\" (UID: \"caba7506-b396-4302-b039-45c432c2b44a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1427b1-787494c786-hs22s"
Sep 10 21:22:07.299124 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:07.299095    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62zsk\" (UniqueName: \"kubernetes.io/projected/caba7506-b396-4302-b039-45c432c2b44a-kube-api-access-62zsk\") pod \"payload-pre-processing-e2e-ipp-b-1427b1-787494c786-hs22s\" (UID: \"caba7506-b396-4302-b039-45c432c2b44a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1427b1-787494c786-hs22s"
Sep 10 21:22:07.392769 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:07.392703    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg2jd\" (UniqueName: \"kubernetes.io/projected/626c8e14-8583-4d7c-9ded-e3048e1018e3-kube-api-access-cg2jd\") pod \"payload-processing-e2e-ipp-b-1427b1-76558d5ddf-bh527\" (UID: \"626c8e14-8583-4d7c-9ded-e3048e1018e3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1427b1-76558d5ddf-bh527"
Sep 10 21:22:07.392769 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:07.392743    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/626c8e14-8583-4d7c-9ded-e3048e1018e3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1427b1-76558d5ddf-bh527\" (UID: \"626c8e14-8583-4d7c-9ded-e3048e1018e3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1427b1-76558d5ddf-bh527"
Sep 10 21:22:07.393361 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:07.393345    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/626c8e14-8583-4d7c-9ded-e3048e1018e3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1427b1-76558d5ddf-bh527\" (UID: \"626c8e14-8583-4d7c-9ded-e3048e1018e3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1427b1-76558d5ddf-bh527"
Sep 10 21:22:07.402289 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:07.402260    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg2jd\" (UniqueName: \"kubernetes.io/projected/626c8e14-8583-4d7c-9ded-e3048e1018e3-kube-api-access-cg2jd\") pod \"payload-processing-e2e-ipp-b-1427b1-76558d5ddf-bh527\" (UID: \"626c8e14-8583-4d7c-9ded-e3048e1018e3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1427b1-76558d5ddf-bh527"
Sep 10 21:22:07.436606 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:07.436584    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1427b1-787494c786-hs22s"
Sep 10 21:22:07.571515 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:07.571484    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1427b1-76558d5ddf-bh527"
Sep 10 21:22:07.706151 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:07.706122    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1427b1-787494c786-hs22s"]
Sep 10 21:22:07.707945 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:22:07.707914    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaba7506_b396_4302_b039_45c432c2b44a.slice/crio-61c274f8ad6fa9d286bb5768e2195a0d161c3d629871cb0ad987481057464515 WatchSource:0}: Error finding container 61c274f8ad6fa9d286bb5768e2195a0d161c3d629871cb0ad987481057464515: Status 404 returned error can't find the container with id 61c274f8ad6fa9d286bb5768e2195a0d161c3d629871cb0ad987481057464515
Sep 10 21:22:07.845007 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:07.844976    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1427b1-76558d5ddf-bh527"]
Sep 10 21:22:07.846714 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:22:07.846691    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod626c8e14_8583_4d7c_9ded_e3048e1018e3.slice/crio-8eb8fa159900079c9596e085ff2847d6a4b478449809372d6c34b6ee005af5b3 WatchSource:0}: Error finding container 8eb8fa159900079c9596e085ff2847d6a4b478449809372d6c34b6ee005af5b3: Status 404 returned error can't find the container with id 8eb8fa159900079c9596e085ff2847d6a4b478449809372d6c34b6ee005af5b3
Sep 10 21:22:08.230582 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:08.230540    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1427b1-787494c786-hs22s" event={"ID":"caba7506-b396-4302-b039-45c432c2b44a","Type":"ContainerStarted","Data":"7e05fad13ce7d9b69b1bfd9e30a618e238d081fb837808039d9ee876d25e05a2"}
Sep 10 21:22:08.231043 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:08.230603    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1427b1-787494c786-hs22s" event={"ID":"caba7506-b396-4302-b039-45c432c2b44a","Type":"ContainerStarted","Data":"61c274f8ad6fa9d286bb5768e2195a0d161c3d629871cb0ad987481057464515"}
Sep 10 21:22:08.231043 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:08.230637    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1427b1-787494c786-hs22s"
Sep 10 21:22:08.232086 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:08.232053    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1427b1-76558d5ddf-bh527" event={"ID":"626c8e14-8583-4d7c-9ded-e3048e1018e3","Type":"ContainerStarted","Data":"756b98a01366316b916f8c4df30886fc4cbe027557a07aad37a8dcbd84e19bd9"}
Sep 10 21:22:08.232215 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:08.232103    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1427b1-76558d5ddf-bh527" event={"ID":"626c8e14-8583-4d7c-9ded-e3048e1018e3","Type":"ContainerStarted","Data":"8eb8fa159900079c9596e085ff2847d6a4b478449809372d6c34b6ee005af5b3"}
Sep 10 21:22:08.232215 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:08.232194    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1427b1-76558d5ddf-bh527"
Sep 10 21:22:08.248086 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:08.248043    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1427b1-787494c786-hs22s" podStartSLOduration=1.2480259089999999 podStartE2EDuration="1.248025909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:22:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:22:08.247341662 +0000 UTC m=+1445.904250435" watchObservedRunningTime="2026-09-10 21:22:08.248025909 +0000 UTC m=+1445.904934680"
Sep 10 21:22:08.266525 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:08.265053    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-1427b1-76558d5ddf-bh527" podStartSLOduration=1.265033197 podStartE2EDuration="1.265033197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:22:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:22:08.263039124 +0000 UTC m=+1445.919947898" watchObservedRunningTime="2026-09-10 21:22:08.265033197 +0000 UTC m=+1445.921941972"
Sep 10 21:22:18.303060 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:18.303024    2581 generic.go:356] "Generic (PLEG): container finished" podID="1ba1f825-34bb-41e6-a692-35d2de203a14" containerID="37e2b7e01776ae1ab6694ea54035222aea50625bc1c4bc9e20467e5bd225924c" exitCode=0
Sep 10 21:22:18.303507 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:18.303101    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml" event={"ID":"1ba1f825-34bb-41e6-a692-35d2de203a14","Type":"ContainerDied","Data":"37e2b7e01776ae1ab6694ea54035222aea50625bc1c4bc9e20467e5bd225924c"}
Sep 10 21:22:18.303577 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:18.303556    2581 scope.go:117] "RemoveContainer" containerID="37e2b7e01776ae1ab6694ea54035222aea50625bc1c4bc9e20467e5bd225924c"
Sep 10 21:22:18.304639 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:18.304607    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 21:22:18.304706 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:18.304679    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 21:22:18.304744 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:18.304709    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 10 21:22:19.202896 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:19.202857    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"
Sep 10 21:22:19.240074 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:19.240047    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1427b1-76558d5ddf-bh527"
Sep 10 21:22:19.240229 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:19.240120    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1427b1-787494c786-hs22s"
Sep 10 21:22:19.312133 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:19.312091    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml" event={"ID":"1ba1f825-34bb-41e6-a692-35d2de203a14","Type":"ContainerStarted","Data":"4566a2dc1e21c91d1876e06333de8f9d9a4963b7d9f08cd16023bf787cfa39df"}
Sep 10 21:22:20.238400 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:20.238359    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"
Sep 10 21:22:20.239704 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:20.239676    2581 patch_prober.go:28] interesting pod/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Sep 10 21:22:20.239819 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:20.239733    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml" podUID="1ba1f825-34bb-41e6-a692-35d2de203a14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Sep 10 21:22:21.238465 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:21.238432    2581 patch_prober.go:28] interesting pod/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Sep 10 21:22:21.238815 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:21.238485    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml" podUID="1ba1f825-34bb-41e6-a692-35d2de203a14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Sep 10 21:22:22.238700 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:22.238661    2581 patch_prober.go:28] interesting pod/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Sep 10 21:22:22.239109 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:22.238717    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml" podUID="1ba1f825-34bb-41e6-a692-35d2de203a14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Sep 10 21:22:23.238409 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:23.238377    2581 patch_prober.go:28] interesting pod/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Sep 10 21:22:23.238578 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:23.238432    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml" podUID="1ba1f825-34bb-41e6-a692-35d2de203a14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Sep 10 21:22:24.237803 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:24.237772    2581 patch_prober.go:28] interesting pod/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Sep 10 21:22:24.238193 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:24.237824    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml" podUID="1ba1f825-34bb-41e6-a692-35d2de203a14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Sep 10 21:22:25.242007 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:25.241973    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"
Sep 10 21:22:25.242486 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:25.242323    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"
Sep 10 21:22:25.242952 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:25.242935    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"
Sep 10 21:22:35.493791 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:22:35.493763    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-1427b1-76558d5ddf-bh527_626c8e14-8583-4d7c-9ded-e3048e1018e3/payload-processing/0.log"
Sep 10 21:23:01.203993 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:01.203961    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-1427b1-76558d5ddf-bh527_626c8e14-8583-4d7c-9ded-e3048e1018e3/payload-processing/0.log"
Sep 10 21:23:02.883733 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:02.883703    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-z6858_c1e06766-b48c-429e-92f1-196bb083360b/console-operator/1.log"
Sep 10 21:23:02.887158 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:02.887133    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-z6858_c1e06766-b48c-429e-92f1-196bb083360b/console-operator/1.log"
Sep 10 21:23:02.887557 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:02.887538    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brrtd_19d8d7ba-45a1-4013-a935-5670abe12b8a/ovn-acl-logging/0.log"
Sep 10 21:23:02.891018 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:02.890999    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brrtd_19d8d7ba-45a1-4013-a935-5670abe12b8a/ovn-acl-logging/0.log"
Sep 10 21:23:07.303189 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:07.303153    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c1753c-7bcbbbf45f-82lh7"]
Sep 10 21:23:07.305591 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:07.305575    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c1753c-7bcbbbf45f-82lh7"
Sep 10 21:23:07.308189 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:07.308172    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-c1753c-dockercfg-ccsz6\""
Sep 10 21:23:07.308286 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:07.308199    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-c1753c\""
Sep 10 21:23:07.318150 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:07.318130    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c1753c-7bcbbbf45f-82lh7"]
Sep 10 21:23:07.448452 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:07.448420    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5cnn\" (UniqueName: \"kubernetes.io/projected/558b51ef-34f6-4e32-b30d-9fa18b2f1b1f-kube-api-access-k5cnn\") pod \"payload-processing-e2e-ipp-cleanup-c1753c-7bcbbbf45f-82lh7\" (UID: \"558b51ef-34f6-4e32-b30d-9fa18b2f1b1f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c1753c-7bcbbbf45f-82lh7"
Sep 10 21:23:07.448648 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:07.448550    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/558b51ef-34f6-4e32-b30d-9fa18b2f1b1f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-c1753c-7bcbbbf45f-82lh7\" (UID: \"558b51ef-34f6-4e32-b30d-9fa18b2f1b1f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c1753c-7bcbbbf45f-82lh7"
Sep 10 21:23:07.549676 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:07.549646    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5cnn\" (UniqueName: \"kubernetes.io/projected/558b51ef-34f6-4e32-b30d-9fa18b2f1b1f-kube-api-access-k5cnn\") pod \"payload-processing-e2e-ipp-cleanup-c1753c-7bcbbbf45f-82lh7\" (UID: \"558b51ef-34f6-4e32-b30d-9fa18b2f1b1f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c1753c-7bcbbbf45f-82lh7"
Sep 10 21:23:07.549837 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:07.549717    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/558b51ef-34f6-4e32-b30d-9fa18b2f1b1f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-c1753c-7bcbbbf45f-82lh7\" (UID: \"558b51ef-34f6-4e32-b30d-9fa18b2f1b1f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c1753c-7bcbbbf45f-82lh7"
Sep 10 21:23:07.550405 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:07.550388    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/558b51ef-34f6-4e32-b30d-9fa18b2f1b1f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-c1753c-7bcbbbf45f-82lh7\" (UID: \"558b51ef-34f6-4e32-b30d-9fa18b2f1b1f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c1753c-7bcbbbf45f-82lh7"
Sep 10 21:23:07.557749 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:07.557690    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5cnn\" (UniqueName: \"kubernetes.io/projected/558b51ef-34f6-4e32-b30d-9fa18b2f1b1f-kube-api-access-k5cnn\") pod \"payload-processing-e2e-ipp-cleanup-c1753c-7bcbbbf45f-82lh7\" (UID: \"558b51ef-34f6-4e32-b30d-9fa18b2f1b1f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c1753c-7bcbbbf45f-82lh7"
Sep 10 21:23:07.616319 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:07.616271    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c1753c-7bcbbbf45f-82lh7"
Sep 10 21:23:07.907559 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:07.907535    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c1753c-7bcbbbf45f-82lh7"]
Sep 10 21:23:07.909725 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:23:07.909694    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod558b51ef_34f6_4e32_b30d_9fa18b2f1b1f.slice/crio-6c200156039381f2bf7d6c8c040355cf13edf3a8cd049ef13ae4e5d773d8c369 WatchSource:0}: Error finding container 6c200156039381f2bf7d6c8c040355cf13edf3a8cd049ef13ae4e5d773d8c369: Status 404 returned error can't find the container with id 6c200156039381f2bf7d6c8c040355cf13edf3a8cd049ef13ae4e5d773d8c369
Sep 10 21:23:08.581189 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:08.581157    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c1753c-7bcbbbf45f-82lh7" event={"ID":"558b51ef-34f6-4e32-b30d-9fa18b2f1b1f","Type":"ContainerStarted","Data":"e5463c84a486cdc59cbe07ce87f520e824a90ffa268d425234cbff7444428dab"}
Sep 10 21:23:08.581189 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:08.581207    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c1753c-7bcbbbf45f-82lh7" event={"ID":"558b51ef-34f6-4e32-b30d-9fa18b2f1b1f","Type":"ContainerStarted","Data":"6c200156039381f2bf7d6c8c040355cf13edf3a8cd049ef13ae4e5d773d8c369"}
Sep 10 21:23:08.581734 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:08.581242    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c1753c-7bcbbbf45f-82lh7"
Sep 10 21:23:08.599520 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:08.599473    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c1753c-7bcbbbf45f-82lh7" podStartSLOduration=1.599456306 podStartE2EDuration="1.599456306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:23:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:23:08.598787158 +0000 UTC m=+1506.255695930" watchObservedRunningTime="2026-09-10 21:23:08.599456306 +0000 UTC m=+1506.256365120"
Sep 10 21:23:19.589018 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:19.588982    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c1753c-7bcbbbf45f-82lh7"
Sep 10 21:23:31.242921 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:31.242836    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c1753c-npt87"]
Sep 10 21:23:31.245744 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:31.245728    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c1753c-npt87"
Sep 10 21:23:31.248329 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:31.248307    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s4qqn\""
Sep 10 21:23:31.248457 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:31.248330    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:23:31.255018 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:31.254992    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c1753c-npt87"]
Sep 10 21:23:31.270520 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:31.270498    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a49f64d7-34c9-4425-8c17-b21989b0f7cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-c1753c-npt87\" (UID: \"a49f64d7-34c9-4425-8c17-b21989b0f7cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c1753c-npt87"
Sep 10 21:23:31.371099 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:31.371073    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a49f64d7-34c9-4425-8c17-b21989b0f7cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-c1753c-npt87\" (UID: \"a49f64d7-34c9-4425-8c17-b21989b0f7cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c1753c-npt87"
Sep 10 21:23:31.371750 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:31.371725    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a49f64d7-34c9-4425-8c17-b21989b0f7cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-c1753c-npt87\" (UID: \"a49f64d7-34c9-4425-8c17-b21989b0f7cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c1753c-npt87"
Sep 10 21:23:31.558417 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:31.558392    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c1753c-npt87"
Sep 10 21:23:31.834290 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:31.834262    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c1753c-npt87"]
Sep 10 21:23:31.836066 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:23:31.836039    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda49f64d7_34c9_4425_8c17_b21989b0f7cc.slice/crio-f118b3b9018f4d5b3849241e0b4702712483e66b2cf15c41acc7dcbdc0059b14 WatchSource:0}: Error finding container f118b3b9018f4d5b3849241e0b4702712483e66b2cf15c41acc7dcbdc0059b14: Status 404 returned error can't find the container with id f118b3b9018f4d5b3849241e0b4702712483e66b2cf15c41acc7dcbdc0059b14
Sep 10 21:23:32.713077 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:32.713039    2581 generic.go:356] "Generic (PLEG): container finished" podID="a49f64d7-34c9-4425-8c17-b21989b0f7cc" containerID="af7f92ecfc0b8d393cee0917a93e1846c03300e64c15db87cb243887cbd6a99d" exitCode=0
Sep 10 21:23:32.713500 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:32.713125    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c1753c-npt87" event={"ID":"a49f64d7-34c9-4425-8c17-b21989b0f7cc","Type":"ContainerDied","Data":"af7f92ecfc0b8d393cee0917a93e1846c03300e64c15db87cb243887cbd6a99d"}
Sep 10 21:23:32.713500 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:32.713190    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c1753c-npt87" event={"ID":"a49f64d7-34c9-4425-8c17-b21989b0f7cc","Type":"ContainerStarted","Data":"f118b3b9018f4d5b3849241e0b4702712483e66b2cf15c41acc7dcbdc0059b14"}
Sep 10 21:23:33.851843 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:33.851819    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c1753c-npt87"
Sep 10 21:23:33.894667 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:33.894643    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a49f64d7-34c9-4425-8c17-b21989b0f7cc-maas-api-service-ca\") pod \"a49f64d7-34c9-4425-8c17-b21989b0f7cc\" (UID: \"a49f64d7-34c9-4425-8c17-b21989b0f7cc\") "
Sep 10 21:23:33.894991 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:33.894966    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a49f64d7-34c9-4425-8c17-b21989b0f7cc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a49f64d7-34c9-4425-8c17-b21989b0f7cc" (UID: "a49f64d7-34c9-4425-8c17-b21989b0f7cc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:23:33.996277 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:33.996220    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a49f64d7-34c9-4425-8c17-b21989b0f7cc-maas-api-service-ca\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:23:34.724698 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:34.724671    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c1753c-npt87"
Sep 10 21:23:34.724869 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:34.724695    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c1753c-npt87" event={"ID":"a49f64d7-34c9-4425-8c17-b21989b0f7cc","Type":"ContainerDied","Data":"f118b3b9018f4d5b3849241e0b4702712483e66b2cf15c41acc7dcbdc0059b14"}
Sep 10 21:23:34.724869 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:34.724739    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f118b3b9018f4d5b3849241e0b4702712483e66b2cf15c41acc7dcbdc0059b14"
Sep 10 21:23:37.407830 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:37.407798    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c1753c-7bcbbbf45f-82lh7"]
Sep 10 21:23:37.408693 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:37.408655    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c1753c-7bcbbbf45f-82lh7" podUID="558b51ef-34f6-4e32-b30d-9fa18b2f1b1f" containerName="payload-processing" containerID="cri-o://e5463c84a486cdc59cbe07ce87f520e824a90ffa268d425234cbff7444428dab" gracePeriod=30
Sep 10 21:23:37.679206 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:37.679180    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c1753c-7bcbbbf45f-82lh7"
Sep 10 21:23:37.683112 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:37.683089    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c1753c-npt87"]
Sep 10 21:23:37.686782 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:37.686759    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-c1753c-npt87"]
Sep 10 21:23:37.730306 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:37.730266    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/558b51ef-34f6-4e32-b30d-9fa18b2f1b1f-plugins-config-volume\") pod \"558b51ef-34f6-4e32-b30d-9fa18b2f1b1f\" (UID: \"558b51ef-34f6-4e32-b30d-9fa18b2f1b1f\") "
Sep 10 21:23:37.730474 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:37.730358    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5cnn\" (UniqueName: \"kubernetes.io/projected/558b51ef-34f6-4e32-b30d-9fa18b2f1b1f-kube-api-access-k5cnn\") pod \"558b51ef-34f6-4e32-b30d-9fa18b2f1b1f\" (UID: \"558b51ef-34f6-4e32-b30d-9fa18b2f1b1f\") "
Sep 10 21:23:37.730642 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:37.730608    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/558b51ef-34f6-4e32-b30d-9fa18b2f1b1f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "558b51ef-34f6-4e32-b30d-9fa18b2f1b1f" (UID: "558b51ef-34f6-4e32-b30d-9fa18b2f1b1f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:23:37.732424 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:37.732402    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/558b51ef-34f6-4e32-b30d-9fa18b2f1b1f-kube-api-access-k5cnn" (OuterVolumeSpecName: "kube-api-access-k5cnn") pod "558b51ef-34f6-4e32-b30d-9fa18b2f1b1f" (UID: "558b51ef-34f6-4e32-b30d-9fa18b2f1b1f"). InnerVolumeSpecName "kube-api-access-k5cnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:23:37.745308 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:37.745269    2581 generic.go:356] "Generic (PLEG): container finished" podID="558b51ef-34f6-4e32-b30d-9fa18b2f1b1f" containerID="e5463c84a486cdc59cbe07ce87f520e824a90ffa268d425234cbff7444428dab" exitCode=0
Sep 10 21:23:37.745436 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:37.745336    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c1753c-7bcbbbf45f-82lh7" event={"ID":"558b51ef-34f6-4e32-b30d-9fa18b2f1b1f","Type":"ContainerDied","Data":"e5463c84a486cdc59cbe07ce87f520e824a90ffa268d425234cbff7444428dab"}
Sep 10 21:23:37.745436 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:37.745408    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c1753c-7bcbbbf45f-82lh7" event={"ID":"558b51ef-34f6-4e32-b30d-9fa18b2f1b1f","Type":"ContainerDied","Data":"6c200156039381f2bf7d6c8c040355cf13edf3a8cd049ef13ae4e5d773d8c369"}
Sep 10 21:23:37.745514 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:37.745345    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-c1753c-7bcbbbf45f-82lh7"
Sep 10 21:23:37.745514 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:37.745443    2581 scope.go:117] "RemoveContainer" containerID="e5463c84a486cdc59cbe07ce87f520e824a90ffa268d425234cbff7444428dab"
Sep 10 21:23:37.755712 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:37.755689    2581 scope.go:117] "RemoveContainer" containerID="e5463c84a486cdc59cbe07ce87f520e824a90ffa268d425234cbff7444428dab"
Sep 10 21:23:37.756497 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:23:37.756464    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5463c84a486cdc59cbe07ce87f520e824a90ffa268d425234cbff7444428dab\": container with ID starting with e5463c84a486cdc59cbe07ce87f520e824a90ffa268d425234cbff7444428dab not found: ID does not exist" containerID="e5463c84a486cdc59cbe07ce87f520e824a90ffa268d425234cbff7444428dab"
Sep 10 21:23:37.756603 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:37.756509    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5463c84a486cdc59cbe07ce87f520e824a90ffa268d425234cbff7444428dab"} err="failed to get container status \"e5463c84a486cdc59cbe07ce87f520e824a90ffa268d425234cbff7444428dab\": rpc error: code = NotFound desc = could not find container \"e5463c84a486cdc59cbe07ce87f520e824a90ffa268d425234cbff7444428dab\": container with ID starting with e5463c84a486cdc59cbe07ce87f520e824a90ffa268d425234cbff7444428dab not found: ID does not exist"
Sep 10 21:23:37.773142 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:37.773116    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c1753c-7bcbbbf45f-82lh7"]
Sep 10 21:23:37.775121 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:37.775094    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-c1753c-7bcbbbf45f-82lh7"]
Sep 10 21:23:37.831918 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:37.831879    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/558b51ef-34f6-4e32-b30d-9fa18b2f1b1f-plugins-config-volume\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:23:37.831918 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:37.831914    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5cnn\" (UniqueName: \"kubernetes.io/projected/558b51ef-34f6-4e32-b30d-9fa18b2f1b1f-kube-api-access-k5cnn\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:23:38.868184 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:38.868153    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="558b51ef-34f6-4e32-b30d-9fa18b2f1b1f" path="/var/lib/kubelet/pods/558b51ef-34f6-4e32-b30d-9fa18b2f1b1f/volumes"
Sep 10 21:23:38.868587 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:38.868570    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a49f64d7-34c9-4425-8c17-b21989b0f7cc" path="/var/lib/kubelet/pods/a49f64d7-34c9-4425-8c17-b21989b0f7cc/volumes"
Sep 10 21:23:48.343731 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:48.343683    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7eb29-8c9tl"]
Sep 10 21:23:48.344206 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:48.344109    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a49f64d7-34c9-4425-8c17-b21989b0f7cc" containerName="revoke-keys"
Sep 10 21:23:48.344206 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:48.344123    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a49f64d7-34c9-4425-8c17-b21989b0f7cc" containerName="revoke-keys"
Sep 10 21:23:48.344206 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:48.344134    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="558b51ef-34f6-4e32-b30d-9fa18b2f1b1f" containerName="payload-processing"
Sep 10 21:23:48.344206 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:48.344139    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="558b51ef-34f6-4e32-b30d-9fa18b2f1b1f" containerName="payload-processing"
Sep 10 21:23:48.344361 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:48.344219    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="558b51ef-34f6-4e32-b30d-9fa18b2f1b1f" containerName="payload-processing"
Sep 10 21:23:48.344361 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:48.344230    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a49f64d7-34c9-4425-8c17-b21989b0f7cc" containerName="revoke-keys"
Sep 10 21:23:48.348944 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:48.348920    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7eb29-8c9tl"
Sep 10 21:23:48.351252 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:48.351207    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:23:48.352021 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:48.352001    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s4qqn\""
Sep 10 21:23:48.355102 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:48.355078    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7eb29-8c9tl"]
Sep 10 21:23:48.424602 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:48.424569    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ba7ceb1-03b3-4cb9-9db0-74a6746bd45b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f7eb29-8c9tl\" (UID: \"1ba7ceb1-03b3-4cb9-9db0-74a6746bd45b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7eb29-8c9tl"
Sep 10 21:23:48.525762 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:48.525730    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ba7ceb1-03b3-4cb9-9db0-74a6746bd45b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f7eb29-8c9tl\" (UID: \"1ba7ceb1-03b3-4cb9-9db0-74a6746bd45b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7eb29-8c9tl"
Sep 10 21:23:48.526350 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:48.526332    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ba7ceb1-03b3-4cb9-9db0-74a6746bd45b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f7eb29-8c9tl\" (UID: \"1ba7ceb1-03b3-4cb9-9db0-74a6746bd45b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7eb29-8c9tl"
Sep 10 21:23:48.661909 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:48.661821    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7eb29-8c9tl"
Sep 10 21:23:48.934971 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:48.934942    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7eb29-8c9tl"]
Sep 10 21:23:48.936621 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:23:48.936587    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ba7ceb1_03b3_4cb9_9db0_74a6746bd45b.slice/crio-9547d6a11ed6f7f3c66c78aa5f2d905eacfe27b196632b7d1c4ac1b012a7af6c WatchSource:0}: Error finding container 9547d6a11ed6f7f3c66c78aa5f2d905eacfe27b196632b7d1c4ac1b012a7af6c: Status 404 returned error can't find the container with id 9547d6a11ed6f7f3c66c78aa5f2d905eacfe27b196632b7d1c4ac1b012a7af6c
Sep 10 21:23:49.814220 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:49.814189    2581 generic.go:356] "Generic (PLEG): container finished" podID="1ba7ceb1-03b3-4cb9-9db0-74a6746bd45b" containerID="17126e689ed4167cccfbd37cd95f7763b554b77adb86a418b93a11a00af3bd78" exitCode=0
Sep 10 21:23:49.814656 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:49.814256    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7eb29-8c9tl" event={"ID":"1ba7ceb1-03b3-4cb9-9db0-74a6746bd45b","Type":"ContainerDied","Data":"17126e689ed4167cccfbd37cd95f7763b554b77adb86a418b93a11a00af3bd78"}
Sep 10 21:23:49.814656 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:49.814323    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7eb29-8c9tl" event={"ID":"1ba7ceb1-03b3-4cb9-9db0-74a6746bd45b","Type":"ContainerStarted","Data":"9547d6a11ed6f7f3c66c78aa5f2d905eacfe27b196632b7d1c4ac1b012a7af6c"}
Sep 10 21:23:50.956744 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:50.956724    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7eb29-8c9tl"
Sep 10 21:23:51.050476 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:51.050449    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ba7ceb1-03b3-4cb9-9db0-74a6746bd45b-maas-api-service-ca\") pod \"1ba7ceb1-03b3-4cb9-9db0-74a6746bd45b\" (UID: \"1ba7ceb1-03b3-4cb9-9db0-74a6746bd45b\") "
Sep 10 21:23:51.050772 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:51.050752    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ba7ceb1-03b3-4cb9-9db0-74a6746bd45b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1ba7ceb1-03b3-4cb9-9db0-74a6746bd45b" (UID: "1ba7ceb1-03b3-4cb9-9db0-74a6746bd45b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:23:51.151836 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:51.151757    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ba7ceb1-03b3-4cb9-9db0-74a6746bd45b-maas-api-service-ca\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:23:51.826378 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:51.826346    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7eb29-8c9tl"
Sep 10 21:23:51.826566 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:51.826392    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7eb29-8c9tl" event={"ID":"1ba7ceb1-03b3-4cb9-9db0-74a6746bd45b","Type":"ContainerDied","Data":"9547d6a11ed6f7f3c66c78aa5f2d905eacfe27b196632b7d1c4ac1b012a7af6c"}
Sep 10 21:23:51.826566 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:23:51.826450    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9547d6a11ed6f7f3c66c78aa5f2d905eacfe27b196632b7d1c4ac1b012a7af6c"
Sep 10 21:24:03.120664 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:24:03.120630    2581 scope.go:117] "RemoveContainer" containerID="af36511b646c6aaab911f476cf7b2bc5e3a387320744f08e83e6a7436a846ba6"
Sep 10 21:24:03.962460 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:24:03.962420    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f7eb29-56d4bb5998-bzd2r"]
Sep 10 21:24:03.962673 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:24:03.962643    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7eb29-56d4bb5998-bzd2r" podUID="cfbf31da-aee8-46bf-8c52-4e8b15be7743" containerName="payload-pre-processing" containerID="cri-o://a86142f81b31aa8bf8748247d5f5e1f42bc5002d95bcb663348f58fe02ddceaf" gracePeriod=30
Sep 10 21:24:04.126818 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:24:04.126775    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7eb29-8c9tl"]
Sep 10 21:24:04.129540 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:24:04.129515    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f7eb29-8c9tl"]
Sep 10 21:24:04.226184 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:24:04.225732    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7eb29-56d4bb5998-bzd2r"
Sep 10 21:24:04.371179 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:24:04.371150    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfbf31da-aee8-46bf-8c52-4e8b15be7743-plugins-config-volume\") pod \"cfbf31da-aee8-46bf-8c52-4e8b15be7743\" (UID: \"cfbf31da-aee8-46bf-8c52-4e8b15be7743\") "
Sep 10 21:24:04.371377 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:24:04.371241    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lphj\" (UniqueName: \"kubernetes.io/projected/cfbf31da-aee8-46bf-8c52-4e8b15be7743-kube-api-access-8lphj\") pod \"cfbf31da-aee8-46bf-8c52-4e8b15be7743\" (UID: \"cfbf31da-aee8-46bf-8c52-4e8b15be7743\") "
Sep 10 21:24:04.371648 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:24:04.371624    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfbf31da-aee8-46bf-8c52-4e8b15be7743-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cfbf31da-aee8-46bf-8c52-4e8b15be7743" (UID: "cfbf31da-aee8-46bf-8c52-4e8b15be7743"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:24:04.373414 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:24:04.373387    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfbf31da-aee8-46bf-8c52-4e8b15be7743-kube-api-access-8lphj" (OuterVolumeSpecName: "kube-api-access-8lphj") pod "cfbf31da-aee8-46bf-8c52-4e8b15be7743" (UID: "cfbf31da-aee8-46bf-8c52-4e8b15be7743"). InnerVolumeSpecName "kube-api-access-8lphj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:24:04.472287 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:24:04.472256    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfbf31da-aee8-46bf-8c52-4e8b15be7743-plugins-config-volume\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:24:04.472287 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:24:04.472284    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lphj\" (UniqueName: \"kubernetes.io/projected/cfbf31da-aee8-46bf-8c52-4e8b15be7743-kube-api-access-8lphj\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:24:04.870133 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:24:04.870100    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ba7ceb1-03b3-4cb9-9db0-74a6746bd45b" path="/var/lib/kubelet/pods/1ba7ceb1-03b3-4cb9-9db0-74a6746bd45b/volumes"
Sep 10 21:24:04.904049 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:24:04.904019    2581 generic.go:356] "Generic (PLEG): container finished" podID="cfbf31da-aee8-46bf-8c52-4e8b15be7743" containerID="a86142f81b31aa8bf8748247d5f5e1f42bc5002d95bcb663348f58fe02ddceaf" exitCode=0
Sep 10 21:24:04.904176 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:24:04.904069    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7eb29-56d4bb5998-bzd2r"
Sep 10 21:24:04.904176 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:24:04.904100    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7eb29-56d4bb5998-bzd2r" event={"ID":"cfbf31da-aee8-46bf-8c52-4e8b15be7743","Type":"ContainerDied","Data":"a86142f81b31aa8bf8748247d5f5e1f42bc5002d95bcb663348f58fe02ddceaf"}
Sep 10 21:24:04.904176 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:24:04.904145    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7eb29-56d4bb5998-bzd2r" event={"ID":"cfbf31da-aee8-46bf-8c52-4e8b15be7743","Type":"ContainerDied","Data":"fcf2d5883c657a39150e571b2fd3eacfd618f9ab7b7cd85134b8cf055a2cbd81"}
Sep 10 21:24:04.904176 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:24:04.904168    2581 scope.go:117] "RemoveContainer" containerID="a86142f81b31aa8bf8748247d5f5e1f42bc5002d95bcb663348f58fe02ddceaf"
Sep 10 21:24:04.912987 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:24:04.912971    2581 scope.go:117] "RemoveContainer" containerID="a86142f81b31aa8bf8748247d5f5e1f42bc5002d95bcb663348f58fe02ddceaf"
Sep 10 21:24:04.913248 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:24:04.913229    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a86142f81b31aa8bf8748247d5f5e1f42bc5002d95bcb663348f58fe02ddceaf\": container with ID starting with a86142f81b31aa8bf8748247d5f5e1f42bc5002d95bcb663348f58fe02ddceaf not found: ID does not exist" containerID="a86142f81b31aa8bf8748247d5f5e1f42bc5002d95bcb663348f58fe02ddceaf"
Sep 10 21:24:04.913308 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:24:04.913256    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a86142f81b31aa8bf8748247d5f5e1f42bc5002d95bcb663348f58fe02ddceaf"} err="failed to get container status \"a86142f81b31aa8bf8748247d5f5e1f42bc5002d95bcb663348f58fe02ddceaf\": rpc error: code = NotFound desc = could not find container \"a86142f81b31aa8bf8748247d5f5e1f42bc5002d95bcb663348f58fe02ddceaf\": container with ID starting with a86142f81b31aa8bf8748247d5f5e1f42bc5002d95bcb663348f58fe02ddceaf not found: ID does not exist"
Sep 10 21:24:04.920082 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:24:04.920060    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f7eb29-56d4bb5998-bzd2r"]
Sep 10 21:24:04.924580 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:24:04.924562    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f7eb29-56d4bb5998-bzd2r"]
Sep 10 21:24:06.869248 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:24:06.869212    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfbf31da-aee8-46bf-8c52-4e8b15be7743" path="/var/lib/kubelet/pods/cfbf31da-aee8-46bf-8c52-4e8b15be7743/volumes"
Sep 10 21:25:24.459063 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:24.459022    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1427b1-qrjfp"]
Sep 10 21:25:24.459736 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:24.459689    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfbf31da-aee8-46bf-8c52-4e8b15be7743" containerName="payload-pre-processing"
Sep 10 21:25:24.459736 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:24.459711    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfbf31da-aee8-46bf-8c52-4e8b15be7743" containerName="payload-pre-processing"
Sep 10 21:25:24.459868 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:24.459737    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ba7ceb1-03b3-4cb9-9db0-74a6746bd45b" containerName="revoke-keys"
Sep 10 21:25:24.459868 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:24.459747    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ba7ceb1-03b3-4cb9-9db0-74a6746bd45b" containerName="revoke-keys"
Sep 10 21:25:24.459868 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:24.459867    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ba7ceb1-03b3-4cb9-9db0-74a6746bd45b" containerName="revoke-keys"
Sep 10 21:25:24.460028 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:24.459881    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfbf31da-aee8-46bf-8c52-4e8b15be7743" containerName="payload-pre-processing"
Sep 10 21:25:24.463090 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:24.463066    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1427b1-qrjfp"
Sep 10 21:25:24.465530 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:24.465505    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s4qqn\""
Sep 10 21:25:24.465665 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:24.465514    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:25:24.473881 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:24.473857    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1427b1-qrjfp"]
Sep 10 21:25:24.622739 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:24.622706    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae7167c9-7ce1-4717-83eb-60eb108c82b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1427b1-qrjfp\" (UID: \"ae7167c9-7ce1-4717-83eb-60eb108c82b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1427b1-qrjfp"
Sep 10 21:25:24.723531 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:24.723453    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae7167c9-7ce1-4717-83eb-60eb108c82b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1427b1-qrjfp\" (UID: \"ae7167c9-7ce1-4717-83eb-60eb108c82b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1427b1-qrjfp"
Sep 10 21:25:24.724084 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:24.724063    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae7167c9-7ce1-4717-83eb-60eb108c82b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1427b1-qrjfp\" (UID: \"ae7167c9-7ce1-4717-83eb-60eb108c82b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1427b1-qrjfp"
Sep 10 21:25:24.775497 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:24.775465    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1427b1-qrjfp"
Sep 10 21:25:25.047392 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:25.047359    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1427b1-qrjfp"]
Sep 10 21:25:25.050582 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:25:25.050557    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae7167c9_7ce1_4717_83eb_60eb108c82b2.slice/crio-a1bb454dd80d82895640512bd4e847d0a17d4928586b732ea4de6582b8785bc1 WatchSource:0}: Error finding container a1bb454dd80d82895640512bd4e847d0a17d4928586b732ea4de6582b8785bc1: Status 404 returned error can't find the container with id a1bb454dd80d82895640512bd4e847d0a17d4928586b732ea4de6582b8785bc1
Sep 10 21:25:25.330766 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:25.330690    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1427b1-qrjfp" event={"ID":"ae7167c9-7ce1-4717-83eb-60eb108c82b2","Type":"ContainerStarted","Data":"a1bb454dd80d82895640512bd4e847d0a17d4928586b732ea4de6582b8785bc1"}
Sep 10 21:25:26.338097 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:26.338062    2581 generic.go:356] "Generic (PLEG): container finished" podID="ae7167c9-7ce1-4717-83eb-60eb108c82b2" containerID="f3dd3ed9db5d5c3423f308be00f44d5796dc0a4dd4697be4bb956acf3d49b752" exitCode=0
Sep 10 21:25:26.338557 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:26.338148    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1427b1-qrjfp" event={"ID":"ae7167c9-7ce1-4717-83eb-60eb108c82b2","Type":"ContainerDied","Data":"f3dd3ed9db5d5c3423f308be00f44d5796dc0a4dd4697be4bb956acf3d49b752"}
Sep 10 21:25:27.483192 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:27.483169    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1427b1-qrjfp"
Sep 10 21:25:27.651429 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:27.651348    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae7167c9-7ce1-4717-83eb-60eb108c82b2-maas-api-service-ca\") pod \"ae7167c9-7ce1-4717-83eb-60eb108c82b2\" (UID: \"ae7167c9-7ce1-4717-83eb-60eb108c82b2\") "
Sep 10 21:25:27.651746 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:27.651725    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae7167c9-7ce1-4717-83eb-60eb108c82b2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ae7167c9-7ce1-4717-83eb-60eb108c82b2" (UID: "ae7167c9-7ce1-4717-83eb-60eb108c82b2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:25:27.752317 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:27.752256    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae7167c9-7ce1-4717-83eb-60eb108c82b2-maas-api-service-ca\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:25:28.350496 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:28.350459    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1427b1-qrjfp"
Sep 10 21:25:28.350496 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:28.350486    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1427b1-qrjfp" event={"ID":"ae7167c9-7ce1-4717-83eb-60eb108c82b2","Type":"ContainerDied","Data":"a1bb454dd80d82895640512bd4e847d0a17d4928586b732ea4de6582b8785bc1"}
Sep 10 21:25:28.350709 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:28.350538    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1bb454dd80d82895640512bd4e847d0a17d4928586b732ea4de6582b8785bc1"
Sep 10 21:25:35.418393 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:35.418360    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1427b1-76558d5ddf-bh527"]
Sep 10 21:25:35.418893 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:35.418626    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-1427b1-76558d5ddf-bh527" podUID="626c8e14-8583-4d7c-9ded-e3048e1018e3" containerName="payload-processing" containerID="cri-o://756b98a01366316b916f8c4df30886fc4cbe027557a07aad37a8dcbd84e19bd9" gracePeriod=30
Sep 10 21:25:35.430738 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:35.430709    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1427b1-787494c786-hs22s"]
Sep 10 21:25:35.430970 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:35.430943    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1427b1-787494c786-hs22s" podUID="caba7506-b396-4302-b039-45c432c2b44a" containerName="payload-pre-processing" containerID="cri-o://7e05fad13ce7d9b69b1bfd9e30a618e238d081fb837808039d9ee876d25e05a2" gracePeriod=30
Sep 10 21:25:35.718880 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:35.718860    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1427b1-76558d5ddf-bh527"
Sep 10 21:25:35.721717 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:35.721699    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1427b1-787494c786-hs22s"
Sep 10 21:25:35.755827 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:35.755798    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1427b1-qrjfp"]
Sep 10 21:25:35.760671 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:35.760645    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1427b1-qrjfp"]
Sep 10 21:25:35.817421 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:35.817385    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cg2jd\" (UniqueName: \"kubernetes.io/projected/626c8e14-8583-4d7c-9ded-e3048e1018e3-kube-api-access-cg2jd\") pod \"626c8e14-8583-4d7c-9ded-e3048e1018e3\" (UID: \"626c8e14-8583-4d7c-9ded-e3048e1018e3\") "
Sep 10 21:25:35.817584 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:35.817430    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/626c8e14-8583-4d7c-9ded-e3048e1018e3-plugins-config-volume\") pod \"626c8e14-8583-4d7c-9ded-e3048e1018e3\" (UID: \"626c8e14-8583-4d7c-9ded-e3048e1018e3\") "
Sep 10 21:25:35.817584 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:35.817459    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caba7506-b396-4302-b039-45c432c2b44a-plugins-config-volume\") pod \"caba7506-b396-4302-b039-45c432c2b44a\" (UID: \"caba7506-b396-4302-b039-45c432c2b44a\") "
Sep 10 21:25:35.817584 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:35.817486    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62zsk\" (UniqueName: \"kubernetes.io/projected/caba7506-b396-4302-b039-45c432c2b44a-kube-api-access-62zsk\") pod \"caba7506-b396-4302-b039-45c432c2b44a\" (UID: \"caba7506-b396-4302-b039-45c432c2b44a\") "
Sep 10 21:25:35.817870 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:35.817839    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/626c8e14-8583-4d7c-9ded-e3048e1018e3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "626c8e14-8583-4d7c-9ded-e3048e1018e3" (UID: "626c8e14-8583-4d7c-9ded-e3048e1018e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:25:35.817870 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:35.817856    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caba7506-b396-4302-b039-45c432c2b44a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "caba7506-b396-4302-b039-45c432c2b44a" (UID: "caba7506-b396-4302-b039-45c432c2b44a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:25:35.819562 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:35.819539    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caba7506-b396-4302-b039-45c432c2b44a-kube-api-access-62zsk" (OuterVolumeSpecName: "kube-api-access-62zsk") pod "caba7506-b396-4302-b039-45c432c2b44a" (UID: "caba7506-b396-4302-b039-45c432c2b44a"). InnerVolumeSpecName "kube-api-access-62zsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:25:35.819646 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:35.819572    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/626c8e14-8583-4d7c-9ded-e3048e1018e3-kube-api-access-cg2jd" (OuterVolumeSpecName: "kube-api-access-cg2jd") pod "626c8e14-8583-4d7c-9ded-e3048e1018e3" (UID: "626c8e14-8583-4d7c-9ded-e3048e1018e3"). InnerVolumeSpecName "kube-api-access-cg2jd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:25:35.924001 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:35.919025    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg2jd\" (UniqueName: \"kubernetes.io/projected/626c8e14-8583-4d7c-9ded-e3048e1018e3-kube-api-access-cg2jd\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:25:35.924563 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:35.924527    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/626c8e14-8583-4d7c-9ded-e3048e1018e3-plugins-config-volume\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:25:35.924700 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:35.924586    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caba7506-b396-4302-b039-45c432c2b44a-plugins-config-volume\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:25:35.924700 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:35.924603    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62zsk\" (UniqueName: \"kubernetes.io/projected/caba7506-b396-4302-b039-45c432c2b44a-kube-api-access-62zsk\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:25:36.392146 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:36.392112    2581 generic.go:356] "Generic (PLEG): container finished" podID="626c8e14-8583-4d7c-9ded-e3048e1018e3" containerID="756b98a01366316b916f8c4df30886fc4cbe027557a07aad37a8dcbd84e19bd9" exitCode=0
Sep 10 21:25:36.392343 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:36.392164    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1427b1-76558d5ddf-bh527"
Sep 10 21:25:36.392343 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:36.392199    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1427b1-76558d5ddf-bh527" event={"ID":"626c8e14-8583-4d7c-9ded-e3048e1018e3","Type":"ContainerDied","Data":"756b98a01366316b916f8c4df30886fc4cbe027557a07aad37a8dcbd84e19bd9"}
Sep 10 21:25:36.392343 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:36.392278    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1427b1-76558d5ddf-bh527" event={"ID":"626c8e14-8583-4d7c-9ded-e3048e1018e3","Type":"ContainerDied","Data":"8eb8fa159900079c9596e085ff2847d6a4b478449809372d6c34b6ee005af5b3"}
Sep 10 21:25:36.392343 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:36.392318    2581 scope.go:117] "RemoveContainer" containerID="756b98a01366316b916f8c4df30886fc4cbe027557a07aad37a8dcbd84e19bd9"
Sep 10 21:25:36.393567 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:36.393546    2581 generic.go:356] "Generic (PLEG): container finished" podID="caba7506-b396-4302-b039-45c432c2b44a" containerID="7e05fad13ce7d9b69b1bfd9e30a618e238d081fb837808039d9ee876d25e05a2" exitCode=0
Sep 10 21:25:36.393648 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:36.393608    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1427b1-787494c786-hs22s" event={"ID":"caba7506-b396-4302-b039-45c432c2b44a","Type":"ContainerDied","Data":"7e05fad13ce7d9b69b1bfd9e30a618e238d081fb837808039d9ee876d25e05a2"}
Sep 10 21:25:36.393648 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:36.393621    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1427b1-787494c786-hs22s"
Sep 10 21:25:36.393648 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:36.393642    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1427b1-787494c786-hs22s" event={"ID":"caba7506-b396-4302-b039-45c432c2b44a","Type":"ContainerDied","Data":"61c274f8ad6fa9d286bb5768e2195a0d161c3d629871cb0ad987481057464515"}
Sep 10 21:25:36.402641 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:36.402621    2581 scope.go:117] "RemoveContainer" containerID="756b98a01366316b916f8c4df30886fc4cbe027557a07aad37a8dcbd84e19bd9"
Sep 10 21:25:36.402969 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:25:36.402949    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"756b98a01366316b916f8c4df30886fc4cbe027557a07aad37a8dcbd84e19bd9\": container with ID starting with 756b98a01366316b916f8c4df30886fc4cbe027557a07aad37a8dcbd84e19bd9 not found: ID does not exist" containerID="756b98a01366316b916f8c4df30886fc4cbe027557a07aad37a8dcbd84e19bd9"
Sep 10 21:25:36.403043 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:36.402974    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"756b98a01366316b916f8c4df30886fc4cbe027557a07aad37a8dcbd84e19bd9"} err="failed to get container status \"756b98a01366316b916f8c4df30886fc4cbe027557a07aad37a8dcbd84e19bd9\": rpc error: code = NotFound desc = could not find container \"756b98a01366316b916f8c4df30886fc4cbe027557a07aad37a8dcbd84e19bd9\": container with ID starting with 756b98a01366316b916f8c4df30886fc4cbe027557a07aad37a8dcbd84e19bd9 not found: ID does not exist"
Sep 10 21:25:36.403043 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:36.402989    2581 scope.go:117] "RemoveContainer" containerID="7e05fad13ce7d9b69b1bfd9e30a618e238d081fb837808039d9ee876d25e05a2"
Sep 10 21:25:36.410754 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:36.410738    2581 scope.go:117] "RemoveContainer" containerID="7e05fad13ce7d9b69b1bfd9e30a618e238d081fb837808039d9ee876d25e05a2"
Sep 10 21:25:36.410988 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:25:36.410970    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e05fad13ce7d9b69b1bfd9e30a618e238d081fb837808039d9ee876d25e05a2\": container with ID starting with 7e05fad13ce7d9b69b1bfd9e30a618e238d081fb837808039d9ee876d25e05a2 not found: ID does not exist" containerID="7e05fad13ce7d9b69b1bfd9e30a618e238d081fb837808039d9ee876d25e05a2"
Sep 10 21:25:36.411029 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:36.410994    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e05fad13ce7d9b69b1bfd9e30a618e238d081fb837808039d9ee876d25e05a2"} err="failed to get container status \"7e05fad13ce7d9b69b1bfd9e30a618e238d081fb837808039d9ee876d25e05a2\": rpc error: code = NotFound desc = could not find container \"7e05fad13ce7d9b69b1bfd9e30a618e238d081fb837808039d9ee876d25e05a2\": container with ID starting with 7e05fad13ce7d9b69b1bfd9e30a618e238d081fb837808039d9ee876d25e05a2 not found: ID does not exist"
Sep 10 21:25:36.415733 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:36.415714    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1427b1-76558d5ddf-bh527"]
Sep 10 21:25:36.420424 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:36.420402    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1427b1-76558d5ddf-bh527"]
Sep 10 21:25:36.431168 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:36.431146    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1427b1-787494c786-hs22s"]
Sep 10 21:25:36.438106 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:36.438087    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1427b1-787494c786-hs22s"]
Sep 10 21:25:36.868530 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:36.868487    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="626c8e14-8583-4d7c-9ded-e3048e1018e3" path="/var/lib/kubelet/pods/626c8e14-8583-4d7c-9ded-e3048e1018e3/volumes"
Sep 10 21:25:36.868914 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:36.868892    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae7167c9-7ce1-4717-83eb-60eb108c82b2" path="/var/lib/kubelet/pods/ae7167c9-7ce1-4717-83eb-60eb108c82b2/volumes"
Sep 10 21:25:36.869293 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:36.869275    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="caba7506-b396-4302-b039-45c432c2b44a" path="/var/lib/kubelet/pods/caba7506-b396-4302-b039-45c432c2b44a/volumes"
Sep 10 21:25:43.133883 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:43.133848    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"]
Sep 10 21:25:43.134271 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:43.134041    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml" podUID="1ba1f825-34bb-41e6-a692-35d2de203a14" containerName="istio-proxy" containerID="cri-o://4566a2dc1e21c91d1876e06333de8f9d9a4963b7d9f08cd16023bf787cfa39df" gracePeriod=30
Sep 10 21:25:43.570432 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:43.570396    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-828c7c-z28r6"]
Sep 10 21:25:43.570966 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:43.570946    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caba7506-b396-4302-b039-45c432c2b44a" containerName="payload-pre-processing"
Sep 10 21:25:43.571015 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:43.570975    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="caba7506-b396-4302-b039-45c432c2b44a" containerName="payload-pre-processing"
Sep 10 21:25:43.571015 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:43.571007    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="626c8e14-8583-4d7c-9ded-e3048e1018e3" containerName="payload-processing"
Sep 10 21:25:43.571085 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:43.571016    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="626c8e14-8583-4d7c-9ded-e3048e1018e3" containerName="payload-processing"
Sep 10 21:25:43.571085 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:43.571040    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae7167c9-7ce1-4717-83eb-60eb108c82b2" containerName="revoke-keys"
Sep 10 21:25:43.571085 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:43.571048    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae7167c9-7ce1-4717-83eb-60eb108c82b2" containerName="revoke-keys"
Sep 10 21:25:43.571180 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:43.571150    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="caba7506-b396-4302-b039-45c432c2b44a" containerName="payload-pre-processing"
Sep 10 21:25:43.571180 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:43.571164    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="626c8e14-8583-4d7c-9ded-e3048e1018e3" containerName="payload-processing"
Sep 10 21:25:43.571180 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:43.571175    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae7167c9-7ce1-4717-83eb-60eb108c82b2" containerName="revoke-keys"
Sep 10 21:25:43.576251 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:43.576228    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-828c7c-z28r6"
Sep 10 21:25:43.600708 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:43.600685    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s4qqn\""
Sep 10 21:25:43.606736 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:43.606717    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:25:43.614896 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:43.614869    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-828c7c-z28r6"]
Sep 10 21:25:43.695936 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:43.695907    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9837fd76-a25d-4f7b-bdc7-034809caf8a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-828c7c-z28r6\" (UID: \"9837fd76-a25d-4f7b-bdc7-034809caf8a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-828c7c-z28r6"
Sep 10 21:25:43.796875 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:43.796845    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9837fd76-a25d-4f7b-bdc7-034809caf8a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-828c7c-z28r6\" (UID: \"9837fd76-a25d-4f7b-bdc7-034809caf8a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-828c7c-z28r6"
Sep 10 21:25:43.797432 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:43.797414    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9837fd76-a25d-4f7b-bdc7-034809caf8a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-828c7c-z28r6\" (UID: \"9837fd76-a25d-4f7b-bdc7-034809caf8a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-828c7c-z28r6"
Sep 10 21:25:43.886875 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:43.886789    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-828c7c-z28r6"
Sep 10 21:25:44.166399 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:44.166371    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-828c7c-z28r6"]
Sep 10 21:25:44.168466 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:25:44.168430    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9837fd76_a25d_4f7b_bdc7_034809caf8a0.slice/crio-f40f8db587ef80ccc44251bc23f5738a52bddd61eb2c91a4071870320ece46b8 WatchSource:0}: Error finding container f40f8db587ef80ccc44251bc23f5738a52bddd61eb2c91a4071870320ece46b8: Status 404 returned error can't find the container with id f40f8db587ef80ccc44251bc23f5738a52bddd61eb2c91a4071870320ece46b8
Sep 10 21:25:44.441707 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:44.441627    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-828c7c-z28r6" event={"ID":"9837fd76-a25d-4f7b-bdc7-034809caf8a0","Type":"ContainerStarted","Data":"f40f8db587ef80ccc44251bc23f5738a52bddd61eb2c91a4071870320ece46b8"}
Sep 10 21:25:45.448485 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:45.448454    2581 generic.go:356] "Generic (PLEG): container finished" podID="9837fd76-a25d-4f7b-bdc7-034809caf8a0" containerID="d1612543ea6bcceddb1a444fa8707b208fbfe1e8349f2c68677dce4589cac1bf" exitCode=0
Sep 10 21:25:45.448839 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:45.448528    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-828c7c-z28r6" event={"ID":"9837fd76-a25d-4f7b-bdc7-034809caf8a0","Type":"ContainerDied","Data":"d1612543ea6bcceddb1a444fa8707b208fbfe1e8349f2c68677dce4589cac1bf"}
Sep 10 21:25:46.579047 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:46.579023    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-828c7c-z28r6"
Sep 10 21:25:46.622624 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:46.622592    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9837fd76-a25d-4f7b-bdc7-034809caf8a0-maas-api-service-ca\") pod \"9837fd76-a25d-4f7b-bdc7-034809caf8a0\" (UID: \"9837fd76-a25d-4f7b-bdc7-034809caf8a0\") "
Sep 10 21:25:46.622952 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:46.622927    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9837fd76-a25d-4f7b-bdc7-034809caf8a0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9837fd76-a25d-4f7b-bdc7-034809caf8a0" (UID: "9837fd76-a25d-4f7b-bdc7-034809caf8a0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:25:46.723498 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:46.723428    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9837fd76-a25d-4f7b-bdc7-034809caf8a0-maas-api-service-ca\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:25:47.460253 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:47.460221    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-828c7c-z28r6" event={"ID":"9837fd76-a25d-4f7b-bdc7-034809caf8a0","Type":"ContainerDied","Data":"f40f8db587ef80ccc44251bc23f5738a52bddd61eb2c91a4071870320ece46b8"}
Sep 10 21:25:47.460253 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:47.460255    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-828c7c-z28r6"
Sep 10 21:25:47.460465 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:47.460269    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f40f8db587ef80ccc44251bc23f5738a52bddd61eb2c91a4071870320ece46b8"
Sep 10 21:25:48.382651 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:48.382626    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"
Sep 10 21:25:48.440236 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:48.440147    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ba1f825-34bb-41e6-a692-35d2de203a14-istio-token\") pod \"1ba1f825-34bb-41e6-a692-35d2de203a14\" (UID: \"1ba1f825-34bb-41e6-a692-35d2de203a14\") "
Sep 10 21:25:48.440236 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:48.440196    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ba1f825-34bb-41e6-a692-35d2de203a14-workload-certs\") pod \"1ba1f825-34bb-41e6-a692-35d2de203a14\" (UID: \"1ba1f825-34bb-41e6-a692-35d2de203a14\") "
Sep 10 21:25:48.440236 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:48.440235    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ba1f825-34bb-41e6-a692-35d2de203a14-istiod-ca-cert\") pod \"1ba1f825-34bb-41e6-a692-35d2de203a14\" (UID: \"1ba1f825-34bb-41e6-a692-35d2de203a14\") "
Sep 10 21:25:48.440522 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:48.440276    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba1f825-34bb-41e6-a692-35d2de203a14-workload-socket\") pod \"1ba1f825-34bb-41e6-a692-35d2de203a14\" (UID: \"1ba1f825-34bb-41e6-a692-35d2de203a14\") "
Sep 10 21:25:48.440522 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:48.440352    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4b794\" (UniqueName: \"kubernetes.io/projected/1ba1f825-34bb-41e6-a692-35d2de203a14-kube-api-access-4b794\") pod \"1ba1f825-34bb-41e6-a692-35d2de203a14\" (UID: \"1ba1f825-34bb-41e6-a692-35d2de203a14\") "
Sep 10 21:25:48.440522 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:48.440392    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ba1f825-34bb-41e6-a692-35d2de203a14-istio-envoy\") pod \"1ba1f825-34bb-41e6-a692-35d2de203a14\" (UID: \"1ba1f825-34bb-41e6-a692-35d2de203a14\") "
Sep 10 21:25:48.440522 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:48.440458    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ba1f825-34bb-41e6-a692-35d2de203a14-istio-podinfo\") pod \"1ba1f825-34bb-41e6-a692-35d2de203a14\" (UID: \"1ba1f825-34bb-41e6-a692-35d2de203a14\") "
Sep 10 21:25:48.440522 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:48.440471    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba1f825-34bb-41e6-a692-35d2de203a14-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1ba1f825-34bb-41e6-a692-35d2de203a14" (UID: "1ba1f825-34bb-41e6-a692-35d2de203a14"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:25:48.440522 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:48.440486    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba1f825-34bb-41e6-a692-35d2de203a14-credential-socket\") pod \"1ba1f825-34bb-41e6-a692-35d2de203a14\" (UID: \"1ba1f825-34bb-41e6-a692-35d2de203a14\") "
Sep 10 21:25:48.440829 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:48.440561    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ba1f825-34bb-41e6-a692-35d2de203a14-istio-data\") pod \"1ba1f825-34bb-41e6-a692-35d2de203a14\" (UID: \"1ba1f825-34bb-41e6-a692-35d2de203a14\") "
Sep 10 21:25:48.440829 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:48.440647    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba1f825-34bb-41e6-a692-35d2de203a14-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1ba1f825-34bb-41e6-a692-35d2de203a14" (UID: "1ba1f825-34bb-41e6-a692-35d2de203a14"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:25:48.440931 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:48.440891    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ba1f825-34bb-41e6-a692-35d2de203a14-workload-certs\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:25:48.440931 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:48.440911    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba1f825-34bb-41e6-a692-35d2de203a14-workload-socket\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:25:48.440931 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:48.440912    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba1f825-34bb-41e6-a692-35d2de203a14-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1ba1f825-34bb-41e6-a692-35d2de203a14" (UID: "1ba1f825-34bb-41e6-a692-35d2de203a14"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:25:48.441100 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:48.440933    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ba1f825-34bb-41e6-a692-35d2de203a14-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1ba1f825-34bb-41e6-a692-35d2de203a14" (UID: "1ba1f825-34bb-41e6-a692-35d2de203a14"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:25:48.441100 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:48.441055    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba1f825-34bb-41e6-a692-35d2de203a14-istio-data" (OuterVolumeSpecName: "istio-data") pod "1ba1f825-34bb-41e6-a692-35d2de203a14" (UID: "1ba1f825-34bb-41e6-a692-35d2de203a14"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:25:48.442515 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:48.442486    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ba1f825-34bb-41e6-a692-35d2de203a14-kube-api-access-4b794" (OuterVolumeSpecName: "kube-api-access-4b794") pod "1ba1f825-34bb-41e6-a692-35d2de203a14" (UID: "1ba1f825-34bb-41e6-a692-35d2de203a14"). InnerVolumeSpecName "kube-api-access-4b794". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:25:48.442618 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:48.442531    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ba1f825-34bb-41e6-a692-35d2de203a14-istio-token" (OuterVolumeSpecName: "istio-token") pod "1ba1f825-34bb-41e6-a692-35d2de203a14" (UID: "1ba1f825-34bb-41e6-a692-35d2de203a14"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:25:48.442673 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:48.442656    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ba1f825-34bb-41e6-a692-35d2de203a14-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1ba1f825-34bb-41e6-a692-35d2de203a14" (UID: "1ba1f825-34bb-41e6-a692-35d2de203a14"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 21:25:48.442772 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:48.442745    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba1f825-34bb-41e6-a692-35d2de203a14-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1ba1f825-34bb-41e6-a692-35d2de203a14" (UID: "1ba1f825-34bb-41e6-a692-35d2de203a14"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:25:48.467582 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:48.467548    2581 generic.go:356] "Generic (PLEG): container finished" podID="1ba1f825-34bb-41e6-a692-35d2de203a14" containerID="4566a2dc1e21c91d1876e06333de8f9d9a4963b7d9f08cd16023bf787cfa39df" exitCode=0
Sep 10 21:25:48.467707 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:48.467607    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"
Sep 10 21:25:48.467707 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:48.467633    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml" event={"ID":"1ba1f825-34bb-41e6-a692-35d2de203a14","Type":"ContainerDied","Data":"4566a2dc1e21c91d1876e06333de8f9d9a4963b7d9f08cd16023bf787cfa39df"}
Sep 10 21:25:48.467707 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:48.467697    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml" event={"ID":"1ba1f825-34bb-41e6-a692-35d2de203a14","Type":"ContainerDied","Data":"2850a1fc49e0279adc00e264d57e39985b2d25041dc5645c521ef07d20613e62"}
Sep 10 21:25:48.467893 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:48.467728    2581 scope.go:117] "RemoveContainer" containerID="4566a2dc1e21c91d1876e06333de8f9d9a4963b7d9f08cd16023bf787cfa39df"
Sep 10 21:25:48.479011 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:48.478994    2581 scope.go:117] "RemoveContainer" containerID="37e2b7e01776ae1ab6694ea54035222aea50625bc1c4bc9e20467e5bd225924c"
Sep 10 21:25:48.487041 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:48.487020    2581 scope.go:117] "RemoveContainer" containerID="4566a2dc1e21c91d1876e06333de8f9d9a4963b7d9f08cd16023bf787cfa39df"
Sep 10 21:25:48.487279 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:25:48.487261    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4566a2dc1e21c91d1876e06333de8f9d9a4963b7d9f08cd16023bf787cfa39df\": container with ID starting with 4566a2dc1e21c91d1876e06333de8f9d9a4963b7d9f08cd16023bf787cfa39df not found: ID does not exist" containerID="4566a2dc1e21c91d1876e06333de8f9d9a4963b7d9f08cd16023bf787cfa39df"
Sep 10 21:25:48.487437 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:48.487287    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4566a2dc1e21c91d1876e06333de8f9d9a4963b7d9f08cd16023bf787cfa39df"} err="failed to get container status \"4566a2dc1e21c91d1876e06333de8f9d9a4963b7d9f08cd16023bf787cfa39df\": rpc error: code = NotFound desc = could not find container \"4566a2dc1e21c91d1876e06333de8f9d9a4963b7d9f08cd16023bf787cfa39df\": container with ID starting with 4566a2dc1e21c91d1876e06333de8f9d9a4963b7d9f08cd16023bf787cfa39df not found: ID does not exist"
Sep 10 21:25:48.487437 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:48.487321    2581 scope.go:117] "RemoveContainer" containerID="37e2b7e01776ae1ab6694ea54035222aea50625bc1c4bc9e20467e5bd225924c"
Sep 10 21:25:48.487591 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:25:48.487574    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37e2b7e01776ae1ab6694ea54035222aea50625bc1c4bc9e20467e5bd225924c\": container with ID starting with 37e2b7e01776ae1ab6694ea54035222aea50625bc1c4bc9e20467e5bd225924c not found: ID does not exist" containerID="37e2b7e01776ae1ab6694ea54035222aea50625bc1c4bc9e20467e5bd225924c"
Sep 10 21:25:48.487636 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:48.487596    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37e2b7e01776ae1ab6694ea54035222aea50625bc1c4bc9e20467e5bd225924c"} err="failed to get container status \"37e2b7e01776ae1ab6694ea54035222aea50625bc1c4bc9e20467e5bd225924c\": rpc error: code = NotFound desc = could not find container \"37e2b7e01776ae1ab6694ea54035222aea50625bc1c4bc9e20467e5bd225924c\": container with ID starting with 37e2b7e01776ae1ab6694ea54035222aea50625bc1c4bc9e20467e5bd225924c not found: ID does not exist"
Sep 10 21:25:48.492435 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:48.492413    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"]
Sep 10 21:25:48.496221 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:48.496198    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-1427b1-openshift-default-8ff59c5bb-srhml"]
Sep 10 21:25:48.541733 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:48.541706    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b794\" (UniqueName: \"kubernetes.io/projected/1ba1f825-34bb-41e6-a692-35d2de203a14-kube-api-access-4b794\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:25:48.541733 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:48.541730    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ba1f825-34bb-41e6-a692-35d2de203a14-istio-envoy\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:25:48.541733 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:48.541739    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ba1f825-34bb-41e6-a692-35d2de203a14-istio-podinfo\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:25:48.541913 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:48.541747    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba1f825-34bb-41e6-a692-35d2de203a14-credential-socket\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:25:48.541913 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:48.541757    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ba1f825-34bb-41e6-a692-35d2de203a14-istio-data\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:25:48.541913 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:48.541770    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ba1f825-34bb-41e6-a692-35d2de203a14-istio-token\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:25:48.541913 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:48.541779    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ba1f825-34bb-41e6-a692-35d2de203a14-istiod-ca-cert\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:25:48.868793 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:48.868762    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ba1f825-34bb-41e6-a692-35d2de203a14" path="/var/lib/kubelet/pods/1ba1f825-34bb-41e6-a692-35d2de203a14/volumes"
Sep 10 21:25:54.023095 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:54.023054    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-828c7c-29817915-hzkvj"]
Sep 10 21:25:54.025682 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:54.025652    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-828c7c-29817915-hzkvj"]
Sep 10 21:25:54.049811 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:54.049778    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-828c7c-79f4cf7bfd-5f4t2"]
Sep 10 21:25:54.050005 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:54.049977    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-828c7c-79f4cf7bfd-5f4t2" podUID="c29d5c7c-8766-490a-b1e5-e453245d8b42" containerName="payload-pre-processing" containerID="cri-o://2d785e9ce21d017468cd1e9b973ca88e3e6c301fd76e66d9602d09bb8e50a001" gracePeriod=30
Sep 10 21:25:54.191065 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:54.191034    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-828c7c-z28r6"]
Sep 10 21:25:54.196372 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:54.196348    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-828c7c-z28r6"]
Sep 10 21:25:54.306210 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:54.306185    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-828c7c-79f4cf7bfd-5f4t2"
Sep 10 21:25:54.391643 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:54.391612    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c29d5c7c-8766-490a-b1e5-e453245d8b42-plugins-config-volume\") pod \"c29d5c7c-8766-490a-b1e5-e453245d8b42\" (UID: \"c29d5c7c-8766-490a-b1e5-e453245d8b42\") "
Sep 10 21:25:54.391830 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:54.391671    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7b6r\" (UniqueName: \"kubernetes.io/projected/c29d5c7c-8766-490a-b1e5-e453245d8b42-kube-api-access-n7b6r\") pod \"c29d5c7c-8766-490a-b1e5-e453245d8b42\" (UID: \"c29d5c7c-8766-490a-b1e5-e453245d8b42\") "
Sep 10 21:25:54.392032 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:54.392006    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c29d5c7c-8766-490a-b1e5-e453245d8b42-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c29d5c7c-8766-490a-b1e5-e453245d8b42" (UID: "c29d5c7c-8766-490a-b1e5-e453245d8b42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:25:54.393783 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:54.393764    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c29d5c7c-8766-490a-b1e5-e453245d8b42-kube-api-access-n7b6r" (OuterVolumeSpecName: "kube-api-access-n7b6r") pod "c29d5c7c-8766-490a-b1e5-e453245d8b42" (UID: "c29d5c7c-8766-490a-b1e5-e453245d8b42"). InnerVolumeSpecName "kube-api-access-n7b6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:25:54.492865 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:54.492839    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c29d5c7c-8766-490a-b1e5-e453245d8b42-plugins-config-volume\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:25:54.492865 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:54.492862    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7b6r\" (UniqueName: \"kubernetes.io/projected/c29d5c7c-8766-490a-b1e5-e453245d8b42-kube-api-access-n7b6r\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:25:54.501448 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:54.501421    2581 generic.go:356] "Generic (PLEG): container finished" podID="c29d5c7c-8766-490a-b1e5-e453245d8b42" containerID="2d785e9ce21d017468cd1e9b973ca88e3e6c301fd76e66d9602d09bb8e50a001" exitCode=0
Sep 10 21:25:54.501554 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:54.501459    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-828c7c-79f4cf7bfd-5f4t2"
Sep 10 21:25:54.501554 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:54.501504    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-828c7c-79f4cf7bfd-5f4t2" event={"ID":"c29d5c7c-8766-490a-b1e5-e453245d8b42","Type":"ContainerDied","Data":"2d785e9ce21d017468cd1e9b973ca88e3e6c301fd76e66d9602d09bb8e50a001"}
Sep 10 21:25:54.501554 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:54.501553    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-828c7c-79f4cf7bfd-5f4t2" event={"ID":"c29d5c7c-8766-490a-b1e5-e453245d8b42","Type":"ContainerDied","Data":"f640be2ac09695a880d60bcec0538e26974022d567641df91d8415ac7a3926f4"}
Sep 10 21:25:54.501664 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:54.501575    2581 scope.go:117] "RemoveContainer" containerID="2d785e9ce21d017468cd1e9b973ca88e3e6c301fd76e66d9602d09bb8e50a001"
Sep 10 21:25:54.510629 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:54.510613    2581 scope.go:117] "RemoveContainer" containerID="2d785e9ce21d017468cd1e9b973ca88e3e6c301fd76e66d9602d09bb8e50a001"
Sep 10 21:25:54.510887 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:25:54.510872    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d785e9ce21d017468cd1e9b973ca88e3e6c301fd76e66d9602d09bb8e50a001\": container with ID starting with 2d785e9ce21d017468cd1e9b973ca88e3e6c301fd76e66d9602d09bb8e50a001 not found: ID does not exist" containerID="2d785e9ce21d017468cd1e9b973ca88e3e6c301fd76e66d9602d09bb8e50a001"
Sep 10 21:25:54.510925 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:54.510895    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d785e9ce21d017468cd1e9b973ca88e3e6c301fd76e66d9602d09bb8e50a001"} err="failed to get container status \"2d785e9ce21d017468cd1e9b973ca88e3e6c301fd76e66d9602d09bb8e50a001\": rpc error: code = NotFound desc = could not find container \"2d785e9ce21d017468cd1e9b973ca88e3e6c301fd76e66d9602d09bb8e50a001\": container with ID starting with 2d785e9ce21d017468cd1e9b973ca88e3e6c301fd76e66d9602d09bb8e50a001 not found: ID does not exist"
Sep 10 21:25:54.521475 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:54.521452    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-828c7c-79f4cf7bfd-5f4t2"]
Sep 10 21:25:54.525670 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:54.525651    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-828c7c-79f4cf7bfd-5f4t2"]
Sep 10 21:25:54.868236 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:54.868201    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9837fd76-a25d-4f7b-bdc7-034809caf8a0" path="/var/lib/kubelet/pods/9837fd76-a25d-4f7b-bdc7-034809caf8a0/volumes"
Sep 10 21:25:54.868637 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:54.868622    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c29d5c7c-8766-490a-b1e5-e453245d8b42" path="/var/lib/kubelet/pods/c29d5c7c-8766-490a-b1e5-e453245d8b42/volumes"
Sep 10 21:25:54.868873 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:25:54.868862    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3ee754b-56e0-4483-85ba-02e2bd77769a" path="/var/lib/kubelet/pods/e3ee754b-56e0-4483-85ba-02e2bd77769a/volumes"
Sep 10 21:26:01.770160 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:01.770123    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9f6475-h47cg"]
Sep 10 21:26:01.770577 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:01.770495    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ba1f825-34bb-41e6-a692-35d2de203a14" containerName="istio-proxy"
Sep 10 21:26:01.770577 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:01.770507    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ba1f825-34bb-41e6-a692-35d2de203a14" containerName="istio-proxy"
Sep 10 21:26:01.770577 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:01.770516    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9837fd76-a25d-4f7b-bdc7-034809caf8a0" containerName="revoke-keys"
Sep 10 21:26:01.770577 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:01.770522    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9837fd76-a25d-4f7b-bdc7-034809caf8a0" containerName="revoke-keys"
Sep 10 21:26:01.770577 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:01.770535    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c29d5c7c-8766-490a-b1e5-e453245d8b42" containerName="payload-pre-processing"
Sep 10 21:26:01.770577 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:01.770541    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c29d5c7c-8766-490a-b1e5-e453245d8b42" containerName="payload-pre-processing"
Sep 10 21:26:01.770792 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:01.770595    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9837fd76-a25d-4f7b-bdc7-034809caf8a0" containerName="revoke-keys"
Sep 10 21:26:01.770792 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:01.770604    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c29d5c7c-8766-490a-b1e5-e453245d8b42" containerName="payload-pre-processing"
Sep 10 21:26:01.770792 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:01.770610    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ba1f825-34bb-41e6-a692-35d2de203a14" containerName="istio-proxy"
Sep 10 21:26:01.775041 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:01.775023    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9f6475-h47cg"
Sep 10 21:26:01.777752 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:01.777724    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s4qqn\""
Sep 10 21:26:01.777887 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:01.777775    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:26:01.782408 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:01.782383    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9f6475-h47cg"]
Sep 10 21:26:01.859190 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:01.859152    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/358842c1-984a-4753-989f-14072506470d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-9f6475-h47cg\" (UID: \"358842c1-984a-4753-989f-14072506470d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9f6475-h47cg"
Sep 10 21:26:01.960591 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:01.960557    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/358842c1-984a-4753-989f-14072506470d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-9f6475-h47cg\" (UID: \"358842c1-984a-4753-989f-14072506470d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9f6475-h47cg"
Sep 10 21:26:01.961187 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:01.961165    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/358842c1-984a-4753-989f-14072506470d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-9f6475-h47cg\" (UID: \"358842c1-984a-4753-989f-14072506470d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9f6475-h47cg"
Sep 10 21:26:02.088263 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:02.088233    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9f6475-h47cg"
Sep 10 21:26:02.358373 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:02.358273    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9f6475-h47cg"]
Sep 10 21:26:02.361861 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:26:02.361837    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod358842c1_984a_4753_989f_14072506470d.slice/crio-64e6c26dbc8354417c009e2fb1a00b8a418652aaa601977b0863013c6eea9395 WatchSource:0}: Error finding container 64e6c26dbc8354417c009e2fb1a00b8a418652aaa601977b0863013c6eea9395: Status 404 returned error can't find the container with id 64e6c26dbc8354417c009e2fb1a00b8a418652aaa601977b0863013c6eea9395
Sep 10 21:26:02.543406 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:02.543369    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9f6475-h47cg" event={"ID":"358842c1-984a-4753-989f-14072506470d","Type":"ContainerStarted","Data":"64e6c26dbc8354417c009e2fb1a00b8a418652aaa601977b0863013c6eea9395"}
Sep 10 21:26:03.160839 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:03.160812    2581 scope.go:117] "RemoveContainer" containerID="5f6380fc890cad81f0a43f59fda1bcffbf326eb4cef1ce7615cbff0a4b829a8e"
Sep 10 21:26:03.550525 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:03.550492    2581 generic.go:356] "Generic (PLEG): container finished" podID="358842c1-984a-4753-989f-14072506470d" containerID="7c9ebb0d5c8ba105e3f321ba9b548b44f3035e62eb9b7d43b99f4f3a1c38c9fd" exitCode=0
Sep 10 21:26:03.550704 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:03.550577    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9f6475-h47cg" event={"ID":"358842c1-984a-4753-989f-14072506470d","Type":"ContainerDied","Data":"7c9ebb0d5c8ba105e3f321ba9b548b44f3035e62eb9b7d43b99f4f3a1c38c9fd"}
Sep 10 21:26:04.688188 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:04.688165    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9f6475-h47cg"
Sep 10 21:26:04.785653 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:04.785617    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/358842c1-984a-4753-989f-14072506470d-maas-api-service-ca\") pod \"358842c1-984a-4753-989f-14072506470d\" (UID: \"358842c1-984a-4753-989f-14072506470d\") "
Sep 10 21:26:04.786034 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:04.786012    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/358842c1-984a-4753-989f-14072506470d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "358842c1-984a-4753-989f-14072506470d" (UID: "358842c1-984a-4753-989f-14072506470d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:26:04.886821 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:04.886792    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/358842c1-984a-4753-989f-14072506470d-maas-api-service-ca\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:26:05.563873 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:05.563846    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9f6475-h47cg"
Sep 10 21:26:05.563873 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:05.563854    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9f6475-h47cg" event={"ID":"358842c1-984a-4753-989f-14072506470d","Type":"ContainerDied","Data":"64e6c26dbc8354417c009e2fb1a00b8a418652aaa601977b0863013c6eea9395"}
Sep 10 21:26:05.564061 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:05.563904    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64e6c26dbc8354417c009e2fb1a00b8a418652aaa601977b0863013c6eea9395"
Sep 10 21:26:12.441557 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:12.441524    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-9f6475-5d7d84f979-nkxnj"]
Sep 10 21:26:12.442024 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:12.441761    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9f6475-5d7d84f979-nkxnj" podUID="8a65e733-908c-4846-97b6-d6b6b4f1c094" containerName="payload-processing" containerID="cri-o://411a8bc53f90f1072857e5284e1a9ce33dd287c039e63dfc1f93f1775291dfd1" gracePeriod=30
Sep 10 21:26:12.465618 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:12.465590    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9f6475-85fb54bc6b-m6ktj"]
Sep 10 21:26:12.465913 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:12.465871    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9f6475-85fb54bc6b-m6ktj" podUID="b6895de4-48ac-4813-98ce-52b92ebfa04f" containerName="payload-pre-processing" containerID="cri-o://2238f0b824aacd7b29ba56649daa877ae700e22d0e2d11ed68ed8df2ef959c25" gracePeriod=30
Sep 10 21:26:12.604267 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:12.604235    2581 generic.go:356] "Generic (PLEG): container finished" podID="b6895de4-48ac-4813-98ce-52b92ebfa04f" containerID="2238f0b824aacd7b29ba56649daa877ae700e22d0e2d11ed68ed8df2ef959c25" exitCode=0
Sep 10 21:26:12.604449 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:12.604335    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9f6475-85fb54bc6b-m6ktj" event={"ID":"b6895de4-48ac-4813-98ce-52b92ebfa04f","Type":"ContainerDied","Data":"2238f0b824aacd7b29ba56649daa877ae700e22d0e2d11ed68ed8df2ef959c25"}
Sep 10 21:26:12.606313 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:12.606277    2581 generic.go:356] "Generic (PLEG): container finished" podID="8a65e733-908c-4846-97b6-d6b6b4f1c094" containerID="411a8bc53f90f1072857e5284e1a9ce33dd287c039e63dfc1f93f1775291dfd1" exitCode=0
Sep 10 21:26:12.606428 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:12.606336    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9f6475-5d7d84f979-nkxnj" event={"ID":"8a65e733-908c-4846-97b6-d6b6b4f1c094","Type":"ContainerDied","Data":"411a8bc53f90f1072857e5284e1a9ce33dd287c039e63dfc1f93f1775291dfd1"}
Sep 10 21:26:12.747120 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:12.747097    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9f6475-5d7d84f979-nkxnj"
Sep 10 21:26:12.750072 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:12.750048    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9f6475-85fb54bc6b-m6ktj"
Sep 10 21:26:12.757218 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:12.757192    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9f6475-h47cg"]
Sep 10 21:26:12.762839 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:12.762817    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9f6475-h47cg"]
Sep 10 21:26:12.856117 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:12.856072    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbphr\" (UniqueName: \"kubernetes.io/projected/b6895de4-48ac-4813-98ce-52b92ebfa04f-kube-api-access-wbphr\") pod \"b6895de4-48ac-4813-98ce-52b92ebfa04f\" (UID: \"b6895de4-48ac-4813-98ce-52b92ebfa04f\") "
Sep 10 21:26:12.856277 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:12.856139    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a65e733-908c-4846-97b6-d6b6b4f1c094-plugins-config-volume\") pod \"8a65e733-908c-4846-97b6-d6b6b4f1c094\" (UID: \"8a65e733-908c-4846-97b6-d6b6b4f1c094\") "
Sep 10 21:26:12.856277 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:12.856163    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6895de4-48ac-4813-98ce-52b92ebfa04f-plugins-config-volume\") pod \"b6895de4-48ac-4813-98ce-52b92ebfa04f\" (UID: \"b6895de4-48ac-4813-98ce-52b92ebfa04f\") "
Sep 10 21:26:12.856277 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:12.856254    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfvwq\" (UniqueName: \"kubernetes.io/projected/8a65e733-908c-4846-97b6-d6b6b4f1c094-kube-api-access-hfvwq\") pod \"8a65e733-908c-4846-97b6-d6b6b4f1c094\" (UID: \"8a65e733-908c-4846-97b6-d6b6b4f1c094\") "
Sep 10 21:26:12.856718 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:12.856638    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a65e733-908c-4846-97b6-d6b6b4f1c094-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8a65e733-908c-4846-97b6-d6b6b4f1c094" (UID: "8a65e733-908c-4846-97b6-d6b6b4f1c094"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:26:12.856856 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:12.856817    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6895de4-48ac-4813-98ce-52b92ebfa04f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b6895de4-48ac-4813-98ce-52b92ebfa04f" (UID: "b6895de4-48ac-4813-98ce-52b92ebfa04f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:26:12.858551 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:12.858528    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6895de4-48ac-4813-98ce-52b92ebfa04f-kube-api-access-wbphr" (OuterVolumeSpecName: "kube-api-access-wbphr") pod "b6895de4-48ac-4813-98ce-52b92ebfa04f" (UID: "b6895de4-48ac-4813-98ce-52b92ebfa04f"). InnerVolumeSpecName "kube-api-access-wbphr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:26:12.858790 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:12.858765    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a65e733-908c-4846-97b6-d6b6b4f1c094-kube-api-access-hfvwq" (OuterVolumeSpecName: "kube-api-access-hfvwq") pod "8a65e733-908c-4846-97b6-d6b6b4f1c094" (UID: "8a65e733-908c-4846-97b6-d6b6b4f1c094"). InnerVolumeSpecName "kube-api-access-hfvwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:26:12.868282 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:12.868237    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="358842c1-984a-4753-989f-14072506470d" path="/var/lib/kubelet/pods/358842c1-984a-4753-989f-14072506470d/volumes"
Sep 10 21:26:12.957426 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:12.957388    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfvwq\" (UniqueName: \"kubernetes.io/projected/8a65e733-908c-4846-97b6-d6b6b4f1c094-kube-api-access-hfvwq\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:26:12.957426 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:12.957419    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbphr\" (UniqueName: \"kubernetes.io/projected/b6895de4-48ac-4813-98ce-52b92ebfa04f-kube-api-access-wbphr\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:26:12.957426 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:12.957431    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a65e733-908c-4846-97b6-d6b6b4f1c094-plugins-config-volume\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:26:12.957647 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:12.957444    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6895de4-48ac-4813-98ce-52b92ebfa04f-plugins-config-volume\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:26:13.613764 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:13.613724    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9f6475-85fb54bc6b-m6ktj" event={"ID":"b6895de4-48ac-4813-98ce-52b92ebfa04f","Type":"ContainerDied","Data":"9a6f78b52cb1b55ce2295796edb1635408c3acec08d2529222ec6ad7c9b34353"}
Sep 10 21:26:13.614231 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:13.613790    2581 scope.go:117] "RemoveContainer" containerID="2238f0b824aacd7b29ba56649daa877ae700e22d0e2d11ed68ed8df2ef959c25"
Sep 10 21:26:13.614231 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:13.613738    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9f6475-85fb54bc6b-m6ktj"
Sep 10 21:26:13.615327 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:13.615288    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9f6475-5d7d84f979-nkxnj"
Sep 10 21:26:13.615444 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:13.615291    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-9f6475-5d7d84f979-nkxnj" event={"ID":"8a65e733-908c-4846-97b6-d6b6b4f1c094","Type":"ContainerDied","Data":"127de3d3db57bf7fc23abe0cd625be15b8dae2f1f723d08b200cfc789d74d1af"}
Sep 10 21:26:13.623053 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:13.623033    2581 scope.go:117] "RemoveContainer" containerID="411a8bc53f90f1072857e5284e1a9ce33dd287c039e63dfc1f93f1775291dfd1"
Sep 10 21:26:13.633211 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:13.633190    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9f6475-85fb54bc6b-m6ktj"]
Sep 10 21:26:13.639149 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:13.639119    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-9f6475-85fb54bc6b-m6ktj"]
Sep 10 21:26:13.652663 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:13.652640    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-9f6475-5d7d84f979-nkxnj"]
Sep 10 21:26:13.655919 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:13.655899    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-9f6475-5d7d84f979-nkxnj"]
Sep 10 21:26:14.869210 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:14.869173    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a65e733-908c-4846-97b6-d6b6b4f1c094" path="/var/lib/kubelet/pods/8a65e733-908c-4846-97b6-d6b6b4f1c094/volumes"
Sep 10 21:26:14.869595 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:14.869559    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6895de4-48ac-4813-98ce-52b92ebfa04f" path="/var/lib/kubelet/pods/b6895de4-48ac-4813-98ce-52b92ebfa04f/volumes"
Sep 10 21:26:22.547310 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:22.547262    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d795c6fb-r5g2k"]
Sep 10 21:26:22.547860 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:22.547468    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6d795c6fb-r5g2k" podUID="e9192d60-ff3d-486d-b34b-9de531284fb7" containerName="manager" containerID="cri-o://8661e8307f006c62989da69497b92d8397046a632a0c83133bf4ffa97eb3e2ec" gracePeriod=30
Sep 10 21:26:22.675112 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:22.675075    2581 generic.go:356] "Generic (PLEG): container finished" podID="e9192d60-ff3d-486d-b34b-9de531284fb7" containerID="8661e8307f006c62989da69497b92d8397046a632a0c83133bf4ffa97eb3e2ec" exitCode=0
Sep 10 21:26:22.675252 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:22.675143    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d795c6fb-r5g2k" event={"ID":"e9192d60-ff3d-486d-b34b-9de531284fb7","Type":"ContainerDied","Data":"8661e8307f006c62989da69497b92d8397046a632a0c83133bf4ffa97eb3e2ec"}
Sep 10 21:26:22.791205 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:22.791183    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d795c6fb-r5g2k"
Sep 10 21:26:22.833219 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:22.833136    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9192d60-ff3d-486d-b34b-9de531284fb7-metrics-certs\") pod \"e9192d60-ff3d-486d-b34b-9de531284fb7\" (UID: \"e9192d60-ff3d-486d-b34b-9de531284fb7\") "
Sep 10 21:26:22.833396 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:22.833221    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7dpn\" (UniqueName: \"kubernetes.io/projected/e9192d60-ff3d-486d-b34b-9de531284fb7-kube-api-access-p7dpn\") pod \"e9192d60-ff3d-486d-b34b-9de531284fb7\" (UID: \"e9192d60-ff3d-486d-b34b-9de531284fb7\") "
Sep 10 21:26:22.833396 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:22.833281    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e9192d60-ff3d-486d-b34b-9de531284fb7-webhook-cert\") pod \"e9192d60-ff3d-486d-b34b-9de531284fb7\" (UID: \"e9192d60-ff3d-486d-b34b-9de531284fb7\") "
Sep 10 21:26:22.835258 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:22.835227    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9192d60-ff3d-486d-b34b-9de531284fb7-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e9192d60-ff3d-486d-b34b-9de531284fb7" (UID: "e9192d60-ff3d-486d-b34b-9de531284fb7"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:26:22.835704 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:22.835685    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9192d60-ff3d-486d-b34b-9de531284fb7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e9192d60-ff3d-486d-b34b-9de531284fb7" (UID: "e9192d60-ff3d-486d-b34b-9de531284fb7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:26:22.835791 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:22.835709    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9192d60-ff3d-486d-b34b-9de531284fb7-kube-api-access-p7dpn" (OuterVolumeSpecName: "kube-api-access-p7dpn") pod "e9192d60-ff3d-486d-b34b-9de531284fb7" (UID: "e9192d60-ff3d-486d-b34b-9de531284fb7"). InnerVolumeSpecName "kube-api-access-p7dpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:26:22.934361 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:22.934331    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7dpn\" (UniqueName: \"kubernetes.io/projected/e9192d60-ff3d-486d-b34b-9de531284fb7-kube-api-access-p7dpn\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:26:22.934513 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:22.934386    2581 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e9192d60-ff3d-486d-b34b-9de531284fb7-webhook-cert\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:26:22.934513 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:22.934442    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9192d60-ff3d-486d-b34b-9de531284fb7-metrics-certs\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:26:23.681343 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:23.681284    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d795c6fb-r5g2k" event={"ID":"e9192d60-ff3d-486d-b34b-9de531284fb7","Type":"ContainerDied","Data":"9fc57a3704c479c9749498662840b4ffb05778bb7c765a29c171802910b3a4ce"}
Sep 10 21:26:23.681720 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:23.681375    2581 scope.go:117] "RemoveContainer" containerID="8661e8307f006c62989da69497b92d8397046a632a0c83133bf4ffa97eb3e2ec"
Sep 10 21:26:23.681720 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:23.681293    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d795c6fb-r5g2k"
Sep 10 21:26:23.706021 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:23.705993    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d795c6fb-r5g2k"]
Sep 10 21:26:23.714607 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:23.714579    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6d795c6fb-r5g2k"]
Sep 10 21:26:24.868712 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:26:24.868671    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9192d60-ff3d-486d-b34b-9de531284fb7" path="/var/lib/kubelet/pods/e9192d60-ff3d-486d-b34b-9de531284fb7/volumes"
Sep 10 21:27:03.200558 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:03.200526    2581 scope.go:117] "RemoveContainer" containerID="0b3baeafb2ca126ad0b82c79b2469f9bc94189f5c3a5ac3b12738eaf4669f2f0"
Sep 10 21:27:03.208706 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:03.208688    2581 scope.go:117] "RemoveContainer" containerID="f9502361630a4696eec6cf37df934b901153ce7897a95d0de875151ac6e999bd"
Sep 10 21:27:24.600458 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:24.600422    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6d795c6fb-mnf78"]
Sep 10 21:27:24.602797 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:24.600772    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6895de4-48ac-4813-98ce-52b92ebfa04f" containerName="payload-pre-processing"
Sep 10 21:27:24.602797 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:24.600783    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6895de4-48ac-4813-98ce-52b92ebfa04f" containerName="payload-pre-processing"
Sep 10 21:27:24.602797 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:24.600807    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a65e733-908c-4846-97b6-d6b6b4f1c094" containerName="payload-processing"
Sep 10 21:27:24.602797 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:24.600814    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a65e733-908c-4846-97b6-d6b6b4f1c094" containerName="payload-processing"
Sep 10 21:27:24.602797 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:24.600825    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9192d60-ff3d-486d-b34b-9de531284fb7" containerName="manager"
Sep 10 21:27:24.602797 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:24.600834    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9192d60-ff3d-486d-b34b-9de531284fb7" containerName="manager"
Sep 10 21:27:24.602797 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:24.600843    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ba1f825-34bb-41e6-a692-35d2de203a14" containerName="istio-proxy"
Sep 10 21:27:24.602797 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:24.600848    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ba1f825-34bb-41e6-a692-35d2de203a14" containerName="istio-proxy"
Sep 10 21:27:24.602797 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:24.600867    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="358842c1-984a-4753-989f-14072506470d" containerName="revoke-keys"
Sep 10 21:27:24.602797 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:24.600873    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="358842c1-984a-4753-989f-14072506470d" containerName="revoke-keys"
Sep 10 21:27:24.602797 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:24.600940    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9192d60-ff3d-486d-b34b-9de531284fb7" containerName="manager"
Sep 10 21:27:24.602797 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:24.600948    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="358842c1-984a-4753-989f-14072506470d" containerName="revoke-keys"
Sep 10 21:27:24.602797 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:24.600957    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6895de4-48ac-4813-98ce-52b92ebfa04f" containerName="payload-pre-processing"
Sep 10 21:27:24.602797 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:24.600962    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a65e733-908c-4846-97b6-d6b6b4f1c094" containerName="payload-processing"
Sep 10 21:27:24.602797 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:24.600967    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ba1f825-34bb-41e6-a692-35d2de203a14" containerName="istio-proxy"
Sep 10 21:27:24.603698 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:24.603682    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d795c6fb-mnf78"
Sep 10 21:27:24.606858 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:24.606839    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 10 21:27:24.607110 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:24.607089    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 10 21:27:24.607161 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:24.607119    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2s7xb\""
Sep 10 21:27:24.607224 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:24.607207    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 21:27:24.617029 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:24.617006    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d795c6fb-mnf78"]
Sep 10 21:27:24.668338 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:24.668282    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4qjx\" (UniqueName: \"kubernetes.io/projected/4e6a532f-36ac-4092-bf3b-2d0cf04daf36-kube-api-access-q4qjx\") pod \"maas-controller-6d795c6fb-mnf78\" (UID: \"4e6a532f-36ac-4092-bf3b-2d0cf04daf36\") " pod="opendatahub/maas-controller-6d795c6fb-mnf78"
Sep 10 21:27:24.668497 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:24.668363    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4e6a532f-36ac-4092-bf3b-2d0cf04daf36-webhook-cert\") pod \"maas-controller-6d795c6fb-mnf78\" (UID: \"4e6a532f-36ac-4092-bf3b-2d0cf04daf36\") " pod="opendatahub/maas-controller-6d795c6fb-mnf78"
Sep 10 21:27:24.668497 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:24.668417    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e6a532f-36ac-4092-bf3b-2d0cf04daf36-metrics-certs\") pod \"maas-controller-6d795c6fb-mnf78\" (UID: \"4e6a532f-36ac-4092-bf3b-2d0cf04daf36\") " pod="opendatahub/maas-controller-6d795c6fb-mnf78"
Sep 10 21:27:24.769551 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:24.769498    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4qjx\" (UniqueName: \"kubernetes.io/projected/4e6a532f-36ac-4092-bf3b-2d0cf04daf36-kube-api-access-q4qjx\") pod \"maas-controller-6d795c6fb-mnf78\" (UID: \"4e6a532f-36ac-4092-bf3b-2d0cf04daf36\") " pod="opendatahub/maas-controller-6d795c6fb-mnf78"
Sep 10 21:27:24.769764 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:24.769592    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4e6a532f-36ac-4092-bf3b-2d0cf04daf36-webhook-cert\") pod \"maas-controller-6d795c6fb-mnf78\" (UID: \"4e6a532f-36ac-4092-bf3b-2d0cf04daf36\") " pod="opendatahub/maas-controller-6d795c6fb-mnf78"
Sep 10 21:27:24.769764 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:24.769683    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e6a532f-36ac-4092-bf3b-2d0cf04daf36-metrics-certs\") pod \"maas-controller-6d795c6fb-mnf78\" (UID: \"4e6a532f-36ac-4092-bf3b-2d0cf04daf36\") " pod="opendatahub/maas-controller-6d795c6fb-mnf78"
Sep 10 21:27:24.772206 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:24.772183    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4e6a532f-36ac-4092-bf3b-2d0cf04daf36-webhook-cert\") pod \"maas-controller-6d795c6fb-mnf78\" (UID: \"4e6a532f-36ac-4092-bf3b-2d0cf04daf36\") " pod="opendatahub/maas-controller-6d795c6fb-mnf78"
Sep 10 21:27:24.772206 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:24.772192    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e6a532f-36ac-4092-bf3b-2d0cf04daf36-metrics-certs\") pod \"maas-controller-6d795c6fb-mnf78\" (UID: \"4e6a532f-36ac-4092-bf3b-2d0cf04daf36\") " pod="opendatahub/maas-controller-6d795c6fb-mnf78"
Sep 10 21:27:24.778490 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:24.778467    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4qjx\" (UniqueName: \"kubernetes.io/projected/4e6a532f-36ac-4092-bf3b-2d0cf04daf36-kube-api-access-q4qjx\") pod \"maas-controller-6d795c6fb-mnf78\" (UID: \"4e6a532f-36ac-4092-bf3b-2d0cf04daf36\") " pod="opendatahub/maas-controller-6d795c6fb-mnf78"
Sep 10 21:27:24.914213 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:24.914126    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d795c6fb-mnf78"
Sep 10 21:27:25.190438 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:25.190282    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d795c6fb-mnf78"]
Sep 10 21:27:25.192973 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:27:25.192933    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e6a532f_36ac_4092_bf3b_2d0cf04daf36.slice/crio-bd222708126eb8e3e617bc3f33f087913076e301a3bea05a511a4f4356ae0db7 WatchSource:0}: Error finding container bd222708126eb8e3e617bc3f33f087913076e301a3bea05a511a4f4356ae0db7: Status 404 returned error can't find the container with id bd222708126eb8e3e617bc3f33f087913076e301a3bea05a511a4f4356ae0db7
Sep 10 21:27:25.194670 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:25.194651    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 21:27:26.001467 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:26.001383    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d795c6fb-mnf78" event={"ID":"4e6a532f-36ac-4092-bf3b-2d0cf04daf36","Type":"ContainerStarted","Data":"1b3881068f5cb4d26c29afeb4f6e89932a675caa332f763b8817ee1cbc402984"}
Sep 10 21:27:26.001467 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:26.001437    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d795c6fb-mnf78" event={"ID":"4e6a532f-36ac-4092-bf3b-2d0cf04daf36","Type":"ContainerStarted","Data":"bd222708126eb8e3e617bc3f33f087913076e301a3bea05a511a4f4356ae0db7"}
Sep 10 21:27:26.001846 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:26.001514    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6d795c6fb-mnf78"
Sep 10 21:27:26.019476 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:26.019157    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6d795c6fb-mnf78" podStartSLOduration=1.535163894 podStartE2EDuration="2.019113863s" totalImagesPullingTime="483.949969ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:27:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:27:26.018839744 +0000 UTC m=+1763.675748520" watchObservedRunningTime="2026-09-10 21:27:26.019113863 +0000 UTC m=+1763.676022628"
Sep 10 21:27:37.008720 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:37.008691    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6d795c6fb-mnf78"
Sep 10 21:27:46.451364 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:46.451320    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d795c6fb-mnf78"]
Sep 10 21:27:46.451851 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:46.451619    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6d795c6fb-mnf78" podUID="4e6a532f-36ac-4092-bf3b-2d0cf04daf36" containerName="manager" containerID="cri-o://1b3881068f5cb4d26c29afeb4f6e89932a675caa332f763b8817ee1cbc402984" gracePeriod=30
Sep 10 21:27:46.700505 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:46.700480    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d795c6fb-mnf78"
Sep 10 21:27:46.862284 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:46.862246    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4qjx\" (UniqueName: \"kubernetes.io/projected/4e6a532f-36ac-4092-bf3b-2d0cf04daf36-kube-api-access-q4qjx\") pod \"4e6a532f-36ac-4092-bf3b-2d0cf04daf36\" (UID: \"4e6a532f-36ac-4092-bf3b-2d0cf04daf36\") "
Sep 10 21:27:46.862504 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:46.862291    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4e6a532f-36ac-4092-bf3b-2d0cf04daf36-webhook-cert\") pod \"4e6a532f-36ac-4092-bf3b-2d0cf04daf36\" (UID: \"4e6a532f-36ac-4092-bf3b-2d0cf04daf36\") "
Sep 10 21:27:46.862504 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:46.862466    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e6a532f-36ac-4092-bf3b-2d0cf04daf36-metrics-certs\") pod \"4e6a532f-36ac-4092-bf3b-2d0cf04daf36\" (UID: \"4e6a532f-36ac-4092-bf3b-2d0cf04daf36\") "
Sep 10 21:27:46.864540 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:46.864504    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e6a532f-36ac-4092-bf3b-2d0cf04daf36-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4e6a532f-36ac-4092-bf3b-2d0cf04daf36" (UID: "4e6a532f-36ac-4092-bf3b-2d0cf04daf36"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:27:46.864934 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:46.864914    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e6a532f-36ac-4092-bf3b-2d0cf04daf36-kube-api-access-q4qjx" (OuterVolumeSpecName: "kube-api-access-q4qjx") pod "4e6a532f-36ac-4092-bf3b-2d0cf04daf36" (UID: "4e6a532f-36ac-4092-bf3b-2d0cf04daf36"). InnerVolumeSpecName "kube-api-access-q4qjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:27:46.865069 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:46.864948    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e6a532f-36ac-4092-bf3b-2d0cf04daf36-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "4e6a532f-36ac-4092-bf3b-2d0cf04daf36" (UID: "4e6a532f-36ac-4092-bf3b-2d0cf04daf36"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:27:46.963821 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:46.963773    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4qjx\" (UniqueName: \"kubernetes.io/projected/4e6a532f-36ac-4092-bf3b-2d0cf04daf36-kube-api-access-q4qjx\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:27:46.963821 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:46.963800    2581 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4e6a532f-36ac-4092-bf3b-2d0cf04daf36-webhook-cert\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:27:46.964079 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:46.963883    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e6a532f-36ac-4092-bf3b-2d0cf04daf36-metrics-certs\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:27:47.106682 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:47.106644    2581 generic.go:356] "Generic (PLEG): container finished" podID="4e6a532f-36ac-4092-bf3b-2d0cf04daf36" containerID="1b3881068f5cb4d26c29afeb4f6e89932a675caa332f763b8817ee1cbc402984" exitCode=0
Sep 10 21:27:47.106866 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:47.106689    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d795c6fb-mnf78"
Sep 10 21:27:47.106866 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:47.106727    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d795c6fb-mnf78" event={"ID":"4e6a532f-36ac-4092-bf3b-2d0cf04daf36","Type":"ContainerDied","Data":"1b3881068f5cb4d26c29afeb4f6e89932a675caa332f763b8817ee1cbc402984"}
Sep 10 21:27:47.106866 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:47.106785    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d795c6fb-mnf78" event={"ID":"4e6a532f-36ac-4092-bf3b-2d0cf04daf36","Type":"ContainerDied","Data":"bd222708126eb8e3e617bc3f33f087913076e301a3bea05a511a4f4356ae0db7"}
Sep 10 21:27:47.106866 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:47.106809    2581 scope.go:117] "RemoveContainer" containerID="1b3881068f5cb4d26c29afeb4f6e89932a675caa332f763b8817ee1cbc402984"
Sep 10 21:27:47.115468 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:47.115344    2581 scope.go:117] "RemoveContainer" containerID="1b3881068f5cb4d26c29afeb4f6e89932a675caa332f763b8817ee1cbc402984"
Sep 10 21:27:47.115676 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:27:47.115659    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b3881068f5cb4d26c29afeb4f6e89932a675caa332f763b8817ee1cbc402984\": container with ID starting with 1b3881068f5cb4d26c29afeb4f6e89932a675caa332f763b8817ee1cbc402984 not found: ID does not exist" containerID="1b3881068f5cb4d26c29afeb4f6e89932a675caa332f763b8817ee1cbc402984"
Sep 10 21:27:47.115729 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:47.115685    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b3881068f5cb4d26c29afeb4f6e89932a675caa332f763b8817ee1cbc402984"} err="failed to get container status \"1b3881068f5cb4d26c29afeb4f6e89932a675caa332f763b8817ee1cbc402984\": rpc error: code = NotFound desc = could not find container \"1b3881068f5cb4d26c29afeb4f6e89932a675caa332f763b8817ee1cbc402984\": container with ID starting with 1b3881068f5cb4d26c29afeb4f6e89932a675caa332f763b8817ee1cbc402984 not found: ID does not exist"
Sep 10 21:27:47.122690 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:47.122665    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d795c6fb-mnf78"]
Sep 10 21:27:47.126292 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:47.126267    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6d795c6fb-mnf78"]
Sep 10 21:27:47.955803 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:47.955768    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6d795c6fb-sbvns"]
Sep 10 21:27:47.956195 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:47.956131    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e6a532f-36ac-4092-bf3b-2d0cf04daf36" containerName="manager"
Sep 10 21:27:47.956195 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:47.956142    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e6a532f-36ac-4092-bf3b-2d0cf04daf36" containerName="manager"
Sep 10 21:27:47.956269 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:47.956213    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e6a532f-36ac-4092-bf3b-2d0cf04daf36" containerName="manager"
Sep 10 21:27:47.960288 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:47.960272    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d795c6fb-sbvns"
Sep 10 21:27:47.962492 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:47.962471    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2s7xb\""
Sep 10 21:27:47.963139 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:47.963120    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 10 21:27:47.963254 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:47.963191    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 21:27:47.963254 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:47.963194    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 10 21:27:47.969223 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:47.969200    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d795c6fb-sbvns"]
Sep 10 21:27:48.073241 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:48.073206    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/309b0878-ff1c-4275-919d-af72b34772fb-metrics-certs\") pod \"maas-controller-6d795c6fb-sbvns\" (UID: \"309b0878-ff1c-4275-919d-af72b34772fb\") " pod="opendatahub/maas-controller-6d795c6fb-sbvns"
Sep 10 21:27:48.073428 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:48.073249    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tb82\" (UniqueName: \"kubernetes.io/projected/309b0878-ff1c-4275-919d-af72b34772fb-kube-api-access-7tb82\") pod \"maas-controller-6d795c6fb-sbvns\" (UID: \"309b0878-ff1c-4275-919d-af72b34772fb\") " pod="opendatahub/maas-controller-6d795c6fb-sbvns"
Sep 10 21:27:48.073428 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:48.073397    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/309b0878-ff1c-4275-919d-af72b34772fb-webhook-cert\") pod \"maas-controller-6d795c6fb-sbvns\" (UID: \"309b0878-ff1c-4275-919d-af72b34772fb\") " pod="opendatahub/maas-controller-6d795c6fb-sbvns"
Sep 10 21:27:48.174546 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:48.174511    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/309b0878-ff1c-4275-919d-af72b34772fb-webhook-cert\") pod \"maas-controller-6d795c6fb-sbvns\" (UID: \"309b0878-ff1c-4275-919d-af72b34772fb\") " pod="opendatahub/maas-controller-6d795c6fb-sbvns"
Sep 10 21:27:48.174722 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:48.174558    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/309b0878-ff1c-4275-919d-af72b34772fb-metrics-certs\") pod \"maas-controller-6d795c6fb-sbvns\" (UID: \"309b0878-ff1c-4275-919d-af72b34772fb\") " pod="opendatahub/maas-controller-6d795c6fb-sbvns"
Sep 10 21:27:48.174722 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:48.174618    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tb82\" (UniqueName: \"kubernetes.io/projected/309b0878-ff1c-4275-919d-af72b34772fb-kube-api-access-7tb82\") pod \"maas-controller-6d795c6fb-sbvns\" (UID: \"309b0878-ff1c-4275-919d-af72b34772fb\") " pod="opendatahub/maas-controller-6d795c6fb-sbvns"
Sep 10 21:27:48.176944 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:48.176919    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/309b0878-ff1c-4275-919d-af72b34772fb-metrics-certs\") pod \"maas-controller-6d795c6fb-sbvns\" (UID: \"309b0878-ff1c-4275-919d-af72b34772fb\") " pod="opendatahub/maas-controller-6d795c6fb-sbvns"
Sep 10 21:27:48.177067 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:48.176992    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/309b0878-ff1c-4275-919d-af72b34772fb-webhook-cert\") pod \"maas-controller-6d795c6fb-sbvns\" (UID: \"309b0878-ff1c-4275-919d-af72b34772fb\") " pod="opendatahub/maas-controller-6d795c6fb-sbvns"
Sep 10 21:27:48.182134 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:48.182115    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tb82\" (UniqueName: \"kubernetes.io/projected/309b0878-ff1c-4275-919d-af72b34772fb-kube-api-access-7tb82\") pod \"maas-controller-6d795c6fb-sbvns\" (UID: \"309b0878-ff1c-4275-919d-af72b34772fb\") " pod="opendatahub/maas-controller-6d795c6fb-sbvns"
Sep 10 21:27:48.271594 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:48.271494    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d795c6fb-sbvns"
Sep 10 21:27:48.543464 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:48.543440    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d795c6fb-sbvns"]
Sep 10 21:27:48.545990 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:27:48.545949    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod309b0878_ff1c_4275_919d_af72b34772fb.slice/crio-b850e34920134a6d13f3d5133f65ff1d2df97152680b86f1df4cae46f52c10ac WatchSource:0}: Error finding container b850e34920134a6d13f3d5133f65ff1d2df97152680b86f1df4cae46f52c10ac: Status 404 returned error can't find the container with id b850e34920134a6d13f3d5133f65ff1d2df97152680b86f1df4cae46f52c10ac
Sep 10 21:27:48.869412 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:48.869379    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e6a532f-36ac-4092-bf3b-2d0cf04daf36" path="/var/lib/kubelet/pods/4e6a532f-36ac-4092-bf3b-2d0cf04daf36/volumes"
Sep 10 21:27:49.120868 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:49.120776    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d795c6fb-sbvns" event={"ID":"309b0878-ff1c-4275-919d-af72b34772fb","Type":"ContainerStarted","Data":"f296a23667ada82ab0e2867e1a980073b71b47ddc343786eb01644b2446def34"}
Sep 10 21:27:49.120868 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:49.120828    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d795c6fb-sbvns" event={"ID":"309b0878-ff1c-4275-919d-af72b34772fb","Type":"ContainerStarted","Data":"b850e34920134a6d13f3d5133f65ff1d2df97152680b86f1df4cae46f52c10ac"}
Sep 10 21:27:49.121248 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:49.120909    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6d795c6fb-sbvns"
Sep 10 21:27:49.139733 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:27:49.139687    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6d795c6fb-sbvns" podStartSLOduration=1.797540273 podStartE2EDuration="2.139669817s" totalImagesPullingTime="342.129544ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:27:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:27:49.137992344 +0000 UTC m=+1786.794901115" watchObservedRunningTime="2026-09-10 21:27:49.139669817 +0000 UTC m=+1786.796578663"
Sep 10 21:28:00.127991 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:28:00.127916    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6d795c6fb-sbvns"
Sep 10 21:28:02.919875 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:28:02.919848    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-z6858_c1e06766-b48c-429e-92f1-196bb083360b/console-operator/1.log"
Sep 10 21:28:02.926936 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:28:02.926912    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brrtd_19d8d7ba-45a1-4013-a935-5670abe12b8a/ovn-acl-logging/0.log"
Sep 10 21:28:02.927623 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:28:02.927602    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-z6858_c1e06766-b48c-429e-92f1-196bb083360b/console-operator/1.log"
Sep 10 21:28:02.931686 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:28:02.931665    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-brrtd_19d8d7ba-45a1-4013-a935-5670abe12b8a/ovn-acl-logging/0.log"
Sep 10 21:28:03.240057 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:28:03.239981    2581 scope.go:117] "RemoveContainer" containerID="dd973375df5663608a410d1914c690888f9239aaa014c88ee76aaafbedf26c14"
Sep 10 21:28:03.248074 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:28:03.248055    2581 scope.go:117] "RemoveContainer" containerID="261e614a11c1ab65284dc9c83e5086e3908e8a44a57f86ab1d4e39b126139aa8"
Sep 10 21:28:03.255336 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:28:03.255320    2581 scope.go:117] "RemoveContainer" containerID="1fd6bc98dca3500b5829939771ff0f5a9eb1460736cdbf86f73227e3d8a54b92"
Sep 10 21:30:00.137038 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:00.137000    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817930-94hbz"]
Sep 10 21:30:00.140142 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:00.140123    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817930-94hbz"
Sep 10 21:30:00.143211 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:00.143181    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s4qqn\""
Sep 10 21:30:00.150944 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:00.150913    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817930-94hbz"]
Sep 10 21:30:00.157242 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:00.157216    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817930-94hbz"
Sep 10 21:30:00.428286 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:00.428180    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817930-94hbz"]
Sep 10 21:30:00.431083 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:30:00.431059    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1ef94fe_e618_4537_ba01_75fa4522c606.slice/crio-cd9ef1d5131abd8138a1a7d5966d25caa2c6dbc8026a6b1b979e3a1226c2fdec WatchSource:0}: Error finding container cd9ef1d5131abd8138a1a7d5966d25caa2c6dbc8026a6b1b979e3a1226c2fdec: Status 404 returned error can't find the container with id cd9ef1d5131abd8138a1a7d5966d25caa2c6dbc8026a6b1b979e3a1226c2fdec
Sep 10 21:30:00.769600 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:00.769564    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817930-94hbz" event={"ID":"d1ef94fe-e618-4537-ba01-75fa4522c606","Type":"ContainerStarted","Data":"f7a49c1bcc3fae66790c09cca2f0644aab5e6bf88031da1abfc072df403fdf9c"}
Sep 10 21:30:00.769771 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:00.769615    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817930-94hbz" event={"ID":"d1ef94fe-e618-4537-ba01-75fa4522c606","Type":"ContainerStarted","Data":"cd9ef1d5131abd8138a1a7d5966d25caa2c6dbc8026a6b1b979e3a1226c2fdec"}
Sep 10 21:30:00.801774 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:00.801724    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817930-94hbz" podStartSLOduration=0.801709195 podStartE2EDuration="801.709195ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:30:00.800247977 +0000 UTC m=+1918.457156748" watchObservedRunningTime="2026-09-10 21:30:00.801709195 +0000 UTC m=+1918.458618033"
Sep 10 21:30:01.775743 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:01.775707    2581 generic.go:356] "Generic (PLEG): container finished" podID="d1ef94fe-e618-4537-ba01-75fa4522c606" containerID="f7a49c1bcc3fae66790c09cca2f0644aab5e6bf88031da1abfc072df403fdf9c" exitCode=0
Sep 10 21:30:01.776139 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:01.775748    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817930-94hbz" event={"ID":"d1ef94fe-e618-4537-ba01-75fa4522c606","Type":"ContainerDied","Data":"f7a49c1bcc3fae66790c09cca2f0644aab5e6bf88031da1abfc072df403fdf9c"}
Sep 10 21:30:02.905772 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:02.905747    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817930-94hbz"
Sep 10 21:30:03.291779 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:03.291748    2581 scope.go:117] "RemoveContainer" containerID="af7f92ecfc0b8d393cee0917a93e1846c03300e64c15db87cb243887cbd6a99d"
Sep 10 21:30:03.300376 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:03.300354    2581 scope.go:117] "RemoveContainer" containerID="17126e689ed4167cccfbd37cd95f7763b554b77adb86a418b93a11a00af3bd78"
Sep 10 21:30:03.786111 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:03.786082    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817930-94hbz"
Sep 10 21:30:03.786111 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:03.786099    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817930-94hbz" event={"ID":"d1ef94fe-e618-4537-ba01-75fa4522c606","Type":"ContainerDied","Data":"cd9ef1d5131abd8138a1a7d5966d25caa2c6dbc8026a6b1b979e3a1226c2fdec"}
Sep 10 21:30:03.786367 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:03.786146    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd9ef1d5131abd8138a1a7d5966d25caa2c6dbc8026a6b1b979e3a1226c2fdec"
Sep 10 21:30:50.929292 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:50.929257    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7"]
Sep 10 21:30:50.929861 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:50.929793    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1ef94fe-e618-4537-ba01-75fa4522c606" containerName="cleanup"
Sep 10 21:30:50.929861 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:50.929812    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1ef94fe-e618-4537-ba01-75fa4522c606" containerName="cleanup"
Sep 10 21:30:50.929985 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:50.929920    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1ef94fe-e618-4537-ba01-75fa4522c606" containerName="cleanup"
Sep 10 21:30:50.932675 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:50.932655    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7"
Sep 10 21:30:50.935138 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:50.935118    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-62ecef60-kserve-self-signed-certs\""
Sep 10 21:30:50.948852 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:50.948832    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7"]
Sep 10 21:30:51.110837 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:51.110805    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e48d1e23-c559-4807-b4fa-cfe80af95402-tls-certs\") pod \"e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7\" (UID: \"e48d1e23-c559-4807-b4fa-cfe80af95402\") " pod="llm/e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7"
Sep 10 21:30:51.110837 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:51.110838    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e48d1e23-c559-4807-b4fa-cfe80af95402-model-cache\") pod \"e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7\" (UID: \"e48d1e23-c559-4807-b4fa-cfe80af95402\") " pod="llm/e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7"
Sep 10 21:30:51.111047 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:51.110893    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e48d1e23-c559-4807-b4fa-cfe80af95402-home\") pod \"e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7\" (UID: \"e48d1e23-c559-4807-b4fa-cfe80af95402\") " pod="llm/e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7"
Sep 10 21:30:51.111047 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:51.110935    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxscv\" (UniqueName: \"kubernetes.io/projected/e48d1e23-c559-4807-b4fa-cfe80af95402-kube-api-access-gxscv\") pod \"e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7\" (UID: \"e48d1e23-c559-4807-b4fa-cfe80af95402\") " pod="llm/e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7"
Sep 10 21:30:51.111047 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:51.110959    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e48d1e23-c559-4807-b4fa-cfe80af95402-tmp-dir\") pod \"e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7\" (UID: \"e48d1e23-c559-4807-b4fa-cfe80af95402\") " pod="llm/e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7"
Sep 10 21:30:51.111047 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:51.111005    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e48d1e23-c559-4807-b4fa-cfe80af95402-dshm\") pod \"e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7\" (UID: \"e48d1e23-c559-4807-b4fa-cfe80af95402\") " pod="llm/e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7"
Sep 10 21:30:51.211546 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:51.211475    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e48d1e23-c559-4807-b4fa-cfe80af95402-tls-certs\") pod \"e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7\" (UID: \"e48d1e23-c559-4807-b4fa-cfe80af95402\") " pod="llm/e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7"
Sep 10 21:30:51.211546 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:51.211512    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e48d1e23-c559-4807-b4fa-cfe80af95402-model-cache\") pod \"e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7\" (UID: \"e48d1e23-c559-4807-b4fa-cfe80af95402\") " pod="llm/e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7"
Sep 10 21:30:51.211546 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:51.211536    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e48d1e23-c559-4807-b4fa-cfe80af95402-home\") pod \"e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7\" (UID: \"e48d1e23-c559-4807-b4fa-cfe80af95402\") " pod="llm/e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7"
Sep 10 21:30:51.211778 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:51.211555    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxscv\" (UniqueName: \"kubernetes.io/projected/e48d1e23-c559-4807-b4fa-cfe80af95402-kube-api-access-gxscv\") pod \"e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7\" (UID: \"e48d1e23-c559-4807-b4fa-cfe80af95402\") " pod="llm/e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7"
Sep 10 21:30:51.211778 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:51.211587    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e48d1e23-c559-4807-b4fa-cfe80af95402-tmp-dir\") pod \"e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7\" (UID: \"e48d1e23-c559-4807-b4fa-cfe80af95402\") " pod="llm/e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7"
Sep 10 21:30:51.211778 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:51.211625    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e48d1e23-c559-4807-b4fa-cfe80af95402-dshm\") pod \"e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7\" (UID: \"e48d1e23-c559-4807-b4fa-cfe80af95402\") " pod="llm/e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7"
Sep 10 21:30:51.211950 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:51.211931    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e48d1e23-c559-4807-b4fa-cfe80af95402-model-cache\") pod \"e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7\" (UID: \"e48d1e23-c559-4807-b4fa-cfe80af95402\") " pod="llm/e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7"
Sep 10 21:30:51.212017 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:51.211962    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e48d1e23-c559-4807-b4fa-cfe80af95402-home\") pod \"e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7\" (UID: \"e48d1e23-c559-4807-b4fa-cfe80af95402\") " pod="llm/e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7"
Sep 10 21:30:51.212093 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:51.212069    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e48d1e23-c559-4807-b4fa-cfe80af95402-tmp-dir\") pod \"e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7\" (UID: \"e48d1e23-c559-4807-b4fa-cfe80af95402\") " pod="llm/e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7"
Sep 10 21:30:51.213821 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:51.213801    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e48d1e23-c559-4807-b4fa-cfe80af95402-dshm\") pod \"e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7\" (UID: \"e48d1e23-c559-4807-b4fa-cfe80af95402\") " pod="llm/e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7"
Sep 10 21:30:51.214114 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:51.214098    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e48d1e23-c559-4807-b4fa-cfe80af95402-tls-certs\") pod \"e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7\" (UID: \"e48d1e23-c559-4807-b4fa-cfe80af95402\") " pod="llm/e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7"
Sep 10 21:30:51.220571 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:51.220555    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxscv\" (UniqueName: \"kubernetes.io/projected/e48d1e23-c559-4807-b4fa-cfe80af95402-kube-api-access-gxscv\") pod \"e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7\" (UID: \"e48d1e23-c559-4807-b4fa-cfe80af95402\") " pod="llm/e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7"
Sep 10 21:30:51.241763 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:51.241741    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7"
Sep 10 21:30:51.516142 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:51.516118    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7"]
Sep 10 21:30:51.518461 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:30:51.518432    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode48d1e23_c559_4807_b4fa_cfe80af95402.slice/crio-ab42f6463db98f3dfeffc096610fbb1b518d139772a0253740b6b90c4dfb6d1b WatchSource:0}: Error finding container ab42f6463db98f3dfeffc096610fbb1b518d139772a0253740b6b90c4dfb6d1b: Status 404 returned error can't find the container with id ab42f6463db98f3dfeffc096610fbb1b518d139772a0253740b6b90c4dfb6d1b
Sep 10 21:30:52.025010 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:52.024980    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7" event={"ID":"e48d1e23-c559-4807-b4fa-cfe80af95402","Type":"ContainerStarted","Data":"306042afcf7a2b30dd8b623707a7114aa2c5fcb90091588d58938ef9101e315a"}
Sep 10 21:30:52.025396 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:52.025017    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7" event={"ID":"e48d1e23-c559-4807-b4fa-cfe80af95402","Type":"ContainerStarted","Data":"ab42f6463db98f3dfeffc096610fbb1b518d139772a0253740b6b90c4dfb6d1b"}
Sep 10 21:30:52.042135 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:30:52.042090    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7" podStartSLOduration=2.042075615 podStartE2EDuration="2.042075615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:30:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:30:52.041538939 +0000 UTC m=+1969.698447712" watchObservedRunningTime="2026-09-10 21:30:52.042075615 +0000 UTC m=+1969.698984424"
Sep 10 21:31:01.242518 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:31:01.242429    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7"
Sep 10 21:31:01.242518 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:31:01.242484    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7"
Sep 10 21:31:01.255212 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:31:01.255183    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7"
Sep 10 21:31:02.082257 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:31:02.082228    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7"
Sep 10 21:31:13.582820 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:31:13.582781    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7"]
Sep 10 21:31:28.601915 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:31:28.601851    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7" podUID="e48d1e23-c559-4807-b4fa-cfe80af95402" containerName="main" containerID="cri-o://306042afcf7a2b30dd8b623707a7114aa2c5fcb90091588d58938ef9101e315a" gracePeriod=45
Sep 10 21:31:28.852190 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:31:28.852131    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7"
Sep 10 21:31:28.944198 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:31:28.944165    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e48d1e23-c559-4807-b4fa-cfe80af95402-tls-certs\") pod \"e48d1e23-c559-4807-b4fa-cfe80af95402\" (UID: \"e48d1e23-c559-4807-b4fa-cfe80af95402\") "
Sep 10 21:31:28.944198 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:31:28.944204    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e48d1e23-c559-4807-b4fa-cfe80af95402-tmp-dir\") pod \"e48d1e23-c559-4807-b4fa-cfe80af95402\" (UID: \"e48d1e23-c559-4807-b4fa-cfe80af95402\") "
Sep 10 21:31:28.944460 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:31:28.944233    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxscv\" (UniqueName: \"kubernetes.io/projected/e48d1e23-c559-4807-b4fa-cfe80af95402-kube-api-access-gxscv\") pod \"e48d1e23-c559-4807-b4fa-cfe80af95402\" (UID: \"e48d1e23-c559-4807-b4fa-cfe80af95402\") "
Sep 10 21:31:28.944460 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:31:28.944258    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e48d1e23-c559-4807-b4fa-cfe80af95402-dshm\") pod \"e48d1e23-c559-4807-b4fa-cfe80af95402\" (UID: \"e48d1e23-c559-4807-b4fa-cfe80af95402\") "
Sep 10 21:31:28.944460 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:31:28.944363    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e48d1e23-c559-4807-b4fa-cfe80af95402-home\") pod \"e48d1e23-c559-4807-b4fa-cfe80af95402\" (UID: \"e48d1e23-c559-4807-b4fa-cfe80af95402\") "
Sep 10 21:31:28.944648 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:31:28.944523    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e48d1e23-c559-4807-b4fa-cfe80af95402-model-cache\") pod \"e48d1e23-c559-4807-b4fa-cfe80af95402\" (UID: \"e48d1e23-c559-4807-b4fa-cfe80af95402\") "
Sep 10 21:31:28.944648 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:31:28.944533    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e48d1e23-c559-4807-b4fa-cfe80af95402-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e48d1e23-c559-4807-b4fa-cfe80af95402" (UID: "e48d1e23-c559-4807-b4fa-cfe80af95402"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:31:28.944648 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:31:28.944628    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e48d1e23-c559-4807-b4fa-cfe80af95402-home" (OuterVolumeSpecName: "home") pod "e48d1e23-c559-4807-b4fa-cfe80af95402" (UID: "e48d1e23-c559-4807-b4fa-cfe80af95402"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:31:28.944822 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:31:28.944776    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e48d1e23-c559-4807-b4fa-cfe80af95402-model-cache" (OuterVolumeSpecName: "model-cache") pod "e48d1e23-c559-4807-b4fa-cfe80af95402" (UID: "e48d1e23-c559-4807-b4fa-cfe80af95402"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:31:28.944898 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:31:28.944875    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e48d1e23-c559-4807-b4fa-cfe80af95402-model-cache\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:31:28.944960 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:31:28.944905    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e48d1e23-c559-4807-b4fa-cfe80af95402-tmp-dir\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:31:28.944960 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:31:28.944918    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e48d1e23-c559-4807-b4fa-cfe80af95402-home\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:31:28.946635 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:31:28.946611    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e48d1e23-c559-4807-b4fa-cfe80af95402-kube-api-access-gxscv" (OuterVolumeSpecName: "kube-api-access-gxscv") pod "e48d1e23-c559-4807-b4fa-cfe80af95402" (UID: "e48d1e23-c559-4807-b4fa-cfe80af95402"). InnerVolumeSpecName "kube-api-access-gxscv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:31:28.946755 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:31:28.946649    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e48d1e23-c559-4807-b4fa-cfe80af95402-dshm" (OuterVolumeSpecName: "dshm") pod "e48d1e23-c559-4807-b4fa-cfe80af95402" (UID: "e48d1e23-c559-4807-b4fa-cfe80af95402"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:31:28.946755 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:31:28.946655    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e48d1e23-c559-4807-b4fa-cfe80af95402-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e48d1e23-c559-4807-b4fa-cfe80af95402" (UID: "e48d1e23-c559-4807-b4fa-cfe80af95402"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:31:29.045521 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:31:29.045494    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e48d1e23-c559-4807-b4fa-cfe80af95402-tls-certs\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:31:29.045521 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:31:29.045518    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxscv\" (UniqueName: \"kubernetes.io/projected/e48d1e23-c559-4807-b4fa-cfe80af95402-kube-api-access-gxscv\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:31:29.045521 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:31:29.045527    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e48d1e23-c559-4807-b4fa-cfe80af95402-dshm\") on node \"ip-10-0-139-165.ec2.internal\" DevicePath \"\""
Sep 10 21:31:29.211452 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:31:29.211373    2581 generic.go:356] "Generic (PLEG): container finished" podID="e48d1e23-c559-4807-b4fa-cfe80af95402" containerID="306042afcf7a2b30dd8b623707a7114aa2c5fcb90091588d58938ef9101e315a" exitCode=0
Sep 10 21:31:29.211452 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:31:29.211421    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7"
Sep 10 21:31:29.211452 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:31:29.211438    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7" event={"ID":"e48d1e23-c559-4807-b4fa-cfe80af95402","Type":"ContainerDied","Data":"306042afcf7a2b30dd8b623707a7114aa2c5fcb90091588d58938ef9101e315a"}
Sep 10 21:31:29.211711 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:31:29.211470    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7" event={"ID":"e48d1e23-c559-4807-b4fa-cfe80af95402","Type":"ContainerDied","Data":"ab42f6463db98f3dfeffc096610fbb1b518d139772a0253740b6b90c4dfb6d1b"}
Sep 10 21:31:29.211711 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:31:29.211489    2581 scope.go:117] "RemoveContainer" containerID="306042afcf7a2b30dd8b623707a7114aa2c5fcb90091588d58938ef9101e315a"
Sep 10 21:31:29.220907 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:31:29.220896    2581 scope.go:117] "RemoveContainer" containerID="306042afcf7a2b30dd8b623707a7114aa2c5fcb90091588d58938ef9101e315a"
Sep 10 21:31:29.221181 ip-10-0-139-165 kubenswrapper[2581]: E0910 21:31:29.221157    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"306042afcf7a2b30dd8b623707a7114aa2c5fcb90091588d58938ef9101e315a\": container with ID starting with 306042afcf7a2b30dd8b623707a7114aa2c5fcb90091588d58938ef9101e315a not found: ID does not exist" containerID="306042afcf7a2b30dd8b623707a7114aa2c5fcb90091588d58938ef9101e315a"
Sep 10 21:31:29.221258 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:31:29.221187    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"306042afcf7a2b30dd8b623707a7114aa2c5fcb90091588d58938ef9101e315a"} err="failed to get container status \"306042afcf7a2b30dd8b623707a7114aa2c5fcb90091588d58938ef9101e315a\": rpc error: code = NotFound desc = could not find container \"306042afcf7a2b30dd8b623707a7114aa2c5fcb90091588d58938ef9101e315a\": container with ID starting with 306042afcf7a2b30dd8b623707a7114aa2c5fcb90091588d58938ef9101e315a not found: ID does not exist"
Sep 10 21:31:29.231708 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:31:29.231660    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7"]
Sep 10 21:31:29.233433 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:31:29.233411    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-62ecef60-kserve-7d9cccfcf9-gf9d7"]
Sep 10 21:31:30.867914 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:31:30.867874    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e48d1e23-c559-4807-b4fa-cfe80af95402" path="/var/lib/kubelet/pods/e48d1e23-c559-4807-b4fa-cfe80af95402/volumes"
Sep 10 21:32:03.332421 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:03.332393    2581 scope.go:117] "RemoveContainer" containerID="d1612543ea6bcceddb1a444fa8707b208fbfe1e8349f2c68677dce4589cac1bf"
Sep 10 21:32:03.343735 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:03.343720    2581 scope.go:117] "RemoveContainer" containerID="f3dd3ed9db5d5c3423f308be00f44d5796dc0a4dd4697be4bb956acf3d49b752"
Sep 10 21:32:03.351824 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:03.351807    2581 scope.go:117] "RemoveContainer" containerID="7c9ebb0d5c8ba105e3f321ba9b548b44f3035e62eb9b7d43b99f4f3a1c38c9fd"
Sep 10 21:32:05.554667 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:05.554634    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6d795c6fb-sbvns_309b0878-ff1c-4275-919d-af72b34772fb/manager/0.log"
Sep 10 21:32:05.795471 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:05.795442    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-7blpg_e7b385e1-fa9f-47b8-87bf-006187f9d607/manager/0.log"
Sep 10 21:32:06.071452 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:06.071418    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d4c889df7-ck2h9_f146c084-a4fd-4b66-b74d-2b9887fedd3f/manager/0.log"
Sep 10 21:32:06.079096 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:06.079071    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d4c889df7-ck2h9_f146c084-a4fd-4b66-b74d-2b9887fedd3f/copy-manifests/0.log"
Sep 10 21:32:06.819746 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:06.819719    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29817915-ngwj8_616f2812-d98d-4e01-8197-405863e91fe1/cleanup/0.log"
Sep 10 21:32:06.933210 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:06.933183    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29817930-94hbz_d1ef94fe-e618-4537-ba01-75fa4522c606/cleanup/0.log"
Sep 10 21:32:08.944175 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:08.944145    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cfltqv_d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f/istio-proxy/0.log"
Sep 10 21:32:09.069593 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:09.069555    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-fmnwn_20736e92-a91b-452c-a01f-5a981c3c629a/discovery/0.log"
Sep 10 21:32:09.564669 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:09.564640    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-chrtv_a87366b4-0438-4d4a-8b0c-6f4973b07de9/payload-pre-processing/0.log"
Sep 10 21:32:09.821625 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:09.821550    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64c8fc8856-tfwzb_64a30645-a3fb-4670-89af-a044b3dd00a0/router/0.log"
Sep 10 21:32:10.193970 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:10.193893    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-lhr85_8c1c5f73-c025-4c6c-b610-a55cd7fd5a9d/main/0.log"
Sep 10 21:32:10.322354 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:10.322331    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-xcbjr_b57b2f45-34c2-4ca5-9ce8-148b66cb3b49/main/0.log"
Sep 10 21:32:10.561794 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:10.561763    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-6gpmj_ff227acf-0519-4216-abcf-ece643c1bc05/main/0.log"
Sep 10 21:32:19.943765 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:19.943733    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mgx5w_6e1f126a-f90d-4cf6-b863-d6130c01a622/global-pull-secret-syncer/0.log"
Sep 10 21:32:20.026240 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:20.026213    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-l2gkq_407d670d-5138-4d8b-aba7-96bdd91d9b19/konnectivity-agent/0.log"
Sep 10 21:32:20.147848 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:20.147803    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-165.ec2.internal_d43f2c9998219eadf34d68ce7c7d50dc/haproxy/0.log"
Sep 10 21:32:26.590605 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:26.590523    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-bvh97_da65604c-6689-4ef5-9cdf-0507cbeba512/cluster-monitoring-operator/0.log"
Sep 10 21:32:26.722334 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:26.722286    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-gf7q2_e546d5f4-6eb9-41d4-9244-46f8c9509f7e/monitoring-plugin/0.log"
Sep 10 21:32:26.858246 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:26.858171    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rp4kl_ed90097e-3c6b-476e-ad90-84a73b81a994/node-exporter/0.log"
Sep 10 21:32:26.888130 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:26.888107    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rp4kl_ed90097e-3c6b-476e-ad90-84a73b81a994/kube-rbac-proxy/0.log"
Sep 10 21:32:26.915626 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:26.915601    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rp4kl_ed90097e-3c6b-476e-ad90-84a73b81a994/init-textfile/0.log"
Sep 10 21:32:27.033228 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:27.033197    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-t4vqh_d3e7ffe4-fc2f-4782-a8dc-867199ee980c/kube-rbac-proxy-main/0.log"
Sep 10 21:32:27.059702 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:27.059675    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-t4vqh_d3e7ffe4-fc2f-4782-a8dc-867199ee980c/kube-rbac-proxy-self/0.log"
Sep 10 21:32:27.090253 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:27.090228    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-t4vqh_d3e7ffe4-fc2f-4782-a8dc-867199ee980c/openshift-state-metrics/0.log"
Sep 10 21:32:27.350763 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:27.350733    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-mjx9q_18a79c16-9d5e-49e2-a4b4-d909f665c2b5/prometheus-operator/0.log"
Sep 10 21:32:27.382417 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:27.382363    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-mjx9q_18a79c16-9d5e-49e2-a4b4-d909f665c2b5/kube-rbac-proxy/0.log"
Sep 10 21:32:29.117849 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:29.117804    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-9h26g_1a92d9ed-9fff-4373-8513-be11d25b6a37/networking-console-plugin/0.log"
Sep 10 21:32:29.732890 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:29.732853    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-z6858_c1e06766-b48c-429e-92f1-196bb083360b/console-operator/1.log"
Sep 10 21:32:29.738084 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:29.738064    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-z6858_c1e06766-b48c-429e-92f1-196bb083360b/console-operator/2.log"
Sep 10 21:32:30.182951 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:30.182915    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mst9f/perf-node-gather-daemonset-4629q"]
Sep 10 21:32:30.183340 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:30.183246    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e48d1e23-c559-4807-b4fa-cfe80af95402" containerName="main"
Sep 10 21:32:30.183340 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:30.183257    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e48d1e23-c559-4807-b4fa-cfe80af95402" containerName="main"
Sep 10 21:32:30.183421 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:30.183357    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e48d1e23-c559-4807-b4fa-cfe80af95402" containerName="main"
Sep 10 21:32:30.186392 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:30.186371    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mst9f/perf-node-gather-daemonset-4629q"
Sep 10 21:32:30.188462 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:30.188441    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mst9f\"/\"kube-root-ca.crt\""
Sep 10 21:32:30.188580 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:30.188464    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mst9f\"/\"default-dockercfg-jxlqh\""
Sep 10 21:32:30.189074 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:30.189056    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mst9f\"/\"openshift-service-ca.crt\""
Sep 10 21:32:30.196733 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:30.196713    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mst9f/perf-node-gather-daemonset-4629q"]
Sep 10 21:32:30.275390 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:30.275361    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ee2c00b-edc3-447f-8ab3-2c418ed9faef-sys\") pod \"perf-node-gather-daemonset-4629q\" (UID: \"5ee2c00b-edc3-447f-8ab3-2c418ed9faef\") " pod="openshift-must-gather-mst9f/perf-node-gather-daemonset-4629q"
Sep 10 21:32:30.275572 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:30.275408    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ee2c00b-edc3-447f-8ab3-2c418ed9faef-lib-modules\") pod \"perf-node-gather-daemonset-4629q\" (UID: \"5ee2c00b-edc3-447f-8ab3-2c418ed9faef\") " pod="openshift-must-gather-mst9f/perf-node-gather-daemonset-4629q"
Sep 10 21:32:30.275572 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:30.275429    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5ee2c00b-edc3-447f-8ab3-2c418ed9faef-podres\") pod \"perf-node-gather-daemonset-4629q\" (UID: \"5ee2c00b-edc3-447f-8ab3-2c418ed9faef\") " pod="openshift-must-gather-mst9f/perf-node-gather-daemonset-4629q"
Sep 10 21:32:30.275572 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:30.275445    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5ee2c00b-edc3-447f-8ab3-2c418ed9faef-proc\") pod \"perf-node-gather-daemonset-4629q\" (UID: \"5ee2c00b-edc3-447f-8ab3-2c418ed9faef\") " pod="openshift-must-gather-mst9f/perf-node-gather-daemonset-4629q"
Sep 10 21:32:30.275572 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:30.275513    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br8c2\" (UniqueName: \"kubernetes.io/projected/5ee2c00b-edc3-447f-8ab3-2c418ed9faef-kube-api-access-br8c2\") pod \"perf-node-gather-daemonset-4629q\" (UID: \"5ee2c00b-edc3-447f-8ab3-2c418ed9faef\") " pod="openshift-must-gather-mst9f/perf-node-gather-daemonset-4629q"
Sep 10 21:32:30.376078 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:30.376051    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ee2c00b-edc3-447f-8ab3-2c418ed9faef-sys\") pod \"perf-node-gather-daemonset-4629q\" (UID: \"5ee2c00b-edc3-447f-8ab3-2c418ed9faef\") " pod="openshift-must-gather-mst9f/perf-node-gather-daemonset-4629q"
Sep 10 21:32:30.376215 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:30.376089    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ee2c00b-edc3-447f-8ab3-2c418ed9faef-lib-modules\") pod \"perf-node-gather-daemonset-4629q\" (UID: \"5ee2c00b-edc3-447f-8ab3-2c418ed9faef\") " pod="openshift-must-gather-mst9f/perf-node-gather-daemonset-4629q"
Sep 10 21:32:30.376215 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:30.376109    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5ee2c00b-edc3-447f-8ab3-2c418ed9faef-podres\") pod \"perf-node-gather-daemonset-4629q\" (UID: \"5ee2c00b-edc3-447f-8ab3-2c418ed9faef\") " pod="openshift-must-gather-mst9f/perf-node-gather-daemonset-4629q"
Sep 10 21:32:30.376215 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:30.376134    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5ee2c00b-edc3-447f-8ab3-2c418ed9faef-proc\") pod \"perf-node-gather-daemonset-4629q\" (UID: \"5ee2c00b-edc3-447f-8ab3-2c418ed9faef\") " pod="openshift-must-gather-mst9f/perf-node-gather-daemonset-4629q"
Sep 10 21:32:30.376215 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:30.376179    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ee2c00b-edc3-447f-8ab3-2c418ed9faef-sys\") pod \"perf-node-gather-daemonset-4629q\" (UID: \"5ee2c00b-edc3-447f-8ab3-2c418ed9faef\") " pod="openshift-must-gather-mst9f/perf-node-gather-daemonset-4629q"
Sep 10 21:32:30.376215 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:30.376197    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br8c2\" (UniqueName: \"kubernetes.io/projected/5ee2c00b-edc3-447f-8ab3-2c418ed9faef-kube-api-access-br8c2\") pod \"perf-node-gather-daemonset-4629q\" (UID: \"5ee2c00b-edc3-447f-8ab3-2c418ed9faef\") " pod="openshift-must-gather-mst9f/perf-node-gather-daemonset-4629q"
Sep 10 21:32:30.376417 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:30.376249    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5ee2c00b-edc3-447f-8ab3-2c418ed9faef-lib-modules\") pod \"perf-node-gather-daemonset-4629q\" (UID: \"5ee2c00b-edc3-447f-8ab3-2c418ed9faef\") " pod="openshift-must-gather-mst9f/perf-node-gather-daemonset-4629q"
Sep 10 21:32:30.376417 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:30.376258    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5ee2c00b-edc3-447f-8ab3-2c418ed9faef-podres\") pod \"perf-node-gather-daemonset-4629q\" (UID: \"5ee2c00b-edc3-447f-8ab3-2c418ed9faef\") " pod="openshift-must-gather-mst9f/perf-node-gather-daemonset-4629q"
Sep 10 21:32:30.376417 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:30.376272    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5ee2c00b-edc3-447f-8ab3-2c418ed9faef-proc\") pod \"perf-node-gather-daemonset-4629q\" (UID: \"5ee2c00b-edc3-447f-8ab3-2c418ed9faef\") " pod="openshift-must-gather-mst9f/perf-node-gather-daemonset-4629q"
Sep 10 21:32:30.383986 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:30.383961    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-br8c2\" (UniqueName: \"kubernetes.io/projected/5ee2c00b-edc3-447f-8ab3-2c418ed9faef-kube-api-access-br8c2\") pod \"perf-node-gather-daemonset-4629q\" (UID: \"5ee2c00b-edc3-447f-8ab3-2c418ed9faef\") " pod="openshift-must-gather-mst9f/perf-node-gather-daemonset-4629q"
Sep 10 21:32:30.497638 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:30.497564    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mst9f/perf-node-gather-daemonset-4629q"
Sep 10 21:32:30.769803 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:30.769722    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mst9f/perf-node-gather-daemonset-4629q"]
Sep 10 21:32:30.773605 ip-10-0-139-165 kubenswrapper[2581]: W0910 21:32:30.773576    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5ee2c00b_edc3_447f_8ab3_2c418ed9faef.slice/crio-c0105ab08b7ecf6ce83d9433a506b6af4d2f26243c5e91f2f1e034e126e07377 WatchSource:0}: Error finding container c0105ab08b7ecf6ce83d9433a506b6af4d2f26243c5e91f2f1e034e126e07377: Status 404 returned error can't find the container with id c0105ab08b7ecf6ce83d9433a506b6af4d2f26243c5e91f2f1e034e126e07377
Sep 10 21:32:31.511774 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:31.511738    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mst9f/perf-node-gather-daemonset-4629q" event={"ID":"5ee2c00b-edc3-447f-8ab3-2c418ed9faef","Type":"ContainerStarted","Data":"124746b52794b8d296e5fa6d02d6c332cd4fc498e5b1f01a663babd8ac5cbe3d"}
Sep 10 21:32:31.512156 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:31.511793    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mst9f/perf-node-gather-daemonset-4629q" event={"ID":"5ee2c00b-edc3-447f-8ab3-2c418ed9faef","Type":"ContainerStarted","Data":"c0105ab08b7ecf6ce83d9433a506b6af4d2f26243c5e91f2f1e034e126e07377"}
Sep 10 21:32:31.512156 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:31.511857    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mst9f/perf-node-gather-daemonset-4629q"
Sep 10 21:32:31.530412 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:31.530367    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mst9f/perf-node-gather-daemonset-4629q" podStartSLOduration=1.530353608 podStartE2EDuration="1.530353608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 21:32:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:32:31.529027134 +0000 UTC m=+2069.185935905" watchObservedRunningTime="2026-09-10 21:32:31.530353608 +0000 UTC m=+2069.187262380"
Sep 10 21:32:32.411767 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:32.411735    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wfsbr_378700dd-8e73-4a3e-9c10-35b09ebbdd68/dns/0.log"
Sep 10 21:32:32.431995 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:32.431975    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wfsbr_378700dd-8e73-4a3e-9c10-35b09ebbdd68/kube-rbac-proxy/0.log"
Sep 10 21:32:32.454510 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:32.454483    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-h4md4_0f68613e-8086-4f5f-8a47-af8dcd8e3a76/dns-node-resolver/0.log"
Sep 10 21:32:33.056740 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:33.056694    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-757878586c-489dg_9da9490c-a5bc-40b9-ae71-37fcb67f73ee/registry/0.log"
Sep 10 21:32:33.105254 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:33.105233    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mthkf_7fc96d36-95bc-4159-b504-7fad608e5819/node-ca/0.log"
Sep 10 21:32:34.180148 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:34.180112    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cfltqv_d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f/istio-proxy/0.log"
Sep 10 21:32:34.401244 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:34.401195    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-fmnwn_20736e92-a91b-452c-a01f-5a981c3c629a/discovery/0.log"
Sep 10 21:32:34.532461 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:34.532377    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-chrtv_a87366b4-0438-4d4a-8b0c-6f4973b07de9/payload-pre-processing/0.log"
Sep 10 21:32:34.603787 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:34.603762    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64c8fc8856-tfwzb_64a30645-a3fb-4670-89af-a044b3dd00a0/router/0.log"
Sep 10 21:32:35.498574 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:35.498545    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4bsk5_443d28b9-b165-4d98-b86a-ea4924d54962/serve-healthcheck-canary/0.log"
Sep 10 21:32:36.046920 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:36.046886    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-9d2gj_08f9e62b-ae0b-426a-a306-39f8c4fdb7ff/insights-operator/0.log"
Sep 10 21:32:36.047123 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:36.047105    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-9d2gj_08f9e62b-ae0b-426a-a306-39f8c4fdb7ff/insights-operator/1.log"
Sep 10 21:32:36.224764 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:36.224734    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sxgtk_895ac1f0-9420-486f-8a54-6de076e5ae32/kube-rbac-proxy/0.log"
Sep 10 21:32:36.247072 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:36.247049    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sxgtk_895ac1f0-9420-486f-8a54-6de076e5ae32/exporter/0.log"
Sep 10 21:32:36.271132 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:36.271103    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sxgtk_895ac1f0-9420-486f-8a54-6de076e5ae32/extractor/0.log"
Sep 10 21:32:37.525980 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:37.525948    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mst9f/perf-node-gather-daemonset-4629q"
Sep 10 21:32:38.785367 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:38.785341    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6d795c6fb-sbvns_309b0878-ff1c-4275-919d-af72b34772fb/manager/0.log"
Sep 10 21:32:38.899002 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:38.898967    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-7blpg_e7b385e1-fa9f-47b8-87bf-006187f9d607/manager/0.log"
Sep 10 21:32:39.024886 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:39.024853    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d4c889df7-ck2h9_f146c084-a4fd-4b66-b74d-2b9887fedd3f/manager/0.log"
Sep 10 21:32:39.044208 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:39.044187    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d4c889df7-ck2h9_f146c084-a4fd-4b66-b74d-2b9887fedd3f/copy-manifests/0.log"
Sep 10 21:32:40.567506 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:40.567476    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cfltqv_d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f/istio-proxy/0.log"
Sep 10 21:32:40.679692 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:40.679606    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5d99b4bcb4-cdq2z_7282ee80-78f8-4de9-b3a4-d09f186772b7/manager/0.log"
Sep 10 21:32:40.784638 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:40.784609    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-fmnwn_20736e92-a91b-452c-a01f-5a981c3c629a/discovery/0.log"
Sep 10 21:32:40.871121 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:40.871093    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-chrtv_a87366b4-0438-4d4a-8b0c-6f4973b07de9/payload-pre-processing/0.log"
Sep 10 21:32:40.925003 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:40.924976    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64c8fc8856-tfwzb_64a30645-a3fb-4670-89af-a044b3dd00a0/router/0.log"
Sep 10 21:32:42.458273 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:42.458200    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cfltqv_d04cd5a4-8327-4dae-bfd8-72dafdd2cc4f/istio-proxy/0.log"
Sep 10 21:32:42.677288 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:42.677258    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-fmnwn_20736e92-a91b-452c-a01f-5a981c3c629a/discovery/0.log"
Sep 10 21:32:42.762746 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:42.762669    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-chrtv_a87366b4-0438-4d4a-8b0c-6f4973b07de9/payload-pre-processing/0.log"
Sep 10 21:32:42.823504 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:42.823479    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64c8fc8856-tfwzb_64a30645-a3fb-4670-89af-a044b3dd00a0/router/0.log"
Sep 10 21:32:47.430833 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:47.430805    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5rbwt_432f3e3d-207e-4732-97f7-43182d9582b9/kube-multus-additional-cni-plugins/0.log"
Sep 10 21:32:47.472311 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:47.472271    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5rbwt_432f3e3d-207e-4732-97f7-43182d9582b9/egress-router-binary-copy/0.log"
Sep 10 21:32:47.499261 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:47.499235    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5rbwt_432f3e3d-207e-4732-97f7-43182d9582b9/cni-plugins/0.log"
Sep 10 21:32:47.520817 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:47.520797    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5rbwt_432f3e3d-207e-4732-97f7-43182d9582b9/bond-cni-plugin/0.log"
Sep 10 21:32:47.542593 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:47.542575    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5rbwt_432f3e3d-207e-4732-97f7-43182d9582b9/routeoverride-cni/0.log"
Sep 10 21:32:47.563984 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:47.563962    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5rbwt_432f3e3d-207e-4732-97f7-43182d9582b9/whereabouts-cni-bincopy/0.log"
Sep 10 21:32:47.587620 ip-10-0-139-165 kubenswrapper[2581]: I0910 21:32:47.587596    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5rbwt_432f3e3d-207e-4732-97f7-43182d9582b9/whereabouts-cni/0.log"