Sep 09 02:28:24.483447 ip-10-0-136-89 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 09 02:28:24.484009 ip-10-0-136-89 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 09 02:28:24.484029 ip-10-0-136-89 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 09 02:28:24.484361 ip-10-0-136-89 systemd[1]: Failed to start Kubernetes Kubelet. Sep 09 02:28:34.491069 ip-10-0-136-89 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 09 02:28:34.491086 ip-10-0-136-89 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 051546b3097c45c2a6c87ffb6990851d -- Sep 09 02:30:06.528253 ip-10-0-136-89 systemd[1]: Starting Kubernetes Kubelet... Sep 09 02:30:06.956699 ip-10-0-136-89 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 02:30:06.956699 ip-10-0-136-89 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 09 02:30:06.956699 ip-10-0-136-89 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 02:30:06.956699 ip-10-0-136-89 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 09 02:30:06.956699 ip-10-0-136-89 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 02:30:06.957863 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.957750 2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 09 02:30:06.961221 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961207 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 02:30:06.961221 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961221 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 02:30:06.961285 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961225 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 02:30:06.961285 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961230 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 02:30:06.961285 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961233 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 02:30:06.961285 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961236 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 02:30:06.961285 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961239 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 02:30:06.961285 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961241 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 02:30:06.961285 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961245 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 02:30:06.961285 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961248 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 02:30:06.961285 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961251 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 02:30:06.961285 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961254 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 02:30:06.961285 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961257 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 02:30:06.961285 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961259 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 02:30:06.961285 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961262 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 02:30:06.961285 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961265 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 02:30:06.961285 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961269 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 02:30:06.961285 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961273 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 02:30:06.961285 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961276 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 02:30:06.961285 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961278 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 02:30:06.961285 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961281 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 02:30:06.961285 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961284 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 02:30:06.961771 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961287 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 02:30:06.961771 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961289 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 02:30:06.961771 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961292 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 02:30:06.961771 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961295 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 02:30:06.961771 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961298 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 02:30:06.961771 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961302 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 02:30:06.961771 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961304 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 02:30:06.961771 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961307 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 02:30:06.961771 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961311 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 02:30:06.961771 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961315 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 02:30:06.961771 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961318 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 02:30:06.961771 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961320 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 02:30:06.961771 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961323 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 02:30:06.961771 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961326 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 02:30:06.961771 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961329 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 02:30:06.961771 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961331 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 02:30:06.961771 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961334 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 02:30:06.961771 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961336 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 02:30:06.961771 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961339 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 02:30:06.962252 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961341 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 02:30:06.962252 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961343 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 02:30:06.962252 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961349 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 02:30:06.962252 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961352 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 02:30:06.962252 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961354 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 02:30:06.962252 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961357 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 02:30:06.962252 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961359 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 02:30:06.962252 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961361 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 02:30:06.962252 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961364 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 02:30:06.962252 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961366 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 02:30:06.962252 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961369 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 02:30:06.962252 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961371 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 02:30:06.962252 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961373 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 02:30:06.962252 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961375 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 02:30:06.962252 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961379 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 02:30:06.962252 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961383 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 02:30:06.962252 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961385 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 09 02:30:06.962252 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961388 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 02:30:06.962252 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961390 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 02:30:06.962252 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961393 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 02:30:06.962737 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961396 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 02:30:06.962737 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961398 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 02:30:06.962737 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961400 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 02:30:06.962737 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961403 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 02:30:06.962737 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961405 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 02:30:06.962737 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961407 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 02:30:06.962737 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961410 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 02:30:06.962737 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961412 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 02:30:06.962737 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961415 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 02:30:06.962737 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961417 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 02:30:06.962737 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961420 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 02:30:06.962737 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961422 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 02:30:06.962737 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961424 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 02:30:06.962737 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961427 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 02:30:06.962737 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961429 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 02:30:06.962737 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961441 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 02:30:06.962737 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961444 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 02:30:06.962737 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961446 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 02:30:06.962737 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961449 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 02:30:06.962737 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961451 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 02:30:06.963229 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961454 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 02:30:06.963229 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961456 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 02:30:06.963229 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961459 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 02:30:06.963229 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961461 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 02:30:06.963229 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961464 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 02:30:06.963229 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961466 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 02:30:06.963229 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961471 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 02:30:06.963229 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961475 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 02:30:06.963229 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961478 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 02:30:06.963229 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961480 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 02:30:06.963229 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961483 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 02:30:06.963229 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961485 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 02:30:06.963229 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961488 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 02:30:06.963229 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961490 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 02:30:06.963229 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961493 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 02:30:06.963229 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961495 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 02:30:06.963229 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961498 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 02:30:06.963229 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.961500 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 02:30:06.963229 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962273 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 02:30:06.963781 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962280 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 02:30:06.963781 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962283 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 02:30:06.963781 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962286 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 02:30:06.963781 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962288 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 02:30:06.963781 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962291 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 02:30:06.963781 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962294 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 02:30:06.963781 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962296 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 02:30:06.963781 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962299 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 02:30:06.963781 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962302 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 02:30:06.963781 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962308 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 02:30:06.963781 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962312 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 02:30:06.963781 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962315 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 02:30:06.963781 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962318 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 02:30:06.963781 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962321 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 02:30:06.963781 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962323 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 02:30:06.963781 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962326 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 02:30:06.963781 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962328 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 02:30:06.963781 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962331 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 02:30:06.963781 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962333 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 02:30:06.963781 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962336 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 02:30:06.964568 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962339 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 02:30:06.964568 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962341 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 02:30:06.964568 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962344 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 02:30:06.964568 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962347 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 02:30:06.964568 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962349 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 02:30:06.964568 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962351 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 02:30:06.964568 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962354 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 02:30:06.964568 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962356 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 02:30:06.964568 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962359 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 02:30:06.964568 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962361 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 02:30:06.964568 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962364 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 02:30:06.964568 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962366 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 02:30:06.964568 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962369 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 02:30:06.964568 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962371 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 02:30:06.964568 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962374 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 02:30:06.964568 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962376 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 02:30:06.964568 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962378 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 02:30:06.964568 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962381 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 02:30:06.964568 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962384 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 02:30:06.964568 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962387 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 02:30:06.965089 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962389 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 02:30:06.965089 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962392 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 02:30:06.965089 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962395 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 02:30:06.965089 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962397 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 02:30:06.965089 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962400 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 02:30:06.965089 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962402 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 02:30:06.965089 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962404 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 02:30:06.965089 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962407 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 02:30:06.965089 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962409 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 09 02:30:06.965089 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962412 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 02:30:06.965089 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962414 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 02:30:06.965089 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962418 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 02:30:06.965089 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962421 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 02:30:06.965089 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962423 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 02:30:06.965089 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962426 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 02:30:06.965089 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962429 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 02:30:06.965089 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962433 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 02:30:06.965089 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962435 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 02:30:06.965089 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962438 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 02:30:06.965571 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962440 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 02:30:06.965571 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962443 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 02:30:06.965571 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962445 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 02:30:06.965571 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962447 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 02:30:06.965571 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962450 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 02:30:06.965571 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962452 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 02:30:06.965571 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962455 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 02:30:06.965571 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962457 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 02:30:06.965571 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962459 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 02:30:06.965571 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962463 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 02:30:06.965571 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962465 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 02:30:06.965571 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962468 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 02:30:06.965571 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962470 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 02:30:06.965571 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962472 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 02:30:06.965571 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962475 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 02:30:06.965571 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962477 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 02:30:06.965571 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962480 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 02:30:06.965571 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962482 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 02:30:06.965571 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962485 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 02:30:06.966058 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962488 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 02:30:06.966058 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962490 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 02:30:06.966058 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962493 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 02:30:06.966058 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962496 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 02:30:06.966058 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962500 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 02:30:06.966058 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962504 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 02:30:06.966058 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962507 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 02:30:06.966058 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962510 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 02:30:06.966058 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962512 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 02:30:06.966058 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962515 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 02:30:06.966058 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962517 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 02:30:06.966058 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962519 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 02:30:06.966058 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962522 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 02:30:06.966058 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962525 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 02:30:06.966058 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962527 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 02:30:06.966058 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962529 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 02:30:06.966058 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962532 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 02:30:06.966058 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962534 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 02:30:06.966058 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962537 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 02:30:06.966527 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962539 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 02:30:06.966527 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962889 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 02:30:06.966527 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962893 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 02:30:06.966527 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962896 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 02:30:06.966527 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962900 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 02:30:06.966527 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962905 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 02:30:06.966527 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962908 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 02:30:06.966527 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962910 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 02:30:06.966527 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962914 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 02:30:06.966527 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962917 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 02:30:06.966527 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962921 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 02:30:06.966527 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962923 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 02:30:06.966527 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962926 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 02:30:06.966527 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962929 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 02:30:06.966527 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962931 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 02:30:06.966527 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962933 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 02:30:06.966527 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962936 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 02:30:06.966527 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962940 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 02:30:06.966527 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962944 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 02:30:06.967023 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962947 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 02:30:06.967023 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962950 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 02:30:06.967023 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962952 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 02:30:06.967023 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962971 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 02:30:06.967023 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962974 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 02:30:06.967023 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962977 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 02:30:06.967023 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962979 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 02:30:06.967023 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962984 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 02:30:06.967023 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962986 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 02:30:06.967023 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962989 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 02:30:06.967023 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962991 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 02:30:06.967023 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962994 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 02:30:06.967023 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962996 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 02:30:06.967023 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.962999 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 02:30:06.967023 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963001 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 02:30:06.967023 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963003 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 02:30:06.967023 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963006 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 02:30:06.967023 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963008 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 02:30:06.967023 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963011 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 02:30:06.967023 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963014 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 02:30:06.967501 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963016 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 02:30:06.967501 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963019 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 02:30:06.967501 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963021 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 02:30:06.967501 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963023 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 02:30:06.967501 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963027 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 02:30:06.967501 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963029 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 02:30:06.967501 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963032 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 02:30:06.967501 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963034 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 02:30:06.967501 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963037 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 02:30:06.967501 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963039 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 02:30:06.967501 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963042 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 02:30:06.967501 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963045 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 02:30:06.967501 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963048 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 02:30:06.967501 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963050 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 02:30:06.967501 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963053 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 02:30:06.967501 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963056 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 02:30:06.967501 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963058 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 02:30:06.967501 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963061 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 02:30:06.967501 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963064 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 02:30:06.967973 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963067 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 02:30:06.967973 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963069 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 02:30:06.967973 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963072 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 02:30:06.967973 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963074 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 02:30:06.967973 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963077 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 02:30:06.967973 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963079 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 02:30:06.967973 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963082 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 02:30:06.967973 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963084 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 02:30:06.967973 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963087 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 02:30:06.967973 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963090 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 02:30:06.967973 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963092 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 02:30:06.967973 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963094 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 02:30:06.967973 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963097 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 02:30:06.967973 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963099 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 02:30:06.967973 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963102 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 02:30:06.967973 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963104 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 02:30:06.967973 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963106 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 02:30:06.967973 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963109 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 02:30:06.967973 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963112 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 02:30:06.967973 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963114 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 02:30:06.968479 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963117 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 02:30:06.968479 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963119 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 02:30:06.968479 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963122 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 02:30:06.968479 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963125 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 02:30:06.968479 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963128 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 02:30:06.968479 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963131 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 02:30:06.968479 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963134 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 02:30:06.968479 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963136 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 02:30:06.968479 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963151 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 02:30:06.968479 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963154 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 02:30:06.968479 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963157 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 02:30:06.968479 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963160 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 02:30:06.968479 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963162 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 09 02:30:06.968479 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963165 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 02:30:06.968479 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963167 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 02:30:06.968479 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963170 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 02:30:06.968479 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963173 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 02:30:06.968479 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963175 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 02:30:06.968479 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963177 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 02:30:06.968479 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963180 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 02:30:06.968981 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963182 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 02:30:06.968981 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.963185 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 02:30:06.968981 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963245 2577 flags.go:64] FLAG: --address="0.0.0.0" Sep 09 02:30:06.968981 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963279 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 09 02:30:06.968981 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963285 2577 flags.go:64] FLAG: --anonymous-auth="true" Sep 09 02:30:06.968981 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963292 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 09 02:30:06.968981 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963296 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 09 02:30:06.968981 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963300 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 09 02:30:06.968981 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963308 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 09 02:30:06.968981 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963312 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 09 02:30:06.968981 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963315 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 09 02:30:06.968981 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963318 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 09 02:30:06.968981 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963322 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 09 02:30:06.968981 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963325 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 09 02:30:06.968981 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963328 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 09 02:30:06.968981 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963331 2577 flags.go:64] FLAG: --cgroup-root="" Sep 09 02:30:06.968981 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963335 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 09 02:30:06.968981 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963338 2577 flags.go:64] FLAG: --client-ca-file="" Sep 09 02:30:06.968981 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963341 2577 flags.go:64] FLAG: --cloud-provider="external" Sep 09 02:30:06.968981 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963344 2577 flags.go:64] FLAG: --cluster-dns="[]" Sep 09 02:30:06.968981 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963348 2577 flags.go:64] FLAG: --cluster-domain="" Sep 09 02:30:06.968981 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963351 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 09 02:30:06.968981 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963354 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 09 02:30:06.968981 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963357 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 09 02:30:06.969598 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963360 2577 flags.go:64] FLAG: --container-log-max-files="5" Sep 09 02:30:06.969598 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963364 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 09 02:30:06.969598 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963367 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 09 02:30:06.969598 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963370 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 09 02:30:06.969598 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963373 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 09 02:30:06.969598 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963376 2577 flags.go:64] FLAG: --contention-profiling="false" Sep 09 02:30:06.969598 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963378 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 09 02:30:06.969598 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963381 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 09 02:30:06.969598 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963384 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 09 02:30:06.969598 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963387 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 09 02:30:06.969598 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963391 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 09 02:30:06.969598 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963394 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 09 02:30:06.969598 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963396 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 09 02:30:06.969598 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963399 2577 flags.go:64] FLAG: --enable-load-reader="false" Sep 09 02:30:06.969598 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963402 2577 flags.go:64] FLAG: --enable-server="true" Sep 09 02:30:06.969598 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963405 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 09 02:30:06.969598 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963410 2577 flags.go:64] FLAG: --event-burst="100" Sep 09 02:30:06.969598 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963413 2577 flags.go:64] FLAG: --event-qps="50" Sep 09 02:30:06.969598 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963416 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 09 02:30:06.969598 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963419 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 09 02:30:06.969598 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963422 2577 flags.go:64] FLAG: --eviction-hard="" Sep 09 02:30:06.969598 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963426 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 09 02:30:06.969598 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963429 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 09 02:30:06.969598 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963432 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 09 02:30:06.969598 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963436 2577 flags.go:64] FLAG: --eviction-soft="" Sep 09 02:30:06.970213 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963439 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 09 02:30:06.970213 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963442 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 09 02:30:06.970213 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963445 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 09 02:30:06.970213 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963447 2577 flags.go:64] FLAG: --experimental-mounter-path="" Sep 09 02:30:06.970213 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963450 2577 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 09 02:30:06.970213 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963453 2577 flags.go:64] FLAG: --fail-swap-on="true" Sep 09 02:30:06.970213 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963456 2577 flags.go:64] FLAG: --feature-gates="" Sep 09 02:30:06.970213 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963459 2577 flags.go:64] FLAG: --file-check-frequency="20s" Sep 09 02:30:06.970213 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963462 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 09 02:30:06.970213 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963465 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 09 02:30:06.970213 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963468 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 09 02:30:06.970213 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963471 2577 flags.go:64] FLAG: --healthz-port="10248" Sep 09 02:30:06.970213 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963474 2577 flags.go:64] FLAG: --help="false" Sep 09 02:30:06.970213 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963476 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-136-89.ec2.internal" Sep 09 02:30:06.970213 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963479 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 09 02:30:06.970213 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963482 2577 flags.go:64] FLAG: --http-check-frequency="20s" Sep 09 02:30:06.970213 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963485 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 09 02:30:06.970213 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963488 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 09 02:30:06.970213 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963491 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 09 02:30:06.970213 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963494 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 09 02:30:06.970213 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963496 2577 flags.go:64] FLAG: --image-service-endpoint="" Sep 09 02:30:06.970213 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963499 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 09 02:30:06.970213 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963501 2577 flags.go:64] FLAG: --kube-api-burst="100" Sep 09 02:30:06.970786 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963505 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 09 02:30:06.970786 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963508 2577 flags.go:64] FLAG: --kube-api-qps="50" Sep 09 02:30:06.970786 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963511 2577 flags.go:64] FLAG: --kube-reserved="" Sep 09 02:30:06.970786 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963514 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 09 02:30:06.970786 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963517 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 09 02:30:06.970786 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963557 2577 flags.go:64] FLAG: --kubelet-cgroups="" Sep 09 02:30:06.970786 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963603 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 09 02:30:06.970786 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963608 2577 flags.go:64] FLAG: --lock-file="" Sep 09 02:30:06.970786 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963615 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 09 02:30:06.970786 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963620 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 09 02:30:06.970786 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963625 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 09 02:30:06.970786 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963634 2577 flags.go:64] FLAG: --log-json-split-stream="false" Sep 09 02:30:06.970786 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963639 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 09 02:30:06.970786 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963644 2577 flags.go:64] FLAG: --log-text-split-stream="false" Sep 09 02:30:06.970786 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963649 2577 flags.go:64] FLAG: --logging-format="text" Sep 09 02:30:06.970786 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963737 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 09 02:30:06.970786 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963762 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 09 02:30:06.970786 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963770 2577 flags.go:64] FLAG: --manifest-url="" Sep 09 02:30:06.970786 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963777 2577 flags.go:64] FLAG: --manifest-url-header="" Sep 09 02:30:06.970786 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963787 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 09 02:30:06.970786 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963794 2577 flags.go:64] FLAG: --max-open-files="1000000" Sep 09 02:30:06.970786 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963802 2577 flags.go:64] FLAG: --max-pods="110" Sep 09 02:30:06.970786 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963808 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 09 02:30:06.970786 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963822 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 09 02:30:06.970786 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963827 2577 flags.go:64] FLAG: --memory-manager-policy="None" Sep 09 02:30:06.971450 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963832 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 09 02:30:06.971450 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963839 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 09 02:30:06.971450 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963844 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 09 02:30:06.971450 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963850 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 09 02:30:06.971450 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963863 2577 flags.go:64] FLAG: --node-status-max-images="50" Sep 09 02:30:06.971450 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963877 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 09 02:30:06.971450 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963882 2577 flags.go:64] FLAG: --oom-score-adj="-999" Sep 09 02:30:06.971450 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963888 2577 flags.go:64] FLAG: --pod-cidr="" Sep 09 02:30:06.971450 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963895 2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 09 02:30:06.971450 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963905 2577 flags.go:64] FLAG: --pod-manifest-path="" Sep 09 02:30:06.971450 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963909 2577 flags.go:64] FLAG: --pod-max-pids="-1" Sep 09 02:30:06.971450 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963915 2577 flags.go:64] FLAG: --pods-per-core="0" Sep 09 02:30:06.971450 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963920 2577 flags.go:64] FLAG: --port="10250" Sep 09 02:30:06.971450 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963925 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 09 02:30:06.971450 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963936 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-005aca9d66e6002c9" Sep 09 02:30:06.971450 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963941 2577 flags.go:64] FLAG: --qos-reserved="" Sep 09 02:30:06.971450 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963946 2577 flags.go:64] FLAG: --read-only-port="10255" Sep 09 02:30:06.971450 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963951 2577 flags.go:64] FLAG: --register-node="true" Sep 09 02:30:06.971450 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963973 2577 flags.go:64] FLAG: --register-with-taints="" Sep 09 02:30:06.971450 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963980 2577 flags.go:64] FLAG: --registry-burst="10" Sep 09 02:30:06.971450 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963985 2577 flags.go:64] FLAG: --registry-qps="5" Sep 09 02:30:06.971450 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.963990 2577 flags.go:64] FLAG: --reserved-cpus="" Sep 09 02:30:06.971450 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.964000 2577 flags.go:64] FLAG: --reserved-memory="" Sep 09 02:30:06.971450 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.964007 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 09 02:30:06.971450 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.964013 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 09 02:30:06.972170 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.964018 2577 flags.go:64] FLAG: --rotate-certificates="false" Sep 09 02:30:06.972170 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.964023 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 09 02:30:06.972170 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.964028 2577 flags.go:64] FLAG: --runonce="false" Sep 09 02:30:06.972170 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.964033 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 09 02:30:06.972170 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.964038 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 09 02:30:06.972170 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.964043 2577 flags.go:64] FLAG: --seccomp-default="false" Sep 09 02:30:06.972170 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.964053 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 09 02:30:06.972170 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.964058 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 09 02:30:06.972170 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.964063 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 09 02:30:06.972170 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.964068 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 09 02:30:06.972170 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.964073 2577 flags.go:64] FLAG: --storage-driver-password="root" Sep 09 02:30:06.972170 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.964078 2577 flags.go:64] FLAG: --storage-driver-secure="false" Sep 09 02:30:06.972170 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.964083 2577 flags.go:64] FLAG: --storage-driver-table="stats" Sep 09 02:30:06.972170 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.964088 2577 flags.go:64] FLAG: --storage-driver-user="root" Sep 09 02:30:06.972170 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.964098 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 09 02:30:06.972170 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.964103 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 09 02:30:06.972170 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.964110 2577 flags.go:64] FLAG: --system-cgroups="" Sep 09 02:30:06.972170 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.964114 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 09 02:30:06.972170 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.964125 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 09 02:30:06.972170 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.964130 2577 flags.go:64] FLAG: --tls-cert-file="" Sep 09 02:30:06.972170 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.964135 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 09 02:30:06.972170 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.964149 2577 flags.go:64] FLAG: --tls-min-version="" Sep 09 02:30:06.972170 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.964153 2577 flags.go:64] FLAG: --tls-private-key-file="" Sep 09 02:30:06.972170 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.964158 2577 flags.go:64] FLAG: --topology-manager-policy="none" Sep 09 02:30:06.972170 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.964163 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 09 02:30:06.972768 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.964168 2577 flags.go:64] FLAG: --topology-manager-scope="container" Sep 09 02:30:06.972768 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.964173 2577 flags.go:64] FLAG: --v="2" Sep 09 02:30:06.972768 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.964179 2577 flags.go:64] FLAG: --version="false" Sep 09 02:30:06.972768 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.964185 2577 flags.go:64] FLAG: --vmodule="" Sep 09 02:30:06.972768 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.964197 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 09 02:30:06.972768 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.964202 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 09 02:30:06.972768 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964719 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 02:30:06.972768 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964733 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 02:30:06.972768 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964737 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 02:30:06.972768 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964740 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 02:30:06.972768 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964743 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 02:30:06.972768 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964746 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 02:30:06.972768 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964749 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 02:30:06.972768 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964752 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 02:30:06.972768 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964755 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 02:30:06.972768 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964758 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 02:30:06.972768 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964760 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 02:30:06.972768 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964763 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 02:30:06.972768 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964766 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 02:30:06.972768 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964769 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 02:30:06.972768 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964771 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 02:30:06.972768 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964774 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 02:30:06.973352 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964776 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 02:30:06.973352 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964780 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 02:30:06.973352 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964782 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 02:30:06.973352 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964785 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 02:30:06.973352 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964787 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 02:30:06.973352 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964790 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 02:30:06.973352 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964792 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 02:30:06.973352 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964796 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 02:30:06.973352 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964799 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 02:30:06.973352 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964801 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 02:30:06.973352 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964806 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 02:30:06.973352 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964810 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 02:30:06.973352 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964813 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 02:30:06.973352 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964815 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 02:30:06.973352 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964819 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 02:30:06.973352 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964821 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 02:30:06.973352 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964824 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 02:30:06.973352 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964827 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 02:30:06.973352 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964829 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 02:30:06.973827 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964832 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 02:30:06.973827 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964834 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 02:30:06.973827 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964837 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 02:30:06.973827 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964839 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 02:30:06.973827 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964842 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 02:30:06.973827 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964845 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 09 02:30:06.973827 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964848 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 02:30:06.973827 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964850 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 02:30:06.973827 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964853 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 02:30:06.973827 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964855 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 02:30:06.973827 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964858 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 02:30:06.973827 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964860 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 02:30:06.973827 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964863 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 02:30:06.973827 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964867 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 02:30:06.973827 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964870 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 02:30:06.973827 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964873 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 02:30:06.973827 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964876 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 02:30:06.973827 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964879 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 02:30:06.973827 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964881 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 02:30:06.974299 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964883 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 02:30:06.974299 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964886 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 02:30:06.974299 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964888 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 02:30:06.974299 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964891 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 02:30:06.974299 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964893 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 02:30:06.974299 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964896 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 02:30:06.974299 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964898 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 02:30:06.974299 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964904 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 02:30:06.974299 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964906 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 02:30:06.974299 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964909 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 02:30:06.974299 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964911 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 02:30:06.974299 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964914 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 02:30:06.974299 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964916 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 02:30:06.974299 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964919 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 02:30:06.974299 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964922 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 02:30:06.974299 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964924 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 02:30:06.974299 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964927 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 02:30:06.974299 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964930 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 02:30:06.974299 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964932 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 02:30:06.974299 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964943 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 02:30:06.974785 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964946 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 02:30:06.974785 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964948 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 02:30:06.974785 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964951 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 02:30:06.974785 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964954 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 02:30:06.974785 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964970 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 02:30:06.974785 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964974 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 02:30:06.974785 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964977 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 02:30:06.974785 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964980 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 02:30:06.974785 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964983 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 02:30:06.974785 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964986 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 02:30:06.974785 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964989 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 02:30:06.974785 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964991 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 02:30:06.974785 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964994 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 02:30:06.974785 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964996 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 02:30:06.974785 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.964999 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 02:30:06.974785 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.965001 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 02:30:06.974785 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.965003 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 02:30:06.974785 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.965006 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 02:30:06.974785 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.965008 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 02:30:06.975263 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.965013 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 02:30:06.975263 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.965016 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 02:30:06.975263 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.965018 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 02:30:06.975263 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.965022 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 02:30:06.975263 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.965025 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 02:30:06.975263 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.965027 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 02:30:06.975263 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.965815 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 02:30:06.975263 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.973786 2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 09 02:30:06.975263 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.973801 2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 09 02:30:06.975263 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.974681 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 09 02:30:06.975263 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.974692 2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 09 02:30:06.975263 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974747 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 02:30:06.975263 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974753 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 02:30:06.975263 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974756 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 02:30:06.975263 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974759 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 02:30:06.975645 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974762 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 02:30:06.975645 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974765 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 02:30:06.975645 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974767 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 02:30:06.975645 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974770 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 02:30:06.975645 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974773 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 02:30:06.975645 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974775 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 02:30:06.975645 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974778 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 02:30:06.975645 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974781 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 02:30:06.975645 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974783 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 09 02:30:06.975645 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974786 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 02:30:06.975645 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974788 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 02:30:06.975645 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974791 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 02:30:06.975645 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974794 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 02:30:06.975645 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974797 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 02:30:06.975645 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974800 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 02:30:06.975645 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974802 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 02:30:06.975645 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974805 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 02:30:06.975645 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974807 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 02:30:06.975645 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974810 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 02:30:06.975645 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974813 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 02:30:06.976155 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974815 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 02:30:06.976155 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974818 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 02:30:06.976155 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974821 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 02:30:06.976155 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974823 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 02:30:06.976155 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974826 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 02:30:06.976155 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974829 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 02:30:06.976155 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974831 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 02:30:06.976155 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974834 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 02:30:06.976155 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974836 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 02:30:06.976155 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974839 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 02:30:06.976155 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974842 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 02:30:06.976155 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974844 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 02:30:06.976155 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974847 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 02:30:06.976155 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974849 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 02:30:06.976155 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974852 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 02:30:06.976155 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974854 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 02:30:06.976155 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974856 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 02:30:06.976155 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974859 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 02:30:06.976155 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974862 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 02:30:06.976155 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974864 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 02:30:06.976665 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974868 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 02:30:06.976665 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974871 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 02:30:06.976665 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974874 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 02:30:06.976665 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974877 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 02:30:06.976665 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974879 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 02:30:06.976665 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974882 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 02:30:06.976665 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974884 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 02:30:06.976665 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974887 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 02:30:06.976665 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974889 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 02:30:06.976665 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974891 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 02:30:06.976665 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974894 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 02:30:06.976665 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974896 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 02:30:06.976665 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974899 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 02:30:06.976665 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974901 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 02:30:06.976665 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974904 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 02:30:06.976665 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974906 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 02:30:06.976665 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974909 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 02:30:06.976665 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974912 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 02:30:06.976665 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974914 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 02:30:06.977149 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974917 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 02:30:06.977149 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974920 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 02:30:06.977149 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974922 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 02:30:06.977149 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974925 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 02:30:06.977149 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974928 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 02:30:06.977149 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974930 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 02:30:06.977149 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974933 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 02:30:06.977149 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974937 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 02:30:06.977149 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974940 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 02:30:06.977149 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974943 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 02:30:06.977149 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974945 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 02:30:06.977149 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974949 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 02:30:06.977149 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974951 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 02:30:06.977149 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974953 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 02:30:06.977149 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974967 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 02:30:06.977149 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974970 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 02:30:06.977149 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974973 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 02:30:06.977149 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974976 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 02:30:06.977149 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974978 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 02:30:06.977149 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974981 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 02:30:06.977642 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974983 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 02:30:06.977642 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974986 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 02:30:06.977642 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974989 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 02:30:06.977642 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974991 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 02:30:06.977642 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974994 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 02:30:06.977642 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974996 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 02:30:06.977642 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.974999 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 02:30:06.977642 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975002 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 02:30:06.977642 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975004 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 02:30:06.977642 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975008 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 02:30:06.977642 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975012 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 02:30:06.977642 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975015 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 02:30:06.977642 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975017 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 02:30:06.977642 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975020 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 02:30:06.977642 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975022 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 02:30:06.977642 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975025 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 02:30:06.978050 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.975030 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 02:30:06.978050 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975145 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 02:30:06.978050 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975149 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 02:30:06.978050 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975153 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 02:30:06.978050 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975155 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 02:30:06.978050 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975158 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 02:30:06.978050 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975161 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 02:30:06.978050 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975163 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 02:30:06.978050 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975166 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 02:30:06.978050 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975168 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 02:30:06.978050 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975171 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 02:30:06.978050 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975174 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 02:30:06.978050 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975176 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 02:30:06.978050 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975179 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 02:30:06.978050 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975181 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 02:30:06.978413 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975183 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 02:30:06.978413 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975185 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 02:30:06.978413 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975188 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 02:30:06.978413 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975190 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 02:30:06.978413 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975193 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 02:30:06.978413 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975195 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 02:30:06.978413 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975197 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 02:30:06.978413 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975200 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 02:30:06.978413 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975203 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 02:30:06.978413 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975206 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 02:30:06.978413 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975210 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 02:30:06.978413 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975212 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 02:30:06.978413 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975215 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 02:30:06.978413 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975217 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 02:30:06.978413 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975220 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 02:30:06.978413 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975223 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 02:30:06.978413 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975225 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 02:30:06.978413 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975228 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 02:30:06.978413 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975230 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 02:30:06.978413 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975233 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 02:30:06.978890 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975236 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 02:30:06.978890 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975238 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 02:30:06.978890 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975240 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 02:30:06.978890 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975243 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 02:30:06.978890 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975245 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 02:30:06.978890 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975248 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 02:30:06.978890 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975250 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 02:30:06.978890 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975253 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 02:30:06.978890 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975256 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 02:30:06.978890 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975258 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 02:30:06.978890 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975260 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 02:30:06.978890 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975263 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 02:30:06.978890 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975265 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 02:30:06.978890 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975268 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 02:30:06.978890 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975270 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 02:30:06.978890 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975274 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 02:30:06.978890 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975278 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 02:30:06.978890 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975280 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 02:30:06.978890 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975283 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 02:30:06.979428 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975285 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 02:30:06.979428 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975288 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 02:30:06.979428 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975291 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 02:30:06.979428 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975293 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 02:30:06.979428 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975295 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 02:30:06.979428 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975298 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 02:30:06.979428 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975300 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 02:30:06.979428 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975303 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 02:30:06.979428 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975305 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 02:30:06.979428 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975308 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 02:30:06.979428 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975310 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 02:30:06.979428 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975312 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 02:30:06.979428 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975315 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 02:30:06.979428 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975317 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 02:30:06.979428 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975320 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 02:30:06.979428 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975322 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 02:30:06.979428 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975325 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 02:30:06.979428 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975327 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 02:30:06.979428 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975329 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 02:30:06.979885 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975332 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 02:30:06.979885 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975334 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 02:30:06.979885 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975337 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 02:30:06.979885 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975339 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 02:30:06.979885 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975341 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 02:30:06.979885 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975344 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 02:30:06.979885 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975346 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 02:30:06.979885 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975350 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 02:30:06.979885 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975353 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 02:30:06.979885 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975356 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 02:30:06.979885 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975359 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 02:30:06.979885 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975361 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 02:30:06.979885 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975364 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 09 02:30:06.979885 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975367 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 02:30:06.979885 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975369 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 02:30:06.979885 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975372 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 02:30:06.979885 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975374 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 02:30:06.979885 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975377 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 02:30:06.979885 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975379 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 02:30:06.980349 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975381 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 02:30:06.980349 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975384 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 02:30:06.980349 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975386 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 02:30:06.980349 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975389 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 02:30:06.980349 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975391 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 02:30:06.980349 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975394 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 02:30:06.980349 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975396 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 02:30:06.980349 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:06.975398 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 02:30:06.980349 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.975409 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 02:30:06.980349 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.975525 2577 server.go:962] "Client rotation is on, will bootstrap in background" Sep 09 02:30:06.980349 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.977883 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 09 02:30:06.980349 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.978899 2577 server.go:1019] "Starting client certificate rotation" Sep 09 02:30:06.980349 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.978991 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 09 02:30:06.980349 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.979023 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 09 02:30:06.981733 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.981719 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 09 02:30:06.987550 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.987536 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 09 02:30:06.999556 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:06.999455 2577 log.go:25] "Validated CRI v1 runtime API" Sep 09 02:30:07.004278 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.004262 2577 log.go:25] "Validated CRI v1 image API" Sep 09 02:30:07.005204 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.005189 2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 09 02:30:07.009410 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.009393 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 09 02:30:07.010259 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.010237 2577 fs.go:135] Filesystem UUIDs: map[22a8fb9c-1a1b-48c6-b4d5-dfe91a3e6995:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8253fb58-0928-446f-9215-e09560fa3356:/dev/nvme0n1p4] Sep 09 02:30:07.010328 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.010258 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 09 02:30:07.015598 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.015477 2577 manager.go:216] Machine: {Timestamp:2026-09-09 02:30:07.013853946 +0000 UTC m=+0.378188400 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098844 MemoryCapacity:32812163072 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f7db4afa9f0bdca5f17aeac9cd1b6 SystemUUID:ec2f7db4-afa9-f0bd-ca5f-17aeac9cd1b6 BootID:051546b3-097c-45c2-a6c8-7ffb6990851d Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:9a:a6:a4:c8:97 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:9a:a6:a4:c8:97 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7e:58:61:4e:fe:d0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812163072 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 09 02:30:07.015598 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.015585 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 09 02:30:07.015759 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.015687 2577 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 09 02:30:07.016821 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.016796 2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 09 02:30:07.016947 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.016823 2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-89.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 09 02:30:07.017682 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.017670 2577 topology_manager.go:138] "Creating topology manager with none policy" Sep 09 02:30:07.017717 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.017684 2577 container_manager_linux.go:309] "Creating device plugin manager" Sep 09 02:30:07.017717 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.017692 2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 09 02:30:07.018660 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.018650 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 09 02:30:07.020513 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.020503 2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 09 02:30:07.025905 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.025893 2577 state_mem.go:36] "Initialized new in-memory state store" Sep 09 02:30:07.026042 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.026031 2577 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 09 02:30:07.026085 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.026061 2577 kubelet.go:486] "Attempting to sync node with API server" Sep 09 02:30:07.026085 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.026071 2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 09 02:30:07.026085 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.026080 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 09 02:30:07.026191 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.026088 2577 kubelet.go:398] "Adding apiserver pod source" Sep 09 02:30:07.026191 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.026097 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 09 02:30:07.026512 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.026500 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 09 02:30:07.026551 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.026522 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 09 02:30:07.030446 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.030419 2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 09 02:30:07.033350 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.033337 2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 09 02:30:07.033398 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.033366 2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 09 02:30:07.033550 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.033540 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 09 02:30:07.033581 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.033555 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 09 02:30:07.033581 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.033561 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 09 02:30:07.033581 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.033566 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 09 02:30:07.033581 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.033576 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 09 02:30:07.033691 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.033585 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 09 02:30:07.034245 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.034234 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 09 02:30:07.034275 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.034248 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 09 02:30:07.034275 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.034258 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 09 02:30:07.034275 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.034266 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 09 02:30:07.034275 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.034275 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 09 02:30:07.034380 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.034283 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 09 02:30:07.035203 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.035191 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 09 02:30:07.035233 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.035206 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 09 02:30:07.038124 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:07.038102 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 09 02:30:07.038184 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:07.038104 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-89.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 09 02:30:07.039417 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.039403 2577 server.go:1268] "Started kubelet" Sep 09 02:30:07.039508 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.039485 2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 09 02:30:07.039565 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.039482 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 09 02:30:07.039624 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.039612 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 09 02:30:07.040398 ip-10-0-136-89 systemd[1]: Started Kubernetes Kubelet. Sep 09 02:30:07.041274 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.041175 2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 09 02:30:07.042090 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.042072 2577 server.go:310] "Adding debug handlers to kubelet server" Sep 09 02:30:07.048016 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:07.047728 2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 09 02:30:07.049055 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.049035 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-89.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 09 02:30:07.049455 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.049425 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 09 02:30:07.050064 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.050034 2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 09 02:30:07.050169 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:07.050146 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-89.ec2.internal\" not found" Sep 09 02:30:07.050531 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:07.049091 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-89.ec2.internal.18d3862d83d0e28e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-89.ec2.internal,UID:ip-10-0-136-89.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-89.ec2.internal,},FirstTimestamp:2026-09-09 02:30:07.039373966 +0000 UTC m=+0.403708420,LastTimestamp:2026-09-09 02:30:07.039373966 +0000 UTC m=+0.403708420,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-89.ec2.internal,}" Sep 09 02:30:07.050815 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.050799 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 09 02:30:07.050815 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.050801 2577 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 09 02:30:07.050947 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.050849 2577 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 09 02:30:07.050947 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.050873 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 09 02:30:07.050947 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.050886 2577 factory.go:55] Registering systemd factory Sep 09 02:30:07.050947 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.050896 2577 factory.go:223] Registration of the systemd container factory successfully Sep 09 02:30:07.050947 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.050935 2577 reconstruct.go:97] "Volume reconstruction finished" Sep 09 02:30:07.050947 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.050941 2577 reconciler.go:29] "Reconciler: start to sync state" Sep 09 02:30:07.051212 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.051127 2577 factory.go:153] Registering CRI-O factory Sep 09 02:30:07.051212 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.051136 2577 factory.go:223] Registration of the crio container factory successfully Sep 09 02:30:07.051212 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.051151 2577 factory.go:103] Registering Raw factory Sep 09 02:30:07.051212 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.051163 2577 manager.go:1196] Started watching for new ooms in manager Sep 09 02:30:07.053076 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.052558 2577 manager.go:349] Starting recovery of all containers Sep 09 02:30:07.054249 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:07.054197 2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-89.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 09 02:30:07.054353 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:07.054327 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 09 02:30:07.061333 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.061056 2577 manager.go:354] Recovery completed Sep 09 02:30:07.065865 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.065847 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 02:30:07.067753 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.067731 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasSufficientMemory" Sep 09 02:30:07.067825 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.067769 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 02:30:07.067825 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.067781 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasSufficientPID" Sep 09 02:30:07.068358 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.068345 2577 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 09 02:30:07.068358 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.068358 2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 09 02:30:07.068449 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.068375 2577 state_mem.go:36] "Initialized new in-memory state store" Sep 09 02:30:07.068449 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.068378 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 02:30:07.069025 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.069011 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasSufficientMemory" Sep 09 02:30:07.069086 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.069040 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 02:30:07.069086 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.069054 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasSufficientPID" Sep 09 02:30:07.070524 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.070512 2577 policy_none.go:49] "None policy: Start" Sep 09 02:30:07.070594 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.070529 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 09 02:30:07.070594 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.070541 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 09 02:30:07.070714 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:07.070650 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-89.ec2.internal.18d3862d8581f05a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-89.ec2.internal,UID:ip-10-0-136-89.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-89.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-89.ec2.internal,},FirstTimestamp:2026-09-09 02:30:07.067754586 +0000 UTC m=+0.432089039,LastTimestamp:2026-09-09 02:30:07.067754586 +0000 UTC m=+0.432089039,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-89.ec2.internal,}" Sep 09 02:30:07.072483 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.072472 2577 policy_none.go:47] "Start" Sep 09 02:30:07.079985 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.079950 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fm64m" Sep 09 02:30:07.084566 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:07.084499 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-89.ec2.internal.18d3862d85823ee4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-89.ec2.internal,UID:ip-10-0-136-89.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-89.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-89.ec2.internal,},FirstTimestamp:2026-09-09 02:30:07.067774692 +0000 UTC m=+0.432109146,LastTimestamp:2026-09-09 02:30:07.067774692 +0000 UTC m=+0.432109146,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-89.ec2.internal,}" Sep 09 02:30:07.088701 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.088685 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fm64m" Sep 09 02:30:07.131214 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.111326 2577 manager.go:337] "Starting Device Plugin manager" Sep 09 02:30:07.131214 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:07.111358 2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 09 02:30:07.131214 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.111371 2577 server.go:87] "Starting device plugin registration server" Sep 09 02:30:07.131214 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.111615 2577 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 09 02:30:07.131214 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.111627 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 09 02:30:07.131214 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.111772 2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 09 02:30:07.131214 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.111866 2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 09 02:30:07.131214 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.111877 2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 09 02:30:07.131214 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:07.112357 2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 09 02:30:07.131214 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:07.112413 2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-89.ec2.internal\" not found" Sep 09 02:30:07.157345 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.157321 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 09 02:30:07.158541 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.158525 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 09 02:30:07.158622 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.158571 2577 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 09 02:30:07.158622 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.158592 2577 kubelet.go:2448] "Starting kubelet main sync loop" Sep 09 02:30:07.158706 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:07.158627 2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 09 02:30:07.160708 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.160691 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 02:30:07.212621 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.212530 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 02:30:07.213450 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.213425 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasSufficientMemory" Sep 09 02:30:07.213546 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.213465 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 02:30:07.213546 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.213476 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasSufficientPID" Sep 09 02:30:07.213546 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.213499 2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-89.ec2.internal" Sep 09 02:30:07.222813 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.222782 2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-89.ec2.internal" Sep 09 02:30:07.222813 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:07.222808 2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-89.ec2.internal\": node \"ip-10-0-136-89.ec2.internal\" not found" Sep 09 02:30:07.256234 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:07.256211 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-89.ec2.internal\" not found" Sep 09 02:30:07.259273 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.259256 2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-89.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-89.ec2.internal"] Sep 09 02:30:07.259336 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.259318 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 02:30:07.260247 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.260231 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasSufficientMemory" Sep 09 02:30:07.260317 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.260261 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 02:30:07.260317 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.260272 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasSufficientPID" Sep 09 02:30:07.262708 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.262696 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 02:30:07.262884 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.262870 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-89.ec2.internal" Sep 09 02:30:07.262926 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.262899 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 02:30:07.263457 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.263442 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasSufficientMemory" Sep 09 02:30:07.263513 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.263467 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 02:30:07.263513 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.263479 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasSufficientPID" Sep 09 02:30:07.263513 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.263492 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasSufficientMemory" Sep 09 02:30:07.263513 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.263510 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 02:30:07.263649 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.263522 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasSufficientPID" Sep 09 02:30:07.265743 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.265727 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-89.ec2.internal" Sep 09 02:30:07.265819 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.265753 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 02:30:07.266402 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.266385 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasSufficientMemory" Sep 09 02:30:07.266483 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.266418 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 02:30:07.266483 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.266434 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasSufficientPID" Sep 09 02:30:07.282468 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:07.282451 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-89.ec2.internal\" not found" node="ip-10-0-136-89.ec2.internal" Sep 09 02:30:07.285731 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:07.285717 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-89.ec2.internal\" not found" node="ip-10-0-136-89.ec2.internal" Sep 09 02:30:07.352307 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.352286 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/81bd3a799875c0edff0ebc792a1d03d1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-89.ec2.internal\" (UID: \"81bd3a799875c0edff0ebc792a1d03d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-89.ec2.internal" Sep 09 02:30:07.352397 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.352310 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81bd3a799875c0edff0ebc792a1d03d1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-89.ec2.internal\" (UID: \"81bd3a799875c0edff0ebc792a1d03d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-89.ec2.internal" Sep 09 02:30:07.352397 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.352334 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/38b43ea1a0636b4e4e4c403eeb25802f-config\") pod \"kube-apiserver-proxy-ip-10-0-136-89.ec2.internal\" (UID: \"38b43ea1a0636b4e4e4c403eeb25802f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-89.ec2.internal" Sep 09 02:30:07.356660 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:07.356644 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-89.ec2.internal\" not found" Sep 09 02:30:07.453089 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.453062 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/81bd3a799875c0edff0ebc792a1d03d1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-89.ec2.internal\" (UID: \"81bd3a799875c0edff0ebc792a1d03d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-89.ec2.internal" Sep 09 02:30:07.453200 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.453121 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81bd3a799875c0edff0ebc792a1d03d1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-89.ec2.internal\" (UID: \"81bd3a799875c0edff0ebc792a1d03d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-89.ec2.internal" Sep 09 02:30:07.453200 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.453139 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/38b43ea1a0636b4e4e4c403eeb25802f-config\") pod \"kube-apiserver-proxy-ip-10-0-136-89.ec2.internal\" (UID: \"38b43ea1a0636b4e4e4c403eeb25802f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-89.ec2.internal" Sep 09 02:30:07.453200 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.453180 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/38b43ea1a0636b4e4e4c403eeb25802f-config\") pod \"kube-apiserver-proxy-ip-10-0-136-89.ec2.internal\" (UID: \"38b43ea1a0636b4e4e4c403eeb25802f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-89.ec2.internal" Sep 09 02:30:07.453342 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.453195 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/81bd3a799875c0edff0ebc792a1d03d1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-89.ec2.internal\" (UID: \"81bd3a799875c0edff0ebc792a1d03d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-89.ec2.internal" Sep 09 02:30:07.453342 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.453234 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81bd3a799875c0edff0ebc792a1d03d1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-89.ec2.internal\" (UID: \"81bd3a799875c0edff0ebc792a1d03d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-89.ec2.internal" Sep 09 02:30:07.457160 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:07.457145 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-89.ec2.internal\" not found" Sep 09 02:30:07.558038 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:07.557954 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-89.ec2.internal\" not found" Sep 09 02:30:07.585127 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.585100 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-89.ec2.internal" Sep 09 02:30:07.588271 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.588255 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-89.ec2.internal" Sep 09 02:30:07.658273 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:07.658247 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-89.ec2.internal\" not found" Sep 09 02:30:07.758843 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:07.758813 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-89.ec2.internal\" not found" Sep 09 02:30:07.859337 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:07.859276 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-89.ec2.internal\" not found" Sep 09 02:30:07.959985 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:07.959944 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-89.ec2.internal\" not found" Sep 09 02:30:07.979113 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.979091 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 09 02:30:07.979232 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:07.979214 2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 09 02:30:08.060991 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:08.060950 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-89.ec2.internal\" not found" Sep 09 02:30:08.092296 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:08.092251 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 02:25:07 +0000 UTC" deadline="2028-03-26 12:13:08.759490851 +0000 UTC" Sep 09 02:30:08.092296 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:08.092286 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13545h43m0.667208624s" Sep 09 02:30:08.162069 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:08.162007 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-89.ec2.internal\" not found" Sep 09 02:30:08.196174 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:08.196146 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 02:30:08.197295 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:08.197277 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasSufficientMemory" Sep 09 02:30:08.197384 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:08.197312 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 02:30:08.197384 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:08.197326 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeHasSufficientPID" Sep 09 02:30:08.228888 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:08.228856 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81bd3a799875c0edff0ebc792a1d03d1.slice/crio-870c7351cd307b4317544a301636d51da90ae4e40ddb71f02e2bef3c5dfb16f1 WatchSource:0}: Error finding container 870c7351cd307b4317544a301636d51da90ae4e40ddb71f02e2bef3c5dfb16f1: Status 404 returned error can't find the container with id 870c7351cd307b4317544a301636d51da90ae4e40ddb71f02e2bef3c5dfb16f1 Sep 09 02:30:08.229670 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:08.229649 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38b43ea1a0636b4e4e4c403eeb25802f.slice/crio-2d004e4c181c8b5aa0b6708f80cb4de66a81a792de20d586b7918a2bd15ec3aa WatchSource:0}: Error finding container 2d004e4c181c8b5aa0b6708f80cb4de66a81a792de20d586b7918a2bd15ec3aa: Status 404 returned error can't find the container with id 2d004e4c181c8b5aa0b6708f80cb4de66a81a792de20d586b7918a2bd15ec3aa Sep 09 02:30:08.234476 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:08.234461 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 02:30:08.262631 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:08.262609 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-89.ec2.internal\" not found" Sep 09 02:30:08.322700 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:08.322675 2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 02:30:08.363405 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:08.363382 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-89.ec2.internal\" not found" Sep 09 02:30:08.392813 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:08.392791 2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 02:30:08.451218 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:08.451165 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-89.ec2.internal" Sep 09 02:30:08.459915 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:08.459898 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 09 02:30:08.461029 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:08.461017 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-89.ec2.internal" Sep 09 02:30:08.474743 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:08.474727 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 09 02:30:08.618860 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:08.618832 2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 02:30:09.026973 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.026937 2577 apiserver.go:52] "Watching apiserver" Sep 09 02:30:09.036925 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.036894 2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 09 02:30:09.038261 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.038232 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-tvhb5","openshift-network-diagnostics/network-check-target-65ldr","openshift-network-operator/iptables-alerter-gktjg","openshift-ovn-kubernetes/ovnkube-node-smnqw","kube-system/global-pull-secret-syncer-6lzgq","kube-system/konnectivity-agent-qw4bt","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfpfn","openshift-cluster-node-tuning-operator/tuned-h4jdc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-89.ec2.internal","openshift-multus/multus-additional-cni-plugins-zjstd","openshift-multus/multus-dj8dq","openshift-multus/network-metrics-daemon-mgjfl","kube-system/kube-apiserver-proxy-ip-10-0-136-89.ec2.internal","openshift-dns/node-resolver-pd8dm"] Sep 09 02:30:09.040864 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.040836 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfpfn" Sep 09 02:30:09.043126 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.043102 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.045308 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.045290 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gktjg" Sep 09 02:30:09.045458 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.045438 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 09 02:30:09.045533 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.045473 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 09 02:30:09.045888 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.045872 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zr6gb\"" Sep 09 02:30:09.045968 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.045888 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 09 02:30:09.045968 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.045904 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 09 02:30:09.046080 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.045980 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 09 02:30:09.046416 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.046394 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 09 02:30:09.046507 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.046483 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-hxskh\"" Sep 09 02:30:09.047273 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.047237 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 09 02:30:09.047911 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.047892 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.049749 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.049730 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 09 02:30:09.050650 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.050256 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 09 02:30:09.050650 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.050569 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6lzgq" Sep 09 02:30:09.050771 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:09.050648 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6lzgq" podUID="b974ece6-0b9d-4339-aa6f-203ce53dcd85" Sep 09 02:30:09.052157 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.052138 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 09 02:30:09.052363 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.052340 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 09 02:30:09.052464 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.052449 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 09 02:30:09.052606 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.052590 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 09 02:30:09.052680 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.052666 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 09 02:30:09.052782 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.052345 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 09 02:30:09.052874 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.052842 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-c2dc5\"" Sep 09 02:30:09.053135 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.053114 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qw4bt" Sep 09 02:30:09.053324 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.053309 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 09 02:30:09.053423 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.053410 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 09 02:30:09.053555 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.053539 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vbnll\"" Sep 09 02:30:09.055318 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.055298 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pd8dm" Sep 09 02:30:09.057358 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.057026 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qhx7r\"" Sep 09 02:30:09.057358 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.057111 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 09 02:30:09.057358 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.057248 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 09 02:30:09.058163 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.058095 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" Sep 09 02:30:09.058678 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.058657 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 09 02:30:09.058771 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.058740 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sj7xn\"" Sep 09 02:30:09.058771 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.058750 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 09 02:30:09.060421 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.060400 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65ldr" Sep 09 02:30:09.060515 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:09.060461 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-65ldr" podUID="a50c24dc-cc0a-404a-89d0-77e82c417bb5" Sep 09 02:30:09.062356 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.062335 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 09 02:30:09.063215 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.063126 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6b4bg\"" Sep 09 02:30:09.063321 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.063132 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 09 02:30:09.063321 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.063245 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zjstd" Sep 09 02:30:09.064925 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.063416 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83-socket-dir\") pod \"aws-ebs-csi-driver-node-dfpfn\" (UID: \"cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfpfn" Sep 09 02:30:09.064925 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.063463 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83-device-dir\") pod \"aws-ebs-csi-driver-node-dfpfn\" (UID: \"cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfpfn" Sep 09 02:30:09.064925 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.063563 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lblsl\" (UniqueName: \"kubernetes.io/projected/cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83-kube-api-access-lblsl\") pod \"aws-ebs-csi-driver-node-dfpfn\" (UID: \"cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfpfn" Sep 09 02:30:09.064925 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.063587 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-multus-cni-dir\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.064925 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.063602 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-host-run-netns\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.064925 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.063622 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-systemd-units\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.064925 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.063642 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-log-socket\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.064925 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.063766 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dead4f18-8c41-41a5-9f37-897ddaee8563-agent-certs\") pod \"konnectivity-agent-qw4bt\" (UID: \"dead4f18-8c41-41a5-9f37-897ddaee8563\") " pod="kube-system/konnectivity-agent-qw4bt" Sep 09 02:30:09.064925 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.063786 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-host-var-lib-kubelet\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.064925 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.063804 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbql6\" (UniqueName: \"kubernetes.io/projected/52808c42-8ac8-49fd-96f8-e05e956b42be-kube-api-access-vbql6\") pod \"iptables-alerter-gktjg\" (UID: \"52808c42-8ac8-49fd-96f8-e05e956b42be\") " pod="openshift-network-operator/iptables-alerter-gktjg" Sep 09 02:30:09.064925 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.063824 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-host-run-ovn-kubernetes\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.064925 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.063950 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-host-cni-bin\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.064925 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.063993 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.064925 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.064119 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b974ece6-0b9d-4339-aa6f-203ce53dcd85-kubelet-config\") pod \"global-pull-secret-syncer-6lzgq\" (UID: \"b974ece6-0b9d-4339-aa6f-203ce53dcd85\") " pod="kube-system/global-pull-secret-syncer-6lzgq" Sep 09 02:30:09.064925 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.064179 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83-etc-selinux\") pod \"aws-ebs-csi-driver-node-dfpfn\" (UID: \"cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfpfn" Sep 09 02:30:09.064925 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.064223 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/52808c42-8ac8-49fd-96f8-e05e956b42be-host-slash\") pod \"iptables-alerter-gktjg\" (UID: \"52808c42-8ac8-49fd-96f8-e05e956b42be\") " pod="openshift-network-operator/iptables-alerter-gktjg" Sep 09 02:30:09.065789 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.064281 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-host-run-netns\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.065789 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.064336 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-etc-openvswitch\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.065789 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.064366 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-host-cni-netd\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.065789 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.064400 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b974ece6-0b9d-4339-aa6f-203ce53dcd85-original-pull-secret\") pod \"global-pull-secret-syncer-6lzgq\" (UID: \"b974ece6-0b9d-4339-aa6f-203ce53dcd85\") " pod="kube-system/global-pull-secret-syncer-6lzgq" Sep 09 02:30:09.065789 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.064436 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-host-run-k8s-cni-cncf-io\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.065789 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.064469 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-host-var-lib-cni-bin\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.065789 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.064505 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-hostroot\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.065789 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.064534 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-multus-daemon-config\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.065789 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.064566 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-run-openvswitch\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.065789 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.064597 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-ovnkube-config\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.065789 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.064635 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dead4f18-8c41-41a5-9f37-897ddaee8563-konnectivity-ca\") pod \"konnectivity-agent-qw4bt\" (UID: \"dead4f18-8c41-41a5-9f37-897ddaee8563\") " pod="kube-system/konnectivity-agent-qw4bt" Sep 09 02:30:09.065789 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.064669 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dfpfn\" (UID: \"cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfpfn" Sep 09 02:30:09.065789 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.064699 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83-registration-dir\") pod \"aws-ebs-csi-driver-node-dfpfn\" (UID: \"cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfpfn" Sep 09 02:30:09.065789 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.064734 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-multus-socket-dir-parent\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.065789 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.064838 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-host-var-lib-cni-multus\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.065789 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.064885 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83-sys-fs\") pod \"aws-ebs-csi-driver-node-dfpfn\" (UID: \"cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfpfn" Sep 09 02:30:09.066891 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.064925 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-cnibin\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.066891 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.064971 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-cni-binary-copy\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.066891 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.065020 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-host-run-multus-certs\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.066891 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.065093 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-run-systemd\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.066891 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.065123 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-var-lib-openvswitch\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.066891 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.065169 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whrxm\" (UniqueName: \"kubernetes.io/projected/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-kube-api-access-whrxm\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.066891 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.065196 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b974ece6-0b9d-4339-aa6f-203ce53dcd85-dbus\") pod \"global-pull-secret-syncer-6lzgq\" (UID: \"b974ece6-0b9d-4339-aa6f-203ce53dcd85\") " pod="kube-system/global-pull-secret-syncer-6lzgq" Sep 09 02:30:09.066891 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.065232 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-system-cni-dir\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.066891 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.065267 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-etc-kubernetes\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.066891 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.065299 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-host-kubelet\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.066891 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.065328 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-host-slash\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.066891 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.065358 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-env-overrides\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.066891 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.065400 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-ovn-node-metrics-cert\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.066891 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.065450 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-multus-conf-dir\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.066891 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.065495 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t486\" (UniqueName: \"kubernetes.io/projected/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-kube-api-access-5t486\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.066891 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.065543 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/52808c42-8ac8-49fd-96f8-e05e956b42be-iptables-alerter-script\") pod \"iptables-alerter-gktjg\" (UID: \"52808c42-8ac8-49fd-96f8-e05e956b42be\") " pod="openshift-network-operator/iptables-alerter-gktjg" Sep 09 02:30:09.066891 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.065577 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-run-ovn\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.067692 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.065621 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-node-log\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.067692 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.065654 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-ovnkube-script-lib\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.067692 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.066540 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tvhb5" Sep 09 02:30:09.068080 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.067977 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 09 02:30:09.068170 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.068154 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fs8xm\"" Sep 09 02:30:09.068753 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.068732 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 09 02:30:09.069192 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.069045 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 09 02:30:09.070810 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.069489 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjfl" Sep 09 02:30:09.070810 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:09.069552 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgjfl" podUID="06180cab-e7c8-4fb8-a719-70072ee0ee36" Sep 09 02:30:09.073444 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.073421 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 09 02:30:09.073709 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.073684 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 09 02:30:09.073947 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.073931 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 09 02:30:09.074222 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.074201 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rpqjd\"" Sep 09 02:30:09.101585 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.101555 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-q8wq8" Sep 09 02:30:09.109641 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.109236 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-q8wq8" Sep 09 02:30:09.153022 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.152760 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 09 02:30:09.165341 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.165289 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-89.ec2.internal" event={"ID":"81bd3a799875c0edff0ebc792a1d03d1","Type":"ContainerStarted","Data":"870c7351cd307b4317544a301636d51da90ae4e40ddb71f02e2bef3c5dfb16f1"} Sep 09 02:30:09.165784 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.165759 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-host-run-netns\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.165884 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.165795 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-host-var-lib-cni-bin\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.165884 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.165821 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-hostroot\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.165884 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.165846 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-multus-daemon-config\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.165884 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.165871 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-run-openvswitch\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.165884 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.165875 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-host-run-netns\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.166195 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.165887 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-host-var-lib-cni-bin\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.166195 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.165898 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1-cnibin\") pod \"multus-additional-cni-plugins-zjstd\" (UID: \"95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1\") " pod="openshift-multus/multus-additional-cni-plugins-zjstd" Sep 09 02:30:09.166195 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.165942 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dfpfn\" (UID: \"cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfpfn" Sep 09 02:30:09.166195 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.165971 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-run-openvswitch\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.166195 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.165943 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-hostroot\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.166195 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.165992 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-multus-socket-dir-parent\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.166195 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.166053 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dfpfn\" (UID: \"cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfpfn" Sep 09 02:30:09.166195 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.166093 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e3e028fc-3961-4b04-9875-fa06912e9518-etc-modprobe-d\") pod \"tuned-h4jdc\" (UID: \"e3e028fc-3961-4b04-9875-fa06912e9518\") " pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" Sep 09 02:30:09.166195 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.166142 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84fp8\" (UniqueName: \"kubernetes.io/projected/e3e028fc-3961-4b04-9875-fa06912e9518-kube-api-access-84fp8\") pod \"tuned-h4jdc\" (UID: \"e3e028fc-3961-4b04-9875-fa06912e9518\") " pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" Sep 09 02:30:09.166195 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.166172 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/54c26b32-0189-4a6a-a569-c2f859d87c8b-serviceca\") pod \"node-ca-tvhb5\" (UID: \"54c26b32-0189-4a6a-a569-c2f859d87c8b\") " pod="openshift-image-registry/node-ca-tvhb5" Sep 09 02:30:09.166195 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.166190 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-multus-socket-dir-parent\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.166195 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.166200 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-cnibin\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.166771 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.166227 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-cni-binary-copy\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.166771 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.166253 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-host-run-multus-certs\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.166771 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.166277 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-run-systemd\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.166771 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.166286 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-cnibin\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.166771 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.166302 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-var-lib-openvswitch\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.166771 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.166328 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whrxm\" (UniqueName: \"kubernetes.io/projected/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-kube-api-access-whrxm\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.166771 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.166348 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-host-run-multus-certs\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.166771 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.166351 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b974ece6-0b9d-4339-aa6f-203ce53dcd85-dbus\") pod \"global-pull-secret-syncer-6lzgq\" (UID: \"b974ece6-0b9d-4339-aa6f-203ce53dcd85\") " pod="kube-system/global-pull-secret-syncer-6lzgq" Sep 09 02:30:09.166771 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.166398 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-env-overrides\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.166771 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.166421 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b974ece6-0b9d-4339-aa6f-203ce53dcd85-dbus\") pod \"global-pull-secret-syncer-6lzgq\" (UID: \"b974ece6-0b9d-4339-aa6f-203ce53dcd85\") " pod="kube-system/global-pull-secret-syncer-6lzgq" Sep 09 02:30:09.166771 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.166426 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zptrg\" (UniqueName: \"kubernetes.io/projected/54c26b32-0189-4a6a-a569-c2f859d87c8b-kube-api-access-zptrg\") pod \"node-ca-tvhb5\" (UID: \"54c26b32-0189-4a6a-a569-c2f859d87c8b\") " pod="openshift-image-registry/node-ca-tvhb5" Sep 09 02:30:09.166771 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.166452 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-ovnkube-script-lib\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.166771 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.166469 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-run-systemd\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.166771 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.166476 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e3e028fc-3961-4b04-9875-fa06912e9518-lib-modules\") pod \"tuned-h4jdc\" (UID: \"e3e028fc-3961-4b04-9875-fa06912e9518\") " pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" Sep 09 02:30:09.167472 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.166873 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-var-lib-openvswitch\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.167472 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.166922 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1-cni-binary-copy\") pod \"multus-additional-cni-plugins-zjstd\" (UID: \"95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1\") " pod="openshift-multus/multus-additional-cni-plugins-zjstd" Sep 09 02:30:09.167472 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.166952 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcg5c\" (UniqueName: \"kubernetes.io/projected/95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1-kube-api-access-hcg5c\") pod \"multus-additional-cni-plugins-zjstd\" (UID: \"95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1\") " pod="openshift-multus/multus-additional-cni-plugins-zjstd" Sep 09 02:30:09.167472 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.167001 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83-device-dir\") pod \"aws-ebs-csi-driver-node-dfpfn\" (UID: \"cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfpfn" Sep 09 02:30:09.167472 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.167033 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-env-overrides\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.167472 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.167049 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lblsl\" (UniqueName: \"kubernetes.io/projected/cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83-kube-api-access-lblsl\") pod \"aws-ebs-csi-driver-node-dfpfn\" (UID: \"cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfpfn" Sep 09 02:30:09.167472 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.167136 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-host-run-netns\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.167472 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.167165 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-systemd-units\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.167472 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.167189 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-log-socket\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.167472 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.167258 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-host-cni-bin\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.167472 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.167316 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.167472 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.167330 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-ovnkube-script-lib\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.167472 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.167350 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e3e028fc-3961-4b04-9875-fa06912e9518-etc-systemd\") pod \"tuned-h4jdc\" (UID: \"e3e028fc-3961-4b04-9875-fa06912e9518\") " pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" Sep 09 02:30:09.167472 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.167378 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dead4f18-8c41-41a5-9f37-897ddaee8563-agent-certs\") pod \"konnectivity-agent-qw4bt\" (UID: \"dead4f18-8c41-41a5-9f37-897ddaee8563\") " pod="kube-system/konnectivity-agent-qw4bt" Sep 09 02:30:09.167472 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.167386 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-systemd-units\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.167472 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.167393 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-multus-daemon-config\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.167472 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.167401 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-host-run-netns\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.168321 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.167136 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83-device-dir\") pod \"aws-ebs-csi-driver-node-dfpfn\" (UID: \"cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfpfn" Sep 09 02:30:09.168321 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.167404 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-host-var-lib-kubelet\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.168321 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.167445 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.168321 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.167458 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-host-run-ovn-kubernetes\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.168321 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.167480 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-host-cni-bin\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.168321 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.167485 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e3e028fc-3961-4b04-9875-fa06912e9518-run\") pod \"tuned-h4jdc\" (UID: \"e3e028fc-3961-4b04-9875-fa06912e9518\") " pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" Sep 09 02:30:09.168321 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.167520 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-host-run-ovn-kubernetes\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.168321 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.167527 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e3e028fc-3961-4b04-9875-fa06912e9518-etc-tuned\") pod \"tuned-h4jdc\" (UID: \"e3e028fc-3961-4b04-9875-fa06912e9518\") " pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" Sep 09 02:30:09.168321 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.167460 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-log-socket\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.168321 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.167560 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fdsw\" (UniqueName: \"kubernetes.io/projected/19afdc10-5a52-4e9d-a84f-04df6136a1c0-kube-api-access-4fdsw\") pod \"node-resolver-pd8dm\" (UID: \"19afdc10-5a52-4e9d-a84f-04df6136a1c0\") " pod="openshift-dns/node-resolver-pd8dm" Sep 09 02:30:09.168321 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.167583 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-host-var-lib-kubelet\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.168321 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.167629 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1-system-cni-dir\") pod \"multus-additional-cni-plugins-zjstd\" (UID: \"95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1\") " pod="openshift-multus/multus-additional-cni-plugins-zjstd" Sep 09 02:30:09.168321 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.167661 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zjstd\" (UID: \"95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1\") " pod="openshift-multus/multus-additional-cni-plugins-zjstd" Sep 09 02:30:09.168321 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.167694 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b974ece6-0b9d-4339-aa6f-203ce53dcd85-kubelet-config\") pod \"global-pull-secret-syncer-6lzgq\" (UID: \"b974ece6-0b9d-4339-aa6f-203ce53dcd85\") " pod="kube-system/global-pull-secret-syncer-6lzgq" Sep 09 02:30:09.168321 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.167726 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83-etc-selinux\") pod \"aws-ebs-csi-driver-node-dfpfn\" (UID: \"cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfpfn" Sep 09 02:30:09.168321 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.167752 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-etc-openvswitch\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.168321 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.167776 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-host-cni-netd\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.169204 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.167802 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3e028fc-3961-4b04-9875-fa06912e9518-var-lib-kubelet\") pod \"tuned-h4jdc\" (UID: \"e3e028fc-3961-4b04-9875-fa06912e9518\") " pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" Sep 09 02:30:09.169204 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.167815 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-etc-openvswitch\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.169204 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.167812 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b974ece6-0b9d-4339-aa6f-203ce53dcd85-kubelet-config\") pod \"global-pull-secret-syncer-6lzgq\" (UID: \"b974ece6-0b9d-4339-aa6f-203ce53dcd85\") " pod="kube-system/global-pull-secret-syncer-6lzgq" Sep 09 02:30:09.169204 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.167826 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-cni-binary-copy\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.169204 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.167838 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-host-cni-netd\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.169204 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.167853 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b974ece6-0b9d-4339-aa6f-203ce53dcd85-original-pull-secret\") pod \"global-pull-secret-syncer-6lzgq\" (UID: \"b974ece6-0b9d-4339-aa6f-203ce53dcd85\") " pod="kube-system/global-pull-secret-syncer-6lzgq" Sep 09 02:30:09.169204 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.167870 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83-etc-selinux\") pod \"aws-ebs-csi-driver-node-dfpfn\" (UID: \"cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfpfn" Sep 09 02:30:09.169204 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.167882 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-host-run-k8s-cni-cncf-io\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.169204 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.167906 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-ovnkube-config\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.169204 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.167930 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3e028fc-3961-4b04-9875-fa06912e9518-sys\") pod \"tuned-h4jdc\" (UID: \"e3e028fc-3961-4b04-9875-fa06912e9518\") " pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" Sep 09 02:30:09.169204 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.167934 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-host-run-k8s-cni-cncf-io\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.169204 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:09.167956 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 02:30:09.169204 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.167982 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54c26b32-0189-4a6a-a569-c2f859d87c8b-host\") pod \"node-ca-tvhb5\" (UID: \"54c26b32-0189-4a6a-a569-c2f859d87c8b\") " pod="openshift-image-registry/node-ca-tvhb5" Sep 09 02:30:09.169204 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:09.168040 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b974ece6-0b9d-4339-aa6f-203ce53dcd85-original-pull-secret podName:b974ece6-0b9d-4339-aa6f-203ce53dcd85 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:09.668023333 +0000 UTC m=+3.032357799 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b974ece6-0b9d-4339-aa6f-203ce53dcd85-original-pull-secret") pod "global-pull-secret-syncer-6lzgq" (UID: "b974ece6-0b9d-4339-aa6f-203ce53dcd85") : object "kube-system"/"original-pull-secret" not registered Sep 09 02:30:09.169204 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.168083 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dead4f18-8c41-41a5-9f37-897ddaee8563-konnectivity-ca\") pod \"konnectivity-agent-qw4bt\" (UID: \"dead4f18-8c41-41a5-9f37-897ddaee8563\") " pod="kube-system/konnectivity-agent-qw4bt" Sep 09 02:30:09.169204 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.168085 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-89.ec2.internal" event={"ID":"38b43ea1a0636b4e4e4c403eeb25802f","Type":"ContainerStarted","Data":"2d004e4c181c8b5aa0b6708f80cb4de66a81a792de20d586b7918a2bd15ec3aa"} Sep 09 02:30:09.169204 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.168113 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83-registration-dir\") pod \"aws-ebs-csi-driver-node-dfpfn\" (UID: \"cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfpfn" Sep 09 02:30:09.170123 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.168140 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-host-var-lib-cni-multus\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.170123 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.168172 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83-sys-fs\") pod \"aws-ebs-csi-driver-node-dfpfn\" (UID: \"cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfpfn" Sep 09 02:30:09.170123 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.168178 2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 09 02:30:09.170123 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.168218 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-host-var-lib-cni-multus\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.170123 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.168246 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83-sys-fs\") pod \"aws-ebs-csi-driver-node-dfpfn\" (UID: \"cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfpfn" Sep 09 02:30:09.170123 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.168256 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e3e028fc-3961-4b04-9875-fa06912e9518-etc-sysctl-d\") pod \"tuned-h4jdc\" (UID: \"e3e028fc-3961-4b04-9875-fa06912e9518\") " pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" Sep 09 02:30:09.170123 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.168289 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19afdc10-5a52-4e9d-a84f-04df6136a1c0-tmp-dir\") pod \"node-resolver-pd8dm\" (UID: \"19afdc10-5a52-4e9d-a84f-04df6136a1c0\") " pod="openshift-dns/node-resolver-pd8dm" Sep 09 02:30:09.170123 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.168317 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9ltf\" (UniqueName: \"kubernetes.io/projected/06180cab-e7c8-4fb8-a719-70072ee0ee36-kube-api-access-f9ltf\") pod \"network-metrics-daemon-mgjfl\" (UID: \"06180cab-e7c8-4fb8-a719-70072ee0ee36\") " pod="openshift-multus/network-metrics-daemon-mgjfl" Sep 09 02:30:09.170123 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.168349 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-system-cni-dir\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.170123 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.168360 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-ovnkube-config\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.170123 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.168380 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-etc-kubernetes\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.170123 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.168412 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-host-kubelet\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.170123 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.168440 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-host-slash\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.170123 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.168452 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-etc-kubernetes\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.170123 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.168460 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-system-cni-dir\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.170123 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.168469 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-ovn-node-metrics-cert\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.170123 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.168483 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-host-kubelet\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.170123 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.168508 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e3e028fc-3961-4b04-9875-fa06912e9518-etc-sysconfig\") pod \"tuned-h4jdc\" (UID: \"e3e028fc-3961-4b04-9875-fa06912e9518\") " pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" Sep 09 02:30:09.171074 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.168533 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e3e028fc-3961-4b04-9875-fa06912e9518-host\") pod \"tuned-h4jdc\" (UID: \"e3e028fc-3961-4b04-9875-fa06912e9518\") " pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" Sep 09 02:30:09.171074 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.168557 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxzzf\" (UniqueName: \"kubernetes.io/projected/a50c24dc-cc0a-404a-89d0-77e82c417bb5-kube-api-access-nxzzf\") pod \"network-check-target-65ldr\" (UID: \"a50c24dc-cc0a-404a-89d0-77e82c417bb5\") " pod="openshift-network-diagnostics/network-check-target-65ldr" Sep 09 02:30:09.171074 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.168578 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-multus-conf-dir\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.171074 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.168512 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-host-slash\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.171074 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.168601 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t486\" (UniqueName: \"kubernetes.io/projected/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-kube-api-access-5t486\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.171074 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.168627 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/52808c42-8ac8-49fd-96f8-e05e956b42be-iptables-alerter-script\") pod \"iptables-alerter-gktjg\" (UID: \"52808c42-8ac8-49fd-96f8-e05e956b42be\") " pod="openshift-network-operator/iptables-alerter-gktjg" Sep 09 02:30:09.171074 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.168635 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dead4f18-8c41-41a5-9f37-897ddaee8563-konnectivity-ca\") pod \"konnectivity-agent-qw4bt\" (UID: \"dead4f18-8c41-41a5-9f37-897ddaee8563\") " pod="kube-system/konnectivity-agent-qw4bt" Sep 09 02:30:09.171074 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.168650 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-run-ovn\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.171074 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.168650 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-multus-conf-dir\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.171074 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.168218 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83-registration-dir\") pod \"aws-ebs-csi-driver-node-dfpfn\" (UID: \"cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfpfn" Sep 09 02:30:09.171074 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.168685 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-node-log\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.171074 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.168706 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-node-log\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.171074 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.168708 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-run-ovn\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.171074 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.168724 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zjstd\" (UID: \"95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1\") " pod="openshift-multus/multus-additional-cni-plugins-zjstd" Sep 09 02:30:09.171074 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.168862 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83-socket-dir\") pod \"aws-ebs-csi-driver-node-dfpfn\" (UID: \"cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfpfn" Sep 09 02:30:09.171074 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.168919 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-multus-cni-dir\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.171074 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.168988 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83-socket-dir\") pod \"aws-ebs-csi-driver-node-dfpfn\" (UID: \"cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfpfn" Sep 09 02:30:09.171930 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.169000 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3e028fc-3961-4b04-9875-fa06912e9518-etc-kubernetes\") pod \"tuned-h4jdc\" (UID: \"e3e028fc-3961-4b04-9875-fa06912e9518\") " pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" Sep 09 02:30:09.171930 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.168989 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-multus-cni-dir\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.171930 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.169029 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e3e028fc-3961-4b04-9875-fa06912e9518-etc-sysctl-conf\") pod \"tuned-h4jdc\" (UID: \"e3e028fc-3961-4b04-9875-fa06912e9518\") " pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" Sep 09 02:30:09.171930 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.169052 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3e028fc-3961-4b04-9875-fa06912e9518-tmp\") pod \"tuned-h4jdc\" (UID: \"e3e028fc-3961-4b04-9875-fa06912e9518\") " pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" Sep 09 02:30:09.171930 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.169077 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zjstd\" (UID: \"95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1\") " pod="openshift-multus/multus-additional-cni-plugins-zjstd" Sep 09 02:30:09.171930 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.169126 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbql6\" (UniqueName: \"kubernetes.io/projected/52808c42-8ac8-49fd-96f8-e05e956b42be-kube-api-access-vbql6\") pod \"iptables-alerter-gktjg\" (UID: \"52808c42-8ac8-49fd-96f8-e05e956b42be\") " pod="openshift-network-operator/iptables-alerter-gktjg" Sep 09 02:30:09.171930 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.169150 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/19afdc10-5a52-4e9d-a84f-04df6136a1c0-hosts-file\") pod \"node-resolver-pd8dm\" (UID: \"19afdc10-5a52-4e9d-a84f-04df6136a1c0\") " pod="openshift-dns/node-resolver-pd8dm" Sep 09 02:30:09.171930 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.169174 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06180cab-e7c8-4fb8-a719-70072ee0ee36-metrics-certs\") pod \"network-metrics-daemon-mgjfl\" (UID: \"06180cab-e7c8-4fb8-a719-70072ee0ee36\") " pod="openshift-multus/network-metrics-daemon-mgjfl" Sep 09 02:30:09.171930 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.169200 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/52808c42-8ac8-49fd-96f8-e05e956b42be-host-slash\") pod \"iptables-alerter-gktjg\" (UID: \"52808c42-8ac8-49fd-96f8-e05e956b42be\") " pod="openshift-network-operator/iptables-alerter-gktjg" Sep 09 02:30:09.171930 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.169278 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/52808c42-8ac8-49fd-96f8-e05e956b42be-host-slash\") pod \"iptables-alerter-gktjg\" (UID: \"52808c42-8ac8-49fd-96f8-e05e956b42be\") " pod="openshift-network-operator/iptables-alerter-gktjg" Sep 09 02:30:09.171930 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.169373 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/52808c42-8ac8-49fd-96f8-e05e956b42be-iptables-alerter-script\") pod \"iptables-alerter-gktjg\" (UID: \"52808c42-8ac8-49fd-96f8-e05e956b42be\") " pod="openshift-network-operator/iptables-alerter-gktjg" Sep 09 02:30:09.173400 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.172738 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-ovn-node-metrics-cert\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.173400 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.172798 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dead4f18-8c41-41a5-9f37-897ddaee8563-agent-certs\") pod \"konnectivity-agent-qw4bt\" (UID: \"dead4f18-8c41-41a5-9f37-897ddaee8563\") " pod="kube-system/konnectivity-agent-qw4bt" Sep 09 02:30:09.179174 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.179098 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whrxm\" (UniqueName: \"kubernetes.io/projected/b14ff9af-8bbc-4169-8b5f-5d24df8c8968-kube-api-access-whrxm\") pod \"ovnkube-node-smnqw\" (UID: \"b14ff9af-8bbc-4169-8b5f-5d24df8c8968\") " pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.182084 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.182040 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lblsl\" (UniqueName: \"kubernetes.io/projected/cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83-kube-api-access-lblsl\") pod \"aws-ebs-csi-driver-node-dfpfn\" (UID: \"cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfpfn" Sep 09 02:30:09.185398 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.185256 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbql6\" (UniqueName: \"kubernetes.io/projected/52808c42-8ac8-49fd-96f8-e05e956b42be-kube-api-access-vbql6\") pod \"iptables-alerter-gktjg\" (UID: \"52808c42-8ac8-49fd-96f8-e05e956b42be\") " pod="openshift-network-operator/iptables-alerter-gktjg" Sep 09 02:30:09.185593 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.185568 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t486\" (UniqueName: \"kubernetes.io/projected/b7a1ff67-afbc-4e76-aa24-d1a8cebb3888-kube-api-access-5t486\") pod \"multus-dj8dq\" (UID: \"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888\") " pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.270124 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.270095 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54c26b32-0189-4a6a-a569-c2f859d87c8b-host\") pod \"node-ca-tvhb5\" (UID: \"54c26b32-0189-4a6a-a569-c2f859d87c8b\") " pod="openshift-image-registry/node-ca-tvhb5" Sep 09 02:30:09.270124 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.270130 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e3e028fc-3961-4b04-9875-fa06912e9518-etc-sysctl-d\") pod \"tuned-h4jdc\" (UID: \"e3e028fc-3961-4b04-9875-fa06912e9518\") " pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" Sep 09 02:30:09.270356 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.270146 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19afdc10-5a52-4e9d-a84f-04df6136a1c0-tmp-dir\") pod \"node-resolver-pd8dm\" (UID: \"19afdc10-5a52-4e9d-a84f-04df6136a1c0\") " pod="openshift-dns/node-resolver-pd8dm" Sep 09 02:30:09.270356 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.270162 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9ltf\" (UniqueName: \"kubernetes.io/projected/06180cab-e7c8-4fb8-a719-70072ee0ee36-kube-api-access-f9ltf\") pod \"network-metrics-daemon-mgjfl\" (UID: \"06180cab-e7c8-4fb8-a719-70072ee0ee36\") " pod="openshift-multus/network-metrics-daemon-mgjfl" Sep 09 02:30:09.270356 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.270187 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e3e028fc-3961-4b04-9875-fa06912e9518-etc-sysconfig\") pod \"tuned-h4jdc\" (UID: \"e3e028fc-3961-4b04-9875-fa06912e9518\") " pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" Sep 09 02:30:09.270356 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.270203 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e3e028fc-3961-4b04-9875-fa06912e9518-host\") pod \"tuned-h4jdc\" (UID: \"e3e028fc-3961-4b04-9875-fa06912e9518\") " pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" Sep 09 02:30:09.270356 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.270229 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxzzf\" (UniqueName: \"kubernetes.io/projected/a50c24dc-cc0a-404a-89d0-77e82c417bb5-kube-api-access-nxzzf\") pod \"network-check-target-65ldr\" (UID: \"a50c24dc-cc0a-404a-89d0-77e82c417bb5\") " pod="openshift-network-diagnostics/network-check-target-65ldr" Sep 09 02:30:09.270356 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.270231 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54c26b32-0189-4a6a-a569-c2f859d87c8b-host\") pod \"node-ca-tvhb5\" (UID: \"54c26b32-0189-4a6a-a569-c2f859d87c8b\") " pod="openshift-image-registry/node-ca-tvhb5" Sep 09 02:30:09.270356 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.270248 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zjstd\" (UID: \"95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1\") " pod="openshift-multus/multus-additional-cni-plugins-zjstd" Sep 09 02:30:09.270356 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.270304 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e3e028fc-3961-4b04-9875-fa06912e9518-etc-sysconfig\") pod \"tuned-h4jdc\" (UID: \"e3e028fc-3961-4b04-9875-fa06912e9518\") " pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" Sep 09 02:30:09.270356 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.270308 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e3e028fc-3961-4b04-9875-fa06912e9518-etc-sysctl-d\") pod \"tuned-h4jdc\" (UID: \"e3e028fc-3961-4b04-9875-fa06912e9518\") " pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" Sep 09 02:30:09.270356 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.270336 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e3e028fc-3961-4b04-9875-fa06912e9518-host\") pod \"tuned-h4jdc\" (UID: \"e3e028fc-3961-4b04-9875-fa06912e9518\") " pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" Sep 09 02:30:09.270924 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.270473 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zjstd\" (UID: \"95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1\") " pod="openshift-multus/multus-additional-cni-plugins-zjstd" Sep 09 02:30:09.270924 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.270513 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3e028fc-3961-4b04-9875-fa06912e9518-etc-kubernetes\") pod \"tuned-h4jdc\" (UID: \"e3e028fc-3961-4b04-9875-fa06912e9518\") " pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" Sep 09 02:30:09.270924 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.270543 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e3e028fc-3961-4b04-9875-fa06912e9518-etc-sysctl-conf\") pod \"tuned-h4jdc\" (UID: \"e3e028fc-3961-4b04-9875-fa06912e9518\") " pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" Sep 09 02:30:09.270924 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.270558 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3e028fc-3961-4b04-9875-fa06912e9518-tmp\") pod \"tuned-h4jdc\" (UID: \"e3e028fc-3961-4b04-9875-fa06912e9518\") " pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" Sep 09 02:30:09.270924 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.270564 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19afdc10-5a52-4e9d-a84f-04df6136a1c0-tmp-dir\") pod \"node-resolver-pd8dm\" (UID: \"19afdc10-5a52-4e9d-a84f-04df6136a1c0\") " pod="openshift-dns/node-resolver-pd8dm" Sep 09 02:30:09.270924 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.270573 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zjstd\" (UID: \"95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1\") " pod="openshift-multus/multus-additional-cni-plugins-zjstd" Sep 09 02:30:09.270924 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.270591 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/19afdc10-5a52-4e9d-a84f-04df6136a1c0-hosts-file\") pod \"node-resolver-pd8dm\" (UID: \"19afdc10-5a52-4e9d-a84f-04df6136a1c0\") " pod="openshift-dns/node-resolver-pd8dm" Sep 09 02:30:09.270924 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.270620 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06180cab-e7c8-4fb8-a719-70072ee0ee36-metrics-certs\") pod \"network-metrics-daemon-mgjfl\" (UID: \"06180cab-e7c8-4fb8-a719-70072ee0ee36\") " pod="openshift-multus/network-metrics-daemon-mgjfl" Sep 09 02:30:09.270924 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.270648 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1-cnibin\") pod \"multus-additional-cni-plugins-zjstd\" (UID: \"95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1\") " pod="openshift-multus/multus-additional-cni-plugins-zjstd" Sep 09 02:30:09.270924 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.270674 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e3e028fc-3961-4b04-9875-fa06912e9518-etc-modprobe-d\") pod \"tuned-h4jdc\" (UID: \"e3e028fc-3961-4b04-9875-fa06912e9518\") " pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" Sep 09 02:30:09.270924 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.270697 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84fp8\" (UniqueName: \"kubernetes.io/projected/e3e028fc-3961-4b04-9875-fa06912e9518-kube-api-access-84fp8\") pod \"tuned-h4jdc\" (UID: \"e3e028fc-3961-4b04-9875-fa06912e9518\") " pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" Sep 09 02:30:09.270924 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.270721 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/54c26b32-0189-4a6a-a569-c2f859d87c8b-serviceca\") pod \"node-ca-tvhb5\" (UID: \"54c26b32-0189-4a6a-a569-c2f859d87c8b\") " pod="openshift-image-registry/node-ca-tvhb5" Sep 09 02:30:09.270924 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.270753 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zptrg\" (UniqueName: \"kubernetes.io/projected/54c26b32-0189-4a6a-a569-c2f859d87c8b-kube-api-access-zptrg\") pod \"node-ca-tvhb5\" (UID: \"54c26b32-0189-4a6a-a569-c2f859d87c8b\") " pod="openshift-image-registry/node-ca-tvhb5" Sep 09 02:30:09.270924 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:09.270777 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 02:30:09.270924 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:09.270842 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06180cab-e7c8-4fb8-a719-70072ee0ee36-metrics-certs podName:06180cab-e7c8-4fb8-a719-70072ee0ee36 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:09.770821146 +0000 UTC m=+3.135155588 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06180cab-e7c8-4fb8-a719-70072ee0ee36-metrics-certs") pod "network-metrics-daemon-mgjfl" (UID: "06180cab-e7c8-4fb8-a719-70072ee0ee36") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 02:30:09.270924 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.270878 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e3e028fc-3961-4b04-9875-fa06912e9518-lib-modules\") pod \"tuned-h4jdc\" (UID: \"e3e028fc-3961-4b04-9875-fa06912e9518\") " pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" Sep 09 02:30:09.271806 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.270996 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e3e028fc-3961-4b04-9875-fa06912e9518-etc-modprobe-d\") pod \"tuned-h4jdc\" (UID: \"e3e028fc-3961-4b04-9875-fa06912e9518\") " pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" Sep 09 02:30:09.271806 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.271061 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3e028fc-3961-4b04-9875-fa06912e9518-etc-kubernetes\") pod \"tuned-h4jdc\" (UID: \"e3e028fc-3961-4b04-9875-fa06912e9518\") " pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" Sep 09 02:30:09.271806 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.271252 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1-cnibin\") pod \"multus-additional-cni-plugins-zjstd\" (UID: \"95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1\") " pod="openshift-multus/multus-additional-cni-plugins-zjstd" Sep 09 02:30:09.271806 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.270778 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e3e028fc-3961-4b04-9875-fa06912e9518-lib-modules\") pod \"tuned-h4jdc\" (UID: \"e3e028fc-3961-4b04-9875-fa06912e9518\") " pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" Sep 09 02:30:09.271806 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.271382 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1-cni-binary-copy\") pod \"multus-additional-cni-plugins-zjstd\" (UID: \"95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1\") " pod="openshift-multus/multus-additional-cni-plugins-zjstd" Sep 09 02:30:09.271806 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.271393 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e3e028fc-3961-4b04-9875-fa06912e9518-etc-sysctl-conf\") pod \"tuned-h4jdc\" (UID: \"e3e028fc-3961-4b04-9875-fa06912e9518\") " pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" Sep 09 02:30:09.271806 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.271424 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcg5c\" (UniqueName: \"kubernetes.io/projected/95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1-kube-api-access-hcg5c\") pod \"multus-additional-cni-plugins-zjstd\" (UID: \"95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1\") " pod="openshift-multus/multus-additional-cni-plugins-zjstd" Sep 09 02:30:09.271806 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.271462 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e3e028fc-3961-4b04-9875-fa06912e9518-etc-systemd\") pod \"tuned-h4jdc\" (UID: \"e3e028fc-3961-4b04-9875-fa06912e9518\") " pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" Sep 09 02:30:09.271806 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.271517 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e3e028fc-3961-4b04-9875-fa06912e9518-etc-systemd\") pod \"tuned-h4jdc\" (UID: \"e3e028fc-3961-4b04-9875-fa06912e9518\") " pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" Sep 09 02:30:09.271806 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.271524 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/54c26b32-0189-4a6a-a569-c2f859d87c8b-serviceca\") pod \"node-ca-tvhb5\" (UID: \"54c26b32-0189-4a6a-a569-c2f859d87c8b\") " pod="openshift-image-registry/node-ca-tvhb5" Sep 09 02:30:09.271806 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.271648 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/19afdc10-5a52-4e9d-a84f-04df6136a1c0-hosts-file\") pod \"node-resolver-pd8dm\" (UID: \"19afdc10-5a52-4e9d-a84f-04df6136a1c0\") " pod="openshift-dns/node-resolver-pd8dm" Sep 09 02:30:09.271806 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.271750 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e3e028fc-3961-4b04-9875-fa06912e9518-run\") pod \"tuned-h4jdc\" (UID: \"e3e028fc-3961-4b04-9875-fa06912e9518\") " pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" Sep 09 02:30:09.271806 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.271776 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e3e028fc-3961-4b04-9875-fa06912e9518-etc-tuned\") pod \"tuned-h4jdc\" (UID: \"e3e028fc-3961-4b04-9875-fa06912e9518\") " pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" Sep 09 02:30:09.271806 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.271801 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fdsw\" (UniqueName: \"kubernetes.io/projected/19afdc10-5a52-4e9d-a84f-04df6136a1c0-kube-api-access-4fdsw\") pod \"node-resolver-pd8dm\" (UID: \"19afdc10-5a52-4e9d-a84f-04df6136a1c0\") " pod="openshift-dns/node-resolver-pd8dm" Sep 09 02:30:09.272358 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.271829 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1-system-cni-dir\") pod \"multus-additional-cni-plugins-zjstd\" (UID: \"95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1\") " pod="openshift-multus/multus-additional-cni-plugins-zjstd" Sep 09 02:30:09.272358 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.271854 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zjstd\" (UID: \"95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1\") " pod="openshift-multus/multus-additional-cni-plugins-zjstd" Sep 09 02:30:09.272358 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.271855 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zjstd\" (UID: \"95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1\") " pod="openshift-multus/multus-additional-cni-plugins-zjstd" Sep 09 02:30:09.272358 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.271884 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3e028fc-3961-4b04-9875-fa06912e9518-var-lib-kubelet\") pod \"tuned-h4jdc\" (UID: \"e3e028fc-3961-4b04-9875-fa06912e9518\") " pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" Sep 09 02:30:09.272358 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.271909 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1-cni-binary-copy\") pod \"multus-additional-cni-plugins-zjstd\" (UID: \"95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1\") " pod="openshift-multus/multus-additional-cni-plugins-zjstd" Sep 09 02:30:09.272358 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.271922 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3e028fc-3961-4b04-9875-fa06912e9518-sys\") pod \"tuned-h4jdc\" (UID: \"e3e028fc-3961-4b04-9875-fa06912e9518\") " pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" Sep 09 02:30:09.272358 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.272024 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e3e028fc-3961-4b04-9875-fa06912e9518-run\") pod \"tuned-h4jdc\" (UID: \"e3e028fc-3961-4b04-9875-fa06912e9518\") " pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" Sep 09 02:30:09.272358 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.272069 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1-system-cni-dir\") pod \"multus-additional-cni-plugins-zjstd\" (UID: \"95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1\") " pod="openshift-multus/multus-additional-cni-plugins-zjstd" Sep 09 02:30:09.272358 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.272247 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3e028fc-3961-4b04-9875-fa06912e9518-sys\") pod \"tuned-h4jdc\" (UID: \"e3e028fc-3961-4b04-9875-fa06912e9518\") " pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" Sep 09 02:30:09.272358 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.272300 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3e028fc-3961-4b04-9875-fa06912e9518-var-lib-kubelet\") pod \"tuned-h4jdc\" (UID: \"e3e028fc-3961-4b04-9875-fa06912e9518\") " pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" Sep 09 02:30:09.273118 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.272529 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zjstd\" (UID: \"95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1\") " pod="openshift-multus/multus-additional-cni-plugins-zjstd" Sep 09 02:30:09.273254 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.273226 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3e028fc-3961-4b04-9875-fa06912e9518-tmp\") pod \"tuned-h4jdc\" (UID: \"e3e028fc-3961-4b04-9875-fa06912e9518\") " pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" Sep 09 02:30:09.274209 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.274187 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e3e028fc-3961-4b04-9875-fa06912e9518-etc-tuned\") pod \"tuned-h4jdc\" (UID: \"e3e028fc-3961-4b04-9875-fa06912e9518\") " pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" Sep 09 02:30:09.278780 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:09.278746 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 02:30:09.278780 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:09.278778 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 02:30:09.278950 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:09.278793 2577 projected.go:196] Error preparing data for projected volume kube-api-access-nxzzf for pod openshift-network-diagnostics/network-check-target-65ldr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 02:30:09.278950 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:09.278864 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a50c24dc-cc0a-404a-89d0-77e82c417bb5-kube-api-access-nxzzf podName:a50c24dc-cc0a-404a-89d0-77e82c417bb5 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:09.77884749 +0000 UTC m=+3.143181949 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nxzzf" (UniqueName: "kubernetes.io/projected/a50c24dc-cc0a-404a-89d0-77e82c417bb5-kube-api-access-nxzzf") pod "network-check-target-65ldr" (UID: "a50c24dc-cc0a-404a-89d0-77e82c417bb5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 02:30:09.281787 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.281689 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcg5c\" (UniqueName: \"kubernetes.io/projected/95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1-kube-api-access-hcg5c\") pod \"multus-additional-cni-plugins-zjstd\" (UID: \"95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1\") " pod="openshift-multus/multus-additional-cni-plugins-zjstd" Sep 09 02:30:09.283029 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.282960 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zptrg\" (UniqueName: \"kubernetes.io/projected/54c26b32-0189-4a6a-a569-c2f859d87c8b-kube-api-access-zptrg\") pod \"node-ca-tvhb5\" (UID: \"54c26b32-0189-4a6a-a569-c2f859d87c8b\") " pod="openshift-image-registry/node-ca-tvhb5" Sep 09 02:30:09.283029 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.283019 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fdsw\" (UniqueName: \"kubernetes.io/projected/19afdc10-5a52-4e9d-a84f-04df6136a1c0-kube-api-access-4fdsw\") pod \"node-resolver-pd8dm\" (UID: \"19afdc10-5a52-4e9d-a84f-04df6136a1c0\") " pod="openshift-dns/node-resolver-pd8dm" Sep 09 02:30:09.283702 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.283678 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84fp8\" (UniqueName: \"kubernetes.io/projected/e3e028fc-3961-4b04-9875-fa06912e9518-kube-api-access-84fp8\") pod \"tuned-h4jdc\" (UID: \"e3e028fc-3961-4b04-9875-fa06912e9518\") " pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" Sep 09 02:30:09.283886 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.283862 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9ltf\" (UniqueName: \"kubernetes.io/projected/06180cab-e7c8-4fb8-a719-70072ee0ee36-kube-api-access-f9ltf\") pod \"network-metrics-daemon-mgjfl\" (UID: \"06180cab-e7c8-4fb8-a719-70072ee0ee36\") " pod="openshift-multus/network-metrics-daemon-mgjfl" Sep 09 02:30:09.353586 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.353548 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfpfn" Sep 09 02:30:09.365872 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.365851 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dj8dq" Sep 09 02:30:09.378018 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.377992 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gktjg" Sep 09 02:30:09.385911 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.385887 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:09.393755 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.393732 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qw4bt" Sep 09 02:30:09.402534 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.402512 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pd8dm" Sep 09 02:30:09.411354 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.411336 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" Sep 09 02:30:09.419134 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.419106 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zjstd" Sep 09 02:30:09.426767 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.426751 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tvhb5" Sep 09 02:30:09.505340 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.505311 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 02:30:09.675061 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.674990 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b974ece6-0b9d-4339-aa6f-203ce53dcd85-original-pull-secret\") pod \"global-pull-secret-syncer-6lzgq\" (UID: \"b974ece6-0b9d-4339-aa6f-203ce53dcd85\") " pod="kube-system/global-pull-secret-syncer-6lzgq" Sep 09 02:30:09.675187 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:09.675108 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 02:30:09.675187 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:09.675179 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b974ece6-0b9d-4339-aa6f-203ce53dcd85-original-pull-secret podName:b974ece6-0b9d-4339-aa6f-203ce53dcd85 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:10.67516251 +0000 UTC m=+4.039496951 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b974ece6-0b9d-4339-aa6f-203ce53dcd85-original-pull-secret") pod "global-pull-secret-syncer-6lzgq" (UID: "b974ece6-0b9d-4339-aa6f-203ce53dcd85") : object "kube-system"/"original-pull-secret" not registered Sep 09 02:30:09.776087 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.776050 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06180cab-e7c8-4fb8-a719-70072ee0ee36-metrics-certs\") pod \"network-metrics-daemon-mgjfl\" (UID: \"06180cab-e7c8-4fb8-a719-70072ee0ee36\") " pod="openshift-multus/network-metrics-daemon-mgjfl" Sep 09 02:30:09.776258 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:09.776237 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 02:30:09.776324 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:09.776312 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06180cab-e7c8-4fb8-a719-70072ee0ee36-metrics-certs podName:06180cab-e7c8-4fb8-a719-70072ee0ee36 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:10.776297623 +0000 UTC m=+4.140632065 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06180cab-e7c8-4fb8-a719-70072ee0ee36-metrics-certs") pod "network-metrics-daemon-mgjfl" (UID: "06180cab-e7c8-4fb8-a719-70072ee0ee36") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 02:30:09.876715 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:09.876676 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxzzf\" (UniqueName: \"kubernetes.io/projected/a50c24dc-cc0a-404a-89d0-77e82c417bb5-kube-api-access-nxzzf\") pod \"network-check-target-65ldr\" (UID: \"a50c24dc-cc0a-404a-89d0-77e82c417bb5\") " pod="openshift-network-diagnostics/network-check-target-65ldr" Sep 09 02:30:09.876910 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:09.876824 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 02:30:09.876910 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:09.876842 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 02:30:09.876910 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:09.876857 2577 projected.go:196] Error preparing data for projected volume kube-api-access-nxzzf for pod openshift-network-diagnostics/network-check-target-65ldr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 02:30:09.877108 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:09.876913 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a50c24dc-cc0a-404a-89d0-77e82c417bb5-kube-api-access-nxzzf podName:a50c24dc-cc0a-404a-89d0-77e82c417bb5 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:10.876899901 +0000 UTC m=+4.241234342 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxzzf" (UniqueName: "kubernetes.io/projected/a50c24dc-cc0a-404a-89d0-77e82c417bb5-kube-api-access-nxzzf") pod "network-check-target-65ldr" (UID: "a50c24dc-cc0a-404a-89d0-77e82c417bb5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 02:30:10.113191 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:10.113102 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 02:25:09 +0000 UTC" deadline="2028-02-14 06:33:01.186266607 +0000 UTC" Sep 09 02:30:10.113191 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:10.113138 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12556h2m51.073131437s" Sep 09 02:30:10.381225 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:10.381202 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7a1ff67_afbc_4e76_aa24_d1a8cebb3888.slice/crio-20862fd088210d3a4dc68690b878447b1fa4430a9dbd451990e572ab08748042 WatchSource:0}: Error finding container 20862fd088210d3a4dc68690b878447b1fa4430a9dbd451990e572ab08748042: Status 404 returned error can't find the container with id 20862fd088210d3a4dc68690b878447b1fa4430a9dbd451990e572ab08748042 Sep 09 02:30:10.382539 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:10.382517 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19afdc10_5a52_4e9d_a84f_04df6136a1c0.slice/crio-982fb9d681c9ddcee06d6b6fa9edb71b311de51302cac6add2caea3acf5be2dc WatchSource:0}: Error finding container 982fb9d681c9ddcee06d6b6fa9edb71b311de51302cac6add2caea3acf5be2dc: Status 404 returned error can't find the container with id 982fb9d681c9ddcee06d6b6fa9edb71b311de51302cac6add2caea3acf5be2dc Sep 09 02:30:10.383252 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:10.383077 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52808c42_8ac8_49fd_96f8_e05e956b42be.slice/crio-70a5e38a00d6ede815b6da946cb376129f2e765955b07418280aab9c90fa40f4 WatchSource:0}: Error finding container 70a5e38a00d6ede815b6da946cb376129f2e765955b07418280aab9c90fa40f4: Status 404 returned error can't find the container with id 70a5e38a00d6ede815b6da946cb376129f2e765955b07418280aab9c90fa40f4 Sep 09 02:30:10.386373 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:10.386293 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95b9fbf9_fb9c_44c5_bdde_9306b4be5eb1.slice/crio-aa90d51149ca142bbaf61ee5420f033e707c501e9d2e885358ab08e9e2b0b72e WatchSource:0}: Error finding container aa90d51149ca142bbaf61ee5420f033e707c501e9d2e885358ab08e9e2b0b72e: Status 404 returned error can't find the container with id aa90d51149ca142bbaf61ee5420f033e707c501e9d2e885358ab08e9e2b0b72e Sep 09 02:30:10.387306 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:10.387283 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb14ff9af_8bbc_4169_8b5f_5d24df8c8968.slice/crio-6bfd9838481e0e57f33163f2cccf43555fd7ca45981b9436586398ed12f48446 WatchSource:0}: Error finding container 6bfd9838481e0e57f33163f2cccf43555fd7ca45981b9436586398ed12f48446: Status 404 returned error can't find the container with id 6bfd9838481e0e57f33163f2cccf43555fd7ca45981b9436586398ed12f48446 Sep 09 02:30:10.388385 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:10.388315 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3e028fc_3961_4b04_9875_fa06912e9518.slice/crio-cf710a3a8e065b9eebc670a53d0d350b6b4aff289c04b5c16274501f1f3799b9 WatchSource:0}: Error finding container cf710a3a8e065b9eebc670a53d0d350b6b4aff289c04b5c16274501f1f3799b9: Status 404 returned error can't find the container with id cf710a3a8e065b9eebc670a53d0d350b6b4aff289c04b5c16274501f1f3799b9 Sep 09 02:30:10.389369 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:10.389332 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54c26b32_0189_4a6a_a569_c2f859d87c8b.slice/crio-993879832ae91313d500b021322cf333cd36e5946060c82c03b79b66a108c73b WatchSource:0}: Error finding container 993879832ae91313d500b021322cf333cd36e5946060c82c03b79b66a108c73b: Status 404 returned error can't find the container with id 993879832ae91313d500b021322cf333cd36e5946060c82c03b79b66a108c73b Sep 09 02:30:10.390826 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:10.390802 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcff5b67e_1d8e_4f8a_b9ef_6e7f3e096d83.slice/crio-3460dcebec82da0e7cc1c37fcd3cb7827d19448cacf07d4a23161a6f888df1f5 WatchSource:0}: Error finding container 3460dcebec82da0e7cc1c37fcd3cb7827d19448cacf07d4a23161a6f888df1f5: Status 404 returned error can't find the container with id 3460dcebec82da0e7cc1c37fcd3cb7827d19448cacf07d4a23161a6f888df1f5 Sep 09 02:30:10.392594 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:10.392325 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddead4f18_8c41_41a5_9f37_897ddaee8563.slice/crio-f26fb646f7c379498341c9fc1a797ce1c50a4a0411ff1aa20d0f89507910a48d WatchSource:0}: Error finding container f26fb646f7c379498341c9fc1a797ce1c50a4a0411ff1aa20d0f89507910a48d: Status 404 returned error can't find the container with id f26fb646f7c379498341c9fc1a797ce1c50a4a0411ff1aa20d0f89507910a48d Sep 09 02:30:10.682283 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:10.682259 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b974ece6-0b9d-4339-aa6f-203ce53dcd85-original-pull-secret\") pod \"global-pull-secret-syncer-6lzgq\" (UID: \"b974ece6-0b9d-4339-aa6f-203ce53dcd85\") " pod="kube-system/global-pull-secret-syncer-6lzgq" Sep 09 02:30:10.682391 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:10.682378 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 02:30:10.682442 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:10.682433 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b974ece6-0b9d-4339-aa6f-203ce53dcd85-original-pull-secret podName:b974ece6-0b9d-4339-aa6f-203ce53dcd85 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:12.682418193 +0000 UTC m=+6.046752655 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b974ece6-0b9d-4339-aa6f-203ce53dcd85-original-pull-secret") pod "global-pull-secret-syncer-6lzgq" (UID: "b974ece6-0b9d-4339-aa6f-203ce53dcd85") : object "kube-system"/"original-pull-secret" not registered Sep 09 02:30:10.783375 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:10.783344 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06180cab-e7c8-4fb8-a719-70072ee0ee36-metrics-certs\") pod \"network-metrics-daemon-mgjfl\" (UID: \"06180cab-e7c8-4fb8-a719-70072ee0ee36\") " pod="openshift-multus/network-metrics-daemon-mgjfl" Sep 09 02:30:10.783519 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:10.783468 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 02:30:10.783559 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:10.783522 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06180cab-e7c8-4fb8-a719-70072ee0ee36-metrics-certs podName:06180cab-e7c8-4fb8-a719-70072ee0ee36 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:12.783507587 +0000 UTC m=+6.147842029 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06180cab-e7c8-4fb8-a719-70072ee0ee36-metrics-certs") pod "network-metrics-daemon-mgjfl" (UID: "06180cab-e7c8-4fb8-a719-70072ee0ee36") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 02:30:10.885946 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:10.884207 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxzzf\" (UniqueName: \"kubernetes.io/projected/a50c24dc-cc0a-404a-89d0-77e82c417bb5-kube-api-access-nxzzf\") pod \"network-check-target-65ldr\" (UID: \"a50c24dc-cc0a-404a-89d0-77e82c417bb5\") " pod="openshift-network-diagnostics/network-check-target-65ldr" Sep 09 02:30:10.885946 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:10.884367 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 02:30:10.885946 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:10.884396 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 02:30:10.885946 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:10.884409 2577 projected.go:196] Error preparing data for projected volume kube-api-access-nxzzf for pod openshift-network-diagnostics/network-check-target-65ldr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 02:30:10.885946 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:10.884473 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a50c24dc-cc0a-404a-89d0-77e82c417bb5-kube-api-access-nxzzf podName:a50c24dc-cc0a-404a-89d0-77e82c417bb5 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:12.884454719 +0000 UTC m=+6.248789176 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxzzf" (UniqueName: "kubernetes.io/projected/a50c24dc-cc0a-404a-89d0-77e82c417bb5-kube-api-access-nxzzf") pod "network-check-target-65ldr" (UID: "a50c24dc-cc0a-404a-89d0-77e82c417bb5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 02:30:11.114197 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:11.113998 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 02:25:09 +0000 UTC" deadline="2028-03-20 05:35:21.299010171 +0000 UTC" Sep 09 02:30:11.114197 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:11.114059 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13395h5m10.184955934s" Sep 09 02:30:11.160119 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:11.160070 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6lzgq" Sep 09 02:30:11.160265 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:11.160222 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6lzgq" podUID="b974ece6-0b9d-4339-aa6f-203ce53dcd85" Sep 09 02:30:11.160671 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:11.160613 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65ldr" Sep 09 02:30:11.160768 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:11.160710 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-65ldr" podUID="a50c24dc-cc0a-404a-89d0-77e82c417bb5" Sep 09 02:30:11.162822 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:11.162802 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjfl" Sep 09 02:30:11.162922 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:11.162901 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgjfl" podUID="06180cab-e7c8-4fb8-a719-70072ee0ee36" Sep 09 02:30:11.196046 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:11.196010 2577 generic.go:356] "Generic (PLEG): container finished" podID="81bd3a799875c0edff0ebc792a1d03d1" containerID="1a328816c4f15f13fe38210436cd401a569bbb26064c799d1f039b1f630e10f1" exitCode=0 Sep 09 02:30:11.196199 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:11.196108 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-89.ec2.internal" event={"ID":"81bd3a799875c0edff0ebc792a1d03d1","Type":"ContainerDied","Data":"1a328816c4f15f13fe38210436cd401a569bbb26064c799d1f039b1f630e10f1"} Sep 09 02:30:11.211764 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:11.211732 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfpfn" event={"ID":"cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83","Type":"ContainerStarted","Data":"3460dcebec82da0e7cc1c37fcd3cb7827d19448cacf07d4a23161a6f888df1f5"} Sep 09 02:30:11.222531 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:11.222440 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tvhb5" event={"ID":"54c26b32-0189-4a6a-a569-c2f859d87c8b","Type":"ContainerStarted","Data":"993879832ae91313d500b021322cf333cd36e5946060c82c03b79b66a108c73b"} Sep 09 02:30:11.231162 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:11.231135 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" event={"ID":"b14ff9af-8bbc-4169-8b5f-5d24df8c8968","Type":"ContainerStarted","Data":"6bfd9838481e0e57f33163f2cccf43555fd7ca45981b9436586398ed12f48446"} Sep 09 02:30:11.234309 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:11.234192 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjstd" event={"ID":"95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1","Type":"ContainerStarted","Data":"aa90d51149ca142bbaf61ee5420f033e707c501e9d2e885358ab08e9e2b0b72e"} Sep 09 02:30:11.237369 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:11.237304 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gktjg" event={"ID":"52808c42-8ac8-49fd-96f8-e05e956b42be","Type":"ContainerStarted","Data":"70a5e38a00d6ede815b6da946cb376129f2e765955b07418280aab9c90fa40f4"} Sep 09 02:30:11.239890 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:11.239816 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pd8dm" event={"ID":"19afdc10-5a52-4e9d-a84f-04df6136a1c0","Type":"ContainerStarted","Data":"982fb9d681c9ddcee06d6b6fa9edb71b311de51302cac6add2caea3acf5be2dc"} Sep 09 02:30:11.246337 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:11.246313 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dj8dq" event={"ID":"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888","Type":"ContainerStarted","Data":"20862fd088210d3a4dc68690b878447b1fa4430a9dbd451990e572ab08748042"} Sep 09 02:30:11.255789 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:11.255761 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-89.ec2.internal" event={"ID":"38b43ea1a0636b4e4e4c403eeb25802f","Type":"ContainerStarted","Data":"2036da81466757e5130f6f36a725ad65eb084dbaf5e0d96ca66c1d164d885ed8"} Sep 09 02:30:11.260735 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:11.260710 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qw4bt" event={"ID":"dead4f18-8c41-41a5-9f37-897ddaee8563","Type":"ContainerStarted","Data":"f26fb646f7c379498341c9fc1a797ce1c50a4a0411ff1aa20d0f89507910a48d"} Sep 09 02:30:11.274630 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:11.274441 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" event={"ID":"e3e028fc-3961-4b04-9875-fa06912e9518","Type":"ContainerStarted","Data":"cf710a3a8e065b9eebc670a53d0d350b6b4aff289c04b5c16274501f1f3799b9"} Sep 09 02:30:12.297319 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:12.297281 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-89.ec2.internal" event={"ID":"81bd3a799875c0edff0ebc792a1d03d1","Type":"ContainerStarted","Data":"f26e82509af9e7d99d38738b306e91198aa7ac76618fabf508e9aca88cccfb01"} Sep 09 02:30:12.317720 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:12.317659 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-89.ec2.internal" podStartSLOduration=4.317638731 podStartE2EDuration="4.317638731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 02:30:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:30:12.316865916 +0000 UTC m=+5.681200392" watchObservedRunningTime="2026-09-09 02:30:12.317638731 +0000 UTC m=+5.681973196" Sep 09 02:30:12.318163 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:12.318080 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-89.ec2.internal" podStartSLOduration=4.318058936 podStartE2EDuration="4.318058936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 02:30:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:30:11.273833464 +0000 UTC m=+4.638167929" watchObservedRunningTime="2026-09-09 02:30:12.318058936 +0000 UTC m=+5.682393400" Sep 09 02:30:12.701229 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:12.700605 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b974ece6-0b9d-4339-aa6f-203ce53dcd85-original-pull-secret\") pod \"global-pull-secret-syncer-6lzgq\" (UID: \"b974ece6-0b9d-4339-aa6f-203ce53dcd85\") " pod="kube-system/global-pull-secret-syncer-6lzgq" Sep 09 02:30:12.701229 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:12.700742 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 02:30:12.701229 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:12.700805 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b974ece6-0b9d-4339-aa6f-203ce53dcd85-original-pull-secret podName:b974ece6-0b9d-4339-aa6f-203ce53dcd85 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:16.700787521 +0000 UTC m=+10.065121983 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b974ece6-0b9d-4339-aa6f-203ce53dcd85-original-pull-secret") pod "global-pull-secret-syncer-6lzgq" (UID: "b974ece6-0b9d-4339-aa6f-203ce53dcd85") : object "kube-system"/"original-pull-secret" not registered Sep 09 02:30:12.801452 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:12.801212 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06180cab-e7c8-4fb8-a719-70072ee0ee36-metrics-certs\") pod \"network-metrics-daemon-mgjfl\" (UID: \"06180cab-e7c8-4fb8-a719-70072ee0ee36\") " pod="openshift-multus/network-metrics-daemon-mgjfl" Sep 09 02:30:12.801608 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:12.801395 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 02:30:12.801608 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:12.801532 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06180cab-e7c8-4fb8-a719-70072ee0ee36-metrics-certs podName:06180cab-e7c8-4fb8-a719-70072ee0ee36 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:16.801511691 +0000 UTC m=+10.165846135 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06180cab-e7c8-4fb8-a719-70072ee0ee36-metrics-certs") pod "network-metrics-daemon-mgjfl" (UID: "06180cab-e7c8-4fb8-a719-70072ee0ee36") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 02:30:12.902544 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:12.902505 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxzzf\" (UniqueName: \"kubernetes.io/projected/a50c24dc-cc0a-404a-89d0-77e82c417bb5-kube-api-access-nxzzf\") pod \"network-check-target-65ldr\" (UID: \"a50c24dc-cc0a-404a-89d0-77e82c417bb5\") " pod="openshift-network-diagnostics/network-check-target-65ldr" Sep 09 02:30:12.902748 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:12.902716 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 02:30:12.902748 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:12.902739 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 02:30:12.902871 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:12.902751 2577 projected.go:196] Error preparing data for projected volume kube-api-access-nxzzf for pod openshift-network-diagnostics/network-check-target-65ldr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 02:30:12.902871 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:12.902807 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a50c24dc-cc0a-404a-89d0-77e82c417bb5-kube-api-access-nxzzf podName:a50c24dc-cc0a-404a-89d0-77e82c417bb5 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:16.90278913 +0000 UTC m=+10.267123586 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxzzf" (UniqueName: "kubernetes.io/projected/a50c24dc-cc0a-404a-89d0-77e82c417bb5-kube-api-access-nxzzf") pod "network-check-target-65ldr" (UID: "a50c24dc-cc0a-404a-89d0-77e82c417bb5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 02:30:13.161435 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:13.161382 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6lzgq" Sep 09 02:30:13.161673 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:13.161502 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6lzgq" podUID="b974ece6-0b9d-4339-aa6f-203ce53dcd85" Sep 09 02:30:13.161915 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:13.161864 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65ldr" Sep 09 02:30:13.162012 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:13.161952 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-65ldr" podUID="a50c24dc-cc0a-404a-89d0-77e82c417bb5" Sep 09 02:30:13.162071 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:13.162021 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjfl" Sep 09 02:30:13.162159 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:13.162144 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgjfl" podUID="06180cab-e7c8-4fb8-a719-70072ee0ee36" Sep 09 02:30:15.159842 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:15.159079 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjfl" Sep 09 02:30:15.159842 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:15.159234 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgjfl" podUID="06180cab-e7c8-4fb8-a719-70072ee0ee36" Sep 09 02:30:15.159842 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:15.159582 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65ldr" Sep 09 02:30:15.159842 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:15.159667 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-65ldr" podUID="a50c24dc-cc0a-404a-89d0-77e82c417bb5" Sep 09 02:30:15.159842 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:15.159728 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6lzgq" Sep 09 02:30:15.159842 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:15.159797 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6lzgq" podUID="b974ece6-0b9d-4339-aa6f-203ce53dcd85" Sep 09 02:30:16.738991 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:16.738901 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b974ece6-0b9d-4339-aa6f-203ce53dcd85-original-pull-secret\") pod \"global-pull-secret-syncer-6lzgq\" (UID: \"b974ece6-0b9d-4339-aa6f-203ce53dcd85\") " pod="kube-system/global-pull-secret-syncer-6lzgq" Sep 09 02:30:16.739559 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:16.739087 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 02:30:16.739559 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:16.739158 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b974ece6-0b9d-4339-aa6f-203ce53dcd85-original-pull-secret podName:b974ece6-0b9d-4339-aa6f-203ce53dcd85 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:24.739139307 +0000 UTC m=+18.103473762 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b974ece6-0b9d-4339-aa6f-203ce53dcd85-original-pull-secret") pod "global-pull-secret-syncer-6lzgq" (UID: "b974ece6-0b9d-4339-aa6f-203ce53dcd85") : object "kube-system"/"original-pull-secret" not registered Sep 09 02:30:16.839813 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:16.839777 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06180cab-e7c8-4fb8-a719-70072ee0ee36-metrics-certs\") pod \"network-metrics-daemon-mgjfl\" (UID: \"06180cab-e7c8-4fb8-a719-70072ee0ee36\") " pod="openshift-multus/network-metrics-daemon-mgjfl" Sep 09 02:30:16.840010 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:16.839946 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 02:30:16.840089 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:16.840064 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06180cab-e7c8-4fb8-a719-70072ee0ee36-metrics-certs podName:06180cab-e7c8-4fb8-a719-70072ee0ee36 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:24.840042842 +0000 UTC m=+18.204377297 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06180cab-e7c8-4fb8-a719-70072ee0ee36-metrics-certs") pod "network-metrics-daemon-mgjfl" (UID: "06180cab-e7c8-4fb8-a719-70072ee0ee36") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 02:30:16.941145 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:16.941098 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxzzf\" (UniqueName: \"kubernetes.io/projected/a50c24dc-cc0a-404a-89d0-77e82c417bb5-kube-api-access-nxzzf\") pod \"network-check-target-65ldr\" (UID: \"a50c24dc-cc0a-404a-89d0-77e82c417bb5\") " pod="openshift-network-diagnostics/network-check-target-65ldr" Sep 09 02:30:16.941308 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:16.941290 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 02:30:16.941384 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:16.941310 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 02:30:16.941384 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:16.941322 2577 projected.go:196] Error preparing data for projected volume kube-api-access-nxzzf for pod openshift-network-diagnostics/network-check-target-65ldr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 02:30:16.941384 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:16.941380 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a50c24dc-cc0a-404a-89d0-77e82c417bb5-kube-api-access-nxzzf podName:a50c24dc-cc0a-404a-89d0-77e82c417bb5 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:24.941362724 +0000 UTC m=+18.305697185 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxzzf" (UniqueName: "kubernetes.io/projected/a50c24dc-cc0a-404a-89d0-77e82c417bb5-kube-api-access-nxzzf") pod "network-check-target-65ldr" (UID: "a50c24dc-cc0a-404a-89d0-77e82c417bb5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 02:30:17.160798 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:17.160760 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjfl" Sep 09 02:30:17.160974 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:17.160879 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6lzgq" Sep 09 02:30:17.160974 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:17.160885 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgjfl" podUID="06180cab-e7c8-4fb8-a719-70072ee0ee36" Sep 09 02:30:17.160974 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:17.160898 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65ldr" Sep 09 02:30:17.161114 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:17.160999 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6lzgq" podUID="b974ece6-0b9d-4339-aa6f-203ce53dcd85" Sep 09 02:30:17.161114 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:17.161056 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-65ldr" podUID="a50c24dc-cc0a-404a-89d0-77e82c417bb5" Sep 09 02:30:19.161169 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:19.161084 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6lzgq" Sep 09 02:30:19.161169 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:19.161110 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65ldr" Sep 09 02:30:19.161169 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:19.161110 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjfl" Sep 09 02:30:19.161725 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:19.161213 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6lzgq" podUID="b974ece6-0b9d-4339-aa6f-203ce53dcd85" Sep 09 02:30:19.161725 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:19.161298 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-65ldr" podUID="a50c24dc-cc0a-404a-89d0-77e82c417bb5" Sep 09 02:30:19.161725 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:19.161396 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgjfl" podUID="06180cab-e7c8-4fb8-a719-70072ee0ee36" Sep 09 02:30:21.160147 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:21.159577 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65ldr" Sep 09 02:30:21.160147 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:21.159583 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6lzgq" Sep 09 02:30:21.160147 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:21.159718 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-65ldr" podUID="a50c24dc-cc0a-404a-89d0-77e82c417bb5" Sep 09 02:30:21.160147 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:21.159594 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjfl" Sep 09 02:30:21.160147 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:21.159782 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6lzgq" podUID="b974ece6-0b9d-4339-aa6f-203ce53dcd85" Sep 09 02:30:21.160147 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:21.159885 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgjfl" podUID="06180cab-e7c8-4fb8-a719-70072ee0ee36" Sep 09 02:30:23.159539 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:23.159493 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjfl" Sep 09 02:30:23.159539 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:23.159511 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6lzgq" Sep 09 02:30:23.160107 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:23.159634 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgjfl" podUID="06180cab-e7c8-4fb8-a719-70072ee0ee36" Sep 09 02:30:23.160107 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:23.159682 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6lzgq" podUID="b974ece6-0b9d-4339-aa6f-203ce53dcd85" Sep 09 02:30:23.160107 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:23.159737 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65ldr" Sep 09 02:30:23.160107 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:23.159803 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-65ldr" podUID="a50c24dc-cc0a-404a-89d0-77e82c417bb5" Sep 09 02:30:24.803892 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:24.803854 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b974ece6-0b9d-4339-aa6f-203ce53dcd85-original-pull-secret\") pod \"global-pull-secret-syncer-6lzgq\" (UID: \"b974ece6-0b9d-4339-aa6f-203ce53dcd85\") " pod="kube-system/global-pull-secret-syncer-6lzgq" Sep 09 02:30:24.804345 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:24.804010 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 02:30:24.804345 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:24.804080 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b974ece6-0b9d-4339-aa6f-203ce53dcd85-original-pull-secret podName:b974ece6-0b9d-4339-aa6f-203ce53dcd85 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:40.804061154 +0000 UTC m=+34.168395609 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b974ece6-0b9d-4339-aa6f-203ce53dcd85-original-pull-secret") pod "global-pull-secret-syncer-6lzgq" (UID: "b974ece6-0b9d-4339-aa6f-203ce53dcd85") : object "kube-system"/"original-pull-secret" not registered Sep 09 02:30:24.904192 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:24.904124 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06180cab-e7c8-4fb8-a719-70072ee0ee36-metrics-certs\") pod \"network-metrics-daemon-mgjfl\" (UID: \"06180cab-e7c8-4fb8-a719-70072ee0ee36\") " pod="openshift-multus/network-metrics-daemon-mgjfl" Sep 09 02:30:24.904351 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:24.904313 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 02:30:24.904410 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:24.904386 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06180cab-e7c8-4fb8-a719-70072ee0ee36-metrics-certs podName:06180cab-e7c8-4fb8-a719-70072ee0ee36 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:40.904364024 +0000 UTC m=+34.268698466 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06180cab-e7c8-4fb8-a719-70072ee0ee36-metrics-certs") pod "network-metrics-daemon-mgjfl" (UID: "06180cab-e7c8-4fb8-a719-70072ee0ee36") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 02:30:25.004721 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:25.004686 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxzzf\" (UniqueName: \"kubernetes.io/projected/a50c24dc-cc0a-404a-89d0-77e82c417bb5-kube-api-access-nxzzf\") pod \"network-check-target-65ldr\" (UID: \"a50c24dc-cc0a-404a-89d0-77e82c417bb5\") " pod="openshift-network-diagnostics/network-check-target-65ldr" Sep 09 02:30:25.004922 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:25.004905 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 02:30:25.005022 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:25.004932 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 02:30:25.005022 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:25.004945 2577 projected.go:196] Error preparing data for projected volume kube-api-access-nxzzf for pod openshift-network-diagnostics/network-check-target-65ldr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 02:30:25.005115 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:25.005024 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a50c24dc-cc0a-404a-89d0-77e82c417bb5-kube-api-access-nxzzf podName:a50c24dc-cc0a-404a-89d0-77e82c417bb5 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:41.005002006 +0000 UTC m=+34.369336452 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxzzf" (UniqueName: "kubernetes.io/projected/a50c24dc-cc0a-404a-89d0-77e82c417bb5-kube-api-access-nxzzf") pod "network-check-target-65ldr" (UID: "a50c24dc-cc0a-404a-89d0-77e82c417bb5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 02:30:25.158909 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:25.158821 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6lzgq" Sep 09 02:30:25.159095 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:25.158828 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65ldr" Sep 09 02:30:25.159095 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:25.158837 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjfl" Sep 09 02:30:25.159095 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:25.158977 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6lzgq" podUID="b974ece6-0b9d-4339-aa6f-203ce53dcd85" Sep 09 02:30:25.159095 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:25.159057 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-65ldr" podUID="a50c24dc-cc0a-404a-89d0-77e82c417bb5" Sep 09 02:30:25.159261 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:25.159139 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgjfl" podUID="06180cab-e7c8-4fb8-a719-70072ee0ee36" Sep 09 02:30:27.159643 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:27.159601 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjfl" Sep 09 02:30:27.160079 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:27.159731 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6lzgq" Sep 09 02:30:27.160079 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:27.159837 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6lzgq" podUID="b974ece6-0b9d-4339-aa6f-203ce53dcd85" Sep 09 02:30:27.160079 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:27.159727 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgjfl" podUID="06180cab-e7c8-4fb8-a719-70072ee0ee36" Sep 09 02:30:27.160079 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:27.159897 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65ldr" Sep 09 02:30:27.160079 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:27.159983 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-65ldr" podUID="a50c24dc-cc0a-404a-89d0-77e82c417bb5" Sep 09 02:30:28.346639 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:28.346373 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" event={"ID":"b14ff9af-8bbc-4169-8b5f-5d24df8c8968","Type":"ContainerStarted","Data":"9312153d4fa10e62a803a995b2aacf92271e510329de3188c7ac7346954efb44"} Sep 09 02:30:28.347930 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:28.347903 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjstd" event={"ID":"95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1","Type":"ContainerStarted","Data":"0d1faaae9f5c366fd681be969d6e74d859671e700a64517f1111b9bc1194602a"} Sep 09 02:30:28.349934 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:28.349896 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pd8dm" event={"ID":"19afdc10-5a52-4e9d-a84f-04df6136a1c0","Type":"ContainerStarted","Data":"44fe57bcf5e4fb429844722fb14859987683dcce2ad372cea1f00e5c0af47d5a"} Sep 09 02:30:28.352408 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:28.352385 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dj8dq" event={"ID":"b7a1ff67-afbc-4e76-aa24-d1a8cebb3888","Type":"ContainerStarted","Data":"0724f0948c8a95d3f672bc3fdb30d2186779ee066235bfb5f9d71e1605526b4c"} Sep 09 02:30:28.354209 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:28.354190 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qw4bt" event={"ID":"dead4f18-8c41-41a5-9f37-897ddaee8563","Type":"ContainerStarted","Data":"3e29e57fd971668cd3b423e92332320f62b2bf1ac0d35549c167bc8414c920f9"} Sep 09 02:30:28.355736 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:28.355713 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" event={"ID":"e3e028fc-3961-4b04-9875-fa06912e9518","Type":"ContainerStarted","Data":"b659071f8361d07b1aec14d5ef4b65bf82bc77eac9925a891ba296e0a936a218"} Sep 09 02:30:28.360213 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:28.360177 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfpfn" event={"ID":"cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83","Type":"ContainerStarted","Data":"95a4ccb2081ba5be5fc2402375ea8d080ee35e8151da05c287678ec7aadbf53c"} Sep 09 02:30:28.361565 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:28.361534 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tvhb5" event={"ID":"54c26b32-0189-4a6a-a569-c2f859d87c8b","Type":"ContainerStarted","Data":"43a8a2b12702b0985eb72da3cf9c590fa306e946ca540f0bc1b133aa2bcb2562"} Sep 09 02:30:28.392241 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:28.392191 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-pd8dm" podStartSLOduration=7.215736001 podStartE2EDuration="21.392176485s" totalImagesPullingTime="14.176440484s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 02:30:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:30:28.39146961 +0000 UTC m=+21.755804073" watchObservedRunningTime="2026-09-09 02:30:28.392176485 +0000 UTC m=+21.756510944" Sep 09 02:30:28.412905 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:28.412866 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dj8dq" podStartSLOduration=3.648704803 podStartE2EDuration="21.412852363s" totalImagesPullingTime="17.76414756s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 02:30:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:30:28.412666149 +0000 UTC m=+21.777000612" watchObservedRunningTime="2026-09-09 02:30:28.412852363 +0000 UTC m=+21.777186826" Sep 09 02:30:28.434390 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:28.434339 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qw4bt" podStartSLOduration=7.266754988 podStartE2EDuration="21.434324672s" totalImagesPullingTime="14.167569684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:30:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:30:28.433408619 +0000 UTC m=+21.797743096" watchObservedRunningTime="2026-09-09 02:30:28.434324672 +0000 UTC m=+21.798659134" Sep 09 02:30:28.456950 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:28.456905 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-h4jdc" podStartSLOduration=3.763494582 podStartE2EDuration="21.456888563s" totalImagesPullingTime="17.693393981s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 02:30:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:30:28.456411642 +0000 UTC m=+21.820746106" watchObservedRunningTime="2026-09-09 02:30:28.456888563 +0000 UTC m=+21.821223029" Sep 09 02:30:28.478124 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:28.478075 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tvhb5" podStartSLOduration=3.788072134 podStartE2EDuration="21.478054234s" totalImagesPullingTime="17.6899821s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 02:30:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:30:28.477263543 +0000 UTC m=+21.841598017" watchObservedRunningTime="2026-09-09 02:30:28.478054234 +0000 UTC m=+21.842388698" Sep 09 02:30:29.159297 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:29.159273 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjfl" Sep 09 02:30:29.159476 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:29.159275 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65ldr" Sep 09 02:30:29.159476 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:29.159370 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgjfl" podUID="06180cab-e7c8-4fb8-a719-70072ee0ee36" Sep 09 02:30:29.159476 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:29.159275 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6lzgq" Sep 09 02:30:29.159476 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:29.159440 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-65ldr" podUID="a50c24dc-cc0a-404a-89d0-77e82c417bb5" Sep 09 02:30:29.159695 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:29.159514 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6lzgq" podUID="b974ece6-0b9d-4339-aa6f-203ce53dcd85" Sep 09 02:30:29.366442 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:29.366375 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" event={"ID":"b14ff9af-8bbc-4169-8b5f-5d24df8c8968","Type":"ContainerStarted","Data":"a5606594954f96ec3707bc693cf73683b8a6a224a041ae8308f4769fc3f93727"} Sep 09 02:30:29.367050 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:29.366449 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" event={"ID":"b14ff9af-8bbc-4169-8b5f-5d24df8c8968","Type":"ContainerStarted","Data":"66d8b7239cea9d24afbdd345f47b4347744c09895c2cb4f59025d5d15060c961"} Sep 09 02:30:29.367050 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:29.366494 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" event={"ID":"b14ff9af-8bbc-4169-8b5f-5d24df8c8968","Type":"ContainerStarted","Data":"034bc022546fa59c4ec33ce35f3cdff9506f1f3c2fbad9f687537115cebf4061"} Sep 09 02:30:29.367050 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:29.366536 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" event={"ID":"b14ff9af-8bbc-4169-8b5f-5d24df8c8968","Type":"ContainerStarted","Data":"7fc41f6bab0bd22dbecc4f0227fe37bcb5ddb2b1bbb9a017886531090c14f378"} Sep 09 02:30:29.367050 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:29.366585 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" event={"ID":"b14ff9af-8bbc-4169-8b5f-5d24df8c8968","Type":"ContainerStarted","Data":"21039cb093ad289cce7458ccca909a9b3297f98c5e63887ba4a23bbfb809ab78"} Sep 09 02:30:29.367770 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:29.367750 2577 generic.go:356] "Generic (PLEG): container finished" podID="95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1" containerID="0d1faaae9f5c366fd681be969d6e74d859671e700a64517f1111b9bc1194602a" exitCode=0 Sep 09 02:30:29.367864 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:29.367843 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjstd" event={"ID":"95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1","Type":"ContainerDied","Data":"0d1faaae9f5c366fd681be969d6e74d859671e700a64517f1111b9bc1194602a"} Sep 09 02:30:29.821011 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:29.820816 2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 09 02:30:30.130321 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:30.130232 2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T02:30:29.820981925Z","UUID":"898b4aed-7074-486c-92f9-af107904480d","Handler":null,"Name":"","Endpoint":""} Sep 09 02:30:30.131939 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:30.131918 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 09 02:30:30.131939 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:30.131949 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 09 02:30:30.372135 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:30.372102 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfpfn" event={"ID":"cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83","Type":"ContainerStarted","Data":"2178768a9bacecd3cb6127dedabcd91124f0141c9ab662c7d2fd198c1b51f1dc"} Sep 09 02:30:30.373776 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:30.373739 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gktjg" event={"ID":"52808c42-8ac8-49fd-96f8-e05e956b42be","Type":"ContainerStarted","Data":"ee8f3212c84019fbd0ab124a7322bf40f05b5bc2294ffe807c0b9e3f20ca2a40"} Sep 09 02:30:31.159294 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:31.159270 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65ldr" Sep 09 02:30:31.159294 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:31.159288 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjfl" Sep 09 02:30:31.159470 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:31.159443 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6lzgq" Sep 09 02:30:31.159584 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:31.159557 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6lzgq" podUID="b974ece6-0b9d-4339-aa6f-203ce53dcd85" Sep 09 02:30:31.159584 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:31.159443 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgjfl" podUID="06180cab-e7c8-4fb8-a719-70072ee0ee36" Sep 09 02:30:31.159706 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:31.159652 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-65ldr" podUID="a50c24dc-cc0a-404a-89d0-77e82c417bb5" Sep 09 02:30:31.378231 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:31.378150 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfpfn" event={"ID":"cff5b67e-1d8e-4f8a-b9ef-6e7f3e096d83","Type":"ContainerStarted","Data":"2ddaaf40ae4bca83ce74cf90e7741cb448cf73b11e069d2887a832a71a86c64e"} Sep 09 02:30:31.381519 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:31.381498 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" event={"ID":"b14ff9af-8bbc-4169-8b5f-5d24df8c8968","Type":"ContainerStarted","Data":"3616b99d2ebb42785a0a7adf07d59c379a968ff592dae1101bbe37ca83ea790f"} Sep 09 02:30:31.401303 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:31.401258 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gktjg" podStartSLOduration=6.704877902 podStartE2EDuration="24.401245077s" totalImagesPullingTime="17.696367175s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 02:30:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:30:30.392712816 +0000 UTC m=+23.757047279" watchObservedRunningTime="2026-09-09 02:30:31.401245077 +0000 UTC m=+24.765579540" Sep 09 02:30:31.401564 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:31.401536 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dfpfn" podStartSLOduration=4.081307067 podStartE2EDuration="24.401528843s" totalImagesPullingTime="20.320221776s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 02:30:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:30:31.399834498 +0000 UTC m=+24.764168960" watchObservedRunningTime="2026-09-09 02:30:31.401528843 +0000 UTC m=+24.765863306" Sep 09 02:30:33.159269 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:33.159231 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6lzgq" Sep 09 02:30:33.159704 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:33.159231 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65ldr" Sep 09 02:30:33.159704 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:33.159354 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6lzgq" podUID="b974ece6-0b9d-4339-aa6f-203ce53dcd85" Sep 09 02:30:33.159704 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:33.159238 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjfl" Sep 09 02:30:33.159704 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:33.159449 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-65ldr" podUID="a50c24dc-cc0a-404a-89d0-77e82c417bb5" Sep 09 02:30:33.159704 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:33.159616 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgjfl" podUID="06180cab-e7c8-4fb8-a719-70072ee0ee36" Sep 09 02:30:33.334711 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:33.334677 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qw4bt" Sep 09 02:30:33.335650 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:33.335630 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qw4bt" Sep 09 02:30:33.386052 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:33.386026 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qw4bt" Sep 09 02:30:33.386517 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:33.386499 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qw4bt" Sep 09 02:30:34.392953 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:34.392782 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" event={"ID":"b14ff9af-8bbc-4169-8b5f-5d24df8c8968","Type":"ContainerStarted","Data":"14d5a483eb8b66d7402c89ea40033c608f3ce3cf1ff5cde531fa6f969f49fcad"} Sep 09 02:30:34.393488 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:34.393033 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:34.393488 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:34.393091 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:34.393488 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:34.393133 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:34.395043 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:34.395014 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjstd" event={"ID":"95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1","Type":"ContainerStarted","Data":"691d4f17b9bf90e4399702137c3e84ede40a33b26341193994f4f094b109d4fc"} Sep 09 02:30:34.409976 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:34.409938 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:34.415360 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:34.415344 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:30:34.427107 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:34.427058 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" podStartSLOduration=9.6629561 podStartE2EDuration="27.4270426s" totalImagesPullingTime="17.7640865s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 02:30:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:30:34.426725051 +0000 UTC m=+27.791059514" watchObservedRunningTime="2026-09-09 02:30:34.4270426 +0000 UTC m=+27.791377064" Sep 09 02:30:35.159858 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:35.159826 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6lzgq" Sep 09 02:30:35.160025 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:35.159831 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65ldr" Sep 09 02:30:35.160025 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:35.159938 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6lzgq" podUID="b974ece6-0b9d-4339-aa6f-203ce53dcd85" Sep 09 02:30:35.160172 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:35.160040 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-65ldr" podUID="a50c24dc-cc0a-404a-89d0-77e82c417bb5" Sep 09 02:30:35.160172 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:35.160106 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjfl" Sep 09 02:30:35.160242 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:35.160170 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgjfl" podUID="06180cab-e7c8-4fb8-a719-70072ee0ee36" Sep 09 02:30:35.398573 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:35.398534 2577 generic.go:356] "Generic (PLEG): container finished" podID="95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1" containerID="691d4f17b9bf90e4399702137c3e84ede40a33b26341193994f4f094b109d4fc" exitCode=0 Sep 09 02:30:35.399106 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:35.398619 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjstd" event={"ID":"95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1","Type":"ContainerDied","Data":"691d4f17b9bf90e4399702137c3e84ede40a33b26341193994f4f094b109d4fc"} Sep 09 02:30:36.001120 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:36.001035 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-65ldr"] Sep 09 02:30:36.001272 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:36.001144 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65ldr" Sep 09 02:30:36.001272 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:36.001249 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-65ldr" podUID="a50c24dc-cc0a-404a-89d0-77e82c417bb5" Sep 09 02:30:36.007045 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:36.007019 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mgjfl"] Sep 09 02:30:36.007144 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:36.007119 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjfl" Sep 09 02:30:36.007238 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:36.007220 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgjfl" podUID="06180cab-e7c8-4fb8-a719-70072ee0ee36" Sep 09 02:30:36.008999 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:36.008950 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6lzgq"] Sep 09 02:30:36.009139 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:36.009060 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6lzgq" Sep 09 02:30:36.009196 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:36.009163 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6lzgq" podUID="b974ece6-0b9d-4339-aa6f-203ce53dcd85" Sep 09 02:30:37.162096 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:37.161656 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6lzgq" Sep 09 02:30:37.162096 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:37.161822 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6lzgq" podUID="b974ece6-0b9d-4339-aa6f-203ce53dcd85" Sep 09 02:30:37.405477 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:37.405220 2577 generic.go:356] "Generic (PLEG): container finished" podID="95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1" containerID="b804d061b6836036971ccc2a16ff9ca7c233a3c80c23f549d5d71c7549899d40" exitCode=0 Sep 09 02:30:37.405477 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:37.405295 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjstd" event={"ID":"95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1","Type":"ContainerDied","Data":"b804d061b6836036971ccc2a16ff9ca7c233a3c80c23f549d5d71c7549899d40"} Sep 09 02:30:38.159824 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:38.159749 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjfl" Sep 09 02:30:38.159824 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:38.159781 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65ldr" Sep 09 02:30:38.160032 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:38.159869 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgjfl" podUID="06180cab-e7c8-4fb8-a719-70072ee0ee36" Sep 09 02:30:38.160032 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:38.160018 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-65ldr" podUID="a50c24dc-cc0a-404a-89d0-77e82c417bb5" Sep 09 02:30:38.410163 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:38.410098 2577 generic.go:356] "Generic (PLEG): container finished" podID="95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1" containerID="7346b47bc76c1bd78cf2eb23f8b8915f3d0b9ad20b7bf9c122a7fdc1834f7ba8" exitCode=0 Sep 09 02:30:38.410163 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:38.410138 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjstd" event={"ID":"95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1","Type":"ContainerDied","Data":"7346b47bc76c1bd78cf2eb23f8b8915f3d0b9ad20b7bf9c122a7fdc1834f7ba8"} Sep 09 02:30:39.159528 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:39.159500 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6lzgq" Sep 09 02:30:39.159663 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:39.159596 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6lzgq" podUID="b974ece6-0b9d-4339-aa6f-203ce53dcd85" Sep 09 02:30:40.159572 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:40.159541 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjfl" Sep 09 02:30:40.159572 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:40.159580 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65ldr" Sep 09 02:30:40.160134 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:40.159667 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mgjfl" podUID="06180cab-e7c8-4fb8-a719-70072ee0ee36" Sep 09 02:30:40.160134 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:40.159813 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-65ldr" podUID="a50c24dc-cc0a-404a-89d0-77e82c417bb5" Sep 09 02:30:40.835422 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:40.835384 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b974ece6-0b9d-4339-aa6f-203ce53dcd85-original-pull-secret\") pod \"global-pull-secret-syncer-6lzgq\" (UID: \"b974ece6-0b9d-4339-aa6f-203ce53dcd85\") " pod="kube-system/global-pull-secret-syncer-6lzgq" Sep 09 02:30:40.835612 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:40.835543 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 02:30:40.835677 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:40.835624 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b974ece6-0b9d-4339-aa6f-203ce53dcd85-original-pull-secret podName:b974ece6-0b9d-4339-aa6f-203ce53dcd85 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:12.835603555 +0000 UTC m=+66.199938017 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b974ece6-0b9d-4339-aa6f-203ce53dcd85-original-pull-secret") pod "global-pull-secret-syncer-6lzgq" (UID: "b974ece6-0b9d-4339-aa6f-203ce53dcd85") : object "kube-system"/"original-pull-secret" not registered Sep 09 02:30:40.936371 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:40.936337 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06180cab-e7c8-4fb8-a719-70072ee0ee36-metrics-certs\") pod \"network-metrics-daemon-mgjfl\" (UID: \"06180cab-e7c8-4fb8-a719-70072ee0ee36\") " pod="openshift-multus/network-metrics-daemon-mgjfl" Sep 09 02:30:40.936537 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:40.936471 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 02:30:40.936537 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:40.936528 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06180cab-e7c8-4fb8-a719-70072ee0ee36-metrics-certs podName:06180cab-e7c8-4fb8-a719-70072ee0ee36 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:12.936512487 +0000 UTC m=+66.300846947 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06180cab-e7c8-4fb8-a719-70072ee0ee36-metrics-certs") pod "network-metrics-daemon-mgjfl" (UID: "06180cab-e7c8-4fb8-a719-70072ee0ee36") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 02:30:41.037062 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:41.037029 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxzzf\" (UniqueName: \"kubernetes.io/projected/a50c24dc-cc0a-404a-89d0-77e82c417bb5-kube-api-access-nxzzf\") pod \"network-check-target-65ldr\" (UID: \"a50c24dc-cc0a-404a-89d0-77e82c417bb5\") " pod="openshift-network-diagnostics/network-check-target-65ldr" Sep 09 02:30:41.037202 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:41.037178 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 02:30:41.037202 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:41.037200 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 02:30:41.037309 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:41.037214 2577 projected.go:196] Error preparing data for projected volume kube-api-access-nxzzf for pod openshift-network-diagnostics/network-check-target-65ldr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 02:30:41.037309 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:41.037276 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a50c24dc-cc0a-404a-89d0-77e82c417bb5-kube-api-access-nxzzf podName:a50c24dc-cc0a-404a-89d0-77e82c417bb5 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:13.037259357 +0000 UTC m=+66.401593809 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxzzf" (UniqueName: "kubernetes.io/projected/a50c24dc-cc0a-404a-89d0-77e82c417bb5-kube-api-access-nxzzf") pod "network-check-target-65ldr" (UID: "a50c24dc-cc0a-404a-89d0-77e82c417bb5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 02:30:41.159060 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:41.158993 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6lzgq" Sep 09 02:30:41.159192 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:41.159088 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6lzgq" podUID="b974ece6-0b9d-4339-aa6f-203ce53dcd85" Sep 09 02:30:41.938543 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:41.938459 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-89.ec2.internal" event="NodeReady" Sep 09 02:30:41.938903 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:41.938600 2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 09 02:30:41.980380 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:41.980351 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7c679fdf79-2v4dd"] Sep 09 02:30:41.985150 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:41.985118 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c679fdf79-2v4dd" Sep 09 02:30:41.987939 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:41.987917 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 09 02:30:41.988056 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:41.987983 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 09 02:30:41.988246 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:41.988228 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 09 02:30:41.988340 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:41.988235 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dckqx\"" Sep 09 02:30:41.994778 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:41.994754 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 09 02:30:41.995584 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:41.995563 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-frnzb"] Sep 09 02:30:41.998949 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:41.998931 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-frnzb" Sep 09 02:30:42.001731 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.001708 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 09 02:30:42.001824 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.001743 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 09 02:30:42.002026 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.002008 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-99mnl\"" Sep 09 02:30:42.002631 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.002609 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c679fdf79-2v4dd"] Sep 09 02:30:42.013071 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.013048 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-frnzb"] Sep 09 02:30:42.097011 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.096980 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pkdtz"] Sep 09 02:30:42.100235 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.100206 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pkdtz" Sep 09 02:30:42.107294 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.104064 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 09 02:30:42.107294 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.104135 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 09 02:30:42.107294 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.104376 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 09 02:30:42.107294 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.104749 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-r6wq8\"" Sep 09 02:30:42.114433 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.114412 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pkdtz"] Sep 09 02:30:42.144413 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.144389 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/98d0408e-2648-467e-a2fa-a553b001c70c-config-volume\") pod \"dns-default-frnzb\" (UID: \"98d0408e-2648-467e-a2fa-a553b001c70c\") " pod="openshift-dns/dns-default-frnzb" Sep 09 02:30:42.144532 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.144427 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98d0408e-2648-467e-a2fa-a553b001c70c-metrics-tls\") pod \"dns-default-frnzb\" (UID: \"98d0408e-2648-467e-a2fa-a553b001c70c\") " pod="openshift-dns/dns-default-frnzb" Sep 09 02:30:42.144532 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.144459 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1dc93f45-045a-4969-8db9-3e224316cbd5-trusted-ca\") pod \"image-registry-7c679fdf79-2v4dd\" (UID: \"1dc93f45-045a-4969-8db9-3e224316cbd5\") " pod="openshift-image-registry/image-registry-7c679fdf79-2v4dd" Sep 09 02:30:42.144532 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.144502 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1dc93f45-045a-4969-8db9-3e224316cbd5-installation-pull-secrets\") pod \"image-registry-7c679fdf79-2v4dd\" (UID: \"1dc93f45-045a-4969-8db9-3e224316cbd5\") " pod="openshift-image-registry/image-registry-7c679fdf79-2v4dd" Sep 09 02:30:42.144682 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.144563 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-652g4\" (UniqueName: \"kubernetes.io/projected/98d0408e-2648-467e-a2fa-a553b001c70c-kube-api-access-652g4\") pod \"dns-default-frnzb\" (UID: \"98d0408e-2648-467e-a2fa-a553b001c70c\") " pod="openshift-dns/dns-default-frnzb" Sep 09 02:30:42.144682 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.144597 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1dc93f45-045a-4969-8db9-3e224316cbd5-registry-tls\") pod \"image-registry-7c679fdf79-2v4dd\" (UID: \"1dc93f45-045a-4969-8db9-3e224316cbd5\") " pod="openshift-image-registry/image-registry-7c679fdf79-2v4dd" Sep 09 02:30:42.144682 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.144658 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98d0408e-2648-467e-a2fa-a553b001c70c-tmp-dir\") pod \"dns-default-frnzb\" (UID: \"98d0408e-2648-467e-a2fa-a553b001c70c\") " pod="openshift-dns/dns-default-frnzb" Sep 09 02:30:42.144828 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.144704 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w92bg\" (UniqueName: \"kubernetes.io/projected/1dc93f45-045a-4969-8db9-3e224316cbd5-kube-api-access-w92bg\") pod \"image-registry-7c679fdf79-2v4dd\" (UID: \"1dc93f45-045a-4969-8db9-3e224316cbd5\") " pod="openshift-image-registry/image-registry-7c679fdf79-2v4dd" Sep 09 02:30:42.144828 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.144756 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1dc93f45-045a-4969-8db9-3e224316cbd5-image-registry-private-configuration\") pod \"image-registry-7c679fdf79-2v4dd\" (UID: \"1dc93f45-045a-4969-8db9-3e224316cbd5\") " pod="openshift-image-registry/image-registry-7c679fdf79-2v4dd" Sep 09 02:30:42.144828 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.144783 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1dc93f45-045a-4969-8db9-3e224316cbd5-ca-trust-extracted\") pod \"image-registry-7c679fdf79-2v4dd\" (UID: \"1dc93f45-045a-4969-8db9-3e224316cbd5\") " pod="openshift-image-registry/image-registry-7c679fdf79-2v4dd" Sep 09 02:30:42.144828 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.144813 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1dc93f45-045a-4969-8db9-3e224316cbd5-registry-certificates\") pod \"image-registry-7c679fdf79-2v4dd\" (UID: \"1dc93f45-045a-4969-8db9-3e224316cbd5\") " pod="openshift-image-registry/image-registry-7c679fdf79-2v4dd" Sep 09 02:30:42.145012 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.144835 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1dc93f45-045a-4969-8db9-3e224316cbd5-bound-sa-token\") pod \"image-registry-7c679fdf79-2v4dd\" (UID: \"1dc93f45-045a-4969-8db9-3e224316cbd5\") " pod="openshift-image-registry/image-registry-7c679fdf79-2v4dd" Sep 09 02:30:42.159791 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.159767 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjfl" Sep 09 02:30:42.159896 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.159799 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65ldr" Sep 09 02:30:42.163496 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.163478 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 09 02:30:42.164313 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.164106 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 09 02:30:42.164313 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.164148 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 09 02:30:42.164313 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.164235 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rd54v\"" Sep 09 02:30:42.164313 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.164251 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-shl9x\"" Sep 09 02:30:42.245423 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.245359 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1dc93f45-045a-4969-8db9-3e224316cbd5-image-registry-private-configuration\") pod \"image-registry-7c679fdf79-2v4dd\" (UID: \"1dc93f45-045a-4969-8db9-3e224316cbd5\") " pod="openshift-image-registry/image-registry-7c679fdf79-2v4dd" Sep 09 02:30:42.245423 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.245387 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1dc93f45-045a-4969-8db9-3e224316cbd5-ca-trust-extracted\") pod \"image-registry-7c679fdf79-2v4dd\" (UID: \"1dc93f45-045a-4969-8db9-3e224316cbd5\") " pod="openshift-image-registry/image-registry-7c679fdf79-2v4dd" Sep 09 02:30:42.245423 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.245406 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mj4n\" (UniqueName: \"kubernetes.io/projected/1ceefd67-0082-4ecb-9d62-48f62ac00b58-kube-api-access-5mj4n\") pod \"ingress-canary-pkdtz\" (UID: \"1ceefd67-0082-4ecb-9d62-48f62ac00b58\") " pod="openshift-ingress-canary/ingress-canary-pkdtz" Sep 09 02:30:42.245623 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.245426 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1dc93f45-045a-4969-8db9-3e224316cbd5-registry-certificates\") pod \"image-registry-7c679fdf79-2v4dd\" (UID: \"1dc93f45-045a-4969-8db9-3e224316cbd5\") " pod="openshift-image-registry/image-registry-7c679fdf79-2v4dd" Sep 09 02:30:42.245623 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.245534 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1dc93f45-045a-4969-8db9-3e224316cbd5-bound-sa-token\") pod \"image-registry-7c679fdf79-2v4dd\" (UID: \"1dc93f45-045a-4969-8db9-3e224316cbd5\") " pod="openshift-image-registry/image-registry-7c679fdf79-2v4dd" Sep 09 02:30:42.245623 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.245569 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/98d0408e-2648-467e-a2fa-a553b001c70c-config-volume\") pod \"dns-default-frnzb\" (UID: \"98d0408e-2648-467e-a2fa-a553b001c70c\") " pod="openshift-dns/dns-default-frnzb" Sep 09 02:30:42.245623 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.245597 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98d0408e-2648-467e-a2fa-a553b001c70c-metrics-tls\") pod \"dns-default-frnzb\" (UID: \"98d0408e-2648-467e-a2fa-a553b001c70c\") " pod="openshift-dns/dns-default-frnzb" Sep 09 02:30:42.245825 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.245632 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1dc93f45-045a-4969-8db9-3e224316cbd5-trusted-ca\") pod \"image-registry-7c679fdf79-2v4dd\" (UID: \"1dc93f45-045a-4969-8db9-3e224316cbd5\") " pod="openshift-image-registry/image-registry-7c679fdf79-2v4dd" Sep 09 02:30:42.245825 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.245655 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1dc93f45-045a-4969-8db9-3e224316cbd5-installation-pull-secrets\") pod \"image-registry-7c679fdf79-2v4dd\" (UID: \"1dc93f45-045a-4969-8db9-3e224316cbd5\") " pod="openshift-image-registry/image-registry-7c679fdf79-2v4dd" Sep 09 02:30:42.245825 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.245694 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-652g4\" (UniqueName: \"kubernetes.io/projected/98d0408e-2648-467e-a2fa-a553b001c70c-kube-api-access-652g4\") pod \"dns-default-frnzb\" (UID: \"98d0408e-2648-467e-a2fa-a553b001c70c\") " pod="openshift-dns/dns-default-frnzb" Sep 09 02:30:42.245825 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.245720 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ceefd67-0082-4ecb-9d62-48f62ac00b58-cert\") pod \"ingress-canary-pkdtz\" (UID: \"1ceefd67-0082-4ecb-9d62-48f62ac00b58\") " pod="openshift-ingress-canary/ingress-canary-pkdtz" Sep 09 02:30:42.245825 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:42.245731 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 02:30:42.245825 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.245765 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1dc93f45-045a-4969-8db9-3e224316cbd5-registry-tls\") pod \"image-registry-7c679fdf79-2v4dd\" (UID: \"1dc93f45-045a-4969-8db9-3e224316cbd5\") " pod="openshift-image-registry/image-registry-7c679fdf79-2v4dd" Sep 09 02:30:42.245825 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.245775 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1dc93f45-045a-4969-8db9-3e224316cbd5-ca-trust-extracted\") pod \"image-registry-7c679fdf79-2v4dd\" (UID: \"1dc93f45-045a-4969-8db9-3e224316cbd5\") " pod="openshift-image-registry/image-registry-7c679fdf79-2v4dd" Sep 09 02:30:42.245825 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:42.245787 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98d0408e-2648-467e-a2fa-a553b001c70c-metrics-tls podName:98d0408e-2648-467e-a2fa-a553b001c70c nodeName:}" failed. No retries permitted until 2026-09-09 02:30:42.745768841 +0000 UTC m=+36.110103288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/98d0408e-2648-467e-a2fa-a553b001c70c-metrics-tls") pod "dns-default-frnzb" (UID: "98d0408e-2648-467e-a2fa-a553b001c70c") : secret "dns-default-metrics-tls" not found Sep 09 02:30:42.245825 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.245821 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98d0408e-2648-467e-a2fa-a553b001c70c-tmp-dir\") pod \"dns-default-frnzb\" (UID: \"98d0408e-2648-467e-a2fa-a553b001c70c\") " pod="openshift-dns/dns-default-frnzb" Sep 09 02:30:42.246275 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:42.245840 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 02:30:42.246275 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:42.245851 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c679fdf79-2v4dd: secret "image-registry-tls" not found Sep 09 02:30:42.246275 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.245857 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w92bg\" (UniqueName: \"kubernetes.io/projected/1dc93f45-045a-4969-8db9-3e224316cbd5-kube-api-access-w92bg\") pod \"image-registry-7c679fdf79-2v4dd\" (UID: \"1dc93f45-045a-4969-8db9-3e224316cbd5\") " pod="openshift-image-registry/image-registry-7c679fdf79-2v4dd" Sep 09 02:30:42.246275 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:42.245896 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1dc93f45-045a-4969-8db9-3e224316cbd5-registry-tls podName:1dc93f45-045a-4969-8db9-3e224316cbd5 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:42.745885063 +0000 UTC m=+36.110219523 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1dc93f45-045a-4969-8db9-3e224316cbd5-registry-tls") pod "image-registry-7c679fdf79-2v4dd" (UID: "1dc93f45-045a-4969-8db9-3e224316cbd5") : secret "image-registry-tls" not found Sep 09 02:30:42.246275 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.246182 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98d0408e-2648-467e-a2fa-a553b001c70c-tmp-dir\") pod \"dns-default-frnzb\" (UID: \"98d0408e-2648-467e-a2fa-a553b001c70c\") " pod="openshift-dns/dns-default-frnzb" Sep 09 02:30:42.246275 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.246220 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/98d0408e-2648-467e-a2fa-a553b001c70c-config-volume\") pod \"dns-default-frnzb\" (UID: \"98d0408e-2648-467e-a2fa-a553b001c70c\") " pod="openshift-dns/dns-default-frnzb" Sep 09 02:30:42.246566 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.246549 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1dc93f45-045a-4969-8db9-3e224316cbd5-registry-certificates\") pod \"image-registry-7c679fdf79-2v4dd\" (UID: \"1dc93f45-045a-4969-8db9-3e224316cbd5\") " pod="openshift-image-registry/image-registry-7c679fdf79-2v4dd" Sep 09 02:30:42.247178 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.247151 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1dc93f45-045a-4969-8db9-3e224316cbd5-trusted-ca\") pod \"image-registry-7c679fdf79-2v4dd\" (UID: \"1dc93f45-045a-4969-8db9-3e224316cbd5\") " pod="openshift-image-registry/image-registry-7c679fdf79-2v4dd" Sep 09 02:30:42.250909 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.250883 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1dc93f45-045a-4969-8db9-3e224316cbd5-image-registry-private-configuration\") pod \"image-registry-7c679fdf79-2v4dd\" (UID: \"1dc93f45-045a-4969-8db9-3e224316cbd5\") " pod="openshift-image-registry/image-registry-7c679fdf79-2v4dd" Sep 09 02:30:42.251022 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.250910 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1dc93f45-045a-4969-8db9-3e224316cbd5-installation-pull-secrets\") pod \"image-registry-7c679fdf79-2v4dd\" (UID: \"1dc93f45-045a-4969-8db9-3e224316cbd5\") " pod="openshift-image-registry/image-registry-7c679fdf79-2v4dd" Sep 09 02:30:42.266989 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.266951 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w92bg\" (UniqueName: \"kubernetes.io/projected/1dc93f45-045a-4969-8db9-3e224316cbd5-kube-api-access-w92bg\") pod \"image-registry-7c679fdf79-2v4dd\" (UID: \"1dc93f45-045a-4969-8db9-3e224316cbd5\") " pod="openshift-image-registry/image-registry-7c679fdf79-2v4dd" Sep 09 02:30:42.269131 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.269109 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1dc93f45-045a-4969-8db9-3e224316cbd5-bound-sa-token\") pod \"image-registry-7c679fdf79-2v4dd\" (UID: \"1dc93f45-045a-4969-8db9-3e224316cbd5\") " pod="openshift-image-registry/image-registry-7c679fdf79-2v4dd" Sep 09 02:30:42.269863 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.269842 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-652g4\" (UniqueName: \"kubernetes.io/projected/98d0408e-2648-467e-a2fa-a553b001c70c-kube-api-access-652g4\") pod \"dns-default-frnzb\" (UID: \"98d0408e-2648-467e-a2fa-a553b001c70c\") " pod="openshift-dns/dns-default-frnzb" Sep 09 02:30:42.346464 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.346422 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ceefd67-0082-4ecb-9d62-48f62ac00b58-cert\") pod \"ingress-canary-pkdtz\" (UID: \"1ceefd67-0082-4ecb-9d62-48f62ac00b58\") " pod="openshift-ingress-canary/ingress-canary-pkdtz" Sep 09 02:30:42.346692 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.346668 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mj4n\" (UniqueName: \"kubernetes.io/projected/1ceefd67-0082-4ecb-9d62-48f62ac00b58-kube-api-access-5mj4n\") pod \"ingress-canary-pkdtz\" (UID: \"1ceefd67-0082-4ecb-9d62-48f62ac00b58\") " pod="openshift-ingress-canary/ingress-canary-pkdtz" Sep 09 02:30:42.349140 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.349118 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ceefd67-0082-4ecb-9d62-48f62ac00b58-cert\") pod \"ingress-canary-pkdtz\" (UID: \"1ceefd67-0082-4ecb-9d62-48f62ac00b58\") " pod="openshift-ingress-canary/ingress-canary-pkdtz" Sep 09 02:30:42.356721 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.356693 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mj4n\" (UniqueName: \"kubernetes.io/projected/1ceefd67-0082-4ecb-9d62-48f62ac00b58-kube-api-access-5mj4n\") pod \"ingress-canary-pkdtz\" (UID: \"1ceefd67-0082-4ecb-9d62-48f62ac00b58\") " pod="openshift-ingress-canary/ingress-canary-pkdtz" Sep 09 02:30:42.411483 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.411458 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pkdtz" Sep 09 02:30:42.749943 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.749909 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1dc93f45-045a-4969-8db9-3e224316cbd5-registry-tls\") pod \"image-registry-7c679fdf79-2v4dd\" (UID: \"1dc93f45-045a-4969-8db9-3e224316cbd5\") " pod="openshift-image-registry/image-registry-7c679fdf79-2v4dd" Sep 09 02:30:42.750122 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:42.749988 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98d0408e-2648-467e-a2fa-a553b001c70c-metrics-tls\") pod \"dns-default-frnzb\" (UID: \"98d0408e-2648-467e-a2fa-a553b001c70c\") " pod="openshift-dns/dns-default-frnzb" Sep 09 02:30:42.750122 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:42.750075 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 02:30:42.750122 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:42.750079 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 02:30:42.750122 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:42.750095 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c679fdf79-2v4dd: secret "image-registry-tls" not found Sep 09 02:30:42.750301 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:42.750142 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98d0408e-2648-467e-a2fa-a553b001c70c-metrics-tls podName:98d0408e-2648-467e-a2fa-a553b001c70c nodeName:}" failed. No retries permitted until 2026-09-09 02:30:43.750125014 +0000 UTC m=+37.114459474 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/98d0408e-2648-467e-a2fa-a553b001c70c-metrics-tls") pod "dns-default-frnzb" (UID: "98d0408e-2648-467e-a2fa-a553b001c70c") : secret "dns-default-metrics-tls" not found Sep 09 02:30:42.750301 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:42.750159 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1dc93f45-045a-4969-8db9-3e224316cbd5-registry-tls podName:1dc93f45-045a-4969-8db9-3e224316cbd5 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:43.750151233 +0000 UTC m=+37.114485674 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1dc93f45-045a-4969-8db9-3e224316cbd5-registry-tls") pod "image-registry-7c679fdf79-2v4dd" (UID: "1dc93f45-045a-4969-8db9-3e224316cbd5") : secret "image-registry-tls" not found Sep 09 02:30:43.159698 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:43.159648 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6lzgq" Sep 09 02:30:43.166616 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:43.166590 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 09 02:30:43.425592 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:43.425367 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjstd" event={"ID":"95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1","Type":"ContainerStarted","Data":"e5bf8f424c656c09a707dfa065652bb8f78b820b15a15049f2976f24f27d3b3c"} Sep 09 02:30:43.489260 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:43.489234 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pkdtz"] Sep 09 02:30:43.510849 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:30:43.510819 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ceefd67_0082_4ecb_9d62_48f62ac00b58.slice/crio-4ea7f786ad5d637111eba03a65a69897eff445ddf6df528c0bb92f3925e32e19 WatchSource:0}: Error finding container 4ea7f786ad5d637111eba03a65a69897eff445ddf6df528c0bb92f3925e32e19: Status 404 returned error can't find the container with id 4ea7f786ad5d637111eba03a65a69897eff445ddf6df528c0bb92f3925e32e19 Sep 09 02:30:43.756801 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:43.756741 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98d0408e-2648-467e-a2fa-a553b001c70c-metrics-tls\") pod \"dns-default-frnzb\" (UID: \"98d0408e-2648-467e-a2fa-a553b001c70c\") " pod="openshift-dns/dns-default-frnzb" Sep 09 02:30:43.756923 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:43.756814 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1dc93f45-045a-4969-8db9-3e224316cbd5-registry-tls\") pod \"image-registry-7c679fdf79-2v4dd\" (UID: \"1dc93f45-045a-4969-8db9-3e224316cbd5\") " pod="openshift-image-registry/image-registry-7c679fdf79-2v4dd" Sep 09 02:30:43.756923 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:43.756891 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 02:30:43.756923 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:43.756920 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 02:30:43.757053 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:43.756931 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c679fdf79-2v4dd: secret "image-registry-tls" not found Sep 09 02:30:43.757053 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:43.756973 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98d0408e-2648-467e-a2fa-a553b001c70c-metrics-tls podName:98d0408e-2648-467e-a2fa-a553b001c70c nodeName:}" failed. No retries permitted until 2026-09-09 02:30:45.756940391 +0000 UTC m=+39.121274849 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/98d0408e-2648-467e-a2fa-a553b001c70c-metrics-tls") pod "dns-default-frnzb" (UID: "98d0408e-2648-467e-a2fa-a553b001c70c") : secret "dns-default-metrics-tls" not found Sep 09 02:30:43.757053 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:43.756992 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1dc93f45-045a-4969-8db9-3e224316cbd5-registry-tls podName:1dc93f45-045a-4969-8db9-3e224316cbd5 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:45.756979373 +0000 UTC m=+39.121313814 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1dc93f45-045a-4969-8db9-3e224316cbd5-registry-tls") pod "image-registry-7c679fdf79-2v4dd" (UID: "1dc93f45-045a-4969-8db9-3e224316cbd5") : secret "image-registry-tls" not found Sep 09 02:30:44.431920 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:44.431886 2577 generic.go:356] "Generic (PLEG): container finished" podID="95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1" containerID="e5bf8f424c656c09a707dfa065652bb8f78b820b15a15049f2976f24f27d3b3c" exitCode=0 Sep 09 02:30:44.432622 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:44.431981 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjstd" event={"ID":"95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1","Type":"ContainerDied","Data":"e5bf8f424c656c09a707dfa065652bb8f78b820b15a15049f2976f24f27d3b3c"} Sep 09 02:30:44.433662 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:44.433637 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pkdtz" event={"ID":"1ceefd67-0082-4ecb-9d62-48f62ac00b58","Type":"ContainerStarted","Data":"4ea7f786ad5d637111eba03a65a69897eff445ddf6df528c0bb92f3925e32e19"} Sep 09 02:30:45.439121 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:45.439040 2577 generic.go:356] "Generic (PLEG): container finished" podID="95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1" containerID="6be8b105dddfedb933ea531671624a35e4eb9f9ccf6e55ec9628e9d8c3c3865b" exitCode=0 Sep 09 02:30:45.439121 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:45.439101 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjstd" event={"ID":"95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1","Type":"ContainerDied","Data":"6be8b105dddfedb933ea531671624a35e4eb9f9ccf6e55ec9628e9d8c3c3865b"} Sep 09 02:30:45.775088 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:45.774982 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1dc93f45-045a-4969-8db9-3e224316cbd5-registry-tls\") pod \"image-registry-7c679fdf79-2v4dd\" (UID: \"1dc93f45-045a-4969-8db9-3e224316cbd5\") " pod="openshift-image-registry/image-registry-7c679fdf79-2v4dd" Sep 09 02:30:45.775088 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:45.775073 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98d0408e-2648-467e-a2fa-a553b001c70c-metrics-tls\") pod \"dns-default-frnzb\" (UID: \"98d0408e-2648-467e-a2fa-a553b001c70c\") " pod="openshift-dns/dns-default-frnzb" Sep 09 02:30:45.775312 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:45.775171 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 02:30:45.775312 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:45.775196 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c679fdf79-2v4dd: secret "image-registry-tls" not found Sep 09 02:30:45.775312 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:45.775244 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 02:30:45.775312 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:45.775253 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1dc93f45-045a-4969-8db9-3e224316cbd5-registry-tls podName:1dc93f45-045a-4969-8db9-3e224316cbd5 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:49.775238315 +0000 UTC m=+43.139572756 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1dc93f45-045a-4969-8db9-3e224316cbd5-registry-tls") pod "image-registry-7c679fdf79-2v4dd" (UID: "1dc93f45-045a-4969-8db9-3e224316cbd5") : secret "image-registry-tls" not found Sep 09 02:30:45.775514 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:45.775316 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98d0408e-2648-467e-a2fa-a553b001c70c-metrics-tls podName:98d0408e-2648-467e-a2fa-a553b001c70c nodeName:}" failed. No retries permitted until 2026-09-09 02:30:49.775297162 +0000 UTC m=+43.139631617 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/98d0408e-2648-467e-a2fa-a553b001c70c-metrics-tls") pod "dns-default-frnzb" (UID: "98d0408e-2648-467e-a2fa-a553b001c70c") : secret "dns-default-metrics-tls" not found Sep 09 02:30:46.442940 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:46.442855 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pkdtz" event={"ID":"1ceefd67-0082-4ecb-9d62-48f62ac00b58","Type":"ContainerStarted","Data":"5c5026dbc539e4a588731d9f70f0e318cadc890a5d8f60d66dc5c2c201d16ed9"} Sep 09 02:30:46.446397 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:46.446372 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjstd" event={"ID":"95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1","Type":"ContainerStarted","Data":"c31d382d0162ce1ce13948f4185dfa3c22aad95fd8d1fb93b3b807e694f53265"} Sep 09 02:30:46.461663 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:46.461623 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pkdtz" podStartSLOduration=1.82902857 podStartE2EDuration="4.461610788s" totalImagesPullingTime="2.632582218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:30:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:30:46.460587545 +0000 UTC m=+39.824922008" watchObservedRunningTime="2026-09-09 02:30:46.461610788 +0000 UTC m=+39.825945251" Sep 09 02:30:49.801545 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:49.801506 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1dc93f45-045a-4969-8db9-3e224316cbd5-registry-tls\") pod \"image-registry-7c679fdf79-2v4dd\" (UID: \"1dc93f45-045a-4969-8db9-3e224316cbd5\") " pod="openshift-image-registry/image-registry-7c679fdf79-2v4dd" Sep 09 02:30:49.801925 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:49.801575 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98d0408e-2648-467e-a2fa-a553b001c70c-metrics-tls\") pod \"dns-default-frnzb\" (UID: \"98d0408e-2648-467e-a2fa-a553b001c70c\") " pod="openshift-dns/dns-default-frnzb" Sep 09 02:30:49.801925 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:49.801663 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 02:30:49.801925 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:49.801667 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 02:30:49.801925 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:49.801684 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c679fdf79-2v4dd: secret "image-registry-tls" not found Sep 09 02:30:49.801925 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:49.801741 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98d0408e-2648-467e-a2fa-a553b001c70c-metrics-tls podName:98d0408e-2648-467e-a2fa-a553b001c70c nodeName:}" failed. No retries permitted until 2026-09-09 02:30:57.801723079 +0000 UTC m=+51.166057535 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/98d0408e-2648-467e-a2fa-a553b001c70c-metrics-tls") pod "dns-default-frnzb" (UID: "98d0408e-2648-467e-a2fa-a553b001c70c") : secret "dns-default-metrics-tls" not found Sep 09 02:30:49.801925 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:49.801754 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1dc93f45-045a-4969-8db9-3e224316cbd5-registry-tls podName:1dc93f45-045a-4969-8db9-3e224316cbd5 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:57.801748511 +0000 UTC m=+51.166082951 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1dc93f45-045a-4969-8db9-3e224316cbd5-registry-tls") pod "image-registry-7c679fdf79-2v4dd" (UID: "1dc93f45-045a-4969-8db9-3e224316cbd5") : secret "image-registry-tls" not found Sep 09 02:30:57.859857 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:57.859826 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98d0408e-2648-467e-a2fa-a553b001c70c-metrics-tls\") pod \"dns-default-frnzb\" (UID: \"98d0408e-2648-467e-a2fa-a553b001c70c\") " pod="openshift-dns/dns-default-frnzb" Sep 09 02:30:57.860309 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:30:57.859899 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1dc93f45-045a-4969-8db9-3e224316cbd5-registry-tls\") pod \"image-registry-7c679fdf79-2v4dd\" (UID: \"1dc93f45-045a-4969-8db9-3e224316cbd5\") " pod="openshift-image-registry/image-registry-7c679fdf79-2v4dd" Sep 09 02:30:57.860309 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:57.860020 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 02:30:57.860309 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:57.860024 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 02:30:57.860309 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:57.860042 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c679fdf79-2v4dd: secret "image-registry-tls" not found Sep 09 02:30:57.860309 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:57.860084 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1dc93f45-045a-4969-8db9-3e224316cbd5-registry-tls podName:1dc93f45-045a-4969-8db9-3e224316cbd5 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:13.860067642 +0000 UTC m=+67.224402088 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1dc93f45-045a-4969-8db9-3e224316cbd5-registry-tls") pod "image-registry-7c679fdf79-2v4dd" (UID: "1dc93f45-045a-4969-8db9-3e224316cbd5") : secret "image-registry-tls" not found Sep 09 02:30:57.860309 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:30:57.860099 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98d0408e-2648-467e-a2fa-a553b001c70c-metrics-tls podName:98d0408e-2648-467e-a2fa-a553b001c70c nodeName:}" failed. No retries permitted until 2026-09-09 02:31:13.860091552 +0000 UTC m=+67.224425994 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/98d0408e-2648-467e-a2fa-a553b001c70c-metrics-tls") pod "dns-default-frnzb" (UID: "98d0408e-2648-467e-a2fa-a553b001c70c") : secret "dns-default-metrics-tls" not found Sep 09 02:31:04.213128 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:04.213080 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zjstd" podStartSLOduration=27.801709925 podStartE2EDuration="57.213064794s" totalImagesPullingTime="29.411354869s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 02:30:07 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:30:46.493702617 +0000 UTC m=+39.858037099" watchObservedRunningTime="2026-09-09 02:31:04.213064794 +0000 UTC m=+57.577399265" Sep 09 02:31:04.213702 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:04.213684 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-2lgf6"] Sep 09 02:31:04.217871 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:04.217852 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-2lgf6" Sep 09 02:31:04.221465 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:04.221441 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"config\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-kube-storage-version-migrator-operator\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" type="*v1.ConfigMap" Sep 09 02:31:04.221465 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:04.221439 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kube-storage-version-migrator-operator-dockercfg-mpwmn\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-kube-storage-version-migrator-operator\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-mpwmn\"" type="*v1.Secret" Sep 09 02:31:04.221599 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:04.221528 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-kube-storage-version-migrator-operator\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Sep 09 02:31:04.223894 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:04.223866 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-kube-storage-version-migrator-operator\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Sep 09 02:31:04.225083 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:04.225059 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"serving-cert\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-kube-storage-version-migrator-operator\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" type="*v1.Secret" Sep 09 02:31:04.239873 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:04.239684 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-8s5l5"] Sep 09 02:31:04.242316 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:04.242297 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-2lgf6"] Sep 09 02:31:04.242408 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:04.242383 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-8s5l5" Sep 09 02:31:04.251584 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:04.251564 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 09 02:31:04.251668 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:04.251569 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 09 02:31:04.251668 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:04.251607 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 09 02:31:04.252645 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:04.252632 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-p6jnf\"" Sep 09 02:31:04.266605 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:04.266583 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-8s5l5"] Sep 09 02:31:04.305052 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:04.305032 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb5rw\" (UniqueName: \"kubernetes.io/projected/244d5fd7-a6d4-46fd-baa5-714dbeaacc48-kube-api-access-zb5rw\") pod \"cluster-samples-operator-7cdcfcf76c-8s5l5\" (UID: \"244d5fd7-a6d4-46fd-baa5-714dbeaacc48\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-8s5l5" Sep 09 02:31:04.305162 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:04.305059 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rppcw\" (UniqueName: \"kubernetes.io/projected/d06cd08c-40fb-449c-9b6a-78736d4c362e-kube-api-access-rppcw\") pod \"kube-storage-version-migrator-operator-8d7dd4675-2lgf6\" (UID: \"d06cd08c-40fb-449c-9b6a-78736d4c362e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-2lgf6" Sep 09 02:31:04.305162 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:04.305079 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d06cd08c-40fb-449c-9b6a-78736d4c362e-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-2lgf6\" (UID: \"d06cd08c-40fb-449c-9b6a-78736d4c362e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-2lgf6" Sep 09 02:31:04.305162 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:04.305099 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d06cd08c-40fb-449c-9b6a-78736d4c362e-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-2lgf6\" (UID: \"d06cd08c-40fb-449c-9b6a-78736d4c362e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-2lgf6" Sep 09 02:31:04.305274 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:04.305196 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/244d5fd7-a6d4-46fd-baa5-714dbeaacc48-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-8s5l5\" (UID: \"244d5fd7-a6d4-46fd-baa5-714dbeaacc48\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-8s5l5" Sep 09 02:31:04.405816 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:04.405772 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb5rw\" (UniqueName: \"kubernetes.io/projected/244d5fd7-a6d4-46fd-baa5-714dbeaacc48-kube-api-access-zb5rw\") pod \"cluster-samples-operator-7cdcfcf76c-8s5l5\" (UID: \"244d5fd7-a6d4-46fd-baa5-714dbeaacc48\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-8s5l5" Sep 09 02:31:04.405979 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:04.405825 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rppcw\" (UniqueName: \"kubernetes.io/projected/d06cd08c-40fb-449c-9b6a-78736d4c362e-kube-api-access-rppcw\") pod \"kube-storage-version-migrator-operator-8d7dd4675-2lgf6\" (UID: \"d06cd08c-40fb-449c-9b6a-78736d4c362e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-2lgf6" Sep 09 02:31:04.405979 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:04.405856 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d06cd08c-40fb-449c-9b6a-78736d4c362e-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-2lgf6\" (UID: \"d06cd08c-40fb-449c-9b6a-78736d4c362e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-2lgf6" Sep 09 02:31:04.405979 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:04.405885 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d06cd08c-40fb-449c-9b6a-78736d4c362e-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-2lgf6\" (UID: \"d06cd08c-40fb-449c-9b6a-78736d4c362e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-2lgf6" Sep 09 02:31:04.406285 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:04.406020 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/244d5fd7-a6d4-46fd-baa5-714dbeaacc48-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-8s5l5\" (UID: \"244d5fd7-a6d4-46fd-baa5-714dbeaacc48\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-8s5l5" Sep 09 02:31:04.406285 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:04.406139 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 09 02:31:04.406285 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:04.406208 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/244d5fd7-a6d4-46fd-baa5-714dbeaacc48-samples-operator-tls podName:244d5fd7-a6d4-46fd-baa5-714dbeaacc48 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:04.906192893 +0000 UTC m=+58.270527334 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/244d5fd7-a6d4-46fd-baa5-714dbeaacc48-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-8s5l5" (UID: "244d5fd7-a6d4-46fd-baa5-714dbeaacc48") : secret "samples-operator-tls" not found Sep 09 02:31:04.448121 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:04.448097 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb5rw\" (UniqueName: \"kubernetes.io/projected/244d5fd7-a6d4-46fd-baa5-714dbeaacc48-kube-api-access-zb5rw\") pod \"cluster-samples-operator-7cdcfcf76c-8s5l5\" (UID: \"244d5fd7-a6d4-46fd-baa5-714dbeaacc48\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-8s5l5" Sep 09 02:31:04.909093 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:04.909064 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/244d5fd7-a6d4-46fd-baa5-714dbeaacc48-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-8s5l5\" (UID: \"244d5fd7-a6d4-46fd-baa5-714dbeaacc48\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-8s5l5" Sep 09 02:31:04.909247 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:04.909203 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 09 02:31:04.909295 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:04.909264 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/244d5fd7-a6d4-46fd-baa5-714dbeaacc48-samples-operator-tls podName:244d5fd7-a6d4-46fd-baa5-714dbeaacc48 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:05.909249455 +0000 UTC m=+59.273583922 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/244d5fd7-a6d4-46fd-baa5-714dbeaacc48-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-8s5l5" (UID: "244d5fd7-a6d4-46fd-baa5-714dbeaacc48") : secret "samples-operator-tls" not found Sep 09 02:31:05.168118 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:05.168053 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 09 02:31:05.172437 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:05.172419 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 09 02:31:05.179056 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:05.179037 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d06cd08c-40fb-449c-9b6a-78736d4c362e-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-2lgf6\" (UID: \"d06cd08c-40fb-449c-9b6a-78736d4c362e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-2lgf6" Sep 09 02:31:05.406636 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:05.406605 2577 configmap.go:193] Couldn't get configMap openshift-kube-storage-version-migrator-operator/config: failed to sync configmap cache: timed out waiting for the condition Sep 09 02:31:05.406992 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:05.406687 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d06cd08c-40fb-449c-9b6a-78736d4c362e-config podName:d06cd08c-40fb-449c-9b6a-78736d4c362e nodeName:}" failed. No retries permitted until 2026-09-09 02:31:05.906672608 +0000 UTC m=+59.271007062 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "config" (UniqueName: "kubernetes.io/configmap/d06cd08c-40fb-449c-9b6a-78736d4c362e-config") pod "kube-storage-version-migrator-operator-8d7dd4675-2lgf6" (UID: "d06cd08c-40fb-449c-9b6a-78736d4c362e") : failed to sync configmap cache: timed out waiting for the condition Sep 09 02:31:05.532662 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:05.532634 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-mpwmn\"" Sep 09 02:31:05.649917 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:05.649891 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 09 02:31:05.658564 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:05.658540 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rppcw\" (UniqueName: \"kubernetes.io/projected/d06cd08c-40fb-449c-9b6a-78736d4c362e-kube-api-access-rppcw\") pod \"kube-storage-version-migrator-operator-8d7dd4675-2lgf6\" (UID: \"d06cd08c-40fb-449c-9b6a-78736d4c362e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-2lgf6" Sep 09 02:31:05.807287 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:05.807231 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 09 02:31:05.916408 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:05.916374 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d06cd08c-40fb-449c-9b6a-78736d4c362e-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-2lgf6\" (UID: \"d06cd08c-40fb-449c-9b6a-78736d4c362e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-2lgf6" Sep 09 02:31:05.916544 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:05.916434 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/244d5fd7-a6d4-46fd-baa5-714dbeaacc48-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-8s5l5\" (UID: \"244d5fd7-a6d4-46fd-baa5-714dbeaacc48\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-8s5l5" Sep 09 02:31:05.916617 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:05.916556 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 09 02:31:05.916617 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:05.916612 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/244d5fd7-a6d4-46fd-baa5-714dbeaacc48-samples-operator-tls podName:244d5fd7-a6d4-46fd-baa5-714dbeaacc48 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:07.916596782 +0000 UTC m=+61.280931244 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/244d5fd7-a6d4-46fd-baa5-714dbeaacc48-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-8s5l5" (UID: "244d5fd7-a6d4-46fd-baa5-714dbeaacc48") : secret "samples-operator-tls" not found Sep 09 02:31:05.917034 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:05.917004 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d06cd08c-40fb-449c-9b6a-78736d4c362e-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-2lgf6\" (UID: \"d06cd08c-40fb-449c-9b6a-78736d4c362e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-2lgf6" Sep 09 02:31:06.026517 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:06.026496 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-2lgf6" Sep 09 02:31:06.336866 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:06.336762 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-2lgf6"] Sep 09 02:31:06.340177 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:31:06.340153 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd06cd08c_40fb_449c_9b6a_78736d4c362e.slice/crio-da3ed68671fc371dd436239d501603e467e644949d41dae6ec129412bfc4188a WatchSource:0}: Error finding container da3ed68671fc371dd436239d501603e467e644949d41dae6ec129412bfc4188a: Status 404 returned error can't find the container with id da3ed68671fc371dd436239d501603e467e644949d41dae6ec129412bfc4188a Sep 09 02:31:06.411093 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:06.411072 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-smnqw" Sep 09 02:31:06.497126 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:06.497101 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-2lgf6" event={"ID":"d06cd08c-40fb-449c-9b6a-78736d4c362e","Type":"ContainerStarted","Data":"da3ed68671fc371dd436239d501603e467e644949d41dae6ec129412bfc4188a"} Sep 09 02:31:07.933662 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:07.933630 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/244d5fd7-a6d4-46fd-baa5-714dbeaacc48-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-8s5l5\" (UID: \"244d5fd7-a6d4-46fd-baa5-714dbeaacc48\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-8s5l5" Sep 09 02:31:07.934058 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:07.933744 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 09 02:31:07.934058 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:07.933804 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/244d5fd7-a6d4-46fd-baa5-714dbeaacc48-samples-operator-tls podName:244d5fd7-a6d4-46fd-baa5-714dbeaacc48 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:11.93378317 +0000 UTC m=+65.298117623 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/244d5fd7-a6d4-46fd-baa5-714dbeaacc48-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-8s5l5" (UID: "244d5fd7-a6d4-46fd-baa5-714dbeaacc48") : secret "samples-operator-tls" not found Sep 09 02:31:08.995587 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:08.995506 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-w65x5"] Sep 09 02:31:08.998519 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:08.998503 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w65x5" Sep 09 02:31:09.002152 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:09.002123 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 09 02:31:09.002258 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:09.002203 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 09 02:31:09.002842 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:09.002826 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 09 02:31:09.002908 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:09.002895 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-68fdc\"" Sep 09 02:31:09.003071 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:09.003053 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 09 02:31:09.018006 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:09.017983 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w65x5"] Sep 09 02:31:09.042208 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:09.042188 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9cfp\" (UniqueName: \"kubernetes.io/projected/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-kube-api-access-v9cfp\") pod \"insights-runtime-extractor-w65x5\" (UID: \"d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7\") " pod="openshift-insights/insights-runtime-extractor-w65x5" Sep 09 02:31:09.042304 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:09.042213 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-crio-socket\") pod \"insights-runtime-extractor-w65x5\" (UID: \"d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7\") " pod="openshift-insights/insights-runtime-extractor-w65x5" Sep 09 02:31:09.042304 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:09.042229 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w65x5\" (UID: \"d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7\") " pod="openshift-insights/insights-runtime-extractor-w65x5" Sep 09 02:31:09.042398 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:09.042369 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-data-volume\") pod \"insights-runtime-extractor-w65x5\" (UID: \"d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7\") " pod="openshift-insights/insights-runtime-extractor-w65x5" Sep 09 02:31:09.042442 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:09.042398 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w65x5\" (UID: \"d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7\") " pod="openshift-insights/insights-runtime-extractor-w65x5" Sep 09 02:31:09.143465 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:09.143441 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-data-volume\") pod \"insights-runtime-extractor-w65x5\" (UID: \"d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7\") " pod="openshift-insights/insights-runtime-extractor-w65x5" Sep 09 02:31:09.143557 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:09.143468 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w65x5\" (UID: \"d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7\") " pod="openshift-insights/insights-runtime-extractor-w65x5" Sep 09 02:31:09.143557 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:09.143517 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9cfp\" (UniqueName: \"kubernetes.io/projected/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-kube-api-access-v9cfp\") pod \"insights-runtime-extractor-w65x5\" (UID: \"d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7\") " pod="openshift-insights/insights-runtime-extractor-w65x5" Sep 09 02:31:09.143723 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:09.143599 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 02:31:09.143723 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:09.143628 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-crio-socket\") pod \"insights-runtime-extractor-w65x5\" (UID: \"d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7\") " pod="openshift-insights/insights-runtime-extractor-w65x5" Sep 09 02:31:09.143723 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:09.143674 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-kube-rbac-proxy-cm podName:d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:09.643641553 +0000 UTC m=+63.007975997 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w65x5" (UID: "d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7") : configmap "kube-rbac-proxy" not found Sep 09 02:31:09.143723 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:09.143695 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-crio-socket\") pod \"insights-runtime-extractor-w65x5\" (UID: \"d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7\") " pod="openshift-insights/insights-runtime-extractor-w65x5" Sep 09 02:31:09.143723 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:09.143708 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w65x5\" (UID: \"d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7\") " pod="openshift-insights/insights-runtime-extractor-w65x5" Sep 09 02:31:09.143930 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:09.143776 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 02:31:09.143930 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:09.143798 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-data-volume\") pod \"insights-runtime-extractor-w65x5\" (UID: \"d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7\") " pod="openshift-insights/insights-runtime-extractor-w65x5" Sep 09 02:31:09.143930 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:09.143816 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-insights-runtime-extractor-tls podName:d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:09.643806317 +0000 UTC m=+63.008140757 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w65x5" (UID: "d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7") : secret "insights-runtime-extractor-tls" not found Sep 09 02:31:09.153716 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:09.153696 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9cfp\" (UniqueName: \"kubernetes.io/projected/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-kube-api-access-v9cfp\") pod \"insights-runtime-extractor-w65x5\" (UID: \"d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7\") " pod="openshift-insights/insights-runtime-extractor-w65x5" Sep 09 02:31:09.506061 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:09.506036 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-2lgf6" event={"ID":"d06cd08c-40fb-449c-9b6a-78736d4c362e","Type":"ContainerStarted","Data":"dd242a2c07f9709e098c3e645afce494b6e842980ec353960106f1ffea48852d"} Sep 09 02:31:09.527842 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:09.527799 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-2lgf6" podStartSLOduration=3.189816903 podStartE2EDuration="5.527764825s" totalImagesPullingTime="2.337947922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:31:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:31:09.527232459 +0000 UTC m=+62.891566923" watchObservedRunningTime="2026-09-09 02:31:09.527764825 +0000 UTC m=+62.892099288" Sep 09 02:31:09.647267 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:09.647236 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w65x5\" (UID: \"d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7\") " pod="openshift-insights/insights-runtime-extractor-w65x5" Sep 09 02:31:09.647414 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:09.647338 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w65x5\" (UID: \"d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7\") " pod="openshift-insights/insights-runtime-extractor-w65x5" Sep 09 02:31:09.647414 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:09.647368 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 02:31:09.647502 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:09.647420 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-insights-runtime-extractor-tls podName:d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:10.64740685 +0000 UTC m=+64.011741291 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w65x5" (UID: "d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7") : secret "insights-runtime-extractor-tls" not found Sep 09 02:31:09.647502 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:09.647420 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 02:31:09.647502 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:09.647470 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-kube-rbac-proxy-cm podName:d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:10.647456432 +0000 UTC m=+64.011790890 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w65x5" (UID: "d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7") : configmap "kube-rbac-proxy" not found Sep 09 02:31:10.654489 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:10.654459 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w65x5\" (UID: \"d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7\") " pod="openshift-insights/insights-runtime-extractor-w65x5" Sep 09 02:31:10.654844 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:10.654524 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w65x5\" (UID: \"d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7\") " pod="openshift-insights/insights-runtime-extractor-w65x5" Sep 09 02:31:10.654844 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:10.654589 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 02:31:10.654844 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:10.654607 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 02:31:10.654844 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:10.654653 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-insights-runtime-extractor-tls podName:d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:12.654632999 +0000 UTC m=+66.018967460 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w65x5" (UID: "d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7") : secret "insights-runtime-extractor-tls" not found Sep 09 02:31:10.654844 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:10.654668 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-kube-rbac-proxy-cm podName:d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:12.654661367 +0000 UTC m=+66.018995807 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w65x5" (UID: "d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7") : configmap "kube-rbac-proxy" not found Sep 09 02:31:10.857570 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:10.857551 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pd8dm_19afdc10-5a52-4e9d-a84f-04df6136a1c0/dns-node-resolver/0.log" Sep 09 02:31:11.498709 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:11.498684 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-bf22s"] Sep 09 02:31:11.504419 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:11.504403 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-bf22s" Sep 09 02:31:11.508701 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:11.508680 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 09 02:31:11.508809 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:11.508793 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 09 02:31:11.509905 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:11.509888 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-vvcdb\"" Sep 09 02:31:11.510023 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:11.509910 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 09 02:31:11.510023 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:11.509891 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 09 02:31:11.515005 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:11.514984 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-bf22s"] Sep 09 02:31:11.560651 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:11.560625 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/69d88b34-b509-4d8e-a258-d1927bf2597a-signing-cabundle\") pod \"service-ca-75d99c484-bf22s\" (UID: \"69d88b34-b509-4d8e-a258-d1927bf2597a\") " pod="openshift-service-ca/service-ca-75d99c484-bf22s" Sep 09 02:31:11.560752 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:11.560692 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/69d88b34-b509-4d8e-a258-d1927bf2597a-signing-key\") pod \"service-ca-75d99c484-bf22s\" (UID: \"69d88b34-b509-4d8e-a258-d1927bf2597a\") " pod="openshift-service-ca/service-ca-75d99c484-bf22s" Sep 09 02:31:11.560818 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:11.560795 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls2g6\" (UniqueName: \"kubernetes.io/projected/69d88b34-b509-4d8e-a258-d1927bf2597a-kube-api-access-ls2g6\") pod \"service-ca-75d99c484-bf22s\" (UID: \"69d88b34-b509-4d8e-a258-d1927bf2597a\") " pod="openshift-service-ca/service-ca-75d99c484-bf22s" Sep 09 02:31:11.661177 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:11.661154 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/69d88b34-b509-4d8e-a258-d1927bf2597a-signing-cabundle\") pod \"service-ca-75d99c484-bf22s\" (UID: \"69d88b34-b509-4d8e-a258-d1927bf2597a\") " pod="openshift-service-ca/service-ca-75d99c484-bf22s" Sep 09 02:31:11.661533 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:11.661205 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/69d88b34-b509-4d8e-a258-d1927bf2597a-signing-key\") pod \"service-ca-75d99c484-bf22s\" (UID: \"69d88b34-b509-4d8e-a258-d1927bf2597a\") " pod="openshift-service-ca/service-ca-75d99c484-bf22s" Sep 09 02:31:11.661533 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:11.661406 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls2g6\" (UniqueName: \"kubernetes.io/projected/69d88b34-b509-4d8e-a258-d1927bf2597a-kube-api-access-ls2g6\") pod \"service-ca-75d99c484-bf22s\" (UID: \"69d88b34-b509-4d8e-a258-d1927bf2597a\") " pod="openshift-service-ca/service-ca-75d99c484-bf22s" Sep 09 02:31:11.661868 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:11.661846 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/69d88b34-b509-4d8e-a258-d1927bf2597a-signing-cabundle\") pod \"service-ca-75d99c484-bf22s\" (UID: \"69d88b34-b509-4d8e-a258-d1927bf2597a\") " pod="openshift-service-ca/service-ca-75d99c484-bf22s" Sep 09 02:31:11.663736 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:11.663716 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/69d88b34-b509-4d8e-a258-d1927bf2597a-signing-key\") pod \"service-ca-75d99c484-bf22s\" (UID: \"69d88b34-b509-4d8e-a258-d1927bf2597a\") " pod="openshift-service-ca/service-ca-75d99c484-bf22s" Sep 09 02:31:11.672051 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:11.672033 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls2g6\" (UniqueName: \"kubernetes.io/projected/69d88b34-b509-4d8e-a258-d1927bf2597a-kube-api-access-ls2g6\") pod \"service-ca-75d99c484-bf22s\" (UID: \"69d88b34-b509-4d8e-a258-d1927bf2597a\") " pod="openshift-service-ca/service-ca-75d99c484-bf22s" Sep 09 02:31:11.813119 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:11.813091 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-bf22s" Sep 09 02:31:11.856053 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:11.856013 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tvhb5_54c26b32-0189-4a6a-a569-c2f859d87c8b/node-ca/0.log" Sep 09 02:31:11.963128 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:11.963092 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/244d5fd7-a6d4-46fd-baa5-714dbeaacc48-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-8s5l5\" (UID: \"244d5fd7-a6d4-46fd-baa5-714dbeaacc48\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-8s5l5" Sep 09 02:31:11.963291 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:11.963221 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 09 02:31:11.963355 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:11.963293 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/244d5fd7-a6d4-46fd-baa5-714dbeaacc48-samples-operator-tls podName:244d5fd7-a6d4-46fd-baa5-714dbeaacc48 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:19.96327372 +0000 UTC m=+73.327608174 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/244d5fd7-a6d4-46fd-baa5-714dbeaacc48-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-8s5l5" (UID: "244d5fd7-a6d4-46fd-baa5-714dbeaacc48") : secret "samples-operator-tls" not found Sep 09 02:31:12.088017 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:12.087944 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-bf22s"] Sep 09 02:31:12.090541 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:31:12.090515 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69d88b34_b509_4d8e_a258_d1927bf2597a.slice/crio-60e9cf46332de4d1ef66a3007d2f81cbdf7384520213f90865bf995115a630cb WatchSource:0}: Error finding container 60e9cf46332de4d1ef66a3007d2f81cbdf7384520213f90865bf995115a630cb: Status 404 returned error can't find the container with id 60e9cf46332de4d1ef66a3007d2f81cbdf7384520213f90865bf995115a630cb Sep 09 02:31:12.459654 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:12.459591 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pkdtz_1ceefd67-0082-4ecb-9d62-48f62ac00b58/serve-healthcheck-canary/0.log" Sep 09 02:31:12.514433 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:12.514405 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-bf22s" event={"ID":"69d88b34-b509-4d8e-a258-d1927bf2597a","Type":"ContainerStarted","Data":"60e9cf46332de4d1ef66a3007d2f81cbdf7384520213f90865bf995115a630cb"} Sep 09 02:31:12.669728 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:12.669698 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w65x5\" (UID: \"d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7\") " pod="openshift-insights/insights-runtime-extractor-w65x5" Sep 09 02:31:12.670161 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:12.669786 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w65x5\" (UID: \"d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7\") " pod="openshift-insights/insights-runtime-extractor-w65x5" Sep 09 02:31:12.670161 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:12.669883 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 02:31:12.670161 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:12.669888 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 02:31:12.670161 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:12.669949 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-kube-rbac-proxy-cm podName:d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:16.669933047 +0000 UTC m=+70.034267490 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w65x5" (UID: "d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7") : configmap "kube-rbac-proxy" not found Sep 09 02:31:12.670161 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:12.669990 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-insights-runtime-extractor-tls podName:d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:16.669969724 +0000 UTC m=+70.034304180 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w65x5" (UID: "d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7") : secret "insights-runtime-extractor-tls" not found Sep 09 02:31:12.871235 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:12.871197 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b974ece6-0b9d-4339-aa6f-203ce53dcd85-original-pull-secret\") pod \"global-pull-secret-syncer-6lzgq\" (UID: \"b974ece6-0b9d-4339-aa6f-203ce53dcd85\") " pod="kube-system/global-pull-secret-syncer-6lzgq" Sep 09 02:31:12.874407 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:12.874385 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 09 02:31:12.883764 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:12.883742 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b974ece6-0b9d-4339-aa6f-203ce53dcd85-original-pull-secret\") pod \"global-pull-secret-syncer-6lzgq\" (UID: \"b974ece6-0b9d-4339-aa6f-203ce53dcd85\") " pod="kube-system/global-pull-secret-syncer-6lzgq" Sep 09 02:31:12.971739 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:12.971708 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06180cab-e7c8-4fb8-a719-70072ee0ee36-metrics-certs\") pod \"network-metrics-daemon-mgjfl\" (UID: \"06180cab-e7c8-4fb8-a719-70072ee0ee36\") " pod="openshift-multus/network-metrics-daemon-mgjfl" Sep 09 02:31:12.974581 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:12.974554 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 09 02:31:12.982044 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:12.982021 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 09 02:31:12.982171 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:12.982093 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06180cab-e7c8-4fb8-a719-70072ee0ee36-metrics-certs podName:06180cab-e7c8-4fb8-a719-70072ee0ee36 nodeName:}" failed. No retries permitted until 2026-09-09 02:32:16.982072273 +0000 UTC m=+130.346406731 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06180cab-e7c8-4fb8-a719-70072ee0ee36-metrics-certs") pod "network-metrics-daemon-mgjfl" (UID: "06180cab-e7c8-4fb8-a719-70072ee0ee36") : secret "metrics-daemon-secret" not found Sep 09 02:31:13.072653 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:13.072622 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxzzf\" (UniqueName: \"kubernetes.io/projected/a50c24dc-cc0a-404a-89d0-77e82c417bb5-kube-api-access-nxzzf\") pod \"network-check-target-65ldr\" (UID: \"a50c24dc-cc0a-404a-89d0-77e82c417bb5\") " pod="openshift-network-diagnostics/network-check-target-65ldr" Sep 09 02:31:13.075742 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:13.075725 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 09 02:31:13.085772 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:13.085755 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 09 02:31:13.096707 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:13.096681 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxzzf\" (UniqueName: \"kubernetes.io/projected/a50c24dc-cc0a-404a-89d0-77e82c417bb5-kube-api-access-nxzzf\") pod \"network-check-target-65ldr\" (UID: \"a50c24dc-cc0a-404a-89d0-77e82c417bb5\") " pod="openshift-network-diagnostics/network-check-target-65ldr" Sep 09 02:31:13.170299 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:13.170237 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6lzgq" Sep 09 02:31:13.379704 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:13.379677 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rd54v\"" Sep 09 02:31:13.387895 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:13.387877 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-65ldr" Sep 09 02:31:13.443015 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:13.442578 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6lzgq"] Sep 09 02:31:13.448938 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:31:13.448907 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb974ece6_0b9d_4339_aa6f_203ce53dcd85.slice/crio-921cd573be1b899c2f02c587bf4532d4cc93ea1ec1542b9bf26d1f2b086fdb94 WatchSource:0}: Error finding container 921cd573be1b899c2f02c587bf4532d4cc93ea1ec1542b9bf26d1f2b086fdb94: Status 404 returned error can't find the container with id 921cd573be1b899c2f02c587bf4532d4cc93ea1ec1542b9bf26d1f2b086fdb94 Sep 09 02:31:13.518217 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:13.518190 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6lzgq" event={"ID":"b974ece6-0b9d-4339-aa6f-203ce53dcd85","Type":"ContainerStarted","Data":"921cd573be1b899c2f02c587bf4532d4cc93ea1ec1542b9bf26d1f2b086fdb94"} Sep 09 02:31:13.660278 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:13.660256 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-65ldr"] Sep 09 02:31:13.662287 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:31:13.662262 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda50c24dc_cc0a_404a_89d0_77e82c417bb5.slice/crio-133c48b593a52962bb07e7e32e35e42a61d1bb24c7db95995c46aee30620172b WatchSource:0}: Error finding container 133c48b593a52962bb07e7e32e35e42a61d1bb24c7db95995c46aee30620172b: Status 404 returned error can't find the container with id 133c48b593a52962bb07e7e32e35e42a61d1bb24c7db95995c46aee30620172b Sep 09 02:31:13.880300 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:13.880261 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1dc93f45-045a-4969-8db9-3e224316cbd5-registry-tls\") pod \"image-registry-7c679fdf79-2v4dd\" (UID: \"1dc93f45-045a-4969-8db9-3e224316cbd5\") " pod="openshift-image-registry/image-registry-7c679fdf79-2v4dd" Sep 09 02:31:13.880723 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:13.880345 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98d0408e-2648-467e-a2fa-a553b001c70c-metrics-tls\") pod \"dns-default-frnzb\" (UID: \"98d0408e-2648-467e-a2fa-a553b001c70c\") " pod="openshift-dns/dns-default-frnzb" Sep 09 02:31:13.880723 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:13.880390 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 02:31:13.880723 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:13.880411 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c679fdf79-2v4dd: secret "image-registry-tls" not found Sep 09 02:31:13.880723 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:13.880457 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 02:31:13.880723 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:13.880472 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1dc93f45-045a-4969-8db9-3e224316cbd5-registry-tls podName:1dc93f45-045a-4969-8db9-3e224316cbd5 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:45.880456165 +0000 UTC m=+99.244790606 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1dc93f45-045a-4969-8db9-3e224316cbd5-registry-tls") pod "image-registry-7c679fdf79-2v4dd" (UID: "1dc93f45-045a-4969-8db9-3e224316cbd5") : secret "image-registry-tls" not found Sep 09 02:31:13.880723 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:13.880500 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98d0408e-2648-467e-a2fa-a553b001c70c-metrics-tls podName:98d0408e-2648-467e-a2fa-a553b001c70c nodeName:}" failed. No retries permitted until 2026-09-09 02:31:45.880487694 +0000 UTC m=+99.244822150 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/98d0408e-2648-467e-a2fa-a553b001c70c-metrics-tls") pod "dns-default-frnzb" (UID: "98d0408e-2648-467e-a2fa-a553b001c70c") : secret "dns-default-metrics-tls" not found Sep 09 02:31:14.522820 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:14.522731 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-65ldr" event={"ID":"a50c24dc-cc0a-404a-89d0-77e82c417bb5","Type":"ContainerStarted","Data":"133c48b593a52962bb07e7e32e35e42a61d1bb24c7db95995c46aee30620172b"} Sep 09 02:31:14.524945 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:14.524908 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-bf22s" event={"ID":"69d88b34-b509-4d8e-a258-d1927bf2597a","Type":"ContainerStarted","Data":"e735bb9eaf013109e8af5681edbcf75a24f25389401d6bb10754da2aeef4b6f5"} Sep 09 02:31:14.547727 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:14.547674 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-bf22s" podStartSLOduration=1.430213904 podStartE2EDuration="3.547653167s" totalImagesPullingTime="2.117439263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:31:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:31:14.547222939 +0000 UTC m=+67.911557413" watchObservedRunningTime="2026-09-09 02:31:14.547653167 +0000 UTC m=+67.911987638" Sep 09 02:31:16.714982 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:16.714931 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w65x5\" (UID: \"d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7\") " pod="openshift-insights/insights-runtime-extractor-w65x5" Sep 09 02:31:16.715421 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:16.715184 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 02:31:16.715421 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:16.715250 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-kube-rbac-proxy-cm podName:d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:24.715232474 +0000 UTC m=+78.079566934 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-w65x5" (UID: "d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7") : configmap "kube-rbac-proxy" not found Sep 09 02:31:16.715631 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:16.715092 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w65x5\" (UID: \"d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7\") " pod="openshift-insights/insights-runtime-extractor-w65x5" Sep 09 02:31:16.715856 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:16.715837 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 02:31:16.715933 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:16.715902 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-insights-runtime-extractor-tls podName:d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:24.715887438 +0000 UTC m=+78.080221893 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w65x5" (UID: "d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7") : secret "insights-runtime-extractor-tls" not found Sep 09 02:31:19.556083 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:19.555858 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6lzgq" event={"ID":"b974ece6-0b9d-4339-aa6f-203ce53dcd85","Type":"ContainerStarted","Data":"6dbafa6c96810e06b5431fe8a009856285151dbd973af79dd0fd918500dae518"} Sep 09 02:31:19.557734 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:19.557700 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-65ldr" event={"ID":"a50c24dc-cc0a-404a-89d0-77e82c417bb5","Type":"ContainerStarted","Data":"978c6792595bced8baddb6337e158eb2b3b15808310ca8e98f4a12b79c7d1e17"} Sep 09 02:31:19.557890 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:19.557873 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-65ldr" Sep 09 02:31:19.613039 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:19.612992 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-65ldr" podStartSLOduration=67.01942626 podStartE2EDuration="1m12.61297654s" totalImagesPullingTime="5.59355028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:30:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:31:19.612436769 +0000 UTC m=+72.976771233" watchObservedRunningTime="2026-09-09 02:31:19.61297654 +0000 UTC m=+72.977310994" Sep 09 02:31:19.613380 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:19.613352 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6lzgq" podStartSLOduration=66.788809377 podStartE2EDuration="1m12.613343762s" totalImagesPullingTime="5.824534385s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 02:30:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:31:19.581106026 +0000 UTC m=+72.945440500" watchObservedRunningTime="2026-09-09 02:31:19.613343762 +0000 UTC m=+72.977678233" Sep 09 02:31:20.042973 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:20.042935 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/244d5fd7-a6d4-46fd-baa5-714dbeaacc48-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-8s5l5\" (UID: \"244d5fd7-a6d4-46fd-baa5-714dbeaacc48\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-8s5l5" Sep 09 02:31:20.045393 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:20.045373 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/244d5fd7-a6d4-46fd-baa5-714dbeaacc48-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-8s5l5\" (UID: \"244d5fd7-a6d4-46fd-baa5-714dbeaacc48\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-8s5l5" Sep 09 02:31:20.153179 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:20.153148 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-p6jnf\"" Sep 09 02:31:20.160640 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:20.160613 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-8s5l5" Sep 09 02:31:20.431807 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:20.431722 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-8s5l5"] Sep 09 02:31:20.562160 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:20.562122 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-8s5l5" event={"ID":"244d5fd7-a6d4-46fd-baa5-714dbeaacc48","Type":"ContainerStarted","Data":"3fdfe315f09e60dda49c433aa5db33877885deecbdcee60f0c963f2d757ba67c"} Sep 09 02:31:22.570619 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:22.570534 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-8s5l5" event={"ID":"244d5fd7-a6d4-46fd-baa5-714dbeaacc48","Type":"ContainerStarted","Data":"e788b17744e98ffe85e1954bd4fdef67ee8ec531e6c898c6bb7fad37ed29d7de"} Sep 09 02:31:22.570619 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:22.570615 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-8s5l5" event={"ID":"244d5fd7-a6d4-46fd-baa5-714dbeaacc48","Type":"ContainerStarted","Data":"9d9f83b5b7417d5400f0b12ad34fe94a5b23679a80b5af0e6bada25a0c12eed9"} Sep 09 02:31:22.603131 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:22.603082 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-8s5l5" podStartSLOduration=16.77814069 podStartE2EDuration="18.603064929s" totalImagesPullingTime="1.824924239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:31:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:31:22.602985081 +0000 UTC m=+75.967319564" watchObservedRunningTime="2026-09-09 02:31:22.603064929 +0000 UTC m=+75.967399399" Sep 09 02:31:24.780811 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:24.780773 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w65x5\" (UID: \"d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7\") " pod="openshift-insights/insights-runtime-extractor-w65x5" Sep 09 02:31:24.781194 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:24.780859 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w65x5\" (UID: \"d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7\") " pod="openshift-insights/insights-runtime-extractor-w65x5" Sep 09 02:31:24.781194 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:24.780912 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 02:31:24.781194 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:24.781003 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-insights-runtime-extractor-tls podName:d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:40.780981943 +0000 UTC m=+94.145316399 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-w65x5" (UID: "d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7") : secret "insights-runtime-extractor-tls" not found Sep 09 02:31:24.781373 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:24.781356 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w65x5\" (UID: \"d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7\") " pod="openshift-insights/insights-runtime-extractor-w65x5" Sep 09 02:31:34.105987 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:34.105862 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f89b59b6d-jhjp5"] Sep 09 02:31:34.110279 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:34.110249 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f89b59b6d-jhjp5" Sep 09 02:31:34.111625 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:34.111599 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79cb5cfd46-cf8sp"] Sep 09 02:31:34.114263 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:34.114235 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 09 02:31:34.114444 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:34.114426 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79cb5cfd46-cf8sp" Sep 09 02:31:34.114444 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:34.114436 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 09 02:31:34.114591 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:34.114577 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 09 02:31:34.114893 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:34.114878 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-pr2d8\"" Sep 09 02:31:34.116042 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:34.116025 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 09 02:31:34.118008 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:34.117993 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Sep 09 02:31:34.118245 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:34.118216 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Sep 09 02:31:34.118443 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:34.118428 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Sep 09 02:31:34.125801 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:34.125780 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f89b59b6d-jhjp5"] Sep 09 02:31:34.127302 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:34.127285 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Sep 09 02:31:34.132302 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:34.132282 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79cb5cfd46-cf8sp"] Sep 09 02:31:34.249601 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:34.249567 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62xqw\" (UniqueName: \"kubernetes.io/projected/61f4978b-c4bf-4581-8684-a4bcafed9ad9-kube-api-access-62xqw\") pod \"managed-serviceaccount-addon-agent-5f89b59b6d-jhjp5\" (UID: \"61f4978b-c4bf-4581-8684-a4bcafed9ad9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f89b59b6d-jhjp5" Sep 09 02:31:34.249601 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:34.249604 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/273e6953-291e-49db-bf1f-af8155c5a4ae-additional-service-ca\") pod \"cluster-proxy-proxy-agent-79cb5cfd46-cf8sp\" (UID: \"273e6953-291e-49db-bf1f-af8155c5a4ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79cb5cfd46-cf8sp" Sep 09 02:31:34.249797 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:34.249627 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/273e6953-291e-49db-bf1f-af8155c5a4ae-hub\") pod \"cluster-proxy-proxy-agent-79cb5cfd46-cf8sp\" (UID: \"273e6953-291e-49db-bf1f-af8155c5a4ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79cb5cfd46-cf8sp" Sep 09 02:31:34.249797 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:34.249720 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/273e6953-291e-49db-bf1f-af8155c5a4ae-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-79cb5cfd46-cf8sp\" (UID: \"273e6953-291e-49db-bf1f-af8155c5a4ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79cb5cfd46-cf8sp" Sep 09 02:31:34.249797 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:34.249762 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/273e6953-291e-49db-bf1f-af8155c5a4ae-ca\") pod \"cluster-proxy-proxy-agent-79cb5cfd46-cf8sp\" (UID: \"273e6953-291e-49db-bf1f-af8155c5a4ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79cb5cfd46-cf8sp" Sep 09 02:31:34.249797 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:34.249787 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/273e6953-291e-49db-bf1f-af8155c5a4ae-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-79cb5cfd46-cf8sp\" (UID: \"273e6953-291e-49db-bf1f-af8155c5a4ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79cb5cfd46-cf8sp" Sep 09 02:31:34.249949 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:34.249816 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njb25\" (UniqueName: \"kubernetes.io/projected/273e6953-291e-49db-bf1f-af8155c5a4ae-kube-api-access-njb25\") pod \"cluster-proxy-proxy-agent-79cb5cfd46-cf8sp\" (UID: \"273e6953-291e-49db-bf1f-af8155c5a4ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79cb5cfd46-cf8sp" Sep 09 02:31:34.249949 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:34.249873 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/61f4978b-c4bf-4581-8684-a4bcafed9ad9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5f89b59b6d-jhjp5\" (UID: \"61f4978b-c4bf-4581-8684-a4bcafed9ad9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f89b59b6d-jhjp5" Sep 09 02:31:34.350269 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:34.350234 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/273e6953-291e-49db-bf1f-af8155c5a4ae-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-79cb5cfd46-cf8sp\" (UID: \"273e6953-291e-49db-bf1f-af8155c5a4ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79cb5cfd46-cf8sp" Sep 09 02:31:34.350448 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:34.350294 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/273e6953-291e-49db-bf1f-af8155c5a4ae-ca\") pod \"cluster-proxy-proxy-agent-79cb5cfd46-cf8sp\" (UID: \"273e6953-291e-49db-bf1f-af8155c5a4ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79cb5cfd46-cf8sp" Sep 09 02:31:34.350448 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:34.350325 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/273e6953-291e-49db-bf1f-af8155c5a4ae-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-79cb5cfd46-cf8sp\" (UID: \"273e6953-291e-49db-bf1f-af8155c5a4ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79cb5cfd46-cf8sp" Sep 09 02:31:34.350448 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:34.350343 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njb25\" (UniqueName: \"kubernetes.io/projected/273e6953-291e-49db-bf1f-af8155c5a4ae-kube-api-access-njb25\") pod \"cluster-proxy-proxy-agent-79cb5cfd46-cf8sp\" (UID: \"273e6953-291e-49db-bf1f-af8155c5a4ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79cb5cfd46-cf8sp" Sep 09 02:31:34.350448 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:34.350371 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/61f4978b-c4bf-4581-8684-a4bcafed9ad9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5f89b59b6d-jhjp5\" (UID: \"61f4978b-c4bf-4581-8684-a4bcafed9ad9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f89b59b6d-jhjp5" Sep 09 02:31:34.350448 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:34.350405 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62xqw\" (UniqueName: \"kubernetes.io/projected/61f4978b-c4bf-4581-8684-a4bcafed9ad9-kube-api-access-62xqw\") pod \"managed-serviceaccount-addon-agent-5f89b59b6d-jhjp5\" (UID: \"61f4978b-c4bf-4581-8684-a4bcafed9ad9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f89b59b6d-jhjp5" Sep 09 02:31:34.350685 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:34.350509 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/273e6953-291e-49db-bf1f-af8155c5a4ae-additional-service-ca\") pod \"cluster-proxy-proxy-agent-79cb5cfd46-cf8sp\" (UID: \"273e6953-291e-49db-bf1f-af8155c5a4ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79cb5cfd46-cf8sp" Sep 09 02:31:34.350685 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:34.350625 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/273e6953-291e-49db-bf1f-af8155c5a4ae-hub\") pod \"cluster-proxy-proxy-agent-79cb5cfd46-cf8sp\" (UID: \"273e6953-291e-49db-bf1f-af8155c5a4ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79cb5cfd46-cf8sp" Sep 09 02:31:34.351293 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:34.351251 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/273e6953-291e-49db-bf1f-af8155c5a4ae-additional-service-ca\") pod \"cluster-proxy-proxy-agent-79cb5cfd46-cf8sp\" (UID: \"273e6953-291e-49db-bf1f-af8155c5a4ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79cb5cfd46-cf8sp" Sep 09 02:31:34.353291 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:34.353257 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/273e6953-291e-49db-bf1f-af8155c5a4ae-hub\") pod \"cluster-proxy-proxy-agent-79cb5cfd46-cf8sp\" (UID: \"273e6953-291e-49db-bf1f-af8155c5a4ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79cb5cfd46-cf8sp" Sep 09 02:31:34.353623 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:34.353603 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/61f4978b-c4bf-4581-8684-a4bcafed9ad9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5f89b59b6d-jhjp5\" (UID: \"61f4978b-c4bf-4581-8684-a4bcafed9ad9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f89b59b6d-jhjp5" Sep 09 02:31:34.353741 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:34.353675 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/273e6953-291e-49db-bf1f-af8155c5a4ae-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-79cb5cfd46-cf8sp\" (UID: \"273e6953-291e-49db-bf1f-af8155c5a4ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79cb5cfd46-cf8sp" Sep 09 02:31:34.353741 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:34.353687 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/273e6953-291e-49db-bf1f-af8155c5a4ae-ca\") pod \"cluster-proxy-proxy-agent-79cb5cfd46-cf8sp\" (UID: \"273e6953-291e-49db-bf1f-af8155c5a4ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79cb5cfd46-cf8sp" Sep 09 02:31:34.353858 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:34.353835 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/273e6953-291e-49db-bf1f-af8155c5a4ae-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-79cb5cfd46-cf8sp\" (UID: \"273e6953-291e-49db-bf1f-af8155c5a4ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79cb5cfd46-cf8sp" Sep 09 02:31:34.365791 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:34.365730 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62xqw\" (UniqueName: \"kubernetes.io/projected/61f4978b-c4bf-4581-8684-a4bcafed9ad9-kube-api-access-62xqw\") pod \"managed-serviceaccount-addon-agent-5f89b59b6d-jhjp5\" (UID: \"61f4978b-c4bf-4581-8684-a4bcafed9ad9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f89b59b6d-jhjp5" Sep 09 02:31:34.371430 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:34.371408 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njb25\" (UniqueName: \"kubernetes.io/projected/273e6953-291e-49db-bf1f-af8155c5a4ae-kube-api-access-njb25\") pod \"cluster-proxy-proxy-agent-79cb5cfd46-cf8sp\" (UID: \"273e6953-291e-49db-bf1f-af8155c5a4ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79cb5cfd46-cf8sp" Sep 09 02:31:34.429467 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:34.429437 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f89b59b6d-jhjp5" Sep 09 02:31:34.435641 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:34.435620 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79cb5cfd46-cf8sp" Sep 09 02:31:34.706246 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:34.706166 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f89b59b6d-jhjp5"] Sep 09 02:31:34.708800 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:31:34.708771 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61f4978b_c4bf_4581_8684_a4bcafed9ad9.slice/crio-d1e0fb4908a64f8fb7f553a06dbc978e78f538205c9e11d567b926337aa00879 WatchSource:0}: Error finding container d1e0fb4908a64f8fb7f553a06dbc978e78f538205c9e11d567b926337aa00879: Status 404 returned error can't find the container with id d1e0fb4908a64f8fb7f553a06dbc978e78f538205c9e11d567b926337aa00879 Sep 09 02:31:34.715264 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:34.715246 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79cb5cfd46-cf8sp"] Sep 09 02:31:34.716641 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:31:34.716619 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod273e6953_291e_49db_bf1f_af8155c5a4ae.slice/crio-40a5677d5570dec865d7abc6cb5d56c8e0828e60446328f558d338681dea07c5 WatchSource:0}: Error finding container 40a5677d5570dec865d7abc6cb5d56c8e0828e60446328f558d338681dea07c5: Status 404 returned error can't find the container with id 40a5677d5570dec865d7abc6cb5d56c8e0828e60446328f558d338681dea07c5 Sep 09 02:31:35.614226 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:35.611385 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f89b59b6d-jhjp5" event={"ID":"61f4978b-c4bf-4581-8684-a4bcafed9ad9","Type":"ContainerStarted","Data":"d1e0fb4908a64f8fb7f553a06dbc978e78f538205c9e11d567b926337aa00879"} Sep 09 02:31:35.615630 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:35.615603 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79cb5cfd46-cf8sp" event={"ID":"273e6953-291e-49db-bf1f-af8155c5a4ae","Type":"ContainerStarted","Data":"40a5677d5570dec865d7abc6cb5d56c8e0828e60446328f558d338681dea07c5"} Sep 09 02:31:39.630182 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:39.630147 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f89b59b6d-jhjp5" event={"ID":"61f4978b-c4bf-4581-8684-a4bcafed9ad9","Type":"ContainerStarted","Data":"9011fb3fbc49175460b1f0a429114a8a4f220cb2245a5e594ecb2cd5be61479c"} Sep 09 02:31:39.648204 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:39.648149 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f89b59b6d-jhjp5" podStartSLOduration=0.833499872 podStartE2EDuration="5.64812955s" totalImagesPullingTime="4.814629678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:31:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:31:39.647874883 +0000 UTC m=+93.012209349" watchObservedRunningTime="2026-09-09 02:31:39.64812955 +0000 UTC m=+93.012464013" Sep 09 02:31:40.807219 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:40.807154 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w65x5\" (UID: \"d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7\") " pod="openshift-insights/insights-runtime-extractor-w65x5" Sep 09 02:31:40.809750 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:40.809725 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w65x5\" (UID: \"d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7\") " pod="openshift-insights/insights-runtime-extractor-w65x5" Sep 09 02:31:41.108385 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:41.108301 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w65x5" Sep 09 02:31:41.386339 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:41.386262 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w65x5"] Sep 09 02:31:42.276335 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:31:42.276297 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4c6d9d2_8b3b_4f28_beff_d4c2936e4df7.slice/crio-32ec86b81c6d8ea664c506d8286a665214061d1d162276b6d4f72e085af2351a WatchSource:0}: Error finding container 32ec86b81c6d8ea664c506d8286a665214061d1d162276b6d4f72e085af2351a: Status 404 returned error can't find the container with id 32ec86b81c6d8ea664c506d8286a665214061d1d162276b6d4f72e085af2351a Sep 09 02:31:42.645559 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:42.645026 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79cb5cfd46-cf8sp" event={"ID":"273e6953-291e-49db-bf1f-af8155c5a4ae","Type":"ContainerStarted","Data":"750438904ce24fa8cc389f14f374073b02c582c723f2dd06962130eaebcd6f34"} Sep 09 02:31:42.645559 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:42.645079 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79cb5cfd46-cf8sp" event={"ID":"273e6953-291e-49db-bf1f-af8155c5a4ae","Type":"ContainerStarted","Data":"78be577565138353b9d0fb98f5a9a6d10722d8bd3bb1faae8f4d31736c0a730e"} Sep 09 02:31:42.645559 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:42.645096 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79cb5cfd46-cf8sp" event={"ID":"273e6953-291e-49db-bf1f-af8155c5a4ae","Type":"ContainerStarted","Data":"d288cefd33e4590c0d68c82a191f659fc69c377751f2601d5d2590d5ddfd03c8"} Sep 09 02:31:42.646677 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:42.646651 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w65x5" event={"ID":"d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7","Type":"ContainerStarted","Data":"ec13b83e8041a6ec8a07a6cca190d33e3715232ba7d03de1ae8a914f09e7f802"} Sep 09 02:31:42.646785 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:42.646703 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w65x5" event={"ID":"d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7","Type":"ContainerStarted","Data":"32ec86b81c6d8ea664c506d8286a665214061d1d162276b6d4f72e085af2351a"} Sep 09 02:31:42.670905 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:42.670853 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79cb5cfd46-cf8sp" podStartSLOduration=1.033121023 podStartE2EDuration="8.670837043s" totalImagesPullingTime="7.63771602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:31:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:31:42.670331801 +0000 UTC m=+96.034666264" watchObservedRunningTime="2026-09-09 02:31:42.670837043 +0000 UTC m=+96.035171506" Sep 09 02:31:43.652733 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:43.652697 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w65x5" event={"ID":"d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7","Type":"ContainerStarted","Data":"3454df713823c1a67e5f85331f7ab5095ccd5bf735060972a74baab629d42aa0"} Sep 09 02:31:44.658090 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:44.658057 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w65x5" event={"ID":"d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7","Type":"ContainerStarted","Data":"aee8e7295544414911adeba1ac9e9840af74bb70713cfb29cbb177865808b274"} Sep 09 02:31:44.680036 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:44.679987 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-w65x5" podStartSLOduration=34.841682254 podStartE2EDuration="36.679952028s" totalImagesPullingTime="1.838269774s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 02:31:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:31:44.678622011 +0000 UTC m=+98.042956474" watchObservedRunningTime="2026-09-09 02:31:44.679952028 +0000 UTC m=+98.044286488" Sep 09 02:31:45.948889 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:45.948797 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1dc93f45-045a-4969-8db9-3e224316cbd5-registry-tls\") pod \"image-registry-7c679fdf79-2v4dd\" (UID: \"1dc93f45-045a-4969-8db9-3e224316cbd5\") " pod="openshift-image-registry/image-registry-7c679fdf79-2v4dd" Sep 09 02:31:45.948889 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:45.948854 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98d0408e-2648-467e-a2fa-a553b001c70c-metrics-tls\") pod \"dns-default-frnzb\" (UID: \"98d0408e-2648-467e-a2fa-a553b001c70c\") " pod="openshift-dns/dns-default-frnzb" Sep 09 02:31:45.951243 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:45.951218 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/98d0408e-2648-467e-a2fa-a553b001c70c-metrics-tls\") pod \"dns-default-frnzb\" (UID: \"98d0408e-2648-467e-a2fa-a553b001c70c\") " pod="openshift-dns/dns-default-frnzb" Sep 09 02:31:45.951371 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:45.951353 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1dc93f45-045a-4969-8db9-3e224316cbd5-registry-tls\") pod \"image-registry-7c679fdf79-2v4dd\" (UID: \"1dc93f45-045a-4969-8db9-3e224316cbd5\") " pod="openshift-image-registry/image-registry-7c679fdf79-2v4dd" Sep 09 02:31:46.200619 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:46.200549 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dckqx\"" Sep 09 02:31:46.208188 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:46.208164 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c679fdf79-2v4dd" Sep 09 02:31:46.212312 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:46.212290 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-99mnl\"" Sep 09 02:31:46.219797 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:46.219776 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-frnzb" Sep 09 02:31:46.489116 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:46.489094 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c679fdf79-2v4dd"] Sep 09 02:31:46.491269 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:31:46.491245 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dc93f45_045a_4969_8db9_3e224316cbd5.slice/crio-ccc8f036ddbc9f0cfb92dc22538259357bfce02b75c7ad5b902bc3c29515338e WatchSource:0}: Error finding container ccc8f036ddbc9f0cfb92dc22538259357bfce02b75c7ad5b902bc3c29515338e: Status 404 returned error can't find the container with id ccc8f036ddbc9f0cfb92dc22538259357bfce02b75c7ad5b902bc3c29515338e Sep 09 02:31:46.510498 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:46.510475 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-frnzb"] Sep 09 02:31:46.513688 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:31:46.513665 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98d0408e_2648_467e_a2fa_a553b001c70c.slice/crio-1c6a9160b43bdc694b7d031e7d230e3a8b20a7931253ff4acb206e52b5910402 WatchSource:0}: Error finding container 1c6a9160b43bdc694b7d031e7d230e3a8b20a7931253ff4acb206e52b5910402: Status 404 returned error can't find the container with id 1c6a9160b43bdc694b7d031e7d230e3a8b20a7931253ff4acb206e52b5910402 Sep 09 02:31:46.665334 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:46.665301 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-frnzb" event={"ID":"98d0408e-2648-467e-a2fa-a553b001c70c","Type":"ContainerStarted","Data":"1c6a9160b43bdc694b7d031e7d230e3a8b20a7931253ff4acb206e52b5910402"} Sep 09 02:31:46.666902 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:46.666871 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c679fdf79-2v4dd" event={"ID":"1dc93f45-045a-4969-8db9-3e224316cbd5","Type":"ContainerStarted","Data":"13aeec05d29f00c7f78e754b68a7c08c0b4b437a7657e3998397a65dca43ccdd"} Sep 09 02:31:46.667035 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:46.666909 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c679fdf79-2v4dd" event={"ID":"1dc93f45-045a-4969-8db9-3e224316cbd5","Type":"ContainerStarted","Data":"ccc8f036ddbc9f0cfb92dc22538259357bfce02b75c7ad5b902bc3c29515338e"} Sep 09 02:31:46.667035 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:46.666938 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7c679fdf79-2v4dd" Sep 09 02:31:46.689594 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:46.689544 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7c679fdf79-2v4dd" podStartSLOduration=99.68952835 podStartE2EDuration="1m39.68952835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:31:46.689001673 +0000 UTC m=+100.053336135" watchObservedRunningTime="2026-09-09 02:31:46.68952835 +0000 UTC m=+100.053862836" Sep 09 02:31:47.793992 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:47.792811 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-h7tvf"] Sep 09 02:31:47.810916 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:47.810262 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-h7tvf" Sep 09 02:31:47.818195 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:47.814751 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 09 02:31:47.818195 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:47.814771 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 09 02:31:47.818195 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:47.815138 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 09 02:31:47.818195 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:47.815404 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tj6kt\"" Sep 09 02:31:47.818195 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:47.816350 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 09 02:31:47.818195 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:47.817205 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 09 02:31:47.820200 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:47.819947 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 09 02:31:47.864855 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:47.864822 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b3ae68f-235f-41fe-ba91-9af0909ccdcf-metrics-client-ca\") pod \"node-exporter-h7tvf\" (UID: \"1b3ae68f-235f-41fe-ba91-9af0909ccdcf\") " pod="openshift-monitoring/node-exporter-h7tvf" Sep 09 02:31:47.865046 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:47.864871 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1b3ae68f-235f-41fe-ba91-9af0909ccdcf-node-exporter-accelerators-collector-config\") pod \"node-exporter-h7tvf\" (UID: \"1b3ae68f-235f-41fe-ba91-9af0909ccdcf\") " pod="openshift-monitoring/node-exporter-h7tvf" Sep 09 02:31:47.865046 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:47.864908 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b3ae68f-235f-41fe-ba91-9af0909ccdcf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h7tvf\" (UID: \"1b3ae68f-235f-41fe-ba91-9af0909ccdcf\") " pod="openshift-monitoring/node-exporter-h7tvf" Sep 09 02:31:47.865046 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:47.864947 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1b3ae68f-235f-41fe-ba91-9af0909ccdcf-root\") pod \"node-exporter-h7tvf\" (UID: \"1b3ae68f-235f-41fe-ba91-9af0909ccdcf\") " pod="openshift-monitoring/node-exporter-h7tvf" Sep 09 02:31:47.865046 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:47.864988 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1b3ae68f-235f-41fe-ba91-9af0909ccdcf-node-exporter-textfile\") pod \"node-exporter-h7tvf\" (UID: \"1b3ae68f-235f-41fe-ba91-9af0909ccdcf\") " pod="openshift-monitoring/node-exporter-h7tvf" Sep 09 02:31:47.865046 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:47.865017 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1b3ae68f-235f-41fe-ba91-9af0909ccdcf-node-exporter-wtmp\") pod \"node-exporter-h7tvf\" (UID: \"1b3ae68f-235f-41fe-ba91-9af0909ccdcf\") " pod="openshift-monitoring/node-exporter-h7tvf" Sep 09 02:31:47.865277 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:47.865058 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b3ae68f-235f-41fe-ba91-9af0909ccdcf-sys\") pod \"node-exporter-h7tvf\" (UID: \"1b3ae68f-235f-41fe-ba91-9af0909ccdcf\") " pod="openshift-monitoring/node-exporter-h7tvf" Sep 09 02:31:47.865277 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:47.865088 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b3ae68f-235f-41fe-ba91-9af0909ccdcf-node-exporter-tls\") pod \"node-exporter-h7tvf\" (UID: \"1b3ae68f-235f-41fe-ba91-9af0909ccdcf\") " pod="openshift-monitoring/node-exporter-h7tvf" Sep 09 02:31:47.865277 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:47.865114 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkmfv\" (UniqueName: \"kubernetes.io/projected/1b3ae68f-235f-41fe-ba91-9af0909ccdcf-kube-api-access-vkmfv\") pod \"node-exporter-h7tvf\" (UID: \"1b3ae68f-235f-41fe-ba91-9af0909ccdcf\") " pod="openshift-monitoring/node-exporter-h7tvf" Sep 09 02:31:47.966100 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:47.966060 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b3ae68f-235f-41fe-ba91-9af0909ccdcf-metrics-client-ca\") pod \"node-exporter-h7tvf\" (UID: \"1b3ae68f-235f-41fe-ba91-9af0909ccdcf\") " pod="openshift-monitoring/node-exporter-h7tvf" Sep 09 02:31:47.966299 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:47.966117 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1b3ae68f-235f-41fe-ba91-9af0909ccdcf-node-exporter-accelerators-collector-config\") pod \"node-exporter-h7tvf\" (UID: \"1b3ae68f-235f-41fe-ba91-9af0909ccdcf\") " pod="openshift-monitoring/node-exporter-h7tvf" Sep 09 02:31:47.966299 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:47.966158 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b3ae68f-235f-41fe-ba91-9af0909ccdcf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h7tvf\" (UID: \"1b3ae68f-235f-41fe-ba91-9af0909ccdcf\") " pod="openshift-monitoring/node-exporter-h7tvf" Sep 09 02:31:47.966299 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:47.966197 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1b3ae68f-235f-41fe-ba91-9af0909ccdcf-root\") pod \"node-exporter-h7tvf\" (UID: \"1b3ae68f-235f-41fe-ba91-9af0909ccdcf\") " pod="openshift-monitoring/node-exporter-h7tvf" Sep 09 02:31:47.966299 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:47.966218 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1b3ae68f-235f-41fe-ba91-9af0909ccdcf-node-exporter-textfile\") pod \"node-exporter-h7tvf\" (UID: \"1b3ae68f-235f-41fe-ba91-9af0909ccdcf\") " pod="openshift-monitoring/node-exporter-h7tvf" Sep 09 02:31:47.966299 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:47.966241 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1b3ae68f-235f-41fe-ba91-9af0909ccdcf-node-exporter-wtmp\") pod \"node-exporter-h7tvf\" (UID: \"1b3ae68f-235f-41fe-ba91-9af0909ccdcf\") " pod="openshift-monitoring/node-exporter-h7tvf" Sep 09 02:31:47.966299 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:47.966278 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b3ae68f-235f-41fe-ba91-9af0909ccdcf-sys\") pod \"node-exporter-h7tvf\" (UID: \"1b3ae68f-235f-41fe-ba91-9af0909ccdcf\") " pod="openshift-monitoring/node-exporter-h7tvf" Sep 09 02:31:47.966574 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:47.966311 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b3ae68f-235f-41fe-ba91-9af0909ccdcf-node-exporter-tls\") pod \"node-exporter-h7tvf\" (UID: \"1b3ae68f-235f-41fe-ba91-9af0909ccdcf\") " pod="openshift-monitoring/node-exporter-h7tvf" Sep 09 02:31:47.966574 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:47.966340 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkmfv\" (UniqueName: \"kubernetes.io/projected/1b3ae68f-235f-41fe-ba91-9af0909ccdcf-kube-api-access-vkmfv\") pod \"node-exporter-h7tvf\" (UID: \"1b3ae68f-235f-41fe-ba91-9af0909ccdcf\") " pod="openshift-monitoring/node-exporter-h7tvf" Sep 09 02:31:47.967020 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:47.966989 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1b3ae68f-235f-41fe-ba91-9af0909ccdcf-node-exporter-textfile\") pod \"node-exporter-h7tvf\" (UID: \"1b3ae68f-235f-41fe-ba91-9af0909ccdcf\") " pod="openshift-monitoring/node-exporter-h7tvf" Sep 09 02:31:47.967144 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:47.967120 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1b3ae68f-235f-41fe-ba91-9af0909ccdcf-node-exporter-wtmp\") pod \"node-exporter-h7tvf\" (UID: \"1b3ae68f-235f-41fe-ba91-9af0909ccdcf\") " pod="openshift-monitoring/node-exporter-h7tvf" Sep 09 02:31:47.967220 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:47.967166 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b3ae68f-235f-41fe-ba91-9af0909ccdcf-sys\") pod \"node-exporter-h7tvf\" (UID: \"1b3ae68f-235f-41fe-ba91-9af0909ccdcf\") " pod="openshift-monitoring/node-exporter-h7tvf" Sep 09 02:31:47.967275 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:47.967247 2577 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 09 02:31:47.967325 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:47.967294 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b3ae68f-235f-41fe-ba91-9af0909ccdcf-metrics-client-ca\") pod \"node-exporter-h7tvf\" (UID: \"1b3ae68f-235f-41fe-ba91-9af0909ccdcf\") " pod="openshift-monitoring/node-exporter-h7tvf" Sep 09 02:31:47.967325 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:31:47.967308 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b3ae68f-235f-41fe-ba91-9af0909ccdcf-node-exporter-tls podName:1b3ae68f-235f-41fe-ba91-9af0909ccdcf nodeName:}" failed. No retries permitted until 2026-09-09 02:31:48.467289524 +0000 UTC m=+101.831623982 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/1b3ae68f-235f-41fe-ba91-9af0909ccdcf-node-exporter-tls") pod "node-exporter-h7tvf" (UID: "1b3ae68f-235f-41fe-ba91-9af0909ccdcf") : secret "node-exporter-tls" not found Sep 09 02:31:47.967437 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:47.967366 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1b3ae68f-235f-41fe-ba91-9af0909ccdcf-root\") pod \"node-exporter-h7tvf\" (UID: \"1b3ae68f-235f-41fe-ba91-9af0909ccdcf\") " pod="openshift-monitoring/node-exporter-h7tvf" Sep 09 02:31:47.967575 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:47.967548 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1b3ae68f-235f-41fe-ba91-9af0909ccdcf-node-exporter-accelerators-collector-config\") pod \"node-exporter-h7tvf\" (UID: \"1b3ae68f-235f-41fe-ba91-9af0909ccdcf\") " pod="openshift-monitoring/node-exporter-h7tvf" Sep 09 02:31:47.969429 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:47.969406 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b3ae68f-235f-41fe-ba91-9af0909ccdcf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-h7tvf\" (UID: \"1b3ae68f-235f-41fe-ba91-9af0909ccdcf\") " pod="openshift-monitoring/node-exporter-h7tvf" Sep 09 02:31:47.977066 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:47.977019 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkmfv\" (UniqueName: \"kubernetes.io/projected/1b3ae68f-235f-41fe-ba91-9af0909ccdcf-kube-api-access-vkmfv\") pod \"node-exporter-h7tvf\" (UID: \"1b3ae68f-235f-41fe-ba91-9af0909ccdcf\") " pod="openshift-monitoring/node-exporter-h7tvf" Sep 09 02:31:48.470134 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:48.470099 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b3ae68f-235f-41fe-ba91-9af0909ccdcf-node-exporter-tls\") pod \"node-exporter-h7tvf\" (UID: \"1b3ae68f-235f-41fe-ba91-9af0909ccdcf\") " pod="openshift-monitoring/node-exporter-h7tvf" Sep 09 02:31:48.472409 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:48.472386 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b3ae68f-235f-41fe-ba91-9af0909ccdcf-node-exporter-tls\") pod \"node-exporter-h7tvf\" (UID: \"1b3ae68f-235f-41fe-ba91-9af0909ccdcf\") " pod="openshift-monitoring/node-exporter-h7tvf" Sep 09 02:31:48.729736 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:48.729667 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-h7tvf" Sep 09 02:31:48.772313 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:31:48.772287 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b3ae68f_235f_41fe_ba91_9af0909ccdcf.slice/crio-bf46ec91265b814c9e8b611c4ad8bd134b1a5e1f36149b0d34ad5a2d094475c0 WatchSource:0}: Error finding container bf46ec91265b814c9e8b611c4ad8bd134b1a5e1f36149b0d34ad5a2d094475c0: Status 404 returned error can't find the container with id bf46ec91265b814c9e8b611c4ad8bd134b1a5e1f36149b0d34ad5a2d094475c0 Sep 09 02:31:49.679740 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:49.679697 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-frnzb" event={"ID":"98d0408e-2648-467e-a2fa-a553b001c70c","Type":"ContainerStarted","Data":"dc80a663fa91569720bfc3ec5c267f8309ec7bc36914c2cd7bb7659edd04d49f"} Sep 09 02:31:49.680175 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:49.679777 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-frnzb" Sep 09 02:31:49.680175 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:49.679805 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-frnzb" event={"ID":"98d0408e-2648-467e-a2fa-a553b001c70c","Type":"ContainerStarted","Data":"442ce51636c3262690cedefdd7d6c6d03df9669457e5456132b14259866ae064"} Sep 09 02:31:49.681126 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:49.681100 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h7tvf" event={"ID":"1b3ae68f-235f-41fe-ba91-9af0909ccdcf","Type":"ContainerStarted","Data":"bf46ec91265b814c9e8b611c4ad8bd134b1a5e1f36149b0d34ad5a2d094475c0"} Sep 09 02:31:49.702532 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:49.702488 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-frnzb" podStartSLOduration=66.598424582 podStartE2EDuration="1m8.702472745s" totalImagesPullingTime="2.104048163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:30:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:31:49.701882015 +0000 UTC m=+103.066216497" watchObservedRunningTime="2026-09-09 02:31:49.702472745 +0000 UTC m=+103.066807199" Sep 09 02:31:50.564894 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:50.564863 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-65ldr" Sep 09 02:31:50.686124 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:50.686093 2577 generic.go:356] "Generic (PLEG): container finished" podID="1b3ae68f-235f-41fe-ba91-9af0909ccdcf" containerID="071761c7cebda120abc46a7dd0cf66723c718e5e6f3db0039bce80bb90c8c68b" exitCode=0 Sep 09 02:31:50.686581 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:50.686167 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h7tvf" event={"ID":"1b3ae68f-235f-41fe-ba91-9af0909ccdcf","Type":"ContainerDied","Data":"071761c7cebda120abc46a7dd0cf66723c718e5e6f3db0039bce80bb90c8c68b"} Sep 09 02:31:51.691711 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:51.691673 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h7tvf" event={"ID":"1b3ae68f-235f-41fe-ba91-9af0909ccdcf","Type":"ContainerStarted","Data":"53507213ff6e9f6fb9e811b9768c1724b201dee8152331dfea4ecb4b07cbc45b"} Sep 09 02:31:51.692177 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:51.691734 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-h7tvf" event={"ID":"1b3ae68f-235f-41fe-ba91-9af0909ccdcf","Type":"ContainerStarted","Data":"fada57b381865fab1d9d38301d908d444a993c56dd69376eda9b459277f19705"} Sep 09 02:31:51.719718 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:51.719660 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-h7tvf" podStartSLOduration=3.181381596 podStartE2EDuration="4.719644372s" totalImagesPullingTime="1.444773078s" totalInitContainerRuntime="93.489698ms" isStatefulPod=true podCreationTimestamp="2026-09-09 02:31:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:31:51.717711689 +0000 UTC m=+105.082046151" watchObservedRunningTime="2026-09-09 02:31:51.719644372 +0000 UTC m=+105.083978834" Sep 09 02:31:56.612238 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:56.612205 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c679fdf79-2v4dd"] Sep 09 02:31:59.689014 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:31:59.688979 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-frnzb" Sep 09 02:32:06.616524 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:06.616493 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7c679fdf79-2v4dd" Sep 09 02:32:16.995854 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:16.995817 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06180cab-e7c8-4fb8-a719-70072ee0ee36-metrics-certs\") pod \"network-metrics-daemon-mgjfl\" (UID: \"06180cab-e7c8-4fb8-a719-70072ee0ee36\") " pod="openshift-multus/network-metrics-daemon-mgjfl" Sep 09 02:32:16.998279 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:16.998255 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06180cab-e7c8-4fb8-a719-70072ee0ee36-metrics-certs\") pod \"network-metrics-daemon-mgjfl\" (UID: \"06180cab-e7c8-4fb8-a719-70072ee0ee36\") " pod="openshift-multus/network-metrics-daemon-mgjfl" Sep 09 02:32:17.273988 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:17.273882 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-shl9x\"" Sep 09 02:32:17.281115 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:17.281093 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mgjfl" Sep 09 02:32:17.564075 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:17.564048 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mgjfl"] Sep 09 02:32:17.565873 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:32:17.565844 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06180cab_e7c8_4fb8_a719_70072ee0ee36.slice/crio-a81f0b2d6532d1eb27269f5b28675212e1f6aa01a76ca48adc850d48eedb85cf WatchSource:0}: Error finding container a81f0b2d6532d1eb27269f5b28675212e1f6aa01a76ca48adc850d48eedb85cf: Status 404 returned error can't find the container with id a81f0b2d6532d1eb27269f5b28675212e1f6aa01a76ca48adc850d48eedb85cf Sep 09 02:32:17.777323 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:17.777292 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mgjfl" event={"ID":"06180cab-e7c8-4fb8-a719-70072ee0ee36","Type":"ContainerStarted","Data":"a81f0b2d6532d1eb27269f5b28675212e1f6aa01a76ca48adc850d48eedb85cf"} Sep 09 02:32:18.782805 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:18.782771 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mgjfl" event={"ID":"06180cab-e7c8-4fb8-a719-70072ee0ee36","Type":"ContainerStarted","Data":"3db4a4b32af163ee0d9de56cf3093d35bf414475b3de822750acfddb2f32052e"} Sep 09 02:32:18.783177 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:18.782824 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mgjfl" event={"ID":"06180cab-e7c8-4fb8-a719-70072ee0ee36","Type":"ContainerStarted","Data":"223679d9a087c02c1498cc9427d2d08e1720bb9c34b8c38000c0eb42bc596c30"} Sep 09 02:32:19.787515 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:19.787478 2577 generic.go:356] "Generic (PLEG): container finished" podID="d06cd08c-40fb-449c-9b6a-78736d4c362e" containerID="dd242a2c07f9709e098c3e645afce494b6e842980ec353960106f1ffea48852d" exitCode=0 Sep 09 02:32:19.787928 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:19.787560 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-2lgf6" event={"ID":"d06cd08c-40fb-449c-9b6a-78736d4c362e","Type":"ContainerDied","Data":"dd242a2c07f9709e098c3e645afce494b6e842980ec353960106f1ffea48852d"} Sep 09 02:32:19.788025 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:19.787985 2577 scope.go:117] "RemoveContainer" containerID="dd242a2c07f9709e098c3e645afce494b6e842980ec353960106f1ffea48852d" Sep 09 02:32:19.808012 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:19.807948 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mgjfl" podStartSLOduration=131.821794621 podStartE2EDuration="2m12.807935266s" totalImagesPullingTime="986.140645ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:30:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:32:18.809430515 +0000 UTC m=+132.173764975" watchObservedRunningTime="2026-09-09 02:32:19.807935266 +0000 UTC m=+133.172269728" Sep 09 02:32:20.793902 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:20.793861 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-2lgf6" event={"ID":"d06cd08c-40fb-449c-9b6a-78736d4c362e","Type":"ContainerStarted","Data":"0b901645ba096cecda4dc629e7a991def02927be5bbc4881ecc7124cdff3fb1b"} Sep 09 02:32:21.631756 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:21.631714 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7c679fdf79-2v4dd" podUID="1dc93f45-045a-4969-8db9-3e224316cbd5" containerName="registry" containerID="cri-o://13aeec05d29f00c7f78e754b68a7c08c0b4b437a7657e3998397a65dca43ccdd" gracePeriod=30 Sep 09 02:32:21.799719 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:21.799688 2577 generic.go:356] "Generic (PLEG): container finished" podID="1dc93f45-045a-4969-8db9-3e224316cbd5" containerID="13aeec05d29f00c7f78e754b68a7c08c0b4b437a7657e3998397a65dca43ccdd" exitCode=0 Sep 09 02:32:21.800055 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:21.799759 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c679fdf79-2v4dd" event={"ID":"1dc93f45-045a-4969-8db9-3e224316cbd5","Type":"ContainerDied","Data":"13aeec05d29f00c7f78e754b68a7c08c0b4b437a7657e3998397a65dca43ccdd"} Sep 09 02:32:21.867858 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:21.867824 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c679fdf79-2v4dd" Sep 09 02:32:21.932876 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:21.932781 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1dc93f45-045a-4969-8db9-3e224316cbd5-trusted-ca\") pod \"1dc93f45-045a-4969-8db9-3e224316cbd5\" (UID: \"1dc93f45-045a-4969-8db9-3e224316cbd5\") " Sep 09 02:32:21.932876 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:21.932842 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1dc93f45-045a-4969-8db9-3e224316cbd5-ca-trust-extracted\") pod \"1dc93f45-045a-4969-8db9-3e224316cbd5\" (UID: \"1dc93f45-045a-4969-8db9-3e224316cbd5\") " Sep 09 02:32:21.932876 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:21.932867 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w92bg\" (UniqueName: \"kubernetes.io/projected/1dc93f45-045a-4969-8db9-3e224316cbd5-kube-api-access-w92bg\") pod \"1dc93f45-045a-4969-8db9-3e224316cbd5\" (UID: \"1dc93f45-045a-4969-8db9-3e224316cbd5\") " Sep 09 02:32:21.933191 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:21.932894 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1dc93f45-045a-4969-8db9-3e224316cbd5-installation-pull-secrets\") pod \"1dc93f45-045a-4969-8db9-3e224316cbd5\" (UID: \"1dc93f45-045a-4969-8db9-3e224316cbd5\") " Sep 09 02:32:21.933191 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:21.932913 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1dc93f45-045a-4969-8db9-3e224316cbd5-registry-certificates\") pod \"1dc93f45-045a-4969-8db9-3e224316cbd5\" (UID: \"1dc93f45-045a-4969-8db9-3e224316cbd5\") " Sep 09 02:32:21.933191 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:21.932928 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1dc93f45-045a-4969-8db9-3e224316cbd5-registry-tls\") pod \"1dc93f45-045a-4969-8db9-3e224316cbd5\" (UID: \"1dc93f45-045a-4969-8db9-3e224316cbd5\") " Sep 09 02:32:21.933191 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:21.932944 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1dc93f45-045a-4969-8db9-3e224316cbd5-image-registry-private-configuration\") pod \"1dc93f45-045a-4969-8db9-3e224316cbd5\" (UID: \"1dc93f45-045a-4969-8db9-3e224316cbd5\") " Sep 09 02:32:21.933191 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:21.932999 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1dc93f45-045a-4969-8db9-3e224316cbd5-bound-sa-token\") pod \"1dc93f45-045a-4969-8db9-3e224316cbd5\" (UID: \"1dc93f45-045a-4969-8db9-3e224316cbd5\") " Sep 09 02:32:21.933429 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:21.933379 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dc93f45-045a-4969-8db9-3e224316cbd5-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "1dc93f45-045a-4969-8db9-3e224316cbd5" (UID: "1dc93f45-045a-4969-8db9-3e224316cbd5"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:32:21.933478 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:21.933455 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dc93f45-045a-4969-8db9-3e224316cbd5-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "1dc93f45-045a-4969-8db9-3e224316cbd5" (UID: "1dc93f45-045a-4969-8db9-3e224316cbd5"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:32:21.935630 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:21.935592 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dc93f45-045a-4969-8db9-3e224316cbd5-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1dc93f45-045a-4969-8db9-3e224316cbd5" (UID: "1dc93f45-045a-4969-8db9-3e224316cbd5"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:32:21.935764 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:21.935699 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dc93f45-045a-4969-8db9-3e224316cbd5-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "1dc93f45-045a-4969-8db9-3e224316cbd5" (UID: "1dc93f45-045a-4969-8db9-3e224316cbd5"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:32:21.935830 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:21.935789 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dc93f45-045a-4969-8db9-3e224316cbd5-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "1dc93f45-045a-4969-8db9-3e224316cbd5" (UID: "1dc93f45-045a-4969-8db9-3e224316cbd5"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:32:21.935940 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:21.935923 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dc93f45-045a-4969-8db9-3e224316cbd5-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "1dc93f45-045a-4969-8db9-3e224316cbd5" (UID: "1dc93f45-045a-4969-8db9-3e224316cbd5"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:32:21.936126 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:21.936053 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dc93f45-045a-4969-8db9-3e224316cbd5-kube-api-access-w92bg" (OuterVolumeSpecName: "kube-api-access-w92bg") pod "1dc93f45-045a-4969-8db9-3e224316cbd5" (UID: "1dc93f45-045a-4969-8db9-3e224316cbd5"). InnerVolumeSpecName "kube-api-access-w92bg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:32:21.941658 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:21.941630 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dc93f45-045a-4969-8db9-3e224316cbd5-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "1dc93f45-045a-4969-8db9-3e224316cbd5" (UID: "1dc93f45-045a-4969-8db9-3e224316cbd5"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:32:22.033923 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:22.033873 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1dc93f45-045a-4969-8db9-3e224316cbd5-trusted-ca\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:32:22.033923 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:22.033914 2577 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1dc93f45-045a-4969-8db9-3e224316cbd5-ca-trust-extracted\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:32:22.033923 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:22.033923 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w92bg\" (UniqueName: \"kubernetes.io/projected/1dc93f45-045a-4969-8db9-3e224316cbd5-kube-api-access-w92bg\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:32:22.033923 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:22.033933 2577 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1dc93f45-045a-4969-8db9-3e224316cbd5-installation-pull-secrets\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:32:22.034207 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:22.033942 2577 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1dc93f45-045a-4969-8db9-3e224316cbd5-registry-certificates\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:32:22.034207 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:22.033951 2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1dc93f45-045a-4969-8db9-3e224316cbd5-registry-tls\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:32:22.034207 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:22.033982 2577 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1dc93f45-045a-4969-8db9-3e224316cbd5-image-registry-private-configuration\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:32:22.034207 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:22.033990 2577 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1dc93f45-045a-4969-8db9-3e224316cbd5-bound-sa-token\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:32:22.805017 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:22.804987 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c679fdf79-2v4dd" Sep 09 02:32:22.805516 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:22.804983 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c679fdf79-2v4dd" event={"ID":"1dc93f45-045a-4969-8db9-3e224316cbd5","Type":"ContainerDied","Data":"ccc8f036ddbc9f0cfb92dc22538259357bfce02b75c7ad5b902bc3c29515338e"} Sep 09 02:32:22.805516 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:22.805114 2577 scope.go:117] "RemoveContainer" containerID="13aeec05d29f00c7f78e754b68a7c08c0b4b437a7657e3998397a65dca43ccdd" Sep 09 02:32:22.828128 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:22.828099 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c679fdf79-2v4dd"] Sep 09 02:32:22.837017 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:22.836995 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7c679fdf79-2v4dd"] Sep 09 02:32:23.166849 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:32:23.164682 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dc93f45-045a-4969-8db9-3e224316cbd5" path="/var/lib/kubelet/pods/1dc93f45-045a-4969-8db9-3e224316cbd5/volumes" Sep 09 02:35:07.072100 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:35:07.072072 2577 kubelet.go:1648] "Image garbage collection succeeded" Sep 09 02:35:54.657815 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:35:54.657780 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nx278"] Sep 09 02:35:54.658311 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:35:54.658040 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dc93f45-045a-4969-8db9-3e224316cbd5" containerName="registry" Sep 09 02:35:54.658311 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:35:54.658052 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dc93f45-045a-4969-8db9-3e224316cbd5" containerName="registry" Sep 09 02:35:54.658311 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:35:54.658103 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dc93f45-045a-4969-8db9-3e224316cbd5" containerName="registry" Sep 09 02:35:54.660887 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:35:54.660872 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nx278" Sep 09 02:35:54.664088 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:35:54.664069 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-gdxls\"" Sep 09 02:35:54.664252 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:35:54.664236 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 09 02:35:54.665017 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:35:54.665001 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 09 02:35:54.679382 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:35:54.679359 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nx278"] Sep 09 02:35:54.763854 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:35:54.763820 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b86d9f31-ba26-483e-a864-fb4e04998e63-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nx278\" (UID: \"b86d9f31-ba26-483e-a864-fb4e04998e63\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nx278" Sep 09 02:35:54.764054 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:35:54.763894 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b86d9f31-ba26-483e-a864-fb4e04998e63-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nx278\" (UID: \"b86d9f31-ba26-483e-a864-fb4e04998e63\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nx278" Sep 09 02:35:54.764054 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:35:54.763922 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkd67\" (UniqueName: \"kubernetes.io/projected/b86d9f31-ba26-483e-a864-fb4e04998e63-kube-api-access-vkd67\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nx278\" (UID: \"b86d9f31-ba26-483e-a864-fb4e04998e63\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nx278" Sep 09 02:35:54.865219 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:35:54.865168 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b86d9f31-ba26-483e-a864-fb4e04998e63-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nx278\" (UID: \"b86d9f31-ba26-483e-a864-fb4e04998e63\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nx278" Sep 09 02:35:54.865395 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:35:54.865250 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b86d9f31-ba26-483e-a864-fb4e04998e63-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nx278\" (UID: \"b86d9f31-ba26-483e-a864-fb4e04998e63\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nx278" Sep 09 02:35:54.865395 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:35:54.865273 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkd67\" (UniqueName: \"kubernetes.io/projected/b86d9f31-ba26-483e-a864-fb4e04998e63-kube-api-access-vkd67\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nx278\" (UID: \"b86d9f31-ba26-483e-a864-fb4e04998e63\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nx278" Sep 09 02:35:54.865567 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:35:54.865549 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b86d9f31-ba26-483e-a864-fb4e04998e63-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nx278\" (UID: \"b86d9f31-ba26-483e-a864-fb4e04998e63\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nx278" Sep 09 02:35:54.865605 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:35:54.865591 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b86d9f31-ba26-483e-a864-fb4e04998e63-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nx278\" (UID: \"b86d9f31-ba26-483e-a864-fb4e04998e63\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nx278" Sep 09 02:35:54.875233 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:35:54.875207 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkd67\" (UniqueName: \"kubernetes.io/projected/b86d9f31-ba26-483e-a864-fb4e04998e63-kube-api-access-vkd67\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nx278\" (UID: \"b86d9f31-ba26-483e-a864-fb4e04998e63\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nx278" Sep 09 02:35:54.969298 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:35:54.969239 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nx278" Sep 09 02:35:55.240910 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:35:55.240833 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nx278"] Sep 09 02:35:55.244023 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:35:55.243996 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb86d9f31_ba26_483e_a864_fb4e04998e63.slice/crio-bce64eeb6752145a8d6d8507bdffaa36dd1c1cd5b91f2616b39ccf1b5acddcb4 WatchSource:0}: Error finding container bce64eeb6752145a8d6d8507bdffaa36dd1c1cd5b91f2616b39ccf1b5acddcb4: Status 404 returned error can't find the container with id bce64eeb6752145a8d6d8507bdffaa36dd1c1cd5b91f2616b39ccf1b5acddcb4 Sep 09 02:35:55.245816 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:35:55.245796 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 02:35:55.508643 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:35:55.508570 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nx278" event={"ID":"b86d9f31-ba26-483e-a864-fb4e04998e63","Type":"ContainerStarted","Data":"bce64eeb6752145a8d6d8507bdffaa36dd1c1cd5b91f2616b39ccf1b5acddcb4"} Sep 09 02:36:01.532654 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:01.532609 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nx278" event={"ID":"b86d9f31-ba26-483e-a864-fb4e04998e63","Type":"ContainerStarted","Data":"0dbd7eb4aca3bc5d10d10abe76e1bcc41443e6feccae7c376344972d0890e41e"} Sep 09 02:36:02.536885 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:02.536847 2577 generic.go:356] "Generic (PLEG): container finished" podID="b86d9f31-ba26-483e-a864-fb4e04998e63" containerID="0dbd7eb4aca3bc5d10d10abe76e1bcc41443e6feccae7c376344972d0890e41e" exitCode=0 Sep 09 02:36:02.537374 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:02.536911 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nx278" event={"ID":"b86d9f31-ba26-483e-a864-fb4e04998e63","Type":"ContainerDied","Data":"0dbd7eb4aca3bc5d10d10abe76e1bcc41443e6feccae7c376344972d0890e41e"} Sep 09 02:36:09.561540 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:09.561512 2577 generic.go:356] "Generic (PLEG): container finished" podID="b86d9f31-ba26-483e-a864-fb4e04998e63" containerID="90c0d3b28354319082553e708ae90d8c94165f0637f58125c5305669f5ceeabc" exitCode=0 Sep 09 02:36:09.561837 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:09.561596 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nx278" event={"ID":"b86d9f31-ba26-483e-a864-fb4e04998e63","Type":"ContainerDied","Data":"90c0d3b28354319082553e708ae90d8c94165f0637f58125c5305669f5ceeabc"} Sep 09 02:36:18.597564 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:18.597529 2577 generic.go:356] "Generic (PLEG): container finished" podID="b86d9f31-ba26-483e-a864-fb4e04998e63" containerID="bf63b9ff71124436843e3295a190c830a621525d5100606a710ce48c5a58505e" exitCode=0 Sep 09 02:36:18.597915 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:18.597619 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nx278" event={"ID":"b86d9f31-ba26-483e-a864-fb4e04998e63","Type":"ContainerDied","Data":"bf63b9ff71124436843e3295a190c830a621525d5100606a710ce48c5a58505e"} Sep 09 02:36:19.719016 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:19.718995 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nx278" Sep 09 02:36:19.752373 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:19.752341 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b86d9f31-ba26-483e-a864-fb4e04998e63-util\") pod \"b86d9f31-ba26-483e-a864-fb4e04998e63\" (UID: \"b86d9f31-ba26-483e-a864-fb4e04998e63\") " Sep 09 02:36:19.752540 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:19.752395 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkd67\" (UniqueName: \"kubernetes.io/projected/b86d9f31-ba26-483e-a864-fb4e04998e63-kube-api-access-vkd67\") pod \"b86d9f31-ba26-483e-a864-fb4e04998e63\" (UID: \"b86d9f31-ba26-483e-a864-fb4e04998e63\") " Sep 09 02:36:19.752540 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:19.752422 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b86d9f31-ba26-483e-a864-fb4e04998e63-bundle\") pod \"b86d9f31-ba26-483e-a864-fb4e04998e63\" (UID: \"b86d9f31-ba26-483e-a864-fb4e04998e63\") " Sep 09 02:36:19.753169 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:19.753138 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b86d9f31-ba26-483e-a864-fb4e04998e63-bundle" (OuterVolumeSpecName: "bundle") pod "b86d9f31-ba26-483e-a864-fb4e04998e63" (UID: "b86d9f31-ba26-483e-a864-fb4e04998e63"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:36:19.754716 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:19.754688 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b86d9f31-ba26-483e-a864-fb4e04998e63-kube-api-access-vkd67" (OuterVolumeSpecName: "kube-api-access-vkd67") pod "b86d9f31-ba26-483e-a864-fb4e04998e63" (UID: "b86d9f31-ba26-483e-a864-fb4e04998e63"). InnerVolumeSpecName "kube-api-access-vkd67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:36:19.756577 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:19.756556 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b86d9f31-ba26-483e-a864-fb4e04998e63-util" (OuterVolumeSpecName: "util") pod "b86d9f31-ba26-483e-a864-fb4e04998e63" (UID: "b86d9f31-ba26-483e-a864-fb4e04998e63"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:36:19.853016 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:19.852920 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b86d9f31-ba26-483e-a864-fb4e04998e63-util\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:36:19.853016 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:19.852948 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkd67\" (UniqueName: \"kubernetes.io/projected/b86d9f31-ba26-483e-a864-fb4e04998e63-kube-api-access-vkd67\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:36:19.853016 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:19.852978 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b86d9f31-ba26-483e-a864-fb4e04998e63-bundle\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:36:20.605725 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:20.605692 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nx278" event={"ID":"b86d9f31-ba26-483e-a864-fb4e04998e63","Type":"ContainerDied","Data":"bce64eeb6752145a8d6d8507bdffaa36dd1c1cd5b91f2616b39ccf1b5acddcb4"} Sep 09 02:36:20.605875 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:20.605734 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bce64eeb6752145a8d6d8507bdffaa36dd1c1cd5b91f2616b39ccf1b5acddcb4" Sep 09 02:36:20.605875 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:20.605732 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540nx278" Sep 09 02:36:27.889751 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:27.889702 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cwhq6"] Sep 09 02:36:27.890212 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:27.889975 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b86d9f31-ba26-483e-a864-fb4e04998e63" containerName="extract" Sep 09 02:36:27.890212 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:27.889986 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b86d9f31-ba26-483e-a864-fb4e04998e63" containerName="extract" Sep 09 02:36:27.890212 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:27.889997 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b86d9f31-ba26-483e-a864-fb4e04998e63" containerName="pull" Sep 09 02:36:27.890212 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:27.890001 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b86d9f31-ba26-483e-a864-fb4e04998e63" containerName="pull" Sep 09 02:36:27.890212 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:27.890024 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b86d9f31-ba26-483e-a864-fb4e04998e63" containerName="util" Sep 09 02:36:27.890212 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:27.890030 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b86d9f31-ba26-483e-a864-fb4e04998e63" containerName="util" Sep 09 02:36:27.890212 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:27.890073 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b86d9f31-ba26-483e-a864-fb4e04998e63" containerName="extract" Sep 09 02:36:27.893845 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:27.893829 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cwhq6" Sep 09 02:36:27.897869 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:27.897846 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 09 02:36:27.898015 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:27.897846 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-slvcx\"" Sep 09 02:36:27.898200 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:27.898186 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 09 02:36:27.914463 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:27.914440 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cwhq6"] Sep 09 02:36:28.012429 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:28.012399 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fcda5510-01bf-47aa-9c9d-b238ed2781ab-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-cwhq6\" (UID: \"fcda5510-01bf-47aa-9c9d-b238ed2781ab\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cwhq6" Sep 09 02:36:28.012570 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:28.012437 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxgll\" (UniqueName: \"kubernetes.io/projected/fcda5510-01bf-47aa-9c9d-b238ed2781ab-kube-api-access-lxgll\") pod \"cert-manager-operator-controller-manager-55c444bf86-cwhq6\" (UID: \"fcda5510-01bf-47aa-9c9d-b238ed2781ab\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cwhq6" Sep 09 02:36:28.112863 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:28.112822 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fcda5510-01bf-47aa-9c9d-b238ed2781ab-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-cwhq6\" (UID: \"fcda5510-01bf-47aa-9c9d-b238ed2781ab\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cwhq6" Sep 09 02:36:28.113027 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:28.112872 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxgll\" (UniqueName: \"kubernetes.io/projected/fcda5510-01bf-47aa-9c9d-b238ed2781ab-kube-api-access-lxgll\") pod \"cert-manager-operator-controller-manager-55c444bf86-cwhq6\" (UID: \"fcda5510-01bf-47aa-9c9d-b238ed2781ab\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cwhq6" Sep 09 02:36:28.113325 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:28.113302 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fcda5510-01bf-47aa-9c9d-b238ed2781ab-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-cwhq6\" (UID: \"fcda5510-01bf-47aa-9c9d-b238ed2781ab\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cwhq6" Sep 09 02:36:28.123649 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:28.123628 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxgll\" (UniqueName: \"kubernetes.io/projected/fcda5510-01bf-47aa-9c9d-b238ed2781ab-kube-api-access-lxgll\") pod \"cert-manager-operator-controller-manager-55c444bf86-cwhq6\" (UID: \"fcda5510-01bf-47aa-9c9d-b238ed2781ab\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cwhq6" Sep 09 02:36:28.206863 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:28.206794 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cwhq6" Sep 09 02:36:28.483104 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:28.483011 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cwhq6"] Sep 09 02:36:28.488339 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:36:28.488315 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcda5510_01bf_47aa_9c9d_b238ed2781ab.slice/crio-e81d9dc0d1a76b986247bb3029edd9b2ea3f45e501e8a59d842a30e4e64fe16e WatchSource:0}: Error finding container e81d9dc0d1a76b986247bb3029edd9b2ea3f45e501e8a59d842a30e4e64fe16e: Status 404 returned error can't find the container with id e81d9dc0d1a76b986247bb3029edd9b2ea3f45e501e8a59d842a30e4e64fe16e Sep 09 02:36:28.632988 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:28.632942 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cwhq6" event={"ID":"fcda5510-01bf-47aa-9c9d-b238ed2781ab","Type":"ContainerStarted","Data":"e81d9dc0d1a76b986247bb3029edd9b2ea3f45e501e8a59d842a30e4e64fe16e"} Sep 09 02:36:30.644530 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:30.644494 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cwhq6" event={"ID":"fcda5510-01bf-47aa-9c9d-b238ed2781ab","Type":"ContainerStarted","Data":"8744fdd6439cffab11adf43f175bae69304faea578a335f4e8d2036ed45cd7d0"} Sep 09 02:36:30.673340 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:30.673279 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cwhq6" podStartSLOduration=1.913733954 podStartE2EDuration="3.673255469s" totalImagesPullingTime="1.759521515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:36:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:36:30.672096272 +0000 UTC m=+384.036430745" watchObservedRunningTime="2026-09-09 02:36:30.673255469 +0000 UTC m=+384.037589950" Sep 09 02:36:32.016545 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:32.016513 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzvr5"] Sep 09 02:36:32.020880 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:32.020854 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzvr5" Sep 09 02:36:32.030118 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:32.030096 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-gdxls\"" Sep 09 02:36:32.030238 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:32.030119 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 09 02:36:32.030872 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:32.030855 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 09 02:36:32.054349 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:32.054313 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzvr5"] Sep 09 02:36:32.142737 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:32.142706 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a33873b-8fb5-46d8-8813-a0a07465839b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzvr5\" (UID: \"4a33873b-8fb5-46d8-8813-a0a07465839b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzvr5" Sep 09 02:36:32.142886 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:32.142760 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a33873b-8fb5-46d8-8813-a0a07465839b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzvr5\" (UID: \"4a33873b-8fb5-46d8-8813-a0a07465839b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzvr5" Sep 09 02:36:32.142886 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:32.142784 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kjtx\" (UniqueName: \"kubernetes.io/projected/4a33873b-8fb5-46d8-8813-a0a07465839b-kube-api-access-9kjtx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzvr5\" (UID: \"4a33873b-8fb5-46d8-8813-a0a07465839b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzvr5" Sep 09 02:36:32.243512 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:32.243471 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a33873b-8fb5-46d8-8813-a0a07465839b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzvr5\" (UID: \"4a33873b-8fb5-46d8-8813-a0a07465839b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzvr5" Sep 09 02:36:32.243512 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:32.243520 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kjtx\" (UniqueName: \"kubernetes.io/projected/4a33873b-8fb5-46d8-8813-a0a07465839b-kube-api-access-9kjtx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzvr5\" (UID: \"4a33873b-8fb5-46d8-8813-a0a07465839b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzvr5" Sep 09 02:36:32.243762 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:32.243559 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a33873b-8fb5-46d8-8813-a0a07465839b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzvr5\" (UID: \"4a33873b-8fb5-46d8-8813-a0a07465839b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzvr5" Sep 09 02:36:32.243945 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:32.243922 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a33873b-8fb5-46d8-8813-a0a07465839b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzvr5\" (UID: \"4a33873b-8fb5-46d8-8813-a0a07465839b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzvr5" Sep 09 02:36:32.244027 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:32.243932 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a33873b-8fb5-46d8-8813-a0a07465839b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzvr5\" (UID: \"4a33873b-8fb5-46d8-8813-a0a07465839b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzvr5" Sep 09 02:36:32.264425 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:32.264392 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kjtx\" (UniqueName: \"kubernetes.io/projected/4a33873b-8fb5-46d8-8813-a0a07465839b-kube-api-access-9kjtx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzvr5\" (UID: \"4a33873b-8fb5-46d8-8813-a0a07465839b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzvr5" Sep 09 02:36:32.329390 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:32.329360 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzvr5" Sep 09 02:36:32.635446 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:32.635360 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzvr5"] Sep 09 02:36:32.640113 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:36:32.640082 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a33873b_8fb5_46d8_8813_a0a07465839b.slice/crio-bdbad80001c749c06368459e9b1ea28b1d502910170e20075d49a3d2e70aed2b WatchSource:0}: Error finding container bdbad80001c749c06368459e9b1ea28b1d502910170e20075d49a3d2e70aed2b: Status 404 returned error can't find the container with id bdbad80001c749c06368459e9b1ea28b1d502910170e20075d49a3d2e70aed2b Sep 09 02:36:32.652972 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:32.652939 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzvr5" event={"ID":"4a33873b-8fb5-46d8-8813-a0a07465839b","Type":"ContainerStarted","Data":"bdbad80001c749c06368459e9b1ea28b1d502910170e20075d49a3d2e70aed2b"} Sep 09 02:36:33.657623 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:33.657591 2577 generic.go:356] "Generic (PLEG): container finished" podID="4a33873b-8fb5-46d8-8813-a0a07465839b" containerID="4ca3a8e3249b69da8302b8ae6ec8fe6e20ce429853d4c32468aec5bbcdcd07bc" exitCode=0 Sep 09 02:36:33.658027 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:33.657635 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzvr5" event={"ID":"4a33873b-8fb5-46d8-8813-a0a07465839b","Type":"ContainerDied","Data":"4ca3a8e3249b69da8302b8ae6ec8fe6e20ce429853d4c32468aec5bbcdcd07bc"} Sep 09 02:36:41.687444 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:41.687406 2577 generic.go:356] "Generic (PLEG): container finished" podID="4a33873b-8fb5-46d8-8813-a0a07465839b" containerID="dd70f376d65bd686cb9e26c3a3b877676196643b0bdb9c1a56c2c811b410c683" exitCode=0 Sep 09 02:36:41.687846 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:41.687452 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzvr5" event={"ID":"4a33873b-8fb5-46d8-8813-a0a07465839b","Type":"ContainerDied","Data":"dd70f376d65bd686cb9e26c3a3b877676196643b0bdb9c1a56c2c811b410c683"} Sep 09 02:36:42.252565 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:42.252493 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-vfpv8"] Sep 09 02:36:42.255446 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:42.255428 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-vfpv8" Sep 09 02:36:42.259883 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:42.259864 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-6qnjd\"" Sep 09 02:36:42.260084 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:42.260066 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 09 02:36:42.260135 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:42.260106 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 09 02:36:42.271458 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:42.271434 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-vfpv8"] Sep 09 02:36:42.311462 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:42.311437 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e5a112e1-8ada-4cfa-847e-0601ac2b8b92-bound-sa-token\") pod \"cert-manager-55b4d954d5-vfpv8\" (UID: \"e5a112e1-8ada-4cfa-847e-0601ac2b8b92\") " pod="cert-manager/cert-manager-55b4d954d5-vfpv8" Sep 09 02:36:42.311572 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:42.311495 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9llc8\" (UniqueName: \"kubernetes.io/projected/e5a112e1-8ada-4cfa-847e-0601ac2b8b92-kube-api-access-9llc8\") pod \"cert-manager-55b4d954d5-vfpv8\" (UID: \"e5a112e1-8ada-4cfa-847e-0601ac2b8b92\") " pod="cert-manager/cert-manager-55b4d954d5-vfpv8" Sep 09 02:36:42.411821 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:42.411793 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9llc8\" (UniqueName: \"kubernetes.io/projected/e5a112e1-8ada-4cfa-847e-0601ac2b8b92-kube-api-access-9llc8\") pod \"cert-manager-55b4d954d5-vfpv8\" (UID: \"e5a112e1-8ada-4cfa-847e-0601ac2b8b92\") " pod="cert-manager/cert-manager-55b4d954d5-vfpv8" Sep 09 02:36:42.411925 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:42.411838 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e5a112e1-8ada-4cfa-847e-0601ac2b8b92-bound-sa-token\") pod \"cert-manager-55b4d954d5-vfpv8\" (UID: \"e5a112e1-8ada-4cfa-847e-0601ac2b8b92\") " pod="cert-manager/cert-manager-55b4d954d5-vfpv8" Sep 09 02:36:42.422609 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:42.422583 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e5a112e1-8ada-4cfa-847e-0601ac2b8b92-bound-sa-token\") pod \"cert-manager-55b4d954d5-vfpv8\" (UID: \"e5a112e1-8ada-4cfa-847e-0601ac2b8b92\") " pod="cert-manager/cert-manager-55b4d954d5-vfpv8" Sep 09 02:36:42.422712 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:42.422697 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9llc8\" (UniqueName: \"kubernetes.io/projected/e5a112e1-8ada-4cfa-847e-0601ac2b8b92-kube-api-access-9llc8\") pod \"cert-manager-55b4d954d5-vfpv8\" (UID: \"e5a112e1-8ada-4cfa-847e-0601ac2b8b92\") " pod="cert-manager/cert-manager-55b4d954d5-vfpv8" Sep 09 02:36:42.563728 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:42.563699 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-vfpv8" Sep 09 02:36:42.693235 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:42.693206 2577 generic.go:356] "Generic (PLEG): container finished" podID="4a33873b-8fb5-46d8-8813-a0a07465839b" containerID="63f8000e0815f88623397645082d7de326907b9f4b858e89e05e540d814d96a3" exitCode=0 Sep 09 02:36:42.693560 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:42.693274 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzvr5" event={"ID":"4a33873b-8fb5-46d8-8813-a0a07465839b","Type":"ContainerDied","Data":"63f8000e0815f88623397645082d7de326907b9f4b858e89e05e540d814d96a3"} Sep 09 02:36:42.836004 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:42.835981 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-vfpv8"] Sep 09 02:36:42.838075 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:36:42.838048 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5a112e1_8ada_4cfa_847e_0601ac2b8b92.slice/crio-3e0a9642d1a9ad1520560c5ce131abb32f361ffff2be03b2ffd7f528bf0ec6d5 WatchSource:0}: Error finding container 3e0a9642d1a9ad1520560c5ce131abb32f361ffff2be03b2ffd7f528bf0ec6d5: Status 404 returned error can't find the container with id 3e0a9642d1a9ad1520560c5ce131abb32f361ffff2be03b2ffd7f528bf0ec6d5 Sep 09 02:36:43.698134 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:43.698103 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-vfpv8" event={"ID":"e5a112e1-8ada-4cfa-847e-0601ac2b8b92","Type":"ContainerStarted","Data":"3e0a9642d1a9ad1520560c5ce131abb32f361ffff2be03b2ffd7f528bf0ec6d5"} Sep 09 02:36:43.829615 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:43.829585 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzvr5" Sep 09 02:36:43.922827 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:43.922795 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a33873b-8fb5-46d8-8813-a0a07465839b-bundle\") pod \"4a33873b-8fb5-46d8-8813-a0a07465839b\" (UID: \"4a33873b-8fb5-46d8-8813-a0a07465839b\") " Sep 09 02:36:43.923004 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:43.922841 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kjtx\" (UniqueName: \"kubernetes.io/projected/4a33873b-8fb5-46d8-8813-a0a07465839b-kube-api-access-9kjtx\") pod \"4a33873b-8fb5-46d8-8813-a0a07465839b\" (UID: \"4a33873b-8fb5-46d8-8813-a0a07465839b\") " Sep 09 02:36:43.923004 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:43.922871 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a33873b-8fb5-46d8-8813-a0a07465839b-util\") pod \"4a33873b-8fb5-46d8-8813-a0a07465839b\" (UID: \"4a33873b-8fb5-46d8-8813-a0a07465839b\") " Sep 09 02:36:43.923326 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:43.923298 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a33873b-8fb5-46d8-8813-a0a07465839b-bundle" (OuterVolumeSpecName: "bundle") pod "4a33873b-8fb5-46d8-8813-a0a07465839b" (UID: "4a33873b-8fb5-46d8-8813-a0a07465839b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:36:43.925391 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:43.925366 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a33873b-8fb5-46d8-8813-a0a07465839b-kube-api-access-9kjtx" (OuterVolumeSpecName: "kube-api-access-9kjtx") pod "4a33873b-8fb5-46d8-8813-a0a07465839b" (UID: "4a33873b-8fb5-46d8-8813-a0a07465839b"). InnerVolumeSpecName "kube-api-access-9kjtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:36:43.929932 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:43.929903 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a33873b-8fb5-46d8-8813-a0a07465839b-util" (OuterVolumeSpecName: "util") pod "4a33873b-8fb5-46d8-8813-a0a07465839b" (UID: "4a33873b-8fb5-46d8-8813-a0a07465839b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:36:44.023643 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:44.023555 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a33873b-8fb5-46d8-8813-a0a07465839b-bundle\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:36:44.023643 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:44.023591 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kjtx\" (UniqueName: \"kubernetes.io/projected/4a33873b-8fb5-46d8-8813-a0a07465839b-kube-api-access-9kjtx\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:36:44.023643 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:44.023605 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a33873b-8fb5-46d8-8813-a0a07465839b-util\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:36:44.704602 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:44.704562 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzvr5" event={"ID":"4a33873b-8fb5-46d8-8813-a0a07465839b","Type":"ContainerDied","Data":"bdbad80001c749c06368459e9b1ea28b1d502910170e20075d49a3d2e70aed2b"} Sep 09 02:36:44.705039 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:44.704622 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdbad80001c749c06368459e9b1ea28b1d502910170e20075d49a3d2e70aed2b" Sep 09 02:36:44.705039 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:44.704580 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgzvr5" Sep 09 02:36:46.713688 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:46.713650 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-vfpv8" event={"ID":"e5a112e1-8ada-4cfa-847e-0601ac2b8b92","Type":"ContainerStarted","Data":"df52f08e000acb29bb1924ab54e44c41d6a6f46c8975a418b0becbbb99785e12"} Sep 09 02:36:46.735278 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:46.735233 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-vfpv8" podStartSLOduration=1.8363579429999999 podStartE2EDuration="4.735218424s" totalImagesPullingTime="2.898860481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:36:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:36:46.734251732 +0000 UTC m=+400.098586195" watchObservedRunningTime="2026-09-09 02:36:46.735218424 +0000 UTC m=+400.099552899" Sep 09 02:36:56.822365 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:56.822329 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnfvmv"] Sep 09 02:36:56.822808 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:56.822587 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a33873b-8fb5-46d8-8813-a0a07465839b" containerName="pull" Sep 09 02:36:56.822808 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:56.822598 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a33873b-8fb5-46d8-8813-a0a07465839b" containerName="pull" Sep 09 02:36:56.822808 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:56.822616 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a33873b-8fb5-46d8-8813-a0a07465839b" containerName="extract" Sep 09 02:36:56.822808 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:56.822621 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a33873b-8fb5-46d8-8813-a0a07465839b" containerName="extract" Sep 09 02:36:56.822808 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:56.822632 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a33873b-8fb5-46d8-8813-a0a07465839b" containerName="util" Sep 09 02:36:56.822808 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:56.822638 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a33873b-8fb5-46d8-8813-a0a07465839b" containerName="util" Sep 09 02:36:56.822808 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:56.822679 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a33873b-8fb5-46d8-8813-a0a07465839b" containerName="extract" Sep 09 02:36:56.825675 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:56.825660 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnfvmv" Sep 09 02:36:56.828838 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:56.828803 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 09 02:36:56.830201 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:56.830182 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-gdxls\"" Sep 09 02:36:56.830303 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:56.830206 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 09 02:36:56.837017 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:56.836995 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnfvmv"] Sep 09 02:36:56.915409 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:56.915375 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31a03787-6213-42f8-9ca0-f42d5a7b3f7f-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnfvmv\" (UID: \"31a03787-6213-42f8-9ca0-f42d5a7b3f7f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnfvmv" Sep 09 02:36:56.915409 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:56.915412 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31a03787-6213-42f8-9ca0-f42d5a7b3f7f-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnfvmv\" (UID: \"31a03787-6213-42f8-9ca0-f42d5a7b3f7f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnfvmv" Sep 09 02:36:56.915598 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:56.915460 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x945f\" (UniqueName: \"kubernetes.io/projected/31a03787-6213-42f8-9ca0-f42d5a7b3f7f-kube-api-access-x945f\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnfvmv\" (UID: \"31a03787-6213-42f8-9ca0-f42d5a7b3f7f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnfvmv" Sep 09 02:36:57.016695 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:57.016660 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31a03787-6213-42f8-9ca0-f42d5a7b3f7f-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnfvmv\" (UID: \"31a03787-6213-42f8-9ca0-f42d5a7b3f7f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnfvmv" Sep 09 02:36:57.016695 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:57.016693 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31a03787-6213-42f8-9ca0-f42d5a7b3f7f-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnfvmv\" (UID: \"31a03787-6213-42f8-9ca0-f42d5a7b3f7f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnfvmv" Sep 09 02:36:57.016909 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:57.016722 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x945f\" (UniqueName: \"kubernetes.io/projected/31a03787-6213-42f8-9ca0-f42d5a7b3f7f-kube-api-access-x945f\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnfvmv\" (UID: \"31a03787-6213-42f8-9ca0-f42d5a7b3f7f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnfvmv" Sep 09 02:36:57.017050 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:57.017029 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31a03787-6213-42f8-9ca0-f42d5a7b3f7f-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnfvmv\" (UID: \"31a03787-6213-42f8-9ca0-f42d5a7b3f7f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnfvmv" Sep 09 02:36:57.017135 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:57.017113 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31a03787-6213-42f8-9ca0-f42d5a7b3f7f-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnfvmv\" (UID: \"31a03787-6213-42f8-9ca0-f42d5a7b3f7f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnfvmv" Sep 09 02:36:57.026502 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:57.026481 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x945f\" (UniqueName: \"kubernetes.io/projected/31a03787-6213-42f8-9ca0-f42d5a7b3f7f-kube-api-access-x945f\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnfvmv\" (UID: \"31a03787-6213-42f8-9ca0-f42d5a7b3f7f\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnfvmv" Sep 09 02:36:57.134849 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:57.134767 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnfvmv" Sep 09 02:36:57.407092 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:57.407068 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnfvmv"] Sep 09 02:36:57.409162 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:36:57.409137 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31a03787_6213_42f8_9ca0_f42d5a7b3f7f.slice/crio-42d5d08d6c5a1fb08bd09258e2b106e940854746eb9666e40299233772fba234 WatchSource:0}: Error finding container 42d5d08d6c5a1fb08bd09258e2b106e940854746eb9666e40299233772fba234: Status 404 returned error can't find the container with id 42d5d08d6c5a1fb08bd09258e2b106e940854746eb9666e40299233772fba234 Sep 09 02:36:57.759045 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:57.758939 2577 generic.go:356] "Generic (PLEG): container finished" podID="31a03787-6213-42f8-9ca0-f42d5a7b3f7f" containerID="dbb919b4b88f39033def22ce607c2b23a6708ac74765c2bc5d2d107b2195257f" exitCode=0 Sep 09 02:36:57.759045 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:57.759024 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnfvmv" event={"ID":"31a03787-6213-42f8-9ca0-f42d5a7b3f7f","Type":"ContainerDied","Data":"dbb919b4b88f39033def22ce607c2b23a6708ac74765c2bc5d2d107b2195257f"} Sep 09 02:36:57.759254 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:57.759072 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnfvmv" event={"ID":"31a03787-6213-42f8-9ca0-f42d5a7b3f7f","Type":"ContainerStarted","Data":"42d5d08d6c5a1fb08bd09258e2b106e940854746eb9666e40299233772fba234"} Sep 09 02:36:58.764707 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:58.764622 2577 generic.go:356] "Generic (PLEG): container finished" podID="31a03787-6213-42f8-9ca0-f42d5a7b3f7f" containerID="0d7a1ba94f2b7b72b1fcb08a95ff9186e2fc282583155163d85ab7e3ce31bf74" exitCode=0 Sep 09 02:36:58.765123 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:58.764700 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnfvmv" event={"ID":"31a03787-6213-42f8-9ca0-f42d5a7b3f7f","Type":"ContainerDied","Data":"0d7a1ba94f2b7b72b1fcb08a95ff9186e2fc282583155163d85ab7e3ce31bf74"} Sep 09 02:36:59.770725 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:59.770688 2577 generic.go:356] "Generic (PLEG): container finished" podID="31a03787-6213-42f8-9ca0-f42d5a7b3f7f" containerID="b87dd8682ae62202f57b63c63f9038620217c78d515aeadb90fab3f2a4e49a2b" exitCode=0 Sep 09 02:36:59.771128 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:36:59.770767 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnfvmv" event={"ID":"31a03787-6213-42f8-9ca0-f42d5a7b3f7f","Type":"ContainerDied","Data":"b87dd8682ae62202f57b63c63f9038620217c78d515aeadb90fab3f2a4e49a2b"} Sep 09 02:37:00.896687 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:00.896665 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnfvmv" Sep 09 02:37:00.946008 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:00.945976 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x945f\" (UniqueName: \"kubernetes.io/projected/31a03787-6213-42f8-9ca0-f42d5a7b3f7f-kube-api-access-x945f\") pod \"31a03787-6213-42f8-9ca0-f42d5a7b3f7f\" (UID: \"31a03787-6213-42f8-9ca0-f42d5a7b3f7f\") " Sep 09 02:37:00.946159 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:00.946016 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31a03787-6213-42f8-9ca0-f42d5a7b3f7f-util\") pod \"31a03787-6213-42f8-9ca0-f42d5a7b3f7f\" (UID: \"31a03787-6213-42f8-9ca0-f42d5a7b3f7f\") " Sep 09 02:37:00.946159 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:00.946039 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31a03787-6213-42f8-9ca0-f42d5a7b3f7f-bundle\") pod \"31a03787-6213-42f8-9ca0-f42d5a7b3f7f\" (UID: \"31a03787-6213-42f8-9ca0-f42d5a7b3f7f\") " Sep 09 02:37:00.946674 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:00.946651 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31a03787-6213-42f8-9ca0-f42d5a7b3f7f-bundle" (OuterVolumeSpecName: "bundle") pod "31a03787-6213-42f8-9ca0-f42d5a7b3f7f" (UID: "31a03787-6213-42f8-9ca0-f42d5a7b3f7f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:37:00.948084 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:00.948062 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31a03787-6213-42f8-9ca0-f42d5a7b3f7f-kube-api-access-x945f" (OuterVolumeSpecName: "kube-api-access-x945f") pod "31a03787-6213-42f8-9ca0-f42d5a7b3f7f" (UID: "31a03787-6213-42f8-9ca0-f42d5a7b3f7f"). InnerVolumeSpecName "kube-api-access-x945f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:37:00.951500 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:00.951435 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31a03787-6213-42f8-9ca0-f42d5a7b3f7f-util" (OuterVolumeSpecName: "util") pod "31a03787-6213-42f8-9ca0-f42d5a7b3f7f" (UID: "31a03787-6213-42f8-9ca0-f42d5a7b3f7f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:37:01.047227 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:01.047202 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x945f\" (UniqueName: \"kubernetes.io/projected/31a03787-6213-42f8-9ca0-f42d5a7b3f7f-kube-api-access-x945f\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:37:01.047227 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:01.047226 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31a03787-6213-42f8-9ca0-f42d5a7b3f7f-util\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:37:01.047374 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:01.047237 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31a03787-6213-42f8-9ca0-f42d5a7b3f7f-bundle\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:37:01.782169 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:01.782143 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnfvmv" Sep 09 02:37:01.782334 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:01.782137 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnfvmv" event={"ID":"31a03787-6213-42f8-9ca0-f42d5a7b3f7f","Type":"ContainerDied","Data":"42d5d08d6c5a1fb08bd09258e2b106e940854746eb9666e40299233772fba234"} Sep 09 02:37:01.782334 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:01.782257 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42d5d08d6c5a1fb08bd09258e2b106e940854746eb9666e40299233772fba234" Sep 09 02:37:08.601615 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:08.601580 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rr4sf"] Sep 09 02:37:08.602077 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:08.601847 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31a03787-6213-42f8-9ca0-f42d5a7b3f7f" containerName="pull" Sep 09 02:37:08.602077 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:08.601859 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="31a03787-6213-42f8-9ca0-f42d5a7b3f7f" containerName="pull" Sep 09 02:37:08.602077 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:08.601874 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31a03787-6213-42f8-9ca0-f42d5a7b3f7f" containerName="util" Sep 09 02:37:08.602077 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:08.601879 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="31a03787-6213-42f8-9ca0-f42d5a7b3f7f" containerName="util" Sep 09 02:37:08.602077 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:08.601885 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31a03787-6213-42f8-9ca0-f42d5a7b3f7f" containerName="extract" Sep 09 02:37:08.602077 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:08.601890 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="31a03787-6213-42f8-9ca0-f42d5a7b3f7f" containerName="extract" Sep 09 02:37:08.602077 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:08.601954 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="31a03787-6213-42f8-9ca0-f42d5a7b3f7f" containerName="extract" Sep 09 02:37:08.606029 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:08.606014 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rr4sf" Sep 09 02:37:08.610428 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:08.610407 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-gdxls\"" Sep 09 02:37:08.610528 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:08.610445 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 09 02:37:08.610714 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:08.610696 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 09 02:37:08.617096 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:08.617073 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rr4sf"] Sep 09 02:37:08.704487 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:08.704457 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-972gv\" (UniqueName: \"kubernetes.io/projected/c64740e4-6125-488c-b5e4-27f493e798c4-kube-api-access-972gv\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rr4sf\" (UID: \"c64740e4-6125-488c-b5e4-27f493e798c4\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rr4sf" Sep 09 02:37:08.704664 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:08.704512 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c64740e4-6125-488c-b5e4-27f493e798c4-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rr4sf\" (UID: \"c64740e4-6125-488c-b5e4-27f493e798c4\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rr4sf" Sep 09 02:37:08.704664 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:08.704585 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c64740e4-6125-488c-b5e4-27f493e798c4-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rr4sf\" (UID: \"c64740e4-6125-488c-b5e4-27f493e798c4\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rr4sf" Sep 09 02:37:08.805344 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:08.805312 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-972gv\" (UniqueName: \"kubernetes.io/projected/c64740e4-6125-488c-b5e4-27f493e798c4-kube-api-access-972gv\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rr4sf\" (UID: \"c64740e4-6125-488c-b5e4-27f493e798c4\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rr4sf" Sep 09 02:37:08.805493 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:08.805362 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c64740e4-6125-488c-b5e4-27f493e798c4-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rr4sf\" (UID: \"c64740e4-6125-488c-b5e4-27f493e798c4\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rr4sf" Sep 09 02:37:08.805493 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:08.805394 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c64740e4-6125-488c-b5e4-27f493e798c4-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rr4sf\" (UID: \"c64740e4-6125-488c-b5e4-27f493e798c4\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rr4sf" Sep 09 02:37:08.805747 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:08.805726 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c64740e4-6125-488c-b5e4-27f493e798c4-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rr4sf\" (UID: \"c64740e4-6125-488c-b5e4-27f493e798c4\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rr4sf" Sep 09 02:37:08.805810 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:08.805759 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c64740e4-6125-488c-b5e4-27f493e798c4-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rr4sf\" (UID: \"c64740e4-6125-488c-b5e4-27f493e798c4\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rr4sf" Sep 09 02:37:08.820500 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:08.820469 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-972gv\" (UniqueName: \"kubernetes.io/projected/c64740e4-6125-488c-b5e4-27f493e798c4-kube-api-access-972gv\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rr4sf\" (UID: \"c64740e4-6125-488c-b5e4-27f493e798c4\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rr4sf" Sep 09 02:37:08.914576 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:08.914497 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rr4sf" Sep 09 02:37:09.208371 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:09.208285 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rr4sf"] Sep 09 02:37:09.211193 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:37:09.211167 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc64740e4_6125_488c_b5e4_27f493e798c4.slice/crio-7474c1d61feed24bfec8d8797f8c4e84e4924e83ff4e851c94854399ef2f851a WatchSource:0}: Error finding container 7474c1d61feed24bfec8d8797f8c4e84e4924e83ff4e851c94854399ef2f851a: Status 404 returned error can't find the container with id 7474c1d61feed24bfec8d8797f8c4e84e4924e83ff4e851c94854399ef2f851a Sep 09 02:37:09.816844 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:09.816810 2577 generic.go:356] "Generic (PLEG): container finished" podID="c64740e4-6125-488c-b5e4-27f493e798c4" containerID="12c05997efe142fd7f7ee5fe0ec5ca88171d198771f4b65bb72ec953ebbe0054" exitCode=0 Sep 09 02:37:09.816844 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:09.816847 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rr4sf" event={"ID":"c64740e4-6125-488c-b5e4-27f493e798c4","Type":"ContainerDied","Data":"12c05997efe142fd7f7ee5fe0ec5ca88171d198771f4b65bb72ec953ebbe0054"} Sep 09 02:37:09.817273 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:09.816888 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rr4sf" event={"ID":"c64740e4-6125-488c-b5e4-27f493e798c4","Type":"ContainerStarted","Data":"7474c1d61feed24bfec8d8797f8c4e84e4924e83ff4e851c94854399ef2f851a"} Sep 09 02:37:10.822821 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:10.822739 2577 generic.go:356] "Generic (PLEG): container finished" podID="c64740e4-6125-488c-b5e4-27f493e798c4" containerID="77d797e972c0e0e8309a6250d6c393ff9086b81a4cfe47d28e4df264c10b2606" exitCode=0 Sep 09 02:37:10.822821 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:10.822781 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rr4sf" event={"ID":"c64740e4-6125-488c-b5e4-27f493e798c4","Type":"ContainerDied","Data":"77d797e972c0e0e8309a6250d6c393ff9086b81a4cfe47d28e4df264c10b2606"} Sep 09 02:37:10.949509 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:10.949476 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-bt554"] Sep 09 02:37:10.952841 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:10.952824 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-bt554" Sep 09 02:37:10.961404 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:10.961385 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 09 02:37:10.961940 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:10.961913 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 09 02:37:10.961940 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:10.961926 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 09 02:37:10.962102 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:10.962067 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-b6csz\"" Sep 09 02:37:10.962155 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:10.962107 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 09 02:37:10.982593 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:10.982570 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-bt554"] Sep 09 02:37:11.022916 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:11.022885 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c4dfaff6-f558-4fd8-9e29-ac498d770593-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-bt554\" (UID: \"c4dfaff6-f558-4fd8-9e29-ac498d770593\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-bt554" Sep 09 02:37:11.023044 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:11.022927 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxhxd\" (UniqueName: \"kubernetes.io/projected/c4dfaff6-f558-4fd8-9e29-ac498d770593-kube-api-access-lxhxd\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-bt554\" (UID: \"c4dfaff6-f558-4fd8-9e29-ac498d770593\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-bt554" Sep 09 02:37:11.023044 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:11.022947 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c4dfaff6-f558-4fd8-9e29-ac498d770593-manifests\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-bt554\" (UID: \"c4dfaff6-f558-4fd8-9e29-ac498d770593\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-bt554" Sep 09 02:37:11.023118 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:11.023042 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c4dfaff6-f558-4fd8-9e29-ac498d770593-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-bt554\" (UID: \"c4dfaff6-f558-4fd8-9e29-ac498d770593\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-bt554" Sep 09 02:37:11.023118 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:11.023082 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4dfaff6-f558-4fd8-9e29-ac498d770593-tmp\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-bt554\" (UID: \"c4dfaff6-f558-4fd8-9e29-ac498d770593\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-bt554" Sep 09 02:37:11.123802 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:11.123728 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c4dfaff6-f558-4fd8-9e29-ac498d770593-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-bt554\" (UID: \"c4dfaff6-f558-4fd8-9e29-ac498d770593\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-bt554" Sep 09 02:37:11.123802 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:11.123761 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4dfaff6-f558-4fd8-9e29-ac498d770593-tmp\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-bt554\" (UID: \"c4dfaff6-f558-4fd8-9e29-ac498d770593\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-bt554" Sep 09 02:37:11.124007 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:11.123825 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c4dfaff6-f558-4fd8-9e29-ac498d770593-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-bt554\" (UID: \"c4dfaff6-f558-4fd8-9e29-ac498d770593\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-bt554" Sep 09 02:37:11.124007 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:11.123861 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxhxd\" (UniqueName: \"kubernetes.io/projected/c4dfaff6-f558-4fd8-9e29-ac498d770593-kube-api-access-lxhxd\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-bt554\" (UID: \"c4dfaff6-f558-4fd8-9e29-ac498d770593\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-bt554" Sep 09 02:37:11.124007 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:11.123887 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c4dfaff6-f558-4fd8-9e29-ac498d770593-manifests\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-bt554\" (UID: \"c4dfaff6-f558-4fd8-9e29-ac498d770593\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-bt554" Sep 09 02:37:11.124302 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:11.124275 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c4dfaff6-f558-4fd8-9e29-ac498d770593-manifests\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-bt554\" (UID: \"c4dfaff6-f558-4fd8-9e29-ac498d770593\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-bt554" Sep 09 02:37:11.126243 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:11.126219 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4dfaff6-f558-4fd8-9e29-ac498d770593-tmp\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-bt554\" (UID: \"c4dfaff6-f558-4fd8-9e29-ac498d770593\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-bt554" Sep 09 02:37:11.126375 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:11.126355 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c4dfaff6-f558-4fd8-9e29-ac498d770593-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-bt554\" (UID: \"c4dfaff6-f558-4fd8-9e29-ac498d770593\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-bt554" Sep 09 02:37:11.126441 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:11.126422 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c4dfaff6-f558-4fd8-9e29-ac498d770593-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-bt554\" (UID: \"c4dfaff6-f558-4fd8-9e29-ac498d770593\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-bt554" Sep 09 02:37:11.136907 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:11.136877 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxhxd\" (UniqueName: \"kubernetes.io/projected/c4dfaff6-f558-4fd8-9e29-ac498d770593-kube-api-access-lxhxd\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-bt554\" (UID: \"c4dfaff6-f558-4fd8-9e29-ac498d770593\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-bt554" Sep 09 02:37:11.262573 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:11.262537 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-bt554" Sep 09 02:37:11.546464 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:11.546433 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-bt554"] Sep 09 02:37:11.548010 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:37:11.547983 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4dfaff6_f558_4fd8_9e29_ac498d770593.slice/crio-86455401b91ad978d6eefb845f111a1b56c07a9ce35afcf0bf14dc9d2a0497e8 WatchSource:0}: Error finding container 86455401b91ad978d6eefb845f111a1b56c07a9ce35afcf0bf14dc9d2a0497e8: Status 404 returned error can't find the container with id 86455401b91ad978d6eefb845f111a1b56c07a9ce35afcf0bf14dc9d2a0497e8 Sep 09 02:37:11.828391 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:11.828301 2577 generic.go:356] "Generic (PLEG): container finished" podID="c64740e4-6125-488c-b5e4-27f493e798c4" containerID="a0f1fc995bcce7d45f225bc18b6afd47e782e8a5c25131a98739332d08c3a032" exitCode=0 Sep 09 02:37:11.828391 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:11.828375 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rr4sf" event={"ID":"c64740e4-6125-488c-b5e4-27f493e798c4","Type":"ContainerDied","Data":"a0f1fc995bcce7d45f225bc18b6afd47e782e8a5c25131a98739332d08c3a032"} Sep 09 02:37:11.829733 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:11.829709 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-bt554" event={"ID":"c4dfaff6-f558-4fd8-9e29-ac498d770593","Type":"ContainerStarted","Data":"86455401b91ad978d6eefb845f111a1b56c07a9ce35afcf0bf14dc9d2a0497e8"} Sep 09 02:37:14.553342 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:14.553320 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rr4sf" Sep 09 02:37:14.651891 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:14.651825 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c64740e4-6125-488c-b5e4-27f493e798c4-util\") pod \"c64740e4-6125-488c-b5e4-27f493e798c4\" (UID: \"c64740e4-6125-488c-b5e4-27f493e798c4\") " Sep 09 02:37:14.651891 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:14.651871 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c64740e4-6125-488c-b5e4-27f493e798c4-bundle\") pod \"c64740e4-6125-488c-b5e4-27f493e798c4\" (UID: \"c64740e4-6125-488c-b5e4-27f493e798c4\") " Sep 09 02:37:14.651891 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:14.651923 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-972gv\" (UniqueName: \"kubernetes.io/projected/c64740e4-6125-488c-b5e4-27f493e798c4-kube-api-access-972gv\") pod \"c64740e4-6125-488c-b5e4-27f493e798c4\" (UID: \"c64740e4-6125-488c-b5e4-27f493e798c4\") " Sep 09 02:37:14.652660 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:14.652632 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64740e4-6125-488c-b5e4-27f493e798c4-bundle" (OuterVolumeSpecName: "bundle") pod "c64740e4-6125-488c-b5e4-27f493e798c4" (UID: "c64740e4-6125-488c-b5e4-27f493e798c4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:37:14.654173 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:14.654144 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c64740e4-6125-488c-b5e4-27f493e798c4-kube-api-access-972gv" (OuterVolumeSpecName: "kube-api-access-972gv") pod "c64740e4-6125-488c-b5e4-27f493e798c4" (UID: "c64740e4-6125-488c-b5e4-27f493e798c4"). InnerVolumeSpecName "kube-api-access-972gv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:37:14.656005 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:14.655980 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c64740e4-6125-488c-b5e4-27f493e798c4-util" (OuterVolumeSpecName: "util") pod "c64740e4-6125-488c-b5e4-27f493e798c4" (UID: "c64740e4-6125-488c-b5e4-27f493e798c4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:37:14.752374 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:14.752341 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c64740e4-6125-488c-b5e4-27f493e798c4-bundle\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:37:14.752374 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:14.752369 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-972gv\" (UniqueName: \"kubernetes.io/projected/c64740e4-6125-488c-b5e4-27f493e798c4-kube-api-access-972gv\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:37:14.752374 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:14.752379 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c64740e4-6125-488c-b5e4-27f493e798c4-util\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:37:14.848856 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:14.848786 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rr4sf" event={"ID":"c64740e4-6125-488c-b5e4-27f493e798c4","Type":"ContainerDied","Data":"7474c1d61feed24bfec8d8797f8c4e84e4924e83ff4e851c94854399ef2f851a"} Sep 09 02:37:14.848856 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:14.848838 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7474c1d61feed24bfec8d8797f8c4e84e4924e83ff4e851c94854399ef2f851a" Sep 09 02:37:14.848856 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:14.848800 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rr4sf" Sep 09 02:37:14.850536 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:14.850486 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-bt554" event={"ID":"c4dfaff6-f558-4fd8-9e29-ac498d770593","Type":"ContainerStarted","Data":"467fc4e3a1c9cc9319dd8852a99be3fdbcc0f8d618840d20e0de4be4b8cc1cc0"} Sep 09 02:37:15.856520 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:15.856484 2577 generic.go:356] "Generic (PLEG): container finished" podID="c4dfaff6-f558-4fd8-9e29-ac498d770593" containerID="467fc4e3a1c9cc9319dd8852a99be3fdbcc0f8d618840d20e0de4be4b8cc1cc0" exitCode=0 Sep 09 02:37:15.856927 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:15.856570 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-bt554" event={"ID":"c4dfaff6-f558-4fd8-9e29-ac498d770593","Type":"ContainerDied","Data":"467fc4e3a1c9cc9319dd8852a99be3fdbcc0f8d618840d20e0de4be4b8cc1cc0"} Sep 09 02:37:16.862599 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:16.862565 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-bt554" event={"ID":"c4dfaff6-f558-4fd8-9e29-ac498d770593","Type":"ContainerStarted","Data":"394a4fe3a926ab9201af770643f7a5ec6a828e5d4d834e7a482b6a541775119d"} Sep 09 02:37:16.863002 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:16.862679 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-bt554" Sep 09 02:37:16.888513 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:16.888456 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-bt554" podStartSLOduration=2.456732127 podStartE2EDuration="6.888437037s" totalImagesPullingTime="3.413606868s" totalInitContainerRuntime="1.018098042s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:37:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:37:16.886761149 +0000 UTC m=+430.251095626" watchObservedRunningTime="2026-09-09 02:37:16.888437037 +0000 UTC m=+430.252771511" Sep 09 02:37:27.871892 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:27.871859 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-bt554" Sep 09 02:37:41.455813 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:41.455722 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7dc95696d9-md656"] Sep 09 02:37:41.456368 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:41.456008 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c64740e4-6125-488c-b5e4-27f493e798c4" containerName="extract" Sep 09 02:37:41.456368 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:41.456020 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c64740e4-6125-488c-b5e4-27f493e798c4" containerName="extract" Sep 09 02:37:41.456368 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:41.456038 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c64740e4-6125-488c-b5e4-27f493e798c4" containerName="util" Sep 09 02:37:41.456368 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:41.456043 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c64740e4-6125-488c-b5e4-27f493e798c4" containerName="util" Sep 09 02:37:41.456368 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:41.456050 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c64740e4-6125-488c-b5e4-27f493e798c4" containerName="pull" Sep 09 02:37:41.456368 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:41.456056 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c64740e4-6125-488c-b5e4-27f493e798c4" containerName="pull" Sep 09 02:37:41.456368 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:41.456097 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c64740e4-6125-488c-b5e4-27f493e798c4" containerName="extract" Sep 09 02:37:41.459429 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:41.459272 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7dc95696d9-md656" Sep 09 02:37:41.466415 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:41.466114 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 09 02:37:41.466415 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:41.466143 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 09 02:37:41.467268 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:41.467243 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-c5v6g\"" Sep 09 02:37:41.467374 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:41.467299 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 09 02:37:41.467374 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:41.467328 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 09 02:37:41.477316 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:41.477297 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7dc95696d9-md656"] Sep 09 02:37:41.550532 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:41.550496 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6f2e295-79cb-4143-aa80-8ee4ddb99691-tls-certs\") pod \"kube-auth-proxy-7dc95696d9-md656\" (UID: \"c6f2e295-79cb-4143-aa80-8ee4ddb99691\") " pod="openshift-ingress/kube-auth-proxy-7dc95696d9-md656" Sep 09 02:37:41.550706 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:41.550536 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcsqz\" (UniqueName: \"kubernetes.io/projected/c6f2e295-79cb-4143-aa80-8ee4ddb99691-kube-api-access-xcsqz\") pod \"kube-auth-proxy-7dc95696d9-md656\" (UID: \"c6f2e295-79cb-4143-aa80-8ee4ddb99691\") " pod="openshift-ingress/kube-auth-proxy-7dc95696d9-md656" Sep 09 02:37:41.550706 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:41.550661 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6f2e295-79cb-4143-aa80-8ee4ddb99691-tmp\") pod \"kube-auth-proxy-7dc95696d9-md656\" (UID: \"c6f2e295-79cb-4143-aa80-8ee4ddb99691\") " pod="openshift-ingress/kube-auth-proxy-7dc95696d9-md656" Sep 09 02:37:41.651269 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:41.651232 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6f2e295-79cb-4143-aa80-8ee4ddb99691-tmp\") pod \"kube-auth-proxy-7dc95696d9-md656\" (UID: \"c6f2e295-79cb-4143-aa80-8ee4ddb99691\") " pod="openshift-ingress/kube-auth-proxy-7dc95696d9-md656" Sep 09 02:37:41.651439 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:41.651280 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6f2e295-79cb-4143-aa80-8ee4ddb99691-tls-certs\") pod \"kube-auth-proxy-7dc95696d9-md656\" (UID: \"c6f2e295-79cb-4143-aa80-8ee4ddb99691\") " pod="openshift-ingress/kube-auth-proxy-7dc95696d9-md656" Sep 09 02:37:41.651439 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:41.651301 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcsqz\" (UniqueName: \"kubernetes.io/projected/c6f2e295-79cb-4143-aa80-8ee4ddb99691-kube-api-access-xcsqz\") pod \"kube-auth-proxy-7dc95696d9-md656\" (UID: \"c6f2e295-79cb-4143-aa80-8ee4ddb99691\") " pod="openshift-ingress/kube-auth-proxy-7dc95696d9-md656" Sep 09 02:37:41.653793 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:41.653764 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6f2e295-79cb-4143-aa80-8ee4ddb99691-tmp\") pod \"kube-auth-proxy-7dc95696d9-md656\" (UID: \"c6f2e295-79cb-4143-aa80-8ee4ddb99691\") " pod="openshift-ingress/kube-auth-proxy-7dc95696d9-md656" Sep 09 02:37:41.654024 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:41.654007 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6f2e295-79cb-4143-aa80-8ee4ddb99691-tls-certs\") pod \"kube-auth-proxy-7dc95696d9-md656\" (UID: \"c6f2e295-79cb-4143-aa80-8ee4ddb99691\") " pod="openshift-ingress/kube-auth-proxy-7dc95696d9-md656" Sep 09 02:37:41.660893 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:41.660871 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcsqz\" (UniqueName: \"kubernetes.io/projected/c6f2e295-79cb-4143-aa80-8ee4ddb99691-kube-api-access-xcsqz\") pod \"kube-auth-proxy-7dc95696d9-md656\" (UID: \"c6f2e295-79cb-4143-aa80-8ee4ddb99691\") " pod="openshift-ingress/kube-auth-proxy-7dc95696d9-md656" Sep 09 02:37:41.769038 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:41.768923 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7dc95696d9-md656" Sep 09 02:37:42.040407 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:42.040382 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7dc95696d9-md656"] Sep 09 02:37:42.042015 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:37:42.041985 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f2e295_79cb_4143_aa80_8ee4ddb99691.slice/crio-45459fc7314847326cf9fbe2fe14189dc15015e564b8ce16d071600f896c1a16 WatchSource:0}: Error finding container 45459fc7314847326cf9fbe2fe14189dc15015e564b8ce16d071600f896c1a16: Status 404 returned error can't find the container with id 45459fc7314847326cf9fbe2fe14189dc15015e564b8ce16d071600f896c1a16 Sep 09 02:37:42.983549 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:42.983510 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7dc95696d9-md656" event={"ID":"c6f2e295-79cb-4143-aa80-8ee4ddb99691","Type":"ContainerStarted","Data":"45459fc7314847326cf9fbe2fe14189dc15015e564b8ce16d071600f896c1a16"} Sep 09 02:37:43.036521 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:43.036486 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvbs8"] Sep 09 02:37:43.041496 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:43.041463 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvbs8" Sep 09 02:37:43.045845 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:43.045710 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 09 02:37:43.046347 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:43.046155 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-ppxwv\"" Sep 09 02:37:43.047059 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:43.046994 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 09 02:37:43.047695 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:43.047673 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 09 02:37:43.048656 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:43.048500 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 09 02:37:43.056028 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:43.055930 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvbs8"] Sep 09 02:37:43.163988 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:43.163932 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b19d3ac3-a780-4f36-bf0e-647a0fe5ce53-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-lvbs8\" (UID: \"b19d3ac3-a780-4f36-bf0e-647a0fe5ce53\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvbs8" Sep 09 02:37:43.164175 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:43.164050 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b19d3ac3-a780-4f36-bf0e-647a0fe5ce53-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-lvbs8\" (UID: \"b19d3ac3-a780-4f36-bf0e-647a0fe5ce53\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvbs8" Sep 09 02:37:43.164175 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:43.164081 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b19d3ac3-a780-4f36-bf0e-647a0fe5ce53-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-lvbs8\" (UID: \"b19d3ac3-a780-4f36-bf0e-647a0fe5ce53\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvbs8" Sep 09 02:37:43.164175 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:43.164119 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b19d3ac3-a780-4f36-bf0e-647a0fe5ce53-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-lvbs8\" (UID: \"b19d3ac3-a780-4f36-bf0e-647a0fe5ce53\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvbs8" Sep 09 02:37:43.164175 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:43.164145 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4nln\" (UniqueName: \"kubernetes.io/projected/b19d3ac3-a780-4f36-bf0e-647a0fe5ce53-kube-api-access-g4nln\") pod \"istiod-openshift-gateway-7b7995d5cd-lvbs8\" (UID: \"b19d3ac3-a780-4f36-bf0e-647a0fe5ce53\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvbs8" Sep 09 02:37:43.164175 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:43.164169 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b19d3ac3-a780-4f36-bf0e-647a0fe5ce53-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-lvbs8\" (UID: \"b19d3ac3-a780-4f36-bf0e-647a0fe5ce53\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvbs8" Sep 09 02:37:43.164388 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:43.164198 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b19d3ac3-a780-4f36-bf0e-647a0fe5ce53-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-lvbs8\" (UID: \"b19d3ac3-a780-4f36-bf0e-647a0fe5ce53\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvbs8" Sep 09 02:37:43.265652 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:43.265556 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b19d3ac3-a780-4f36-bf0e-647a0fe5ce53-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-lvbs8\" (UID: \"b19d3ac3-a780-4f36-bf0e-647a0fe5ce53\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvbs8" Sep 09 02:37:43.265652 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:43.265599 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4nln\" (UniqueName: \"kubernetes.io/projected/b19d3ac3-a780-4f36-bf0e-647a0fe5ce53-kube-api-access-g4nln\") pod \"istiod-openshift-gateway-7b7995d5cd-lvbs8\" (UID: \"b19d3ac3-a780-4f36-bf0e-647a0fe5ce53\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvbs8" Sep 09 02:37:43.265652 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:43.265627 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b19d3ac3-a780-4f36-bf0e-647a0fe5ce53-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-lvbs8\" (UID: \"b19d3ac3-a780-4f36-bf0e-647a0fe5ce53\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvbs8" Sep 09 02:37:43.265924 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:43.265659 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b19d3ac3-a780-4f36-bf0e-647a0fe5ce53-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-lvbs8\" (UID: \"b19d3ac3-a780-4f36-bf0e-647a0fe5ce53\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvbs8" Sep 09 02:37:43.265924 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:43.265716 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b19d3ac3-a780-4f36-bf0e-647a0fe5ce53-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-lvbs8\" (UID: \"b19d3ac3-a780-4f36-bf0e-647a0fe5ce53\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvbs8" Sep 09 02:37:43.265924 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:43.265873 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b19d3ac3-a780-4f36-bf0e-647a0fe5ce53-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-lvbs8\" (UID: \"b19d3ac3-a780-4f36-bf0e-647a0fe5ce53\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvbs8" Sep 09 02:37:43.266124 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:43.265923 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b19d3ac3-a780-4f36-bf0e-647a0fe5ce53-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-lvbs8\" (UID: \"b19d3ac3-a780-4f36-bf0e-647a0fe5ce53\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvbs8" Sep 09 02:37:43.266419 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:43.266374 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b19d3ac3-a780-4f36-bf0e-647a0fe5ce53-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-lvbs8\" (UID: \"b19d3ac3-a780-4f36-bf0e-647a0fe5ce53\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvbs8" Sep 09 02:37:43.268681 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:43.268646 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b19d3ac3-a780-4f36-bf0e-647a0fe5ce53-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-lvbs8\" (UID: \"b19d3ac3-a780-4f36-bf0e-647a0fe5ce53\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvbs8" Sep 09 02:37:43.268882 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:43.268859 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b19d3ac3-a780-4f36-bf0e-647a0fe5ce53-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-lvbs8\" (UID: \"b19d3ac3-a780-4f36-bf0e-647a0fe5ce53\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvbs8" Sep 09 02:37:43.268996 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:43.268889 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b19d3ac3-a780-4f36-bf0e-647a0fe5ce53-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-lvbs8\" (UID: \"b19d3ac3-a780-4f36-bf0e-647a0fe5ce53\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvbs8" Sep 09 02:37:43.268996 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:43.268902 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b19d3ac3-a780-4f36-bf0e-647a0fe5ce53-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-lvbs8\" (UID: \"b19d3ac3-a780-4f36-bf0e-647a0fe5ce53\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvbs8" Sep 09 02:37:43.275940 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:43.275919 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b19d3ac3-a780-4f36-bf0e-647a0fe5ce53-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-lvbs8\" (UID: \"b19d3ac3-a780-4f36-bf0e-647a0fe5ce53\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvbs8" Sep 09 02:37:43.276342 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:43.276318 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4nln\" (UniqueName: \"kubernetes.io/projected/b19d3ac3-a780-4f36-bf0e-647a0fe5ce53-kube-api-access-g4nln\") pod \"istiod-openshift-gateway-7b7995d5cd-lvbs8\" (UID: \"b19d3ac3-a780-4f36-bf0e-647a0fe5ce53\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvbs8" Sep 09 02:37:43.357335 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:43.357270 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvbs8" Sep 09 02:37:43.655492 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:43.655452 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvbs8"] Sep 09 02:37:44.878161 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:37:44.878125 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb19d3ac3_a780_4f36_bf0e_647a0fe5ce53.slice/crio-6e526b3cb1c60a9871ca27ba9c5663237788e4728e5950d495ba514fd4fb3cca WatchSource:0}: Error finding container 6e526b3cb1c60a9871ca27ba9c5663237788e4728e5950d495ba514fd4fb3cca: Status 404 returned error can't find the container with id 6e526b3cb1c60a9871ca27ba9c5663237788e4728e5950d495ba514fd4fb3cca Sep 09 02:37:44.995570 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:44.995540 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvbs8" event={"ID":"b19d3ac3-a780-4f36-bf0e-647a0fe5ce53","Type":"ContainerStarted","Data":"6e526b3cb1c60a9871ca27ba9c5663237788e4728e5950d495ba514fd4fb3cca"} Sep 09 02:37:44.997106 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:44.997082 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7dc95696d9-md656" event={"ID":"c6f2e295-79cb-4143-aa80-8ee4ddb99691","Type":"ContainerStarted","Data":"a95aeb6d5c78604e339ebde82efadcb0f9e5110c080d194d560dc58fbe5bd7df"} Sep 09 02:37:45.022366 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:45.022314 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7dc95696d9-md656" podStartSLOduration=1.13467601 podStartE2EDuration="4.022297873s" totalImagesPullingTime="2.887621863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:37:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:37:45.021123726 +0000 UTC m=+458.385458193" watchObservedRunningTime="2026-09-09 02:37:45.022297873 +0000 UTC m=+458.386632335" Sep 09 02:37:47.553858 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:47.553802 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 09 02:37:47.554251 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:47.553902 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 09 02:37:48.016079 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:48.015992 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvbs8" event={"ID":"b19d3ac3-a780-4f36-bf0e-647a0fe5ce53","Type":"ContainerStarted","Data":"081e2c739a7c88e923c627fa06849146e85d33492f92156173fd7f8008c2aa5e"} Sep 09 02:37:48.016254 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:48.016231 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvbs8" Sep 09 02:37:48.017844 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:48.017805 2577 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-lvbs8 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 09 02:37:48.017953 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:48.017873 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvbs8" podUID="b19d3ac3-a780-4f36-bf0e-647a0fe5ce53" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 09 02:37:48.042629 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:48.042586 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvbs8" podStartSLOduration=2.369405368 podStartE2EDuration="5.042571947s" totalImagesPullingTime="2.673166579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:37:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:37:48.041620883 +0000 UTC m=+461.405955347" watchObservedRunningTime="2026-09-09 02:37:48.042571947 +0000 UTC m=+461.406906410" Sep 09 02:37:49.022669 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:49.022639 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvbs8" Sep 09 02:37:58.823405 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:58.819931 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-s2xz8"] Sep 09 02:37:58.830857 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:58.830828 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-s2xz8" Sep 09 02:37:58.844791 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:58.844765 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-s2xz8"] Sep 09 02:37:58.881177 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:58.881139 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/867e65e9-87eb-4a8d-b705-f4ede8a25d39-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-s2xz8\" (UID: \"867e65e9-87eb-4a8d-b705-f4ede8a25d39\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-s2xz8" Sep 09 02:37:58.881177 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:58.881175 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/867e65e9-87eb-4a8d-b705-f4ede8a25d39-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-s2xz8\" (UID: \"867e65e9-87eb-4a8d-b705-f4ede8a25d39\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-s2xz8" Sep 09 02:37:58.881407 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:58.881192 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/867e65e9-87eb-4a8d-b705-f4ede8a25d39-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-s2xz8\" (UID: \"867e65e9-87eb-4a8d-b705-f4ede8a25d39\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-s2xz8" Sep 09 02:37:58.881407 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:58.881254 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/867e65e9-87eb-4a8d-b705-f4ede8a25d39-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-s2xz8\" (UID: \"867e65e9-87eb-4a8d-b705-f4ede8a25d39\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-s2xz8" Sep 09 02:37:58.881407 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:58.881344 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/867e65e9-87eb-4a8d-b705-f4ede8a25d39-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-s2xz8\" (UID: \"867e65e9-87eb-4a8d-b705-f4ede8a25d39\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-s2xz8" Sep 09 02:37:58.881407 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:58.881385 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/867e65e9-87eb-4a8d-b705-f4ede8a25d39-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-s2xz8\" (UID: \"867e65e9-87eb-4a8d-b705-f4ede8a25d39\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-s2xz8" Sep 09 02:37:58.881564 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:58.881419 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q228\" (UniqueName: \"kubernetes.io/projected/867e65e9-87eb-4a8d-b705-f4ede8a25d39-kube-api-access-2q228\") pod \"istiod-openshift-gateway-5677ccb49b-s2xz8\" (UID: \"867e65e9-87eb-4a8d-b705-f4ede8a25d39\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-s2xz8" Sep 09 02:37:58.982278 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:58.982242 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/867e65e9-87eb-4a8d-b705-f4ede8a25d39-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-s2xz8\" (UID: \"867e65e9-87eb-4a8d-b705-f4ede8a25d39\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-s2xz8" Sep 09 02:37:58.982278 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:58.982297 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/867e65e9-87eb-4a8d-b705-f4ede8a25d39-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-s2xz8\" (UID: \"867e65e9-87eb-4a8d-b705-f4ede8a25d39\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-s2xz8" Sep 09 02:37:58.982565 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:58.982338 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q228\" (UniqueName: \"kubernetes.io/projected/867e65e9-87eb-4a8d-b705-f4ede8a25d39-kube-api-access-2q228\") pod \"istiod-openshift-gateway-5677ccb49b-s2xz8\" (UID: \"867e65e9-87eb-4a8d-b705-f4ede8a25d39\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-s2xz8" Sep 09 02:37:58.982565 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:58.982385 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/867e65e9-87eb-4a8d-b705-f4ede8a25d39-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-s2xz8\" (UID: \"867e65e9-87eb-4a8d-b705-f4ede8a25d39\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-s2xz8" Sep 09 02:37:58.982565 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:58.982410 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/867e65e9-87eb-4a8d-b705-f4ede8a25d39-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-s2xz8\" (UID: \"867e65e9-87eb-4a8d-b705-f4ede8a25d39\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-s2xz8" Sep 09 02:37:58.982565 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:58.982433 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/867e65e9-87eb-4a8d-b705-f4ede8a25d39-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-s2xz8\" (UID: \"867e65e9-87eb-4a8d-b705-f4ede8a25d39\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-s2xz8" Sep 09 02:37:58.982739 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:58.982584 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/867e65e9-87eb-4a8d-b705-f4ede8a25d39-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-s2xz8\" (UID: \"867e65e9-87eb-4a8d-b705-f4ede8a25d39\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-s2xz8" Sep 09 02:37:58.983134 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:58.983059 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/867e65e9-87eb-4a8d-b705-f4ede8a25d39-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-s2xz8\" (UID: \"867e65e9-87eb-4a8d-b705-f4ede8a25d39\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-s2xz8" Sep 09 02:37:58.985073 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:58.985044 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/867e65e9-87eb-4a8d-b705-f4ede8a25d39-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-s2xz8\" (UID: \"867e65e9-87eb-4a8d-b705-f4ede8a25d39\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-s2xz8" Sep 09 02:37:58.985073 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:58.985063 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/867e65e9-87eb-4a8d-b705-f4ede8a25d39-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-s2xz8\" (UID: \"867e65e9-87eb-4a8d-b705-f4ede8a25d39\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-s2xz8" Sep 09 02:37:58.985215 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:58.985112 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/867e65e9-87eb-4a8d-b705-f4ede8a25d39-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-s2xz8\" (UID: \"867e65e9-87eb-4a8d-b705-f4ede8a25d39\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-s2xz8" Sep 09 02:37:58.985215 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:58.985173 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/867e65e9-87eb-4a8d-b705-f4ede8a25d39-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-s2xz8\" (UID: \"867e65e9-87eb-4a8d-b705-f4ede8a25d39\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-s2xz8" Sep 09 02:37:59.001656 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:59.001631 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q228\" (UniqueName: \"kubernetes.io/projected/867e65e9-87eb-4a8d-b705-f4ede8a25d39-kube-api-access-2q228\") pod \"istiod-openshift-gateway-5677ccb49b-s2xz8\" (UID: \"867e65e9-87eb-4a8d-b705-f4ede8a25d39\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-s2xz8" Sep 09 02:37:59.003117 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:59.003094 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/867e65e9-87eb-4a8d-b705-f4ede8a25d39-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-s2xz8\" (UID: \"867e65e9-87eb-4a8d-b705-f4ede8a25d39\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-s2xz8" Sep 09 02:37:59.140026 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:59.139907 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-s2xz8" Sep 09 02:37:59.423983 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:59.423871 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-s2xz8"] Sep 09 02:37:59.427304 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:37:59.427279 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod867e65e9_87eb_4a8d_b705_f4ede8a25d39.slice/crio-a2d917280efb67ea56cc0d4e26147150a9c0e5fc0faeb2d131e53ece8dbe0815 WatchSource:0}: Error finding container a2d917280efb67ea56cc0d4e26147150a9c0e5fc0faeb2d131e53ece8dbe0815: Status 404 returned error can't find the container with id a2d917280efb67ea56cc0d4e26147150a9c0e5fc0faeb2d131e53ece8dbe0815 Sep 09 02:37:59.430324 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:59.430292 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 09 02:37:59.430415 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:37:59.430356 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 09 02:38:00.070660 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:00.070618 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-s2xz8" event={"ID":"867e65e9-87eb-4a8d-b705-f4ede8a25d39","Type":"ContainerStarted","Data":"df775ab929730ebfa455d77f3da79f0d9854898679fb9b66fab8b74343d9dfb4"} Sep 09 02:38:00.070660 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:00.070667 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-s2xz8" event={"ID":"867e65e9-87eb-4a8d-b705-f4ede8a25d39","Type":"ContainerStarted","Data":"a2d917280efb67ea56cc0d4e26147150a9c0e5fc0faeb2d131e53ece8dbe0815"} Sep 09 02:38:00.071163 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:00.070862 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-s2xz8" Sep 09 02:38:00.072501 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:00.072480 2577 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-s2xz8 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 09 02:38:00.072608 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:00.072523 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-s2xz8" podUID="867e65e9-87eb-4a8d-b705-f4ede8a25d39" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 09 02:38:00.099866 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:00.099803 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-s2xz8" podStartSLOduration=2.099786641 podStartE2EDuration="2.099786641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:37:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:38:00.098348232 +0000 UTC m=+473.462682714" watchObservedRunningTime="2026-09-09 02:38:00.099786641 +0000 UTC m=+473.464121104" Sep 09 02:38:01.076151 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:01.076123 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-s2xz8" Sep 09 02:38:01.253841 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:01.253809 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvbs8"] Sep 09 02:38:01.254048 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:01.254028 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvbs8" podUID="b19d3ac3-a780-4f36-bf0e-647a0fe5ce53" containerName="discovery" containerID="cri-o://081e2c739a7c88e923c627fa06849146e85d33492f92156173fd7f8008c2aa5e" gracePeriod=30 Sep 09 02:38:01.513815 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:01.513787 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvbs8" Sep 09 02:38:01.598184 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:01.598094 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b19d3ac3-a780-4f36-bf0e-647a0fe5ce53-cacerts\") pod \"b19d3ac3-a780-4f36-bf0e-647a0fe5ce53\" (UID: \"b19d3ac3-a780-4f36-bf0e-647a0fe5ce53\") " Sep 09 02:38:01.598717 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:01.598411 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b19d3ac3-a780-4f36-bf0e-647a0fe5ce53-local-certs\") pod \"b19d3ac3-a780-4f36-bf0e-647a0fe5ce53\" (UID: \"b19d3ac3-a780-4f36-bf0e-647a0fe5ce53\") " Sep 09 02:38:01.598717 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:01.598447 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b19d3ac3-a780-4f36-bf0e-647a0fe5ce53-istio-csr-ca-configmap\") pod \"b19d3ac3-a780-4f36-bf0e-647a0fe5ce53\" (UID: \"b19d3ac3-a780-4f36-bf0e-647a0fe5ce53\") " Sep 09 02:38:01.599113 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:01.599083 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b19d3ac3-a780-4f36-bf0e-647a0fe5ce53-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "b19d3ac3-a780-4f36-bf0e-647a0fe5ce53" (UID: "b19d3ac3-a780-4f36-bf0e-647a0fe5ce53"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:38:01.601261 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:01.601238 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b19d3ac3-a780-4f36-bf0e-647a0fe5ce53-cacerts" (OuterVolumeSpecName: "cacerts") pod "b19d3ac3-a780-4f36-bf0e-647a0fe5ce53" (UID: "b19d3ac3-a780-4f36-bf0e-647a0fe5ce53"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:38:01.602023 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:01.601995 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19d3ac3-a780-4f36-bf0e-647a0fe5ce53-local-certs" (OuterVolumeSpecName: "local-certs") pod "b19d3ac3-a780-4f36-bf0e-647a0fe5ce53" (UID: "b19d3ac3-a780-4f36-bf0e-647a0fe5ce53"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:38:01.699342 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:01.699254 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b19d3ac3-a780-4f36-bf0e-647a0fe5ce53-istio-csr-dns-cert\") pod \"b19d3ac3-a780-4f36-bf0e-647a0fe5ce53\" (UID: \"b19d3ac3-a780-4f36-bf0e-647a0fe5ce53\") " Sep 09 02:38:01.699342 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:01.699291 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4nln\" (UniqueName: \"kubernetes.io/projected/b19d3ac3-a780-4f36-bf0e-647a0fe5ce53-kube-api-access-g4nln\") pod \"b19d3ac3-a780-4f36-bf0e-647a0fe5ce53\" (UID: \"b19d3ac3-a780-4f36-bf0e-647a0fe5ce53\") " Sep 09 02:38:01.699342 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:01.699320 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b19d3ac3-a780-4f36-bf0e-647a0fe5ce53-istio-token\") pod \"b19d3ac3-a780-4f36-bf0e-647a0fe5ce53\" (UID: \"b19d3ac3-a780-4f36-bf0e-647a0fe5ce53\") " Sep 09 02:38:01.699342 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:01.699341 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b19d3ac3-a780-4f36-bf0e-647a0fe5ce53-istio-kubeconfig\") pod \"b19d3ac3-a780-4f36-bf0e-647a0fe5ce53\" (UID: \"b19d3ac3-a780-4f36-bf0e-647a0fe5ce53\") " Sep 09 02:38:01.699642 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:01.699454 2577 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b19d3ac3-a780-4f36-bf0e-647a0fe5ce53-cacerts\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:38:01.699642 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:01.699465 2577 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b19d3ac3-a780-4f36-bf0e-647a0fe5ce53-local-certs\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:38:01.699642 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:01.699474 2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b19d3ac3-a780-4f36-bf0e-647a0fe5ce53-istio-csr-ca-configmap\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:38:01.701580 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:01.701547 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b19d3ac3-a780-4f36-bf0e-647a0fe5ce53-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "b19d3ac3-a780-4f36-bf0e-647a0fe5ce53" (UID: "b19d3ac3-a780-4f36-bf0e-647a0fe5ce53"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:38:01.701580 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:01.701570 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b19d3ac3-a780-4f36-bf0e-647a0fe5ce53-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "b19d3ac3-a780-4f36-bf0e-647a0fe5ce53" (UID: "b19d3ac3-a780-4f36-bf0e-647a0fe5ce53"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:38:01.701728 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:01.701582 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b19d3ac3-a780-4f36-bf0e-647a0fe5ce53-kube-api-access-g4nln" (OuterVolumeSpecName: "kube-api-access-g4nln") pod "b19d3ac3-a780-4f36-bf0e-647a0fe5ce53" (UID: "b19d3ac3-a780-4f36-bf0e-647a0fe5ce53"). InnerVolumeSpecName "kube-api-access-g4nln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:38:01.701728 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:01.701594 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b19d3ac3-a780-4f36-bf0e-647a0fe5ce53-istio-token" (OuterVolumeSpecName: "istio-token") pod "b19d3ac3-a780-4f36-bf0e-647a0fe5ce53" (UID: "b19d3ac3-a780-4f36-bf0e-647a0fe5ce53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:38:01.799730 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:01.799699 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b19d3ac3-a780-4f36-bf0e-647a0fe5ce53-istio-token\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:38:01.799730 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:01.799720 2577 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b19d3ac3-a780-4f36-bf0e-647a0fe5ce53-istio-kubeconfig\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:38:01.799730 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:01.799733 2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b19d3ac3-a780-4f36-bf0e-647a0fe5ce53-istio-csr-dns-cert\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:38:01.799997 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:01.799742 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4nln\" (UniqueName: \"kubernetes.io/projected/b19d3ac3-a780-4f36-bf0e-647a0fe5ce53-kube-api-access-g4nln\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:38:02.081329 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:02.081293 2577 generic.go:356] "Generic (PLEG): container finished" podID="b19d3ac3-a780-4f36-bf0e-647a0fe5ce53" containerID="081e2c739a7c88e923c627fa06849146e85d33492f92156173fd7f8008c2aa5e" exitCode=0 Sep 09 02:38:02.081815 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:02.081336 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvbs8" Sep 09 02:38:02.081815 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:02.081372 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvbs8" event={"ID":"b19d3ac3-a780-4f36-bf0e-647a0fe5ce53","Type":"ContainerDied","Data":"081e2c739a7c88e923c627fa06849146e85d33492f92156173fd7f8008c2aa5e"} Sep 09 02:38:02.081815 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:02.081420 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvbs8" event={"ID":"b19d3ac3-a780-4f36-bf0e-647a0fe5ce53","Type":"ContainerDied","Data":"6e526b3cb1c60a9871ca27ba9c5663237788e4728e5950d495ba514fd4fb3cca"} Sep 09 02:38:02.081815 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:02.081440 2577 scope.go:117] "RemoveContainer" containerID="081e2c739a7c88e923c627fa06849146e85d33492f92156173fd7f8008c2aa5e" Sep 09 02:38:02.090188 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:02.090172 2577 scope.go:117] "RemoveContainer" containerID="081e2c739a7c88e923c627fa06849146e85d33492f92156173fd7f8008c2aa5e" Sep 09 02:38:02.090501 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:38:02.090477 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"081e2c739a7c88e923c627fa06849146e85d33492f92156173fd7f8008c2aa5e\": container with ID starting with 081e2c739a7c88e923c627fa06849146e85d33492f92156173fd7f8008c2aa5e not found: ID does not exist" containerID="081e2c739a7c88e923c627fa06849146e85d33492f92156173fd7f8008c2aa5e" Sep 09 02:38:02.090552 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:02.090514 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"081e2c739a7c88e923c627fa06849146e85d33492f92156173fd7f8008c2aa5e"} err="failed to get container status \"081e2c739a7c88e923c627fa06849146e85d33492f92156173fd7f8008c2aa5e\": rpc error: code = NotFound desc = could not find container \"081e2c739a7c88e923c627fa06849146e85d33492f92156173fd7f8008c2aa5e\": container with ID starting with 081e2c739a7c88e923c627fa06849146e85d33492f92156173fd7f8008c2aa5e not found: ID does not exist" Sep 09 02:38:02.113699 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:02.113672 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvbs8"] Sep 09 02:38:02.120517 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:02.120497 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvbs8"] Sep 09 02:38:02.602782 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:02.602744 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-tbgkg"] Sep 09 02:38:02.603081 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:02.603067 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b19d3ac3-a780-4f36-bf0e-647a0fe5ce53" containerName="discovery" Sep 09 02:38:02.603148 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:02.603085 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b19d3ac3-a780-4f36-bf0e-647a0fe5ce53" containerName="discovery" Sep 09 02:38:02.603148 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:02.603137 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b19d3ac3-a780-4f36-bf0e-647a0fe5ce53" containerName="discovery" Sep 09 02:38:02.607702 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:02.607674 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tbgkg" Sep 09 02:38:02.609859 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:02.609840 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-ntm9t"] Sep 09 02:38:02.611198 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:02.611173 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-859mh\"" Sep 09 02:38:02.611493 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:02.611479 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 09 02:38:02.612845 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:02.612824 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ntm9t" Sep 09 02:38:02.617027 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:02.617007 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-pvp45\"" Sep 09 02:38:02.617134 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:02.617016 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 09 02:38:02.626649 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:02.626627 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-tbgkg"] Sep 09 02:38:02.633330 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:02.633308 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-ntm9t"] Sep 09 02:38:02.704779 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:02.704749 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgphx\" (UniqueName: \"kubernetes.io/projected/77e84108-8c09-487b-add4-4e5e27557030-kube-api-access-fgphx\") pod \"llmisvc-controller-manager-777c6bd7d5-tbgkg\" (UID: \"77e84108-8c09-487b-add4-4e5e27557030\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tbgkg" Sep 09 02:38:02.704986 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:02.704800 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctdmc\" (UniqueName: \"kubernetes.io/projected/0112eec7-e85f-4902-8e6e-18381f89d197-kube-api-access-ctdmc\") pod \"kserve-controller-manager-d6f4b65b8-ntm9t\" (UID: \"0112eec7-e85f-4902-8e6e-18381f89d197\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ntm9t" Sep 09 02:38:02.704986 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:02.704829 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0112eec7-e85f-4902-8e6e-18381f89d197-cert\") pod \"kserve-controller-manager-d6f4b65b8-ntm9t\" (UID: \"0112eec7-e85f-4902-8e6e-18381f89d197\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ntm9t" Sep 09 02:38:02.704986 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:02.704851 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77e84108-8c09-487b-add4-4e5e27557030-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tbgkg\" (UID: \"77e84108-8c09-487b-add4-4e5e27557030\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tbgkg" Sep 09 02:38:02.806047 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:02.806009 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctdmc\" (UniqueName: \"kubernetes.io/projected/0112eec7-e85f-4902-8e6e-18381f89d197-kube-api-access-ctdmc\") pod \"kserve-controller-manager-d6f4b65b8-ntm9t\" (UID: \"0112eec7-e85f-4902-8e6e-18381f89d197\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ntm9t" Sep 09 02:38:02.806047 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:02.806046 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0112eec7-e85f-4902-8e6e-18381f89d197-cert\") pod \"kserve-controller-manager-d6f4b65b8-ntm9t\" (UID: \"0112eec7-e85f-4902-8e6e-18381f89d197\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ntm9t" Sep 09 02:38:02.806310 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:02.806067 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77e84108-8c09-487b-add4-4e5e27557030-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tbgkg\" (UID: \"77e84108-8c09-487b-add4-4e5e27557030\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tbgkg" Sep 09 02:38:02.806310 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:02.806119 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgphx\" (UniqueName: \"kubernetes.io/projected/77e84108-8c09-487b-add4-4e5e27557030-kube-api-access-fgphx\") pod \"llmisvc-controller-manager-777c6bd7d5-tbgkg\" (UID: \"77e84108-8c09-487b-add4-4e5e27557030\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tbgkg" Sep 09 02:38:02.808782 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:02.808756 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0112eec7-e85f-4902-8e6e-18381f89d197-cert\") pod \"kserve-controller-manager-d6f4b65b8-ntm9t\" (UID: \"0112eec7-e85f-4902-8e6e-18381f89d197\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ntm9t" Sep 09 02:38:02.808915 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:02.808759 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77e84108-8c09-487b-add4-4e5e27557030-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-tbgkg\" (UID: \"77e84108-8c09-487b-add4-4e5e27557030\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tbgkg" Sep 09 02:38:02.818679 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:02.818655 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctdmc\" (UniqueName: \"kubernetes.io/projected/0112eec7-e85f-4902-8e6e-18381f89d197-kube-api-access-ctdmc\") pod \"kserve-controller-manager-d6f4b65b8-ntm9t\" (UID: \"0112eec7-e85f-4902-8e6e-18381f89d197\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ntm9t" Sep 09 02:38:02.819308 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:02.819291 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgphx\" (UniqueName: \"kubernetes.io/projected/77e84108-8c09-487b-add4-4e5e27557030-kube-api-access-fgphx\") pod \"llmisvc-controller-manager-777c6bd7d5-tbgkg\" (UID: \"77e84108-8c09-487b-add4-4e5e27557030\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tbgkg" Sep 09 02:38:02.927638 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:02.927540 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tbgkg" Sep 09 02:38:02.935704 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:02.935663 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ntm9t" Sep 09 02:38:03.164142 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:03.164113 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b19d3ac3-a780-4f36-bf0e-647a0fe5ce53" path="/var/lib/kubelet/pods/b19d3ac3-a780-4f36-bf0e-647a0fe5ce53/volumes" Sep 09 02:38:03.215069 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:38:03.215036 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod77e84108_8c09_487b_add4_4e5e27557030.slice/crio-8c333b54a620546ad2103e6197cc4799ff73d1a14395c1c862fff767f349c88b WatchSource:0}: Error finding container 8c333b54a620546ad2103e6197cc4799ff73d1a14395c1c862fff767f349c88b: Status 404 returned error can't find the container with id 8c333b54a620546ad2103e6197cc4799ff73d1a14395c1c862fff767f349c88b Sep 09 02:38:03.215944 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:03.215917 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-tbgkg"] Sep 09 02:38:03.233999 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:03.233951 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-ntm9t"] Sep 09 02:38:03.235869 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:38:03.235844 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0112eec7_e85f_4902_8e6e_18381f89d197.slice/crio-8a5793f512d2daffcf386f0be69f950f251b5abff664605f4b61dcc046a0e2e8 WatchSource:0}: Error finding container 8a5793f512d2daffcf386f0be69f950f251b5abff664605f4b61dcc046a0e2e8: Status 404 returned error can't find the container with id 8a5793f512d2daffcf386f0be69f950f251b5abff664605f4b61dcc046a0e2e8 Sep 09 02:38:04.094234 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:04.094195 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ntm9t" event={"ID":"0112eec7-e85f-4902-8e6e-18381f89d197","Type":"ContainerStarted","Data":"8a5793f512d2daffcf386f0be69f950f251b5abff664605f4b61dcc046a0e2e8"} Sep 09 02:38:04.095466 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:04.095437 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tbgkg" event={"ID":"77e84108-8c09-487b-add4-4e5e27557030","Type":"ContainerStarted","Data":"8c333b54a620546ad2103e6197cc4799ff73d1a14395c1c862fff767f349c88b"} Sep 09 02:38:07.110663 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:07.110627 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ntm9t" event={"ID":"0112eec7-e85f-4902-8e6e-18381f89d197","Type":"ContainerStarted","Data":"c543ba2ede1cf4cf710786f07f9c25afd1a359aaaf4485c812fd5d4c74248859"} Sep 09 02:38:07.111150 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:07.110737 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ntm9t" Sep 09 02:38:07.112375 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:07.112354 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tbgkg" event={"ID":"77e84108-8c09-487b-add4-4e5e27557030","Type":"ContainerStarted","Data":"d4f64fe015db09cf75066bdd9897665f56c6b54d7f17f632d940bed63c60069f"} Sep 09 02:38:07.112505 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:07.112420 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tbgkg" Sep 09 02:38:07.148019 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:07.147947 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ntm9t" podStartSLOduration=2.344763775 podStartE2EDuration="5.147930102s" totalImagesPullingTime="2.803166327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:38:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:38:07.146857366 +0000 UTC m=+480.511191847" watchObservedRunningTime="2026-09-09 02:38:07.147930102 +0000 UTC m=+480.512264556" Sep 09 02:38:07.177030 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:07.176988 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tbgkg" podStartSLOduration=2.309405498 podStartE2EDuration="5.176952608s" totalImagesPullingTime="2.86754711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:38:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:38:07.175656714 +0000 UTC m=+480.539991178" watchObservedRunningTime="2026-09-09 02:38:07.176952608 +0000 UTC m=+480.541287072" Sep 09 02:38:38.118821 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:38.118789 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-tbgkg" Sep 09 02:38:38.121782 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:38.121763 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ntm9t" Sep 09 02:38:58.196144 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.196112 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bfljf"] Sep 09 02:38:58.203305 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.203274 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bfljf" Sep 09 02:38:58.206311 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.206286 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-gdxls\"" Sep 09 02:38:58.206438 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.206289 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 09 02:38:58.207598 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.207573 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 09 02:38:58.211295 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.211273 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bfljf"] Sep 09 02:38:58.287222 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.287196 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl95kg"] Sep 09 02:38:58.290390 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.290374 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl95kg" Sep 09 02:38:58.299845 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.299826 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl95kg"] Sep 09 02:38:58.326484 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.326464 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a5d913f-698c-4801-a0f5-e9c45e8841b5-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bfljf\" (UID: \"9a5d913f-698c-4801-a0f5-e9c45e8841b5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bfljf" Sep 09 02:38:58.326609 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.326488 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a5d913f-698c-4801-a0f5-e9c45e8841b5-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bfljf\" (UID: \"9a5d913f-698c-4801-a0f5-e9c45e8841b5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bfljf" Sep 09 02:38:58.326609 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.326547 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-894wh\" (UniqueName: \"kubernetes.io/projected/9a5d913f-698c-4801-a0f5-e9c45e8841b5-kube-api-access-894wh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bfljf\" (UID: \"9a5d913f-698c-4801-a0f5-e9c45e8841b5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bfljf" Sep 09 02:38:58.386613 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.386588 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gqqr5"] Sep 09 02:38:58.389674 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.389658 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gqqr5" Sep 09 02:38:58.400026 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.400006 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gqqr5"] Sep 09 02:38:58.427410 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.427384 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wglx8\" (UniqueName: \"kubernetes.io/projected/34094536-2e2c-4ed7-8c45-c76b1af518b5-kube-api-access-wglx8\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl95kg\" (UID: \"34094536-2e2c-4ed7-8c45-c76b1af518b5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl95kg" Sep 09 02:38:58.427514 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.427429 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-894wh\" (UniqueName: \"kubernetes.io/projected/9a5d913f-698c-4801-a0f5-e9c45e8841b5-kube-api-access-894wh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bfljf\" (UID: \"9a5d913f-698c-4801-a0f5-e9c45e8841b5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bfljf" Sep 09 02:38:58.427514 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.427503 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a5d913f-698c-4801-a0f5-e9c45e8841b5-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bfljf\" (UID: \"9a5d913f-698c-4801-a0f5-e9c45e8841b5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bfljf" Sep 09 02:38:58.427622 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.427525 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a5d913f-698c-4801-a0f5-e9c45e8841b5-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bfljf\" (UID: \"9a5d913f-698c-4801-a0f5-e9c45e8841b5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bfljf" Sep 09 02:38:58.427622 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.427543 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/34094536-2e2c-4ed7-8c45-c76b1af518b5-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl95kg\" (UID: \"34094536-2e2c-4ed7-8c45-c76b1af518b5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl95kg" Sep 09 02:38:58.427622 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.427588 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/34094536-2e2c-4ed7-8c45-c76b1af518b5-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl95kg\" (UID: \"34094536-2e2c-4ed7-8c45-c76b1af518b5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl95kg" Sep 09 02:38:58.427882 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.427864 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a5d913f-698c-4801-a0f5-e9c45e8841b5-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bfljf\" (UID: \"9a5d913f-698c-4801-a0f5-e9c45e8841b5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bfljf" Sep 09 02:38:58.427919 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.427879 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a5d913f-698c-4801-a0f5-e9c45e8841b5-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bfljf\" (UID: \"9a5d913f-698c-4801-a0f5-e9c45e8841b5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bfljf" Sep 09 02:38:58.437099 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.437051 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-894wh\" (UniqueName: \"kubernetes.io/projected/9a5d913f-698c-4801-a0f5-e9c45e8841b5-kube-api-access-894wh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bfljf\" (UID: \"9a5d913f-698c-4801-a0f5-e9c45e8841b5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bfljf" Sep 09 02:38:58.489462 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.489401 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8pjlr"] Sep 09 02:38:58.492625 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.492611 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8pjlr" Sep 09 02:38:58.503249 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.503231 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8pjlr"] Sep 09 02:38:58.513158 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.513141 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bfljf" Sep 09 02:38:58.528256 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.528234 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a40a859c-dc98-43e1-8fbe-8c3167ad71b0-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gqqr5\" (UID: \"a40a859c-dc98-43e1-8fbe-8c3167ad71b0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gqqr5" Sep 09 02:38:58.528353 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.528283 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/34094536-2e2c-4ed7-8c45-c76b1af518b5-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl95kg\" (UID: \"34094536-2e2c-4ed7-8c45-c76b1af518b5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl95kg" Sep 09 02:38:58.528353 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.528328 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a40a859c-dc98-43e1-8fbe-8c3167ad71b0-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gqqr5\" (UID: \"a40a859c-dc98-43e1-8fbe-8c3167ad71b0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gqqr5" Sep 09 02:38:58.528353 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.528351 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/34094536-2e2c-4ed7-8c45-c76b1af518b5-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl95kg\" (UID: \"34094536-2e2c-4ed7-8c45-c76b1af518b5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl95kg" Sep 09 02:38:58.528482 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.528375 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kfjx\" (UniqueName: \"kubernetes.io/projected/a40a859c-dc98-43e1-8fbe-8c3167ad71b0-kube-api-access-5kfjx\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gqqr5\" (UID: \"a40a859c-dc98-43e1-8fbe-8c3167ad71b0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gqqr5" Sep 09 02:38:58.528482 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.528467 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wglx8\" (UniqueName: \"kubernetes.io/projected/34094536-2e2c-4ed7-8c45-c76b1af518b5-kube-api-access-wglx8\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl95kg\" (UID: \"34094536-2e2c-4ed7-8c45-c76b1af518b5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl95kg" Sep 09 02:38:58.528704 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.528682 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/34094536-2e2c-4ed7-8c45-c76b1af518b5-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl95kg\" (UID: \"34094536-2e2c-4ed7-8c45-c76b1af518b5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl95kg" Sep 09 02:38:58.528747 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.528709 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/34094536-2e2c-4ed7-8c45-c76b1af518b5-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl95kg\" (UID: \"34094536-2e2c-4ed7-8c45-c76b1af518b5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl95kg" Sep 09 02:38:58.538752 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.538719 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wglx8\" (UniqueName: \"kubernetes.io/projected/34094536-2e2c-4ed7-8c45-c76b1af518b5-kube-api-access-wglx8\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl95kg\" (UID: \"34094536-2e2c-4ed7-8c45-c76b1af518b5\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl95kg" Sep 09 02:38:58.598393 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.598356 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl95kg" Sep 09 02:38:58.629766 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.629732 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a40a859c-dc98-43e1-8fbe-8c3167ad71b0-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gqqr5\" (UID: \"a40a859c-dc98-43e1-8fbe-8c3167ad71b0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gqqr5" Sep 09 02:38:58.629945 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.629792 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kfjx\" (UniqueName: \"kubernetes.io/projected/a40a859c-dc98-43e1-8fbe-8c3167ad71b0-kube-api-access-5kfjx\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gqqr5\" (UID: \"a40a859c-dc98-43e1-8fbe-8c3167ad71b0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gqqr5" Sep 09 02:38:58.629945 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.629854 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a40a859c-dc98-43e1-8fbe-8c3167ad71b0-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gqqr5\" (UID: \"a40a859c-dc98-43e1-8fbe-8c3167ad71b0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gqqr5" Sep 09 02:38:58.629945 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.629884 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8pjlr\" (UID: \"1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8pjlr" Sep 09 02:38:58.630145 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.629972 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8pjlr\" (UID: \"1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8pjlr" Sep 09 02:38:58.630145 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.630000 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l74mv\" (UniqueName: \"kubernetes.io/projected/1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc-kube-api-access-l74mv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8pjlr\" (UID: \"1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8pjlr" Sep 09 02:38:58.630254 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.630228 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a40a859c-dc98-43e1-8fbe-8c3167ad71b0-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gqqr5\" (UID: \"a40a859c-dc98-43e1-8fbe-8c3167ad71b0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gqqr5" Sep 09 02:38:58.630309 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.630250 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a40a859c-dc98-43e1-8fbe-8c3167ad71b0-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gqqr5\" (UID: \"a40a859c-dc98-43e1-8fbe-8c3167ad71b0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gqqr5" Sep 09 02:38:58.641584 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.641555 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kfjx\" (UniqueName: \"kubernetes.io/projected/a40a859c-dc98-43e1-8fbe-8c3167ad71b0-kube-api-access-5kfjx\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gqqr5\" (UID: \"a40a859c-dc98-43e1-8fbe-8c3167ad71b0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gqqr5" Sep 09 02:38:58.698536 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.698503 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gqqr5" Sep 09 02:38:58.730982 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.730923 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8pjlr\" (UID: \"1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8pjlr" Sep 09 02:38:58.731115 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.731002 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l74mv\" (UniqueName: \"kubernetes.io/projected/1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc-kube-api-access-l74mv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8pjlr\" (UID: \"1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8pjlr" Sep 09 02:38:58.731115 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.731101 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8pjlr\" (UID: \"1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8pjlr" Sep 09 02:38:58.731373 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.731351 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8pjlr\" (UID: \"1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8pjlr" Sep 09 02:38:58.731413 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.731386 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8pjlr\" (UID: \"1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8pjlr" Sep 09 02:38:58.744335 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.744280 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l74mv\" (UniqueName: \"kubernetes.io/projected/1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc-kube-api-access-l74mv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8pjlr\" (UID: \"1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8pjlr" Sep 09 02:38:58.786771 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.786745 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bfljf"] Sep 09 02:38:58.788429 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:38:58.788404 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a5d913f_698c_4801_a0f5_e9c45e8841b5.slice/crio-3b80f16809b4b82913d3b47b6273549cf9422ceee46aa3426cdf7488e6f78a71 WatchSource:0}: Error finding container 3b80f16809b4b82913d3b47b6273549cf9422ceee46aa3426cdf7488e6f78a71: Status 404 returned error can't find the container with id 3b80f16809b4b82913d3b47b6273549cf9422ceee46aa3426cdf7488e6f78a71 Sep 09 02:38:58.802031 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.802013 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8pjlr" Sep 09 02:38:58.876011 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.875989 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl95kg"] Sep 09 02:38:58.878711 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:38:58.878686 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34094536_2e2c_4ed7_8c45_c76b1af518b5.slice/crio-650fbef02f2ae9bd08f8fb434625843546985e6b26437558bc84c1df3103167c WatchSource:0}: Error finding container 650fbef02f2ae9bd08f8fb434625843546985e6b26437558bc84c1df3103167c: Status 404 returned error can't find the container with id 650fbef02f2ae9bd08f8fb434625843546985e6b26437558bc84c1df3103167c Sep 09 02:38:58.973439 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:58.973417 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gqqr5"] Sep 09 02:38:58.976011 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:38:58.975988 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda40a859c_dc98_43e1_8fbe_8c3167ad71b0.slice/crio-886a50e62a77fb9d1885c15014da2ffe270429990a31de81e175d504cce49d7b WatchSource:0}: Error finding container 886a50e62a77fb9d1885c15014da2ffe270429990a31de81e175d504cce49d7b: Status 404 returned error can't find the container with id 886a50e62a77fb9d1885c15014da2ffe270429990a31de81e175d504cce49d7b Sep 09 02:38:59.093731 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:59.093676 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8pjlr"] Sep 09 02:38:59.140767 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:38:59.140744 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d093e38_1fdc_466b_8f6f_1b4ab03d9fbc.slice/crio-c8f2263c75cf534dd7acd22d87d49c62ef9d3d7fda7ab60a7f56dfd99f8e899b WatchSource:0}: Error finding container c8f2263c75cf534dd7acd22d87d49c62ef9d3d7fda7ab60a7f56dfd99f8e899b: Status 404 returned error can't find the container with id c8f2263c75cf534dd7acd22d87d49c62ef9d3d7fda7ab60a7f56dfd99f8e899b Sep 09 02:38:59.340395 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:59.340367 2577 generic.go:356] "Generic (PLEG): container finished" podID="34094536-2e2c-4ed7-8c45-c76b1af518b5" containerID="39745cd888843ed42a78388c0105eba4b210ff894227d8ef59e7ce8c738f63b6" exitCode=0 Sep 09 02:38:59.340781 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:59.340448 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl95kg" event={"ID":"34094536-2e2c-4ed7-8c45-c76b1af518b5","Type":"ContainerDied","Data":"39745cd888843ed42a78388c0105eba4b210ff894227d8ef59e7ce8c738f63b6"} Sep 09 02:38:59.340781 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:59.340546 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl95kg" event={"ID":"34094536-2e2c-4ed7-8c45-c76b1af518b5","Type":"ContainerStarted","Data":"650fbef02f2ae9bd08f8fb434625843546985e6b26437558bc84c1df3103167c"} Sep 09 02:38:59.342182 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:59.342160 2577 generic.go:356] "Generic (PLEG): container finished" podID="9a5d913f-698c-4801-a0f5-e9c45e8841b5" containerID="db5ad2d6e9a14ecd46969c2579f137b1b027ef55d3eb3cbe5302445798a87532" exitCode=0 Sep 09 02:38:59.342294 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:59.342230 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bfljf" event={"ID":"9a5d913f-698c-4801-a0f5-e9c45e8841b5","Type":"ContainerDied","Data":"db5ad2d6e9a14ecd46969c2579f137b1b027ef55d3eb3cbe5302445798a87532"} Sep 09 02:38:59.342294 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:59.342268 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bfljf" event={"ID":"9a5d913f-698c-4801-a0f5-e9c45e8841b5","Type":"ContainerStarted","Data":"3b80f16809b4b82913d3b47b6273549cf9422ceee46aa3426cdf7488e6f78a71"} Sep 09 02:38:59.343751 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:59.343730 2577 generic.go:356] "Generic (PLEG): container finished" podID="1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc" containerID="7de0a3960878b4c485a6a48172b961da2c91afd96fb9b89b8cbfe5b20728e06c" exitCode=0 Sep 09 02:38:59.343838 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:59.343762 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8pjlr" event={"ID":"1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc","Type":"ContainerDied","Data":"7de0a3960878b4c485a6a48172b961da2c91afd96fb9b89b8cbfe5b20728e06c"} Sep 09 02:38:59.343838 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:59.343808 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8pjlr" event={"ID":"1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc","Type":"ContainerStarted","Data":"c8f2263c75cf534dd7acd22d87d49c62ef9d3d7fda7ab60a7f56dfd99f8e899b"} Sep 09 02:38:59.345577 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:59.345556 2577 generic.go:356] "Generic (PLEG): container finished" podID="a40a859c-dc98-43e1-8fbe-8c3167ad71b0" containerID="bd2ce18b282c2935e721118aa44675029ef4d189146dfc1e40990f7b0807946b" exitCode=0 Sep 09 02:38:59.345666 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:59.345605 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gqqr5" event={"ID":"a40a859c-dc98-43e1-8fbe-8c3167ad71b0","Type":"ContainerDied","Data":"bd2ce18b282c2935e721118aa44675029ef4d189146dfc1e40990f7b0807946b"} Sep 09 02:38:59.345666 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:38:59.345651 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gqqr5" event={"ID":"a40a859c-dc98-43e1-8fbe-8c3167ad71b0","Type":"ContainerStarted","Data":"886a50e62a77fb9d1885c15014da2ffe270429990a31de81e175d504cce49d7b"} Sep 09 02:39:00.353414 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:00.353378 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bfljf" event={"ID":"9a5d913f-698c-4801-a0f5-e9c45e8841b5","Type":"ContainerStarted","Data":"2764615a35c24c8243b1650833aa1098bc1b358c9c88e8b7e892e9bf0e0a3e89"} Sep 09 02:39:00.355331 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:00.355305 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8pjlr" event={"ID":"1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc","Type":"ContainerStarted","Data":"b0e3a5140e7302005c9a106f058873c9491ddfca51683c1b8109b43c5a06e6c2"} Sep 09 02:39:01.361359 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:01.361324 2577 generic.go:356] "Generic (PLEG): container finished" podID="34094536-2e2c-4ed7-8c45-c76b1af518b5" containerID="d3f81b6f22f7b468d4dd8c8fd888a22716c715604f5dbc2eb68cd356525bba46" exitCode=0 Sep 09 02:39:01.361977 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:01.361410 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl95kg" event={"ID":"34094536-2e2c-4ed7-8c45-c76b1af518b5","Type":"ContainerDied","Data":"d3f81b6f22f7b468d4dd8c8fd888a22716c715604f5dbc2eb68cd356525bba46"} Sep 09 02:39:01.363396 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:01.363378 2577 generic.go:356] "Generic (PLEG): container finished" podID="9a5d913f-698c-4801-a0f5-e9c45e8841b5" containerID="2764615a35c24c8243b1650833aa1098bc1b358c9c88e8b7e892e9bf0e0a3e89" exitCode=0 Sep 09 02:39:01.363503 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:01.363445 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bfljf" event={"ID":"9a5d913f-698c-4801-a0f5-e9c45e8841b5","Type":"ContainerDied","Data":"2764615a35c24c8243b1650833aa1098bc1b358c9c88e8b7e892e9bf0e0a3e89"} Sep 09 02:39:01.365389 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:01.365364 2577 generic.go:356] "Generic (PLEG): container finished" podID="1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc" containerID="b0e3a5140e7302005c9a106f058873c9491ddfca51683c1b8109b43c5a06e6c2" exitCode=0 Sep 09 02:39:01.365493 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:01.365431 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8pjlr" event={"ID":"1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc","Type":"ContainerDied","Data":"b0e3a5140e7302005c9a106f058873c9491ddfca51683c1b8109b43c5a06e6c2"} Sep 09 02:39:01.367615 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:01.367597 2577 generic.go:356] "Generic (PLEG): container finished" podID="a40a859c-dc98-43e1-8fbe-8c3167ad71b0" containerID="b01d4c2f2f5b7f36b5925896ed50e9256b710c6829e7a8e92fabc916b468079d" exitCode=0 Sep 09 02:39:01.367696 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:01.367680 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gqqr5" event={"ID":"a40a859c-dc98-43e1-8fbe-8c3167ad71b0","Type":"ContainerDied","Data":"b01d4c2f2f5b7f36b5925896ed50e9256b710c6829e7a8e92fabc916b468079d"} Sep 09 02:39:02.379481 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:02.379439 2577 generic.go:356] "Generic (PLEG): container finished" podID="34094536-2e2c-4ed7-8c45-c76b1af518b5" containerID="40e13f35775a33891b4066ea1e879bb4c3a83ec14cf6b2f0ea685750073aab27" exitCode=0 Sep 09 02:39:02.379887 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:02.379507 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl95kg" event={"ID":"34094536-2e2c-4ed7-8c45-c76b1af518b5","Type":"ContainerDied","Data":"40e13f35775a33891b4066ea1e879bb4c3a83ec14cf6b2f0ea685750073aab27"} Sep 09 02:39:02.381487 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:02.381463 2577 generic.go:356] "Generic (PLEG): container finished" podID="9a5d913f-698c-4801-a0f5-e9c45e8841b5" containerID="139e32cb0c8376211f6de99576d3ffcaf2ddd083969498b5a0a78fa4240c779d" exitCode=0 Sep 09 02:39:02.381622 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:02.381545 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bfljf" event={"ID":"9a5d913f-698c-4801-a0f5-e9c45e8841b5","Type":"ContainerDied","Data":"139e32cb0c8376211f6de99576d3ffcaf2ddd083969498b5a0a78fa4240c779d"} Sep 09 02:39:02.383504 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:02.383484 2577 generic.go:356] "Generic (PLEG): container finished" podID="1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc" containerID="390341b3da3b53656242b8acc289b96f8d6098f8977028135d531ad9c6eef932" exitCode=0 Sep 09 02:39:02.383596 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:02.383559 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8pjlr" event={"ID":"1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc","Type":"ContainerDied","Data":"390341b3da3b53656242b8acc289b96f8d6098f8977028135d531ad9c6eef932"} Sep 09 02:39:02.385529 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:02.385512 2577 generic.go:356] "Generic (PLEG): container finished" podID="a40a859c-dc98-43e1-8fbe-8c3167ad71b0" containerID="c3d51c3ec2c9fd4fffc7a179d26e30dfb34b867d2f8ba590446275332c776f68" exitCode=0 Sep 09 02:39:02.385618 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:02.385573 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gqqr5" event={"ID":"a40a859c-dc98-43e1-8fbe-8c3167ad71b0","Type":"ContainerDied","Data":"c3d51c3ec2c9fd4fffc7a179d26e30dfb34b867d2f8ba590446275332c776f68"} Sep 09 02:39:03.535479 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:03.535450 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8pjlr" Sep 09 02:39:03.541078 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:03.541057 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bfljf" Sep 09 02:39:03.574318 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:03.574295 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gqqr5" Sep 09 02:39:03.582698 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:03.582676 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl95kg" Sep 09 02:39:03.670931 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:03.670860 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a5d913f-698c-4801-a0f5-e9c45e8841b5-util\") pod \"9a5d913f-698c-4801-a0f5-e9c45e8841b5\" (UID: \"9a5d913f-698c-4801-a0f5-e9c45e8841b5\") " Sep 09 02:39:03.670931 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:03.670904 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5kfjx\" (UniqueName: \"kubernetes.io/projected/a40a859c-dc98-43e1-8fbe-8c3167ad71b0-kube-api-access-5kfjx\") pod \"a40a859c-dc98-43e1-8fbe-8c3167ad71b0\" (UID: \"a40a859c-dc98-43e1-8fbe-8c3167ad71b0\") " Sep 09 02:39:03.671171 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:03.670951 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc-bundle\") pod \"1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc\" (UID: \"1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc\") " Sep 09 02:39:03.671171 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:03.671004 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a40a859c-dc98-43e1-8fbe-8c3167ad71b0-bundle\") pod \"a40a859c-dc98-43e1-8fbe-8c3167ad71b0\" (UID: \"a40a859c-dc98-43e1-8fbe-8c3167ad71b0\") " Sep 09 02:39:03.671171 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:03.671041 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a40a859c-dc98-43e1-8fbe-8c3167ad71b0-util\") pod \"a40a859c-dc98-43e1-8fbe-8c3167ad71b0\" (UID: \"a40a859c-dc98-43e1-8fbe-8c3167ad71b0\") " Sep 09 02:39:03.671171 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:03.671073 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc-util\") pod \"1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc\" (UID: \"1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc\") " Sep 09 02:39:03.671171 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:03.671109 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a5d913f-698c-4801-a0f5-e9c45e8841b5-bundle\") pod \"9a5d913f-698c-4801-a0f5-e9c45e8841b5\" (UID: \"9a5d913f-698c-4801-a0f5-e9c45e8841b5\") " Sep 09 02:39:03.671171 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:03.671145 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-894wh\" (UniqueName: \"kubernetes.io/projected/9a5d913f-698c-4801-a0f5-e9c45e8841b5-kube-api-access-894wh\") pod \"9a5d913f-698c-4801-a0f5-e9c45e8841b5\" (UID: \"9a5d913f-698c-4801-a0f5-e9c45e8841b5\") " Sep 09 02:39:03.671464 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:03.671178 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/34094536-2e2c-4ed7-8c45-c76b1af518b5-bundle\") pod \"34094536-2e2c-4ed7-8c45-c76b1af518b5\" (UID: \"34094536-2e2c-4ed7-8c45-c76b1af518b5\") " Sep 09 02:39:03.671464 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:03.671223 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wglx8\" (UniqueName: \"kubernetes.io/projected/34094536-2e2c-4ed7-8c45-c76b1af518b5-kube-api-access-wglx8\") pod \"34094536-2e2c-4ed7-8c45-c76b1af518b5\" (UID: \"34094536-2e2c-4ed7-8c45-c76b1af518b5\") " Sep 09 02:39:03.671464 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:03.671251 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l74mv\" (UniqueName: \"kubernetes.io/projected/1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc-kube-api-access-l74mv\") pod \"1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc\" (UID: \"1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc\") " Sep 09 02:39:03.671464 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:03.671291 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/34094536-2e2c-4ed7-8c45-c76b1af518b5-util\") pod \"34094536-2e2c-4ed7-8c45-c76b1af518b5\" (UID: \"34094536-2e2c-4ed7-8c45-c76b1af518b5\") " Sep 09 02:39:03.673061 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:03.671739 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc-bundle" (OuterVolumeSpecName: "bundle") pod "1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc" (UID: "1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:39:03.673061 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:03.671749 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a40a859c-dc98-43e1-8fbe-8c3167ad71b0-bundle" (OuterVolumeSpecName: "bundle") pod "a40a859c-dc98-43e1-8fbe-8c3167ad71b0" (UID: "a40a859c-dc98-43e1-8fbe-8c3167ad71b0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:39:03.673061 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:03.672199 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5d913f-698c-4801-a0f5-e9c45e8841b5-bundle" (OuterVolumeSpecName: "bundle") pod "9a5d913f-698c-4801-a0f5-e9c45e8841b5" (UID: "9a5d913f-698c-4801-a0f5-e9c45e8841b5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:39:03.673061 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:03.672809 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34094536-2e2c-4ed7-8c45-c76b1af518b5-bundle" (OuterVolumeSpecName: "bundle") pod "34094536-2e2c-4ed7-8c45-c76b1af518b5" (UID: "34094536-2e2c-4ed7-8c45-c76b1af518b5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:39:03.673910 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:03.673878 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a40a859c-dc98-43e1-8fbe-8c3167ad71b0-kube-api-access-5kfjx" (OuterVolumeSpecName: "kube-api-access-5kfjx") pod "a40a859c-dc98-43e1-8fbe-8c3167ad71b0" (UID: "a40a859c-dc98-43e1-8fbe-8c3167ad71b0"). InnerVolumeSpecName "kube-api-access-5kfjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:39:03.674580 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:03.674551 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34094536-2e2c-4ed7-8c45-c76b1af518b5-kube-api-access-wglx8" (OuterVolumeSpecName: "kube-api-access-wglx8") pod "34094536-2e2c-4ed7-8c45-c76b1af518b5" (UID: "34094536-2e2c-4ed7-8c45-c76b1af518b5"). InnerVolumeSpecName "kube-api-access-wglx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:39:03.675749 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:03.675717 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc-kube-api-access-l74mv" (OuterVolumeSpecName: "kube-api-access-l74mv") pod "1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc" (UID: "1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc"). InnerVolumeSpecName "kube-api-access-l74mv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:39:03.675933 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:03.675901 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a5d913f-698c-4801-a0f5-e9c45e8841b5-kube-api-access-894wh" (OuterVolumeSpecName: "kube-api-access-894wh") pod "9a5d913f-698c-4801-a0f5-e9c45e8841b5" (UID: "9a5d913f-698c-4801-a0f5-e9c45e8841b5"). InnerVolumeSpecName "kube-api-access-894wh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:39:03.679557 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:03.679514 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc-util" (OuterVolumeSpecName: "util") pod "1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc" (UID: "1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:39:03.679696 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:03.679678 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34094536-2e2c-4ed7-8c45-c76b1af518b5-util" (OuterVolumeSpecName: "util") pod "34094536-2e2c-4ed7-8c45-c76b1af518b5" (UID: "34094536-2e2c-4ed7-8c45-c76b1af518b5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:39:03.679913 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:03.679893 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5d913f-698c-4801-a0f5-e9c45e8841b5-util" (OuterVolumeSpecName: "util") pod "9a5d913f-698c-4801-a0f5-e9c45e8841b5" (UID: "9a5d913f-698c-4801-a0f5-e9c45e8841b5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:39:03.680694 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:03.680674 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a40a859c-dc98-43e1-8fbe-8c3167ad71b0-util" (OuterVolumeSpecName: "util") pod "a40a859c-dc98-43e1-8fbe-8c3167ad71b0" (UID: "a40a859c-dc98-43e1-8fbe-8c3167ad71b0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:39:03.771890 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:03.771865 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a5d913f-698c-4801-a0f5-e9c45e8841b5-util\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:39:03.771890 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:03.771888 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kfjx\" (UniqueName: \"kubernetes.io/projected/a40a859c-dc98-43e1-8fbe-8c3167ad71b0-kube-api-access-5kfjx\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:39:03.772052 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:03.771899 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc-bundle\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:39:03.772052 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:03.771907 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a40a859c-dc98-43e1-8fbe-8c3167ad71b0-bundle\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:39:03.772052 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:03.771914 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a40a859c-dc98-43e1-8fbe-8c3167ad71b0-util\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:39:03.772052 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:03.771921 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc-util\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:39:03.772052 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:03.771927 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a5d913f-698c-4801-a0f5-e9c45e8841b5-bundle\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:39:03.772052 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:03.771935 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-894wh\" (UniqueName: \"kubernetes.io/projected/9a5d913f-698c-4801-a0f5-e9c45e8841b5-kube-api-access-894wh\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:39:03.772052 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:03.771944 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/34094536-2e2c-4ed7-8c45-c76b1af518b5-bundle\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:39:03.772052 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:03.771951 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wglx8\" (UniqueName: \"kubernetes.io/projected/34094536-2e2c-4ed7-8c45-c76b1af518b5-kube-api-access-wglx8\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:39:03.772052 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:03.771981 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l74mv\" (UniqueName: \"kubernetes.io/projected/1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc-kube-api-access-l74mv\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:39:03.772052 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:03.771990 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/34094536-2e2c-4ed7-8c45-c76b1af518b5-util\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:39:04.399067 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:04.399041 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bfljf" Sep 09 02:39:04.399262 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:04.399035 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bfljf" event={"ID":"9a5d913f-698c-4801-a0f5-e9c45e8841b5","Type":"ContainerDied","Data":"3b80f16809b4b82913d3b47b6273549cf9422ceee46aa3426cdf7488e6f78a71"} Sep 09 02:39:04.399262 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:04.399157 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b80f16809b4b82913d3b47b6273549cf9422ceee46aa3426cdf7488e6f78a71" Sep 09 02:39:04.401401 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:04.401373 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8pjlr" event={"ID":"1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc","Type":"ContainerDied","Data":"c8f2263c75cf534dd7acd22d87d49c62ef9d3d7fda7ab60a7f56dfd99f8e899b"} Sep 09 02:39:04.401534 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:04.401400 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a8pjlr" Sep 09 02:39:04.401534 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:04.401419 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8f2263c75cf534dd7acd22d87d49c62ef9d3d7fda7ab60a7f56dfd99f8e899b" Sep 09 02:39:04.403763 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:04.403742 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gqqr5" Sep 09 02:39:04.403889 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:04.403769 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6gqqr5" event={"ID":"a40a859c-dc98-43e1-8fbe-8c3167ad71b0","Type":"ContainerDied","Data":"886a50e62a77fb9d1885c15014da2ffe270429990a31de81e175d504cce49d7b"} Sep 09 02:39:04.403889 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:04.403808 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="886a50e62a77fb9d1885c15014da2ffe270429990a31de81e175d504cce49d7b" Sep 09 02:39:04.406935 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:04.406909 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl95kg" event={"ID":"34094536-2e2c-4ed7-8c45-c76b1af518b5","Type":"ContainerDied","Data":"650fbef02f2ae9bd08f8fb434625843546985e6b26437558bc84c1df3103167c"} Sep 09 02:39:04.407077 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:04.406950 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="650fbef02f2ae9bd08f8fb434625843546985e6b26437558bc84c1df3103167c" Sep 09 02:39:04.407077 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:04.407038 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl95kg" Sep 09 02:39:16.067043 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:16.067013 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-25rbm"] Sep 09 02:39:16.067524 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:16.067279 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc" containerName="util" Sep 09 02:39:16.067524 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:16.067289 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc" containerName="util" Sep 09 02:39:16.067524 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:16.067302 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34094536-2e2c-4ed7-8c45-c76b1af518b5" containerName="util" Sep 09 02:39:16.067524 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:16.067307 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="34094536-2e2c-4ed7-8c45-c76b1af518b5" containerName="util" Sep 09 02:39:16.067524 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:16.067315 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a40a859c-dc98-43e1-8fbe-8c3167ad71b0" containerName="util" Sep 09 02:39:16.067524 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:16.067320 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a40a859c-dc98-43e1-8fbe-8c3167ad71b0" containerName="util" Sep 09 02:39:16.067524 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:16.067328 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc" containerName="extract" Sep 09 02:39:16.067524 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:16.067333 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc" containerName="extract" Sep 09 02:39:16.067524 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:16.067342 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc" containerName="pull" Sep 09 02:39:16.067524 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:16.067347 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc" containerName="pull" Sep 09 02:39:16.067524 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:16.067353 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34094536-2e2c-4ed7-8c45-c76b1af518b5" containerName="extract" Sep 09 02:39:16.067524 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:16.067358 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="34094536-2e2c-4ed7-8c45-c76b1af518b5" containerName="extract" Sep 09 02:39:16.067524 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:16.067363 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a40a859c-dc98-43e1-8fbe-8c3167ad71b0" containerName="extract" Sep 09 02:39:16.067524 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:16.067367 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a40a859c-dc98-43e1-8fbe-8c3167ad71b0" containerName="extract" Sep 09 02:39:16.067524 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:16.067374 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a5d913f-698c-4801-a0f5-e9c45e8841b5" containerName="extract" Sep 09 02:39:16.067524 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:16.067381 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a5d913f-698c-4801-a0f5-e9c45e8841b5" containerName="extract" Sep 09 02:39:16.067524 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:16.067389 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a5d913f-698c-4801-a0f5-e9c45e8841b5" containerName="pull" Sep 09 02:39:16.067524 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:16.067396 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a5d913f-698c-4801-a0f5-e9c45e8841b5" containerName="pull" Sep 09 02:39:16.067524 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:16.067407 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a5d913f-698c-4801-a0f5-e9c45e8841b5" containerName="util" Sep 09 02:39:16.067524 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:16.067412 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a5d913f-698c-4801-a0f5-e9c45e8841b5" containerName="util" Sep 09 02:39:16.067524 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:16.067417 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34094536-2e2c-4ed7-8c45-c76b1af518b5" containerName="pull" Sep 09 02:39:16.067524 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:16.067422 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="34094536-2e2c-4ed7-8c45-c76b1af518b5" containerName="pull" Sep 09 02:39:16.067524 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:16.067427 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a40a859c-dc98-43e1-8fbe-8c3167ad71b0" containerName="pull" Sep 09 02:39:16.067524 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:16.067432 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a40a859c-dc98-43e1-8fbe-8c3167ad71b0" containerName="pull" Sep 09 02:39:16.067524 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:16.067477 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d093e38-1fdc-466b-8f6f-1b4ab03d9fbc" containerName="extract" Sep 09 02:39:16.067524 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:16.067489 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a40a859c-dc98-43e1-8fbe-8c3167ad71b0" containerName="extract" Sep 09 02:39:16.067524 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:16.067495 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="34094536-2e2c-4ed7-8c45-c76b1af518b5" containerName="extract" Sep 09 02:39:16.067524 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:16.067501 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a5d913f-698c-4801-a0f5-e9c45e8841b5" containerName="extract" Sep 09 02:39:16.071790 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:16.071772 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-25rbm" Sep 09 02:39:16.074660 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:16.074635 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-bh6dl\"" Sep 09 02:39:16.074660 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:16.074647 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 09 02:39:16.074801 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:16.074690 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 09 02:39:16.090206 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:16.090170 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-25rbm"] Sep 09 02:39:16.155890 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:16.155865 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmwcm\" (UniqueName: \"kubernetes.io/projected/1e6d4c87-4be7-4c59-8626-902bfe8a975d-kube-api-access-kmwcm\") pod \"kuadrant-operator-controller-manager-76f8f686bf-25rbm\" (UID: \"1e6d4c87-4be7-4c59-8626-902bfe8a975d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-25rbm" Sep 09 02:39:16.156030 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:16.155911 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1e6d4c87-4be7-4c59-8626-902bfe8a975d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-25rbm\" (UID: \"1e6d4c87-4be7-4c59-8626-902bfe8a975d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-25rbm" Sep 09 02:39:16.256481 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:16.256444 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1e6d4c87-4be7-4c59-8626-902bfe8a975d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-25rbm\" (UID: \"1e6d4c87-4be7-4c59-8626-902bfe8a975d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-25rbm" Sep 09 02:39:16.256649 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:16.256518 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmwcm\" (UniqueName: \"kubernetes.io/projected/1e6d4c87-4be7-4c59-8626-902bfe8a975d-kube-api-access-kmwcm\") pod \"kuadrant-operator-controller-manager-76f8f686bf-25rbm\" (UID: \"1e6d4c87-4be7-4c59-8626-902bfe8a975d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-25rbm" Sep 09 02:39:16.256799 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:16.256781 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1e6d4c87-4be7-4c59-8626-902bfe8a975d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-25rbm\" (UID: \"1e6d4c87-4be7-4c59-8626-902bfe8a975d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-25rbm" Sep 09 02:39:16.286899 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:16.286869 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmwcm\" (UniqueName: \"kubernetes.io/projected/1e6d4c87-4be7-4c59-8626-902bfe8a975d-kube-api-access-kmwcm\") pod \"kuadrant-operator-controller-manager-76f8f686bf-25rbm\" (UID: \"1e6d4c87-4be7-4c59-8626-902bfe8a975d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-25rbm" Sep 09 02:39:16.382065 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:16.381987 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-25rbm" Sep 09 02:39:16.664023 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:16.663995 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-25rbm"] Sep 09 02:39:16.665408 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:39:16.665382 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e6d4c87_4be7_4c59_8626_902bfe8a975d.slice/crio-2f8361419fe378436b77cbe941cc9d5ad42dd51680b099aca604fb33318c35a7 WatchSource:0}: Error finding container 2f8361419fe378436b77cbe941cc9d5ad42dd51680b099aca604fb33318c35a7: Status 404 returned error can't find the container with id 2f8361419fe378436b77cbe941cc9d5ad42dd51680b099aca604fb33318c35a7 Sep 09 02:39:17.466158 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:17.466123 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-25rbm" event={"ID":"1e6d4c87-4be7-4c59-8626-902bfe8a975d","Type":"ContainerStarted","Data":"2f8361419fe378436b77cbe941cc9d5ad42dd51680b099aca604fb33318c35a7"} Sep 09 02:39:18.670583 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:18.669203 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vtrxx"] Sep 09 02:39:18.677552 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:18.677526 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vtrxx" Sep 09 02:39:18.682978 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:18.682940 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-5h28d\"" Sep 09 02:39:18.683387 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:18.683370 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 09 02:39:18.697679 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:18.697651 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vtrxx"] Sep 09 02:39:18.778549 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:18.778511 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd65g\" (UniqueName: \"kubernetes.io/projected/0a4c77b7-3d67-41d7-bae0-1dd2ee694b40-kube-api-access-cd65g\") pod \"dns-operator-controller-manager-7b84cd7c9c-vtrxx\" (UID: \"0a4c77b7-3d67-41d7-bae0-1dd2ee694b40\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vtrxx" Sep 09 02:39:18.879424 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:18.879382 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd65g\" (UniqueName: \"kubernetes.io/projected/0a4c77b7-3d67-41d7-bae0-1dd2ee694b40-kube-api-access-cd65g\") pod \"dns-operator-controller-manager-7b84cd7c9c-vtrxx\" (UID: \"0a4c77b7-3d67-41d7-bae0-1dd2ee694b40\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vtrxx" Sep 09 02:39:18.894344 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:18.894306 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd65g\" (UniqueName: \"kubernetes.io/projected/0a4c77b7-3d67-41d7-bae0-1dd2ee694b40-kube-api-access-cd65g\") pod \"dns-operator-controller-manager-7b84cd7c9c-vtrxx\" (UID: \"0a4c77b7-3d67-41d7-bae0-1dd2ee694b40\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vtrxx" Sep 09 02:39:18.996106 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:18.996016 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vtrxx" Sep 09 02:39:19.894949 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:19.894874 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vtrxx"] Sep 09 02:39:19.897730 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:39:19.897702 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a4c77b7_3d67_41d7_bae0_1dd2ee694b40.slice/crio-caeda144b45436ede2502e681149e58ed16a7f5b51827942c8180fb1802020e9 WatchSource:0}: Error finding container caeda144b45436ede2502e681149e58ed16a7f5b51827942c8180fb1802020e9: Status 404 returned error can't find the container with id caeda144b45436ede2502e681149e58ed16a7f5b51827942c8180fb1802020e9 Sep 09 02:39:20.351688 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:20.351649 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-8szz4"] Sep 09 02:39:20.355915 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:20.355889 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8szz4" Sep 09 02:39:20.358874 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:20.358853 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-hbgps\"" Sep 09 02:39:20.367945 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:20.367925 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-8szz4"] Sep 09 02:39:20.388664 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:20.388638 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbblq\" (UniqueName: \"kubernetes.io/projected/5f566381-7ad6-42a4-bde3-1a4a066a7b3a-kube-api-access-wbblq\") pod \"limitador-operator-controller-manager-74495c69f8-8szz4\" (UID: \"5f566381-7ad6-42a4-bde3-1a4a066a7b3a\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8szz4" Sep 09 02:39:20.483682 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:20.483646 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vtrxx" event={"ID":"0a4c77b7-3d67-41d7-bae0-1dd2ee694b40","Type":"ContainerStarted","Data":"caeda144b45436ede2502e681149e58ed16a7f5b51827942c8180fb1802020e9"} Sep 09 02:39:20.489924 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:20.489886 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbblq\" (UniqueName: \"kubernetes.io/projected/5f566381-7ad6-42a4-bde3-1a4a066a7b3a-kube-api-access-wbblq\") pod \"limitador-operator-controller-manager-74495c69f8-8szz4\" (UID: \"5f566381-7ad6-42a4-bde3-1a4a066a7b3a\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8szz4" Sep 09 02:39:20.505095 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:20.505057 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbblq\" (UniqueName: \"kubernetes.io/projected/5f566381-7ad6-42a4-bde3-1a4a066a7b3a-kube-api-access-wbblq\") pod \"limitador-operator-controller-manager-74495c69f8-8szz4\" (UID: \"5f566381-7ad6-42a4-bde3-1a4a066a7b3a\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8szz4" Sep 09 02:39:20.669288 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:20.669212 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8szz4" Sep 09 02:39:20.966076 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:20.965930 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-8szz4"] Sep 09 02:39:20.969146 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:39:20.969114 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f566381_7ad6_42a4_bde3_1a4a066a7b3a.slice/crio-2f767bbeca256d578af87506867fdd08c64d21ffd613d0d0e9f6c1967d2af670 WatchSource:0}: Error finding container 2f767bbeca256d578af87506867fdd08c64d21ffd613d0d0e9f6c1967d2af670: Status 404 returned error can't find the container with id 2f767bbeca256d578af87506867fdd08c64d21ffd613d0d0e9f6c1967d2af670 Sep 09 02:39:21.491805 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:21.491772 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8szz4" event={"ID":"5f566381-7ad6-42a4-bde3-1a4a066a7b3a","Type":"ContainerStarted","Data":"2f767bbeca256d578af87506867fdd08c64d21ffd613d0d0e9f6c1967d2af670"} Sep 09 02:39:22.500017 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:22.499937 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vtrxx" event={"ID":"0a4c77b7-3d67-41d7-bae0-1dd2ee694b40","Type":"ContainerStarted","Data":"5d2503dc72715e6ad7efeb540eeab59454e532561ee703086c3ba3363e4365b6"} Sep 09 02:39:22.500461 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:22.500108 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vtrxx" Sep 09 02:39:22.534176 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:22.534115 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vtrxx" podStartSLOduration=2.416406738 podStartE2EDuration="4.534094618s" totalImagesPullingTime="2.11768788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:39:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:39:22.532649979 +0000 UTC m=+555.896984442" watchObservedRunningTime="2026-09-09 02:39:22.534094618 +0000 UTC m=+555.898429082" Sep 09 02:39:23.507563 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:23.507523 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8szz4" event={"ID":"5f566381-7ad6-42a4-bde3-1a4a066a7b3a","Type":"ContainerStarted","Data":"4244639d9be7aa66b9d7a595e3a3f3ad5afe1c36ae89cfe8592935d6743e5071"} Sep 09 02:39:23.507994 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:23.507781 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8szz4" Sep 09 02:39:23.534537 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:23.534489 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8szz4" podStartSLOduration=1.195087592 podStartE2EDuration="3.534474801s" totalImagesPullingTime="2.339387209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:39:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:39:23.533131706 +0000 UTC m=+556.897466168" watchObservedRunningTime="2026-09-09 02:39:23.534474801 +0000 UTC m=+556.898809264" Sep 09 02:39:25.520734 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:25.520698 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-25rbm" event={"ID":"1e6d4c87-4be7-4c59-8626-902bfe8a975d","Type":"ContainerStarted","Data":"85650dc501b7994daa25eb0f653334c6d5c0c38dbd5ed5d02f27a4b9a7f9e4fd"} Sep 09 02:39:25.521290 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:25.520781 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-25rbm" Sep 09 02:39:25.546055 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:25.546000 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-25rbm" podStartSLOduration=1.228796208 podStartE2EDuration="9.545982734s" totalImagesPullingTime="8.317186526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:39:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:39:25.544588444 +0000 UTC m=+558.908922918" watchObservedRunningTime="2026-09-09 02:39:25.545982734 +0000 UTC m=+558.910317194" Sep 09 02:39:33.509518 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:33.509489 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vtrxx" Sep 09 02:39:34.517396 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:34.517370 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8szz4" Sep 09 02:39:36.527789 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:36.527757 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-25rbm" Sep 09 02:39:38.930676 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:38.930640 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-25rbm"] Sep 09 02:39:38.931100 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:38.930821 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-25rbm" podUID="1e6d4c87-4be7-4c59-8626-902bfe8a975d" containerName="manager" containerID="cri-o://85650dc501b7994daa25eb0f653334c6d5c0c38dbd5ed5d02f27a4b9a7f9e4fd" gracePeriod=2 Sep 09 02:39:38.945827 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:38.945800 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62mdh"] Sep 09 02:39:38.949544 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:38.949520 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62mdh" Sep 09 02:39:38.950244 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:38.950220 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-25rbm"] Sep 09 02:39:38.951685 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:39:38.951656 2577 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"1e6d4c87-4be7-4c59-8626-902bfe8a975d\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-25rbm\": pods \"kuadrant-operator-controller-manager-76f8f686bf-25rbm\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-25rbm" Sep 09 02:39:38.953829 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:39:38.953795 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-25rbm\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" podUID="1e6d4c87-4be7-4c59-8626-902bfe8a975d" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-25rbm" Sep 09 02:39:38.966155 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:38.966125 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-8szz4"] Sep 09 02:39:38.966404 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:38.966380 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8szz4" podUID="5f566381-7ad6-42a4-bde3-1a4a066a7b3a" containerName="manager" containerID="cri-o://4244639d9be7aa66b9d7a595e3a3f3ad5afe1c36ae89cfe8592935d6743e5071" gracePeriod=2 Sep 09 02:39:38.968329 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:38.968277 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w5nm"] Sep 09 02:39:38.969345 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:38.968696 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e6d4c87-4be7-4c59-8626-902bfe8a975d" containerName="manager" Sep 09 02:39:38.969345 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:38.968717 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e6d4c87-4be7-4c59-8626-902bfe8a975d" containerName="manager" Sep 09 02:39:38.969345 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:38.968820 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e6d4c87-4be7-4c59-8626-902bfe8a975d" containerName="manager" Sep 09 02:39:38.971884 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:38.971863 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62mdh"] Sep 09 02:39:38.972085 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:38.972064 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w5nm" Sep 09 02:39:38.990123 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:38.990089 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-8szz4"] Sep 09 02:39:38.992057 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:38.992034 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w5nm"] Sep 09 02:39:38.997633 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:39:38.997558 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-8szz4\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" podUID="5f566381-7ad6-42a4-bde3-1a4a066a7b3a" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8szz4" Sep 09 02:39:39.002947 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.002928 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-nw6h7"] Sep 09 02:39:39.003239 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.003227 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f566381-7ad6-42a4-bde3-1a4a066a7b3a" containerName="manager" Sep 09 02:39:39.003287 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.003242 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f566381-7ad6-42a4-bde3-1a4a066a7b3a" containerName="manager" Sep 09 02:39:39.003323 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.003304 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f566381-7ad6-42a4-bde3-1a4a066a7b3a" containerName="manager" Sep 09 02:39:39.006293 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.006277 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-nw6h7" Sep 09 02:39:39.025297 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.025260 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-nw6h7"] Sep 09 02:39:39.027121 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:39:39.027094 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-25rbm\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" podUID="1e6d4c87-4be7-4c59-8626-902bfe8a975d" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-25rbm" Sep 09 02:39:39.030661 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:39:39.030636 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-25rbm\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" podUID="1e6d4c87-4be7-4c59-8626-902bfe8a975d" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-25rbm" Sep 09 02:39:39.033466 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:39:39.033440 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-8szz4\" is forbidden: User \"system:node:ip-10-0-136-89.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-89.ec2.internal' and this object" podUID="5f566381-7ad6-42a4-bde3-1a4a066a7b3a" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8szz4" Sep 09 02:39:39.038636 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.038605 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rgm8\" (UniqueName: \"kubernetes.io/projected/6934e092-b7ba-4bad-871d-820a59a621e2-kube-api-access-7rgm8\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4w5nm\" (UID: \"6934e092-b7ba-4bad-871d-820a59a621e2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w5nm" Sep 09 02:39:39.038742 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.038653 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pv5d\" (UniqueName: \"kubernetes.io/projected/2f11071e-a472-4f5c-a935-438d1a845313-kube-api-access-8pv5d\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-62mdh\" (UID: \"2f11071e-a472-4f5c-a935-438d1a845313\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62mdh" Sep 09 02:39:39.038742 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.038688 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f11071e-a472-4f5c-a935-438d1a845313-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-62mdh\" (UID: \"2f11071e-a472-4f5c-a935-438d1a845313\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62mdh" Sep 09 02:39:39.038835 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.038778 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhdqx\" (UniqueName: \"kubernetes.io/projected/c2ece8e4-edcc-4693-bd0a-4b36498c01f0-kube-api-access-mhdqx\") pod \"limitador-operator-controller-manager-74495c69f8-nw6h7\" (UID: \"c2ece8e4-edcc-4693-bd0a-4b36498c01f0\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-nw6h7" Sep 09 02:39:39.038909 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.038831 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6934e092-b7ba-4bad-871d-820a59a621e2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4w5nm\" (UID: \"6934e092-b7ba-4bad-871d-820a59a621e2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w5nm" Sep 09 02:39:39.139528 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.139495 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f11071e-a472-4f5c-a935-438d1a845313-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-62mdh\" (UID: \"2f11071e-a472-4f5c-a935-438d1a845313\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62mdh" Sep 09 02:39:39.139649 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.139542 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhdqx\" (UniqueName: \"kubernetes.io/projected/c2ece8e4-edcc-4693-bd0a-4b36498c01f0-kube-api-access-mhdqx\") pod \"limitador-operator-controller-manager-74495c69f8-nw6h7\" (UID: \"c2ece8e4-edcc-4693-bd0a-4b36498c01f0\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-nw6h7" Sep 09 02:39:39.139649 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.139581 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6934e092-b7ba-4bad-871d-820a59a621e2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4w5nm\" (UID: \"6934e092-b7ba-4bad-871d-820a59a621e2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w5nm" Sep 09 02:39:39.139649 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.139633 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rgm8\" (UniqueName: \"kubernetes.io/projected/6934e092-b7ba-4bad-871d-820a59a621e2-kube-api-access-7rgm8\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4w5nm\" (UID: \"6934e092-b7ba-4bad-871d-820a59a621e2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w5nm" Sep 09 02:39:39.139799 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.139676 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pv5d\" (UniqueName: \"kubernetes.io/projected/2f11071e-a472-4f5c-a935-438d1a845313-kube-api-access-8pv5d\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-62mdh\" (UID: \"2f11071e-a472-4f5c-a935-438d1a845313\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62mdh" Sep 09 02:39:39.140148 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.140118 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6934e092-b7ba-4bad-871d-820a59a621e2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4w5nm\" (UID: \"6934e092-b7ba-4bad-871d-820a59a621e2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w5nm" Sep 09 02:39:39.140290 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.140124 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f11071e-a472-4f5c-a935-438d1a845313-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-62mdh\" (UID: \"2f11071e-a472-4f5c-a935-438d1a845313\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62mdh" Sep 09 02:39:39.148908 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.148828 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rgm8\" (UniqueName: \"kubernetes.io/projected/6934e092-b7ba-4bad-871d-820a59a621e2-kube-api-access-7rgm8\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4w5nm\" (UID: \"6934e092-b7ba-4bad-871d-820a59a621e2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w5nm" Sep 09 02:39:39.154622 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.154204 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pv5d\" (UniqueName: \"kubernetes.io/projected/2f11071e-a472-4f5c-a935-438d1a845313-kube-api-access-8pv5d\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-62mdh\" (UID: \"2f11071e-a472-4f5c-a935-438d1a845313\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62mdh" Sep 09 02:39:39.154622 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.154588 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhdqx\" (UniqueName: \"kubernetes.io/projected/c2ece8e4-edcc-4693-bd0a-4b36498c01f0-kube-api-access-mhdqx\") pod \"limitador-operator-controller-manager-74495c69f8-nw6h7\" (UID: \"c2ece8e4-edcc-4693-bd0a-4b36498c01f0\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-nw6h7" Sep 09 02:39:39.193943 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.193925 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8szz4" Sep 09 02:39:39.200056 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.200030 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-25rbm" Sep 09 02:39:39.240565 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.240541 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbblq\" (UniqueName: \"kubernetes.io/projected/5f566381-7ad6-42a4-bde3-1a4a066a7b3a-kube-api-access-wbblq\") pod \"5f566381-7ad6-42a4-bde3-1a4a066a7b3a\" (UID: \"5f566381-7ad6-42a4-bde3-1a4a066a7b3a\") " Sep 09 02:39:39.240676 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.240583 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmwcm\" (UniqueName: \"kubernetes.io/projected/1e6d4c87-4be7-4c59-8626-902bfe8a975d-kube-api-access-kmwcm\") pod \"1e6d4c87-4be7-4c59-8626-902bfe8a975d\" (UID: \"1e6d4c87-4be7-4c59-8626-902bfe8a975d\") " Sep 09 02:39:39.240676 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.240607 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1e6d4c87-4be7-4c59-8626-902bfe8a975d-extensions-socket-volume\") pod \"1e6d4c87-4be7-4c59-8626-902bfe8a975d\" (UID: \"1e6d4c87-4be7-4c59-8626-902bfe8a975d\") " Sep 09 02:39:39.241165 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.241137 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e6d4c87-4be7-4c59-8626-902bfe8a975d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1e6d4c87-4be7-4c59-8626-902bfe8a975d" (UID: "1e6d4c87-4be7-4c59-8626-902bfe8a975d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:39:39.242856 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.242836 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f566381-7ad6-42a4-bde3-1a4a066a7b3a-kube-api-access-wbblq" (OuterVolumeSpecName: "kube-api-access-wbblq") pod "5f566381-7ad6-42a4-bde3-1a4a066a7b3a" (UID: "5f566381-7ad6-42a4-bde3-1a4a066a7b3a"). InnerVolumeSpecName "kube-api-access-wbblq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:39:39.242938 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.242871 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e6d4c87-4be7-4c59-8626-902bfe8a975d-kube-api-access-kmwcm" (OuterVolumeSpecName: "kube-api-access-kmwcm") pod "1e6d4c87-4be7-4c59-8626-902bfe8a975d" (UID: "1e6d4c87-4be7-4c59-8626-902bfe8a975d"). InnerVolumeSpecName "kube-api-access-kmwcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:39:39.341911 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.341866 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbblq\" (UniqueName: \"kubernetes.io/projected/5f566381-7ad6-42a4-bde3-1a4a066a7b3a-kube-api-access-wbblq\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:39:39.341911 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.341905 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmwcm\" (UniqueName: \"kubernetes.io/projected/1e6d4c87-4be7-4c59-8626-902bfe8a975d-kube-api-access-kmwcm\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:39:39.341911 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.341918 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1e6d4c87-4be7-4c59-8626-902bfe8a975d-extensions-socket-volume\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:39:39.352842 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.352815 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62mdh" Sep 09 02:39:39.359817 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.359795 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w5nm" Sep 09 02:39:39.365722 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.365700 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-nw6h7" Sep 09 02:39:39.590599 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.590566 2577 generic.go:356] "Generic (PLEG): container finished" podID="1e6d4c87-4be7-4c59-8626-902bfe8a975d" containerID="85650dc501b7994daa25eb0f653334c6d5c0c38dbd5ed5d02f27a4b9a7f9e4fd" exitCode=0 Sep 09 02:39:39.590764 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.590615 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-25rbm" Sep 09 02:39:39.590764 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.590666 2577 scope.go:117] "RemoveContainer" containerID="85650dc501b7994daa25eb0f653334c6d5c0c38dbd5ed5d02f27a4b9a7f9e4fd" Sep 09 02:39:39.592241 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.592212 2577 generic.go:356] "Generic (PLEG): container finished" podID="5f566381-7ad6-42a4-bde3-1a4a066a7b3a" containerID="4244639d9be7aa66b9d7a595e3a3f3ad5afe1c36ae89cfe8592935d6743e5071" exitCode=0 Sep 09 02:39:39.592372 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.592256 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8szz4" Sep 09 02:39:39.599398 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.599379 2577 scope.go:117] "RemoveContainer" containerID="85650dc501b7994daa25eb0f653334c6d5c0c38dbd5ed5d02f27a4b9a7f9e4fd" Sep 09 02:39:39.599693 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:39:39.599672 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85650dc501b7994daa25eb0f653334c6d5c0c38dbd5ed5d02f27a4b9a7f9e4fd\": container with ID starting with 85650dc501b7994daa25eb0f653334c6d5c0c38dbd5ed5d02f27a4b9a7f9e4fd not found: ID does not exist" containerID="85650dc501b7994daa25eb0f653334c6d5c0c38dbd5ed5d02f27a4b9a7f9e4fd" Sep 09 02:39:39.599767 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.599702 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85650dc501b7994daa25eb0f653334c6d5c0c38dbd5ed5d02f27a4b9a7f9e4fd"} err="failed to get container status \"85650dc501b7994daa25eb0f653334c6d5c0c38dbd5ed5d02f27a4b9a7f9e4fd\": rpc error: code = NotFound desc = could not find container \"85650dc501b7994daa25eb0f653334c6d5c0c38dbd5ed5d02f27a4b9a7f9e4fd\": container with ID starting with 85650dc501b7994daa25eb0f653334c6d5c0c38dbd5ed5d02f27a4b9a7f9e4fd not found: ID does not exist" Sep 09 02:39:39.599767 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.599720 2577 scope.go:117] "RemoveContainer" containerID="4244639d9be7aa66b9d7a595e3a3f3ad5afe1c36ae89cfe8592935d6743e5071" Sep 09 02:39:39.607724 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.607709 2577 scope.go:117] "RemoveContainer" containerID="4244639d9be7aa66b9d7a595e3a3f3ad5afe1c36ae89cfe8592935d6743e5071" Sep 09 02:39:39.608034 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:39:39.608009 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4244639d9be7aa66b9d7a595e3a3f3ad5afe1c36ae89cfe8592935d6743e5071\": container with ID starting with 4244639d9be7aa66b9d7a595e3a3f3ad5afe1c36ae89cfe8592935d6743e5071 not found: ID does not exist" containerID="4244639d9be7aa66b9d7a595e3a3f3ad5afe1c36ae89cfe8592935d6743e5071" Sep 09 02:39:39.608121 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.608041 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4244639d9be7aa66b9d7a595e3a3f3ad5afe1c36ae89cfe8592935d6743e5071"} err="failed to get container status \"4244639d9be7aa66b9d7a595e3a3f3ad5afe1c36ae89cfe8592935d6743e5071\": rpc error: code = NotFound desc = could not find container \"4244639d9be7aa66b9d7a595e3a3f3ad5afe1c36ae89cfe8592935d6743e5071\": container with ID starting with 4244639d9be7aa66b9d7a595e3a3f3ad5afe1c36ae89cfe8592935d6743e5071 not found: ID does not exist" Sep 09 02:39:39.639027 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.638998 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62mdh"] Sep 09 02:39:39.639713 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:39:39.639689 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f11071e_a472_4f5c_a935_438d1a845313.slice/crio-917eb5cec3541673dd8e03196b37f391cfe679d142e2fcbbe4fd671c9bcdfb55 WatchSource:0}: Error finding container 917eb5cec3541673dd8e03196b37f391cfe679d142e2fcbbe4fd671c9bcdfb55: Status 404 returned error can't find the container with id 917eb5cec3541673dd8e03196b37f391cfe679d142e2fcbbe4fd671c9bcdfb55 Sep 09 02:39:39.657561 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.657535 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-nw6h7"] Sep 09 02:39:39.659430 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:39.659395 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w5nm"] Sep 09 02:39:39.659692 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:39:39.659670 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6934e092_b7ba_4bad_871d_820a59a621e2.slice/crio-20746a818bb1152ec38b531405a373cd66d734d544b25ed76783d107a6f2c2cd WatchSource:0}: Error finding container 20746a818bb1152ec38b531405a373cd66d734d544b25ed76783d107a6f2c2cd: Status 404 returned error can't find the container with id 20746a818bb1152ec38b531405a373cd66d734d544b25ed76783d107a6f2c2cd Sep 09 02:39:39.660571 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:39:39.660549 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2ece8e4_edcc_4693_bd0a_4b36498c01f0.slice/crio-3348a7612aa8857fa4eb5826de5d86cda48cbbeba608fe55be4bafde61ea6e10 WatchSource:0}: Error finding container 3348a7612aa8857fa4eb5826de5d86cda48cbbeba608fe55be4bafde61ea6e10: Status 404 returned error can't find the container with id 3348a7612aa8857fa4eb5826de5d86cda48cbbeba608fe55be4bafde61ea6e10 Sep 09 02:39:40.603522 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:40.603487 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62mdh" event={"ID":"2f11071e-a472-4f5c-a935-438d1a845313","Type":"ContainerStarted","Data":"3aea6420c7b32a7d2a3e78452b7cfce6114f521e69aafd8d54dc0fc4806e2934"} Sep 09 02:39:40.604003 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:40.603548 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62mdh" event={"ID":"2f11071e-a472-4f5c-a935-438d1a845313","Type":"ContainerStarted","Data":"917eb5cec3541673dd8e03196b37f391cfe679d142e2fcbbe4fd671c9bcdfb55"} Sep 09 02:39:40.604003 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:40.603575 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62mdh" Sep 09 02:39:40.605280 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:40.605249 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-nw6h7" event={"ID":"c2ece8e4-edcc-4693-bd0a-4b36498c01f0","Type":"ContainerStarted","Data":"878c06ee6530ee42d51013f3ffcbfc0778bbea5ae3f2d1b21302d224c6bdc81e"} Sep 09 02:39:40.605421 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:40.605306 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-nw6h7" event={"ID":"c2ece8e4-edcc-4693-bd0a-4b36498c01f0","Type":"ContainerStarted","Data":"3348a7612aa8857fa4eb5826de5d86cda48cbbeba608fe55be4bafde61ea6e10"} Sep 09 02:39:40.605421 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:40.605341 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-nw6h7" Sep 09 02:39:40.606913 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:40.606895 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w5nm" event={"ID":"6934e092-b7ba-4bad-871d-820a59a621e2","Type":"ContainerStarted","Data":"9c613249d413272aaaaec552c88267683d07b89867fa89e0ec3e577f248e62ec"} Sep 09 02:39:40.607037 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:40.606927 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w5nm" event={"ID":"6934e092-b7ba-4bad-871d-820a59a621e2","Type":"ContainerStarted","Data":"20746a818bb1152ec38b531405a373cd66d734d544b25ed76783d107a6f2c2cd"} Sep 09 02:39:40.607037 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:40.606979 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w5nm" Sep 09 02:39:40.638149 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:40.638098 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62mdh" podStartSLOduration=2.638081653 podStartE2EDuration="2.638081653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:39:40.635338111 +0000 UTC m=+573.999672575" watchObservedRunningTime="2026-09-09 02:39:40.638081653 +0000 UTC m=+574.002416115" Sep 09 02:39:40.657528 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:40.657482 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w5nm" podStartSLOduration=2.65746496 podStartE2EDuration="2.65746496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:39:40.655788553 +0000 UTC m=+574.020123015" watchObservedRunningTime="2026-09-09 02:39:40.65746496 +0000 UTC m=+574.021799434" Sep 09 02:39:40.678932 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:40.678881 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-nw6h7" podStartSLOduration=2.67886542 podStartE2EDuration="2.67886542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:39:40.675760335 +0000 UTC m=+574.040094798" watchObservedRunningTime="2026-09-09 02:39:40.67886542 +0000 UTC m=+574.043199882" Sep 09 02:39:41.164950 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:41.164919 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e6d4c87-4be7-4c59-8626-902bfe8a975d" path="/var/lib/kubelet/pods/1e6d4c87-4be7-4c59-8626-902bfe8a975d/volumes" Sep 09 02:39:41.165247 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:41.165233 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f566381-7ad6-42a4-bde3-1a4a066a7b3a" path="/var/lib/kubelet/pods/5f566381-7ad6-42a4-bde3-1a4a066a7b3a/volumes" Sep 09 02:39:51.615171 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:51.615136 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-nw6h7" Sep 09 02:39:51.615630 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:51.615206 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w5nm" Sep 09 02:39:51.615630 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:51.615237 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62mdh" Sep 09 02:39:51.717269 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:51.717241 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w5nm"] Sep 09 02:39:51.717446 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:51.717419 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w5nm" podUID="6934e092-b7ba-4bad-871d-820a59a621e2" containerName="manager" containerID="cri-o://9c613249d413272aaaaec552c88267683d07b89867fa89e0ec3e577f248e62ec" gracePeriod=10 Sep 09 02:39:51.955562 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:51.955538 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w5nm" Sep 09 02:39:52.043374 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:52.043346 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6934e092-b7ba-4bad-871d-820a59a621e2-extensions-socket-volume\") pod \"6934e092-b7ba-4bad-871d-820a59a621e2\" (UID: \"6934e092-b7ba-4bad-871d-820a59a621e2\") " Sep 09 02:39:52.043506 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:52.043417 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rgm8\" (UniqueName: \"kubernetes.io/projected/6934e092-b7ba-4bad-871d-820a59a621e2-kube-api-access-7rgm8\") pod \"6934e092-b7ba-4bad-871d-820a59a621e2\" (UID: \"6934e092-b7ba-4bad-871d-820a59a621e2\") " Sep 09 02:39:52.043717 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:52.043695 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6934e092-b7ba-4bad-871d-820a59a621e2-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6934e092-b7ba-4bad-871d-820a59a621e2" (UID: "6934e092-b7ba-4bad-871d-820a59a621e2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:39:52.045589 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:52.045558 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6934e092-b7ba-4bad-871d-820a59a621e2-kube-api-access-7rgm8" (OuterVolumeSpecName: "kube-api-access-7rgm8") pod "6934e092-b7ba-4bad-871d-820a59a621e2" (UID: "6934e092-b7ba-4bad-871d-820a59a621e2"). InnerVolumeSpecName "kube-api-access-7rgm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:39:52.144812 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:52.144739 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6934e092-b7ba-4bad-871d-820a59a621e2-extensions-socket-volume\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:39:52.144812 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:52.144763 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rgm8\" (UniqueName: \"kubernetes.io/projected/6934e092-b7ba-4bad-871d-820a59a621e2-kube-api-access-7rgm8\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:39:52.253662 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:52.253622 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zjkq5"] Sep 09 02:39:52.254062 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:52.254043 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6934e092-b7ba-4bad-871d-820a59a621e2" containerName="manager" Sep 09 02:39:52.254062 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:52.254063 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6934e092-b7ba-4bad-871d-820a59a621e2" containerName="manager" Sep 09 02:39:52.254224 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:52.254130 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6934e092-b7ba-4bad-871d-820a59a621e2" containerName="manager" Sep 09 02:39:52.257387 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:52.257367 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zjkq5" Sep 09 02:39:52.279539 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:52.279517 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zjkq5"] Sep 09 02:39:52.346105 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:52.346080 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c9a73be-b0a4-4c21-9fb0-e0f94826170b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-zjkq5\" (UID: \"9c9a73be-b0a4-4c21-9fb0-e0f94826170b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zjkq5" Sep 09 02:39:52.346278 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:52.346125 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7fhc\" (UniqueName: \"kubernetes.io/projected/9c9a73be-b0a4-4c21-9fb0-e0f94826170b-kube-api-access-j7fhc\") pod \"kuadrant-operator-controller-manager-5769f99d9f-zjkq5\" (UID: \"9c9a73be-b0a4-4c21-9fb0-e0f94826170b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zjkq5" Sep 09 02:39:52.446999 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:52.446939 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c9a73be-b0a4-4c21-9fb0-e0f94826170b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-zjkq5\" (UID: \"9c9a73be-b0a4-4c21-9fb0-e0f94826170b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zjkq5" Sep 09 02:39:52.447089 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:52.446999 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7fhc\" (UniqueName: \"kubernetes.io/projected/9c9a73be-b0a4-4c21-9fb0-e0f94826170b-kube-api-access-j7fhc\") pod \"kuadrant-operator-controller-manager-5769f99d9f-zjkq5\" (UID: \"9c9a73be-b0a4-4c21-9fb0-e0f94826170b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zjkq5" Sep 09 02:39:52.447282 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:52.447263 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c9a73be-b0a4-4c21-9fb0-e0f94826170b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-zjkq5\" (UID: \"9c9a73be-b0a4-4c21-9fb0-e0f94826170b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zjkq5" Sep 09 02:39:52.458394 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:52.458368 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7fhc\" (UniqueName: \"kubernetes.io/projected/9c9a73be-b0a4-4c21-9fb0-e0f94826170b-kube-api-access-j7fhc\") pod \"kuadrant-operator-controller-manager-5769f99d9f-zjkq5\" (UID: \"9c9a73be-b0a4-4c21-9fb0-e0f94826170b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zjkq5" Sep 09 02:39:52.567143 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:52.567118 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zjkq5" Sep 09 02:39:52.670080 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:52.670048 2577 generic.go:356] "Generic (PLEG): container finished" podID="6934e092-b7ba-4bad-871d-820a59a621e2" containerID="9c613249d413272aaaaec552c88267683d07b89867fa89e0ec3e577f248e62ec" exitCode=0 Sep 09 02:39:52.670427 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:52.670096 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w5nm" Sep 09 02:39:52.670427 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:52.670136 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w5nm" event={"ID":"6934e092-b7ba-4bad-871d-820a59a621e2","Type":"ContainerDied","Data":"9c613249d413272aaaaec552c88267683d07b89867fa89e0ec3e577f248e62ec"} Sep 09 02:39:52.670427 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:52.670190 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w5nm" event={"ID":"6934e092-b7ba-4bad-871d-820a59a621e2","Type":"ContainerDied","Data":"20746a818bb1152ec38b531405a373cd66d734d544b25ed76783d107a6f2c2cd"} Sep 09 02:39:52.670427 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:52.670215 2577 scope.go:117] "RemoveContainer" containerID="9c613249d413272aaaaec552c88267683d07b89867fa89e0ec3e577f248e62ec" Sep 09 02:39:52.678941 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:52.678923 2577 scope.go:117] "RemoveContainer" containerID="9c613249d413272aaaaec552c88267683d07b89867fa89e0ec3e577f248e62ec" Sep 09 02:39:52.679233 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:39:52.679215 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c613249d413272aaaaec552c88267683d07b89867fa89e0ec3e577f248e62ec\": container with ID starting with 9c613249d413272aaaaec552c88267683d07b89867fa89e0ec3e577f248e62ec not found: ID does not exist" containerID="9c613249d413272aaaaec552c88267683d07b89867fa89e0ec3e577f248e62ec" Sep 09 02:39:52.679289 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:52.679252 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c613249d413272aaaaec552c88267683d07b89867fa89e0ec3e577f248e62ec"} err="failed to get container status \"9c613249d413272aaaaec552c88267683d07b89867fa89e0ec3e577f248e62ec\": rpc error: code = NotFound desc = could not find container \"9c613249d413272aaaaec552c88267683d07b89867fa89e0ec3e577f248e62ec\": container with ID starting with 9c613249d413272aaaaec552c88267683d07b89867fa89e0ec3e577f248e62ec not found: ID does not exist" Sep 09 02:39:52.697662 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:52.697609 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w5nm"] Sep 09 02:39:52.702085 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:52.702066 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4w5nm"] Sep 09 02:39:52.852605 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:52.852584 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zjkq5"] Sep 09 02:39:52.853941 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:39:52.853916 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c9a73be_b0a4_4c21_9fb0_e0f94826170b.slice/crio-53963a2ada0462dabb9c06ec9a1740e439898daae4e9c9264dc0b793f9e8362e WatchSource:0}: Error finding container 53963a2ada0462dabb9c06ec9a1740e439898daae4e9c9264dc0b793f9e8362e: Status 404 returned error can't find the container with id 53963a2ada0462dabb9c06ec9a1740e439898daae4e9c9264dc0b793f9e8362e Sep 09 02:39:53.165413 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:53.165383 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6934e092-b7ba-4bad-871d-820a59a621e2" path="/var/lib/kubelet/pods/6934e092-b7ba-4bad-871d-820a59a621e2/volumes" Sep 09 02:39:53.677099 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:53.677055 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zjkq5" event={"ID":"9c9a73be-b0a4-4c21-9fb0-e0f94826170b","Type":"ContainerStarted","Data":"9484f87cb08a0b3ad944796d65598020eb4963edc698b97d617ba7a038b95c3f"} Sep 09 02:39:53.677527 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:53.677121 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zjkq5" event={"ID":"9c9a73be-b0a4-4c21-9fb0-e0f94826170b","Type":"ContainerStarted","Data":"53963a2ada0462dabb9c06ec9a1740e439898daae4e9c9264dc0b793f9e8362e"} Sep 09 02:39:53.677527 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:53.677176 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zjkq5" Sep 09 02:39:53.702774 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:39:53.702703 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zjkq5" podStartSLOduration=1.702682598 podStartE2EDuration="1.702682598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:39:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:39:53.698172504 +0000 UTC m=+587.062506967" watchObservedRunningTime="2026-09-09 02:39:53.702682598 +0000 UTC m=+587.067017064" Sep 09 02:40:04.686342 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:40:04.686309 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zjkq5" Sep 09 02:40:04.747348 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:40:04.747319 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62mdh"] Sep 09 02:40:04.747621 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:40:04.747591 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62mdh" podUID="2f11071e-a472-4f5c-a935-438d1a845313" containerName="manager" containerID="cri-o://3aea6420c7b32a7d2a3e78452b7cfce6114f521e69aafd8d54dc0fc4806e2934" gracePeriod=10 Sep 09 02:40:04.991293 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:40:04.991271 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62mdh" Sep 09 02:40:05.148336 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:40:05.148303 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pv5d\" (UniqueName: \"kubernetes.io/projected/2f11071e-a472-4f5c-a935-438d1a845313-kube-api-access-8pv5d\") pod \"2f11071e-a472-4f5c-a935-438d1a845313\" (UID: \"2f11071e-a472-4f5c-a935-438d1a845313\") " Sep 09 02:40:05.148496 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:40:05.148347 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f11071e-a472-4f5c-a935-438d1a845313-extensions-socket-volume\") pod \"2f11071e-a472-4f5c-a935-438d1a845313\" (UID: \"2f11071e-a472-4f5c-a935-438d1a845313\") " Sep 09 02:40:05.148780 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:40:05.148756 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f11071e-a472-4f5c-a935-438d1a845313-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2f11071e-a472-4f5c-a935-438d1a845313" (UID: "2f11071e-a472-4f5c-a935-438d1a845313"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 02:40:05.150589 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:40:05.150569 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f11071e-a472-4f5c-a935-438d1a845313-kube-api-access-8pv5d" (OuterVolumeSpecName: "kube-api-access-8pv5d") pod "2f11071e-a472-4f5c-a935-438d1a845313" (UID: "2f11071e-a472-4f5c-a935-438d1a845313"). InnerVolumeSpecName "kube-api-access-8pv5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:40:05.249209 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:40:05.249147 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pv5d\" (UniqueName: \"kubernetes.io/projected/2f11071e-a472-4f5c-a935-438d1a845313-kube-api-access-8pv5d\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:40:05.249209 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:40:05.249174 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2f11071e-a472-4f5c-a935-438d1a845313-extensions-socket-volume\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:40:05.743328 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:40:05.743288 2577 generic.go:356] "Generic (PLEG): container finished" podID="2f11071e-a472-4f5c-a935-438d1a845313" containerID="3aea6420c7b32a7d2a3e78452b7cfce6114f521e69aafd8d54dc0fc4806e2934" exitCode=0 Sep 09 02:40:05.743728 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:40:05.743352 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62mdh" Sep 09 02:40:05.743728 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:40:05.743372 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62mdh" event={"ID":"2f11071e-a472-4f5c-a935-438d1a845313","Type":"ContainerDied","Data":"3aea6420c7b32a7d2a3e78452b7cfce6114f521e69aafd8d54dc0fc4806e2934"} Sep 09 02:40:05.743728 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:40:05.743429 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62mdh" event={"ID":"2f11071e-a472-4f5c-a935-438d1a845313","Type":"ContainerDied","Data":"917eb5cec3541673dd8e03196b37f391cfe679d142e2fcbbe4fd671c9bcdfb55"} Sep 09 02:40:05.743728 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:40:05.743451 2577 scope.go:117] "RemoveContainer" containerID="3aea6420c7b32a7d2a3e78452b7cfce6114f521e69aafd8d54dc0fc4806e2934" Sep 09 02:40:05.752409 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:40:05.752287 2577 scope.go:117] "RemoveContainer" containerID="3aea6420c7b32a7d2a3e78452b7cfce6114f521e69aafd8d54dc0fc4806e2934" Sep 09 02:40:05.752611 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:40:05.752589 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aea6420c7b32a7d2a3e78452b7cfce6114f521e69aafd8d54dc0fc4806e2934\": container with ID starting with 3aea6420c7b32a7d2a3e78452b7cfce6114f521e69aafd8d54dc0fc4806e2934 not found: ID does not exist" containerID="3aea6420c7b32a7d2a3e78452b7cfce6114f521e69aafd8d54dc0fc4806e2934" Sep 09 02:40:05.752683 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:40:05.752632 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aea6420c7b32a7d2a3e78452b7cfce6114f521e69aafd8d54dc0fc4806e2934"} err="failed to get container status \"3aea6420c7b32a7d2a3e78452b7cfce6114f521e69aafd8d54dc0fc4806e2934\": rpc error: code = NotFound desc = could not find container \"3aea6420c7b32a7d2a3e78452b7cfce6114f521e69aafd8d54dc0fc4806e2934\": container with ID starting with 3aea6420c7b32a7d2a3e78452b7cfce6114f521e69aafd8d54dc0fc4806e2934 not found: ID does not exist" Sep 09 02:40:05.767027 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:40:05.767002 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62mdh"] Sep 09 02:40:05.771734 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:40:05.771712 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-62mdh"] Sep 09 02:40:07.165574 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:40:07.165544 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f11071e-a472-4f5c-a935-438d1a845313" path="/var/lib/kubelet/pods/2f11071e-a472-4f5c-a935-438d1a845313/volumes" Sep 09 02:41:03.888435 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:03.888403 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-c9dbb95fc-llhcc"] Sep 09 02:41:03.890779 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:03.888822 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f11071e-a472-4f5c-a935-438d1a845313" containerName="manager" Sep 09 02:41:03.890779 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:03.888836 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f11071e-a472-4f5c-a935-438d1a845313" containerName="manager" Sep 09 02:41:03.890779 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:03.888944 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f11071e-a472-4f5c-a935-438d1a845313" containerName="manager" Sep 09 02:41:03.892237 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:03.892217 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c9dbb95fc-llhcc" Sep 09 02:41:03.896305 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:03.896278 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 09 02:41:03.896424 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:03.896374 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 09 02:41:03.896609 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:03.896594 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 09 02:41:03.897530 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:03.897501 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zlr72\"" Sep 09 02:41:03.906542 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:03.906517 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c9dbb95fc-llhcc"] Sep 09 02:41:04.026500 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:04.026472 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/683607d6-1890-4724-81f7-37f8d1bc8d0b-webhook-cert\") pod \"maas-controller-c9dbb95fc-llhcc\" (UID: \"683607d6-1890-4724-81f7-37f8d1bc8d0b\") " pod="opendatahub/maas-controller-c9dbb95fc-llhcc" Sep 09 02:41:04.026685 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:04.026504 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67r4t\" (UniqueName: \"kubernetes.io/projected/683607d6-1890-4724-81f7-37f8d1bc8d0b-kube-api-access-67r4t\") pod \"maas-controller-c9dbb95fc-llhcc\" (UID: \"683607d6-1890-4724-81f7-37f8d1bc8d0b\") " pod="opendatahub/maas-controller-c9dbb95fc-llhcc" Sep 09 02:41:04.026685 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:04.026533 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/683607d6-1890-4724-81f7-37f8d1bc8d0b-metrics-certs\") pod \"maas-controller-c9dbb95fc-llhcc\" (UID: \"683607d6-1890-4724-81f7-37f8d1bc8d0b\") " pod="opendatahub/maas-controller-c9dbb95fc-llhcc" Sep 09 02:41:04.127932 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:04.127893 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/683607d6-1890-4724-81f7-37f8d1bc8d0b-webhook-cert\") pod \"maas-controller-c9dbb95fc-llhcc\" (UID: \"683607d6-1890-4724-81f7-37f8d1bc8d0b\") " pod="opendatahub/maas-controller-c9dbb95fc-llhcc" Sep 09 02:41:04.127932 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:04.127932 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67r4t\" (UniqueName: \"kubernetes.io/projected/683607d6-1890-4724-81f7-37f8d1bc8d0b-kube-api-access-67r4t\") pod \"maas-controller-c9dbb95fc-llhcc\" (UID: \"683607d6-1890-4724-81f7-37f8d1bc8d0b\") " pod="opendatahub/maas-controller-c9dbb95fc-llhcc" Sep 09 02:41:04.128154 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:04.127954 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/683607d6-1890-4724-81f7-37f8d1bc8d0b-metrics-certs\") pod \"maas-controller-c9dbb95fc-llhcc\" (UID: \"683607d6-1890-4724-81f7-37f8d1bc8d0b\") " pod="opendatahub/maas-controller-c9dbb95fc-llhcc" Sep 09 02:41:04.128154 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:41:04.128070 2577 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found Sep 09 02:41:04.128154 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:41:04.128140 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/683607d6-1890-4724-81f7-37f8d1bc8d0b-metrics-certs podName:683607d6-1890-4724-81f7-37f8d1bc8d0b nodeName:}" failed. No retries permitted until 2026-09-09 02:41:04.62812208 +0000 UTC m=+657.992456529 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/683607d6-1890-4724-81f7-37f8d1bc8d0b-metrics-certs") pod "maas-controller-c9dbb95fc-llhcc" (UID: "683607d6-1890-4724-81f7-37f8d1bc8d0b") : secret "maas-controller-metrics-tls" not found Sep 09 02:41:04.130798 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:04.130770 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/683607d6-1890-4724-81f7-37f8d1bc8d0b-webhook-cert\") pod \"maas-controller-c9dbb95fc-llhcc\" (UID: \"683607d6-1890-4724-81f7-37f8d1bc8d0b\") " pod="opendatahub/maas-controller-c9dbb95fc-llhcc" Sep 09 02:41:04.139466 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:04.139404 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67r4t\" (UniqueName: \"kubernetes.io/projected/683607d6-1890-4724-81f7-37f8d1bc8d0b-kube-api-access-67r4t\") pod \"maas-controller-c9dbb95fc-llhcc\" (UID: \"683607d6-1890-4724-81f7-37f8d1bc8d0b\") " pod="opendatahub/maas-controller-c9dbb95fc-llhcc" Sep 09 02:41:04.632774 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:04.632738 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/683607d6-1890-4724-81f7-37f8d1bc8d0b-metrics-certs\") pod \"maas-controller-c9dbb95fc-llhcc\" (UID: \"683607d6-1890-4724-81f7-37f8d1bc8d0b\") " pod="opendatahub/maas-controller-c9dbb95fc-llhcc" Sep 09 02:41:04.635307 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:04.635280 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/683607d6-1890-4724-81f7-37f8d1bc8d0b-metrics-certs\") pod \"maas-controller-c9dbb95fc-llhcc\" (UID: \"683607d6-1890-4724-81f7-37f8d1bc8d0b\") " pod="opendatahub/maas-controller-c9dbb95fc-llhcc" Sep 09 02:41:04.804481 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:04.804442 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c9dbb95fc-llhcc" Sep 09 02:41:05.076714 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:05.076690 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-c9dbb95fc-llhcc"] Sep 09 02:41:05.079257 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:41:05.079234 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod683607d6_1890_4724_81f7_37f8d1bc8d0b.slice/crio-582d3750f0f752aa4b218d4bd0692fe73ef8c5e8dd18575e0913732304e59e46 WatchSource:0}: Error finding container 582d3750f0f752aa4b218d4bd0692fe73ef8c5e8dd18575e0913732304e59e46: Status 404 returned error can't find the container with id 582d3750f0f752aa4b218d4bd0692fe73ef8c5e8dd18575e0913732304e59e46 Sep 09 02:41:05.080496 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:05.080479 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 02:41:06.047095 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:06.047058 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c9dbb95fc-llhcc" event={"ID":"683607d6-1890-4724-81f7-37f8d1bc8d0b","Type":"ContainerStarted","Data":"582d3750f0f752aa4b218d4bd0692fe73ef8c5e8dd18575e0913732304e59e46"} Sep 09 02:41:07.053019 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:07.052953 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c9dbb95fc-llhcc" event={"ID":"683607d6-1890-4724-81f7-37f8d1bc8d0b","Type":"ContainerStarted","Data":"4d8092815b00e94f164546901b5d3d9bc16b5a0875fce5ca72ffb0018f7a5f50"} Sep 09 02:41:07.053433 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:07.053071 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-c9dbb95fc-llhcc" Sep 09 02:41:07.074764 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:07.074717 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-c9dbb95fc-llhcc" podStartSLOduration=2.407472851 podStartE2EDuration="4.074701972s" totalImagesPullingTime="1.667229121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:41:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:41:07.0719107 +0000 UTC m=+660.436245207" watchObservedRunningTime="2026-09-09 02:41:07.074701972 +0000 UTC m=+660.439036436" Sep 09 02:41:18.060922 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:18.060885 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-c9dbb95fc-llhcc" Sep 09 02:41:18.586348 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:18.586312 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c9dbb95fc-llhcc"] Sep 09 02:41:18.586524 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:18.586501 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-c9dbb95fc-llhcc" podUID="683607d6-1890-4724-81f7-37f8d1bc8d0b" containerName="manager" containerID="cri-o://4d8092815b00e94f164546901b5d3d9bc16b5a0875fce5ca72ffb0018f7a5f50" gracePeriod=30 Sep 09 02:41:18.832496 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:18.832475 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c9dbb95fc-llhcc" Sep 09 02:41:18.950759 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:18.950669 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/683607d6-1890-4724-81f7-37f8d1bc8d0b-webhook-cert\") pod \"683607d6-1890-4724-81f7-37f8d1bc8d0b\" (UID: \"683607d6-1890-4724-81f7-37f8d1bc8d0b\") " Sep 09 02:41:18.950759 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:18.950751 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67r4t\" (UniqueName: \"kubernetes.io/projected/683607d6-1890-4724-81f7-37f8d1bc8d0b-kube-api-access-67r4t\") pod \"683607d6-1890-4724-81f7-37f8d1bc8d0b\" (UID: \"683607d6-1890-4724-81f7-37f8d1bc8d0b\") " Sep 09 02:41:18.950979 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:18.950771 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/683607d6-1890-4724-81f7-37f8d1bc8d0b-metrics-certs\") pod \"683607d6-1890-4724-81f7-37f8d1bc8d0b\" (UID: \"683607d6-1890-4724-81f7-37f8d1bc8d0b\") " Sep 09 02:41:18.952949 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:18.952920 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/683607d6-1890-4724-81f7-37f8d1bc8d0b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "683607d6-1890-4724-81f7-37f8d1bc8d0b" (UID: "683607d6-1890-4724-81f7-37f8d1bc8d0b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:41:18.953101 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:18.952983 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/683607d6-1890-4724-81f7-37f8d1bc8d0b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "683607d6-1890-4724-81f7-37f8d1bc8d0b" (UID: "683607d6-1890-4724-81f7-37f8d1bc8d0b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 02:41:18.953101 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:18.953061 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/683607d6-1890-4724-81f7-37f8d1bc8d0b-kube-api-access-67r4t" (OuterVolumeSpecName: "kube-api-access-67r4t") pod "683607d6-1890-4724-81f7-37f8d1bc8d0b" (UID: "683607d6-1890-4724-81f7-37f8d1bc8d0b"). InnerVolumeSpecName "kube-api-access-67r4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:41:19.052020 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:19.051987 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67r4t\" (UniqueName: \"kubernetes.io/projected/683607d6-1890-4724-81f7-37f8d1bc8d0b-kube-api-access-67r4t\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:41:19.052020 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:19.052013 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/683607d6-1890-4724-81f7-37f8d1bc8d0b-metrics-certs\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:41:19.052020 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:19.052025 2577 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/683607d6-1890-4724-81f7-37f8d1bc8d0b-webhook-cert\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:41:19.115950 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:19.115911 2577 generic.go:356] "Generic (PLEG): container finished" podID="683607d6-1890-4724-81f7-37f8d1bc8d0b" containerID="4d8092815b00e94f164546901b5d3d9bc16b5a0875fce5ca72ffb0018f7a5f50" exitCode=0 Sep 09 02:41:19.116287 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:19.116001 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c9dbb95fc-llhcc" event={"ID":"683607d6-1890-4724-81f7-37f8d1bc8d0b","Type":"ContainerDied","Data":"4d8092815b00e94f164546901b5d3d9bc16b5a0875fce5ca72ffb0018f7a5f50"} Sep 09 02:41:19.116287 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:19.116015 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-c9dbb95fc-llhcc" Sep 09 02:41:19.116287 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:19.116091 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-c9dbb95fc-llhcc" event={"ID":"683607d6-1890-4724-81f7-37f8d1bc8d0b","Type":"ContainerDied","Data":"582d3750f0f752aa4b218d4bd0692fe73ef8c5e8dd18575e0913732304e59e46"} Sep 09 02:41:19.116287 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:19.116122 2577 scope.go:117] "RemoveContainer" containerID="4d8092815b00e94f164546901b5d3d9bc16b5a0875fce5ca72ffb0018f7a5f50" Sep 09 02:41:19.124941 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:19.124925 2577 scope.go:117] "RemoveContainer" containerID="4d8092815b00e94f164546901b5d3d9bc16b5a0875fce5ca72ffb0018f7a5f50" Sep 09 02:41:19.125233 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:41:19.125217 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d8092815b00e94f164546901b5d3d9bc16b5a0875fce5ca72ffb0018f7a5f50\": container with ID starting with 4d8092815b00e94f164546901b5d3d9bc16b5a0875fce5ca72ffb0018f7a5f50 not found: ID does not exist" containerID="4d8092815b00e94f164546901b5d3d9bc16b5a0875fce5ca72ffb0018f7a5f50" Sep 09 02:41:19.125301 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:19.125244 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d8092815b00e94f164546901b5d3d9bc16b5a0875fce5ca72ffb0018f7a5f50"} err="failed to get container status \"4d8092815b00e94f164546901b5d3d9bc16b5a0875fce5ca72ffb0018f7a5f50\": rpc error: code = NotFound desc = could not find container \"4d8092815b00e94f164546901b5d3d9bc16b5a0875fce5ca72ffb0018f7a5f50\": container with ID starting with 4d8092815b00e94f164546901b5d3d9bc16b5a0875fce5ca72ffb0018f7a5f50 not found: ID does not exist" Sep 09 02:41:19.139169 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:19.139143 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-c9dbb95fc-llhcc"] Sep 09 02:41:19.143090 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:19.143068 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-c9dbb95fc-llhcc"] Sep 09 02:41:19.164816 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:19.164796 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="683607d6-1890-4724-81f7-37f8d1bc8d0b" path="/var/lib/kubelet/pods/683607d6-1890-4724-81f7-37f8d1bc8d0b/volumes" Sep 09 02:41:40.876799 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:40.876760 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-qd7xc"] Sep 09 02:41:40.877278 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:40.877262 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="683607d6-1890-4724-81f7-37f8d1bc8d0b" containerName="manager" Sep 09 02:41:40.877347 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:40.877281 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="683607d6-1890-4724-81f7-37f8d1bc8d0b" containerName="manager" Sep 09 02:41:40.877396 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:40.877370 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="683607d6-1890-4724-81f7-37f8d1bc8d0b" containerName="manager" Sep 09 02:41:40.883100 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:40.883080 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-qd7xc" Sep 09 02:41:40.888207 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:40.888175 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 09 02:41:40.888322 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:40.888272 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-pfmqc\"" Sep 09 02:41:40.907040 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:40.907016 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-qd7xc"] Sep 09 02:41:40.916202 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:40.916180 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgxl4\" (UniqueName: \"kubernetes.io/projected/04c796d2-3ea0-4a3c-9635-4a5d5ef41cbe-kube-api-access-dgxl4\") pod \"payload-processing-5b54b7fbb7-qd7xc\" (UID: \"04c796d2-3ea0-4a3c-9635-4a5d5ef41cbe\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-qd7xc" Sep 09 02:41:40.916326 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:40.916217 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04c796d2-3ea0-4a3c-9635-4a5d5ef41cbe-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-qd7xc\" (UID: \"04c796d2-3ea0-4a3c-9635-4a5d5ef41cbe\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-qd7xc" Sep 09 02:41:41.017428 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:41.017381 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgxl4\" (UniqueName: \"kubernetes.io/projected/04c796d2-3ea0-4a3c-9635-4a5d5ef41cbe-kube-api-access-dgxl4\") pod \"payload-processing-5b54b7fbb7-qd7xc\" (UID: \"04c796d2-3ea0-4a3c-9635-4a5d5ef41cbe\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-qd7xc" Sep 09 02:41:41.017428 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:41.017433 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04c796d2-3ea0-4a3c-9635-4a5d5ef41cbe-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-qd7xc\" (UID: \"04c796d2-3ea0-4a3c-9635-4a5d5ef41cbe\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-qd7xc" Sep 09 02:41:41.018180 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:41.018158 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04c796d2-3ea0-4a3c-9635-4a5d5ef41cbe-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-qd7xc\" (UID: \"04c796d2-3ea0-4a3c-9635-4a5d5ef41cbe\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-qd7xc" Sep 09 02:41:41.027611 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:41.027587 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgxl4\" (UniqueName: \"kubernetes.io/projected/04c796d2-3ea0-4a3c-9635-4a5d5ef41cbe-kube-api-access-dgxl4\") pod \"payload-processing-5b54b7fbb7-qd7xc\" (UID: \"04c796d2-3ea0-4a3c-9635-4a5d5ef41cbe\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-qd7xc" Sep 09 02:41:41.192569 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:41.192493 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-qd7xc" Sep 09 02:41:41.468886 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:41.468856 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-qd7xc"] Sep 09 02:41:41.471509 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:41:41.471479 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04c796d2_3ea0_4a3c_9635_4a5d5ef41cbe.slice/crio-cbeb49a21495cc421cf558e80ad129c97ac5ce22be2fbc96f82b83478e6db190 WatchSource:0}: Error finding container cbeb49a21495cc421cf558e80ad129c97ac5ce22be2fbc96f82b83478e6db190: Status 404 returned error can't find the container with id cbeb49a21495cc421cf558e80ad129c97ac5ce22be2fbc96f82b83478e6db190 Sep 09 02:41:42.235247 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:42.235199 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-qd7xc" event={"ID":"04c796d2-3ea0-4a3c-9635-4a5d5ef41cbe","Type":"ContainerStarted","Data":"cbeb49a21495cc421cf558e80ad129c97ac5ce22be2fbc96f82b83478e6db190"} Sep 09 02:41:43.242108 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:43.242074 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-qd7xc" event={"ID":"04c796d2-3ea0-4a3c-9635-4a5d5ef41cbe","Type":"ContainerStarted","Data":"6657bc61de2269a95cb10579781129c15543bdf3264539ad5f2246fc5729b2bb"} Sep 09 02:41:43.242482 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:43.242137 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-qd7xc" Sep 09 02:41:43.261245 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:43.261192 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-qd7xc" podStartSLOduration=1.605711004 podStartE2EDuration="3.261176008s" totalImagesPullingTime="1.655465004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:41:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:41:43.260384074 +0000 UTC m=+696.624718537" watchObservedRunningTime="2026-09-09 02:41:43.261176008 +0000 UTC m=+696.625510470" Sep 09 02:41:54.250050 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:41:54.250015 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-qd7xc" Sep 09 02:42:12.443943 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:12.443899 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-xptr8"] Sep 09 02:42:12.448499 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:12.448476 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xptr8" Sep 09 02:42:12.452481 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:12.452457 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 09 02:42:12.452611 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:12.452458 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 09 02:42:12.452611 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:12.452460 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 09 02:42:12.452611 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:12.452521 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-t6s5c\"" Sep 09 02:42:12.463625 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:12.463598 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-xptr8"] Sep 09 02:42:12.562378 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:12.562343 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd44bea0-3d05-4a14-a882-acb11ccef0e8-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xptr8\" (UID: \"cd44bea0-3d05-4a14-a882-acb11ccef0e8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xptr8" Sep 09 02:42:12.562378 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:12.562377 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd44bea0-3d05-4a14-a882-acb11ccef0e8-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xptr8\" (UID: \"cd44bea0-3d05-4a14-a882-acb11ccef0e8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xptr8" Sep 09 02:42:12.562611 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:12.562469 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx76z\" (UniqueName: \"kubernetes.io/projected/cd44bea0-3d05-4a14-a882-acb11ccef0e8-kube-api-access-rx76z\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xptr8\" (UID: \"cd44bea0-3d05-4a14-a882-acb11ccef0e8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xptr8" Sep 09 02:42:12.562611 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:12.562504 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd44bea0-3d05-4a14-a882-acb11ccef0e8-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xptr8\" (UID: \"cd44bea0-3d05-4a14-a882-acb11ccef0e8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xptr8" Sep 09 02:42:12.562611 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:12.562589 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd44bea0-3d05-4a14-a882-acb11ccef0e8-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xptr8\" (UID: \"cd44bea0-3d05-4a14-a882-acb11ccef0e8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xptr8" Sep 09 02:42:12.562755 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:12.562640 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd44bea0-3d05-4a14-a882-acb11ccef0e8-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xptr8\" (UID: \"cd44bea0-3d05-4a14-a882-acb11ccef0e8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xptr8" Sep 09 02:42:12.663735 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:12.663696 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx76z\" (UniqueName: \"kubernetes.io/projected/cd44bea0-3d05-4a14-a882-acb11ccef0e8-kube-api-access-rx76z\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xptr8\" (UID: \"cd44bea0-3d05-4a14-a882-acb11ccef0e8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xptr8" Sep 09 02:42:12.663735 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:12.663742 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd44bea0-3d05-4a14-a882-acb11ccef0e8-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xptr8\" (UID: \"cd44bea0-3d05-4a14-a882-acb11ccef0e8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xptr8" Sep 09 02:42:12.664008 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:12.663770 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd44bea0-3d05-4a14-a882-acb11ccef0e8-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xptr8\" (UID: \"cd44bea0-3d05-4a14-a882-acb11ccef0e8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xptr8" Sep 09 02:42:12.664008 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:12.663790 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd44bea0-3d05-4a14-a882-acb11ccef0e8-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xptr8\" (UID: \"cd44bea0-3d05-4a14-a882-acb11ccef0e8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xptr8" Sep 09 02:42:12.664008 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:12.663853 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd44bea0-3d05-4a14-a882-acb11ccef0e8-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xptr8\" (UID: \"cd44bea0-3d05-4a14-a882-acb11ccef0e8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xptr8" Sep 09 02:42:12.664008 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:12.663876 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd44bea0-3d05-4a14-a882-acb11ccef0e8-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xptr8\" (UID: \"cd44bea0-3d05-4a14-a882-acb11ccef0e8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xptr8" Sep 09 02:42:12.664231 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:12.664200 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd44bea0-3d05-4a14-a882-acb11ccef0e8-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xptr8\" (UID: \"cd44bea0-3d05-4a14-a882-acb11ccef0e8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xptr8" Sep 09 02:42:12.664374 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:12.664288 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd44bea0-3d05-4a14-a882-acb11ccef0e8-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xptr8\" (UID: \"cd44bea0-3d05-4a14-a882-acb11ccef0e8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xptr8" Sep 09 02:42:12.664374 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:12.664316 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd44bea0-3d05-4a14-a882-acb11ccef0e8-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xptr8\" (UID: \"cd44bea0-3d05-4a14-a882-acb11ccef0e8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xptr8" Sep 09 02:42:12.666173 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:12.666156 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd44bea0-3d05-4a14-a882-acb11ccef0e8-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xptr8\" (UID: \"cd44bea0-3d05-4a14-a882-acb11ccef0e8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xptr8" Sep 09 02:42:12.666499 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:12.666484 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd44bea0-3d05-4a14-a882-acb11ccef0e8-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xptr8\" (UID: \"cd44bea0-3d05-4a14-a882-acb11ccef0e8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xptr8" Sep 09 02:42:12.673733 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:12.673708 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx76z\" (UniqueName: \"kubernetes.io/projected/cd44bea0-3d05-4a14-a882-acb11ccef0e8-kube-api-access-rx76z\") pod \"e2e-distinct-simulated-kserve-59975d69f9-xptr8\" (UID: \"cd44bea0-3d05-4a14-a882-acb11ccef0e8\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xptr8" Sep 09 02:42:12.759341 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:12.759267 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xptr8" Sep 09 02:42:13.039062 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:13.039027 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-xptr8"] Sep 09 02:42:13.042253 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:42:13.042226 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd44bea0_3d05_4a14_a882_acb11ccef0e8.slice/crio-f4ca52b792dc6f61dedb5683de75a382f9b6ab04a317818a91fea9a42df4f35c WatchSource:0}: Error finding container f4ca52b792dc6f61dedb5683de75a382f9b6ab04a317818a91fea9a42df4f35c: Status 404 returned error can't find the container with id f4ca52b792dc6f61dedb5683de75a382f9b6ab04a317818a91fea9a42df4f35c Sep 09 02:42:13.389422 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:13.389335 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xptr8" event={"ID":"cd44bea0-3d05-4a14-a882-acb11ccef0e8","Type":"ContainerStarted","Data":"f4ca52b792dc6f61dedb5683de75a382f9b6ab04a317818a91fea9a42df4f35c"} Sep 09 02:42:14.400086 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:14.399989 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xptr8" event={"ID":"cd44bea0-3d05-4a14-a882-acb11ccef0e8","Type":"ContainerStarted","Data":"604d4888329c4a362237c62298251767a39d9f93dabb669dd2ff5dc99917ec87"} Sep 09 02:42:14.420599 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:14.420548 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xptr8" podStartSLOduration=1.427554185 podStartE2EDuration="2.420531048s" totalImagesPullingTime="992.976863ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:42:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:42:14.41910869 +0000 UTC m=+727.783443148" watchObservedRunningTime="2026-09-09 02:42:14.420531048 +0000 UTC m=+727.784865511" Sep 09 02:42:22.759975 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:22.759926 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xptr8" Sep 09 02:42:22.760429 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:22.760011 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xptr8" Sep 09 02:42:22.772186 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:22.772165 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xptr8" Sep 09 02:42:23.458065 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:23.458032 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-xptr8" Sep 09 02:42:36.878342 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:36.878305 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-k6924"] Sep 09 02:42:36.881504 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:36.881483 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k6924" Sep 09 02:42:36.884808 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:36.884785 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 09 02:42:36.905091 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:36.905071 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-k6924"] Sep 09 02:42:36.963798 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:36.963779 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1173daaa-3545-412d-965e-ecff86901bba-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k6924\" (UID: \"1173daaa-3545-412d-965e-ecff86901bba\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k6924" Sep 09 02:42:36.963899 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:36.963803 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1173daaa-3545-412d-965e-ecff86901bba-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k6924\" (UID: \"1173daaa-3545-412d-965e-ecff86901bba\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k6924" Sep 09 02:42:36.963899 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:36.963837 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1173daaa-3545-412d-965e-ecff86901bba-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k6924\" (UID: \"1173daaa-3545-412d-965e-ecff86901bba\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k6924" Sep 09 02:42:36.963899 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:36.963891 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1173daaa-3545-412d-965e-ecff86901bba-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k6924\" (UID: \"1173daaa-3545-412d-965e-ecff86901bba\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k6924" Sep 09 02:42:36.964028 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:36.963923 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1173daaa-3545-412d-965e-ecff86901bba-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k6924\" (UID: \"1173daaa-3545-412d-965e-ecff86901bba\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k6924" Sep 09 02:42:36.964028 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:36.963995 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bks2\" (UniqueName: \"kubernetes.io/projected/1173daaa-3545-412d-965e-ecff86901bba-kube-api-access-9bks2\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k6924\" (UID: \"1173daaa-3545-412d-965e-ecff86901bba\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k6924" Sep 09 02:42:37.064736 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:37.064704 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1173daaa-3545-412d-965e-ecff86901bba-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k6924\" (UID: \"1173daaa-3545-412d-965e-ecff86901bba\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k6924" Sep 09 02:42:37.064899 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:37.064745 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1173daaa-3545-412d-965e-ecff86901bba-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k6924\" (UID: \"1173daaa-3545-412d-965e-ecff86901bba\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k6924" Sep 09 02:42:37.064899 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:37.064796 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bks2\" (UniqueName: \"kubernetes.io/projected/1173daaa-3545-412d-965e-ecff86901bba-kube-api-access-9bks2\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k6924\" (UID: \"1173daaa-3545-412d-965e-ecff86901bba\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k6924" Sep 09 02:42:37.064899 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:37.064828 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1173daaa-3545-412d-965e-ecff86901bba-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k6924\" (UID: \"1173daaa-3545-412d-965e-ecff86901bba\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k6924" Sep 09 02:42:37.064899 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:37.064851 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1173daaa-3545-412d-965e-ecff86901bba-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k6924\" (UID: \"1173daaa-3545-412d-965e-ecff86901bba\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k6924" Sep 09 02:42:37.065142 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:37.064911 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1173daaa-3545-412d-965e-ecff86901bba-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k6924\" (UID: \"1173daaa-3545-412d-965e-ecff86901bba\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k6924" Sep 09 02:42:37.065204 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:37.065150 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1173daaa-3545-412d-965e-ecff86901bba-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k6924\" (UID: \"1173daaa-3545-412d-965e-ecff86901bba\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k6924" Sep 09 02:42:37.065260 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:37.065219 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1173daaa-3545-412d-965e-ecff86901bba-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k6924\" (UID: \"1173daaa-3545-412d-965e-ecff86901bba\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k6924" Sep 09 02:42:37.065260 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:37.065251 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1173daaa-3545-412d-965e-ecff86901bba-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k6924\" (UID: \"1173daaa-3545-412d-965e-ecff86901bba\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k6924" Sep 09 02:42:37.067361 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:37.067333 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1173daaa-3545-412d-965e-ecff86901bba-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k6924\" (UID: \"1173daaa-3545-412d-965e-ecff86901bba\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k6924" Sep 09 02:42:37.067505 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:37.067486 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1173daaa-3545-412d-965e-ecff86901bba-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k6924\" (UID: \"1173daaa-3545-412d-965e-ecff86901bba\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k6924" Sep 09 02:42:37.075145 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:37.075123 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bks2\" (UniqueName: \"kubernetes.io/projected/1173daaa-3545-412d-965e-ecff86901bba-kube-api-access-9bks2\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k6924\" (UID: \"1173daaa-3545-412d-965e-ecff86901bba\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k6924" Sep 09 02:42:37.191655 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:37.191579 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k6924" Sep 09 02:42:37.474692 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:37.474622 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-k6924"] Sep 09 02:42:37.481230 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:42:37.481206 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1173daaa_3545_412d_965e_ecff86901bba.slice/crio-d5f5fd7c955e0da11a0aef6337f3ee9c96462268e9f609193c1b9f87e662e0c9 WatchSource:0}: Error finding container d5f5fd7c955e0da11a0aef6337f3ee9c96462268e9f609193c1b9f87e662e0c9: Status 404 returned error can't find the container with id d5f5fd7c955e0da11a0aef6337f3ee9c96462268e9f609193c1b9f87e662e0c9 Sep 09 02:42:37.519330 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:37.519300 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k6924" event={"ID":"1173daaa-3545-412d-965e-ecff86901bba","Type":"ContainerStarted","Data":"d5f5fd7c955e0da11a0aef6337f3ee9c96462268e9f609193c1b9f87e662e0c9"} Sep 09 02:42:38.528916 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:38.528886 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k6924" event={"ID":"1173daaa-3545-412d-965e-ecff86901bba","Type":"ContainerStarted","Data":"2de9cdc7fc1992c4ff1cc645509984932aa63f91f5d56dc59099980d88fe5d05"} Sep 09 02:42:38.550192 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:38.550145 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k6924" podStartSLOduration=2.362889632 podStartE2EDuration="2.550129781s" totalImagesPullingTime="187.240149ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:42:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:42:38.549454741 +0000 UTC m=+751.913789205" watchObservedRunningTime="2026-09-09 02:42:38.550129781 +0000 UTC m=+751.914464244" Sep 09 02:42:47.192527 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:47.192490 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k6924" Sep 09 02:42:47.192985 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:47.192558 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k6924" Sep 09 02:42:47.209263 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:47.209231 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k6924" Sep 09 02:42:47.588221 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:47.588186 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k6924" Sep 09 02:42:54.137217 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:54.137181 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-t25pr"] Sep 09 02:42:54.140653 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:54.140637 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t25pr" Sep 09 02:42:54.143871 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:54.143853 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 09 02:42:54.168772 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:54.168740 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-t25pr"] Sep 09 02:42:54.201398 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:54.201359 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eaf20f7-574a-4fbd-9559-6bcf0f575df2-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t25pr\" (UID: \"3eaf20f7-574a-4fbd-9559-6bcf0f575df2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t25pr" Sep 09 02:42:54.201568 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:54.201408 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eaf20f7-574a-4fbd-9559-6bcf0f575df2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t25pr\" (UID: \"3eaf20f7-574a-4fbd-9559-6bcf0f575df2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t25pr" Sep 09 02:42:54.201568 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:54.201427 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eaf20f7-574a-4fbd-9559-6bcf0f575df2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t25pr\" (UID: \"3eaf20f7-574a-4fbd-9559-6bcf0f575df2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t25pr" Sep 09 02:42:54.201568 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:54.201480 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eaf20f7-574a-4fbd-9559-6bcf0f575df2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t25pr\" (UID: \"3eaf20f7-574a-4fbd-9559-6bcf0f575df2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t25pr" Sep 09 02:42:54.201741 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:54.201596 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs25f\" (UniqueName: \"kubernetes.io/projected/3eaf20f7-574a-4fbd-9559-6bcf0f575df2-kube-api-access-hs25f\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t25pr\" (UID: \"3eaf20f7-574a-4fbd-9559-6bcf0f575df2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t25pr" Sep 09 02:42:54.201741 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:54.201637 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eaf20f7-574a-4fbd-9559-6bcf0f575df2-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t25pr\" (UID: \"3eaf20f7-574a-4fbd-9559-6bcf0f575df2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t25pr" Sep 09 02:42:54.302379 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:54.302341 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs25f\" (UniqueName: \"kubernetes.io/projected/3eaf20f7-574a-4fbd-9559-6bcf0f575df2-kube-api-access-hs25f\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t25pr\" (UID: \"3eaf20f7-574a-4fbd-9559-6bcf0f575df2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t25pr" Sep 09 02:42:54.302379 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:54.302379 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eaf20f7-574a-4fbd-9559-6bcf0f575df2-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t25pr\" (UID: \"3eaf20f7-574a-4fbd-9559-6bcf0f575df2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t25pr" Sep 09 02:42:54.302616 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:54.302428 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eaf20f7-574a-4fbd-9559-6bcf0f575df2-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t25pr\" (UID: \"3eaf20f7-574a-4fbd-9559-6bcf0f575df2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t25pr" Sep 09 02:42:54.302616 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:54.302452 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eaf20f7-574a-4fbd-9559-6bcf0f575df2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t25pr\" (UID: \"3eaf20f7-574a-4fbd-9559-6bcf0f575df2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t25pr" Sep 09 02:42:54.302616 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:54.302471 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eaf20f7-574a-4fbd-9559-6bcf0f575df2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t25pr\" (UID: \"3eaf20f7-574a-4fbd-9559-6bcf0f575df2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t25pr" Sep 09 02:42:54.302616 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:54.302487 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eaf20f7-574a-4fbd-9559-6bcf0f575df2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t25pr\" (UID: \"3eaf20f7-574a-4fbd-9559-6bcf0f575df2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t25pr" Sep 09 02:42:54.302894 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:54.302868 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eaf20f7-574a-4fbd-9559-6bcf0f575df2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t25pr\" (UID: \"3eaf20f7-574a-4fbd-9559-6bcf0f575df2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t25pr" Sep 09 02:42:54.302990 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:54.302938 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eaf20f7-574a-4fbd-9559-6bcf0f575df2-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t25pr\" (UID: \"3eaf20f7-574a-4fbd-9559-6bcf0f575df2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t25pr" Sep 09 02:42:54.303036 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:54.302985 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eaf20f7-574a-4fbd-9559-6bcf0f575df2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t25pr\" (UID: \"3eaf20f7-574a-4fbd-9559-6bcf0f575df2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t25pr" Sep 09 02:42:54.304880 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:54.304859 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eaf20f7-574a-4fbd-9559-6bcf0f575df2-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t25pr\" (UID: \"3eaf20f7-574a-4fbd-9559-6bcf0f575df2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t25pr" Sep 09 02:42:54.305185 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:54.305165 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eaf20f7-574a-4fbd-9559-6bcf0f575df2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t25pr\" (UID: \"3eaf20f7-574a-4fbd-9559-6bcf0f575df2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t25pr" Sep 09 02:42:54.312621 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:54.312599 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs25f\" (UniqueName: \"kubernetes.io/projected/3eaf20f7-574a-4fbd-9559-6bcf0f575df2-kube-api-access-hs25f\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t25pr\" (UID: \"3eaf20f7-574a-4fbd-9559-6bcf0f575df2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t25pr" Sep 09 02:42:54.450439 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:54.450355 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t25pr" Sep 09 02:42:54.722928 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:54.722896 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-t25pr"] Sep 09 02:42:54.724216 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:42:54.724188 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eaf20f7_574a_4fbd_9559_6bcf0f575df2.slice/crio-337c1c80e144e9cbdb010b58e85935e1c8c3d98206553033afc4680a855be763 WatchSource:0}: Error finding container 337c1c80e144e9cbdb010b58e85935e1c8c3d98206553033afc4680a855be763: Status 404 returned error can't find the container with id 337c1c80e144e9cbdb010b58e85935e1c8c3d98206553033afc4680a855be763 Sep 09 02:42:55.618803 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:55.618772 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t25pr" event={"ID":"3eaf20f7-574a-4fbd-9559-6bcf0f575df2","Type":"ContainerStarted","Data":"cf9b68c2d0023d4e85f6bfe6cd94e4da087da7ebcbd1a778317b7ec4491fe85f"} Sep 09 02:42:55.619343 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:55.618822 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t25pr" event={"ID":"3eaf20f7-574a-4fbd-9559-6bcf0f575df2","Type":"ContainerStarted","Data":"337c1c80e144e9cbdb010b58e85935e1c8c3d98206553033afc4680a855be763"} Sep 09 02:42:55.638873 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:42:55.638819 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t25pr" podStartSLOduration=1.458204647 podStartE2EDuration="1.638802817s" totalImagesPullingTime="180.59817ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:42:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:42:55.636761675 +0000 UTC m=+769.001096140" watchObservedRunningTime="2026-09-09 02:42:55.638802817 +0000 UTC m=+769.003137280" Sep 09 02:43:04.450644 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:43:04.450615 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t25pr" Sep 09 02:43:04.451042 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:43:04.450673 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t25pr" Sep 09 02:43:04.462627 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:43:04.462604 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t25pr" Sep 09 02:43:04.674911 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:43:04.674887 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t25pr" Sep 09 02:45:00.144143 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:00.144063 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29815365-qthmt"] Sep 09 02:45:00.147446 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:00.147425 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815365-qthmt" Sep 09 02:45:00.150159 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:00.150134 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbr52\"" Sep 09 02:45:00.152623 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:00.152601 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29815365-qthmt"] Sep 09 02:45:00.174758 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:00.174737 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815365-qthmt" Sep 09 02:45:00.446914 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:00.446889 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29815365-qthmt"] Sep 09 02:45:00.448135 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:45:00.448106 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd398c6a_778c_456f_b230_b41f1c9a6ac1.slice/crio-37f5ec7beb690d1583484ebc954628e4279840f3d7cf1253d1aa0d1310f1975e WatchSource:0}: Error finding container 37f5ec7beb690d1583484ebc954628e4279840f3d7cf1253d1aa0d1310f1975e: Status 404 returned error can't find the container with id 37f5ec7beb690d1583484ebc954628e4279840f3d7cf1253d1aa0d1310f1975e Sep 09 02:45:01.263921 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:01.263885 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815365-qthmt" event={"ID":"dd398c6a-778c-456f-b230-b41f1c9a6ac1","Type":"ContainerStarted","Data":"37f5ec7beb690d1583484ebc954628e4279840f3d7cf1253d1aa0d1310f1975e"} Sep 09 02:45:04.283270 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:04.283238 2577 generic.go:356] "Generic (PLEG): container finished" podID="dd398c6a-778c-456f-b230-b41f1c9a6ac1" containerID="5eeb696243b2b99ede52cc1c7acc675ca346cbaebd767d5e0d8f857d585f3f54" exitCode=0 Sep 09 02:45:04.283641 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:04.283330 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815365-qthmt" event={"ID":"dd398c6a-778c-456f-b230-b41f1c9a6ac1","Type":"ContainerDied","Data":"5eeb696243b2b99ede52cc1c7acc675ca346cbaebd767d5e0d8f857d585f3f54"} Sep 09 02:45:05.423131 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:05.423104 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815365-qthmt" Sep 09 02:45:06.296523 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:06.296494 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815365-qthmt" Sep 09 02:45:06.296726 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:06.296489 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815365-qthmt" event={"ID":"dd398c6a-778c-456f-b230-b41f1c9a6ac1","Type":"ContainerDied","Data":"37f5ec7beb690d1583484ebc954628e4279840f3d7cf1253d1aa0d1310f1975e"} Sep 09 02:45:06.296726 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:06.296618 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37f5ec7beb690d1583484ebc954628e4279840f3d7cf1253d1aa0d1310f1975e" Sep 09 02:45:12.627448 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:12.627404 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b18109-55d84bf57d-cm2wr"] Sep 09 02:45:12.628172 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:12.628148 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd398c6a-778c-456f-b230-b41f1c9a6ac1" containerName="cleanup" Sep 09 02:45:12.628172 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:12.628174 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd398c6a-778c-456f-b230-b41f1c9a6ac1" containerName="cleanup" Sep 09 02:45:12.628400 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:12.628384 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd398c6a-778c-456f-b230-b41f1c9a6ac1" containerName="cleanup" Sep 09 02:45:12.634087 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:12.634062 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b18109-55d84bf57d-cm2wr" Sep 09 02:45:12.636971 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:12.636933 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-b18109\"" Sep 09 02:45:12.644086 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:12.644062 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b18109-55d84bf57d-cm2wr"] Sep 09 02:45:12.647165 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:12.647146 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b18109-7d498769dc-sgr4m"] Sep 09 02:45:12.650911 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:12.650882 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b18109-7d498769dc-sgr4m" Sep 09 02:45:12.653398 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:12.653379 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-b18109-dockercfg-m28j9\"" Sep 09 02:45:12.658899 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:12.658850 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b18109-7d498769dc-sgr4m"] Sep 09 02:45:12.775668 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:12.775632 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75d2b491-13b2-443a-8a19-4e8e7b4f35e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b18109-55d84bf57d-cm2wr\" (UID: \"75d2b491-13b2-443a-8a19-4e8e7b4f35e9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b18109-55d84bf57d-cm2wr" Sep 09 02:45:12.775842 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:12.775677 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7b57\" (UniqueName: \"kubernetes.io/projected/8d288425-ef73-4564-b67c-1e7d7c544efc-kube-api-access-d7b57\") pod \"payload-processing-e2e-inf-a-b18109-7d498769dc-sgr4m\" (UID: \"8d288425-ef73-4564-b67c-1e7d7c544efc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b18109-7d498769dc-sgr4m" Sep 09 02:45:12.775842 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:12.775724 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9djqb\" (UniqueName: \"kubernetes.io/projected/75d2b491-13b2-443a-8a19-4e8e7b4f35e9-kube-api-access-9djqb\") pod \"payload-pre-processing-e2e-inf-a-b18109-55d84bf57d-cm2wr\" (UID: \"75d2b491-13b2-443a-8a19-4e8e7b4f35e9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b18109-55d84bf57d-cm2wr" Sep 09 02:45:12.775842 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:12.775766 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d288425-ef73-4564-b67c-1e7d7c544efc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b18109-7d498769dc-sgr4m\" (UID: \"8d288425-ef73-4564-b67c-1e7d7c544efc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b18109-7d498769dc-sgr4m" Sep 09 02:45:12.877026 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:12.876995 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7b57\" (UniqueName: \"kubernetes.io/projected/8d288425-ef73-4564-b67c-1e7d7c544efc-kube-api-access-d7b57\") pod \"payload-processing-e2e-inf-a-b18109-7d498769dc-sgr4m\" (UID: \"8d288425-ef73-4564-b67c-1e7d7c544efc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b18109-7d498769dc-sgr4m" Sep 09 02:45:12.877209 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:12.877039 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9djqb\" (UniqueName: \"kubernetes.io/projected/75d2b491-13b2-443a-8a19-4e8e7b4f35e9-kube-api-access-9djqb\") pod \"payload-pre-processing-e2e-inf-a-b18109-55d84bf57d-cm2wr\" (UID: \"75d2b491-13b2-443a-8a19-4e8e7b4f35e9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b18109-55d84bf57d-cm2wr" Sep 09 02:45:12.877209 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:12.877082 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d288425-ef73-4564-b67c-1e7d7c544efc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b18109-7d498769dc-sgr4m\" (UID: \"8d288425-ef73-4564-b67c-1e7d7c544efc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b18109-7d498769dc-sgr4m" Sep 09 02:45:12.877209 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:12.877149 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75d2b491-13b2-443a-8a19-4e8e7b4f35e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b18109-55d84bf57d-cm2wr\" (UID: \"75d2b491-13b2-443a-8a19-4e8e7b4f35e9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b18109-55d84bf57d-cm2wr" Sep 09 02:45:12.877805 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:12.877760 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d288425-ef73-4564-b67c-1e7d7c544efc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b18109-7d498769dc-sgr4m\" (UID: \"8d288425-ef73-4564-b67c-1e7d7c544efc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b18109-7d498769dc-sgr4m" Sep 09 02:45:12.877848 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:12.877825 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75d2b491-13b2-443a-8a19-4e8e7b4f35e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b18109-55d84bf57d-cm2wr\" (UID: \"75d2b491-13b2-443a-8a19-4e8e7b4f35e9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b18109-55d84bf57d-cm2wr" Sep 09 02:45:12.886894 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:12.886799 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9djqb\" (UniqueName: \"kubernetes.io/projected/75d2b491-13b2-443a-8a19-4e8e7b4f35e9-kube-api-access-9djqb\") pod \"payload-pre-processing-e2e-inf-a-b18109-55d84bf57d-cm2wr\" (UID: \"75d2b491-13b2-443a-8a19-4e8e7b4f35e9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b18109-55d84bf57d-cm2wr" Sep 09 02:45:12.888301 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:12.888277 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7b57\" (UniqueName: \"kubernetes.io/projected/8d288425-ef73-4564-b67c-1e7d7c544efc-kube-api-access-d7b57\") pod \"payload-processing-e2e-inf-a-b18109-7d498769dc-sgr4m\" (UID: \"8d288425-ef73-4564-b67c-1e7d7c544efc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b18109-7d498769dc-sgr4m" Sep 09 02:45:12.947687 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:12.947662 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b18109-55d84bf57d-cm2wr" Sep 09 02:45:12.961923 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:12.961897 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b18109-7d498769dc-sgr4m" Sep 09 02:45:13.226560 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:13.226538 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b18109-55d84bf57d-cm2wr"] Sep 09 02:45:13.227865 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:45:13.227839 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75d2b491_13b2_443a_8a19_4e8e7b4f35e9.slice/crio-ced0b4445e32bdac8e96d00d783a95013617aa75c823894345cdf89f64857975 WatchSource:0}: Error finding container ced0b4445e32bdac8e96d00d783a95013617aa75c823894345cdf89f64857975: Status 404 returned error can't find the container with id ced0b4445e32bdac8e96d00d783a95013617aa75c823894345cdf89f64857975 Sep 09 02:45:13.252438 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:13.252419 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b18109-7d498769dc-sgr4m"] Sep 09 02:45:13.263061 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:45:13.263034 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d288425_ef73_4564_b67c_1e7d7c544efc.slice/crio-5de5714c99c2ae833ebc75a0f71143182285ca3bc19528adabf0dcb5e2052b97 WatchSource:0}: Error finding container 5de5714c99c2ae833ebc75a0f71143182285ca3bc19528adabf0dcb5e2052b97: Status 404 returned error can't find the container with id 5de5714c99c2ae833ebc75a0f71143182285ca3bc19528adabf0dcb5e2052b97 Sep 09 02:45:13.334838 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:13.334809 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b18109-7d498769dc-sgr4m" event={"ID":"8d288425-ef73-4564-b67c-1e7d7c544efc","Type":"ContainerStarted","Data":"714b58baa6d18efb4260ae2d5c85521e57848f5d34a76256b285569abac32251"} Sep 09 02:45:13.334988 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:13.334857 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b18109-7d498769dc-sgr4m" event={"ID":"8d288425-ef73-4564-b67c-1e7d7c544efc","Type":"ContainerStarted","Data":"5de5714c99c2ae833ebc75a0f71143182285ca3bc19528adabf0dcb5e2052b97"} Sep 09 02:45:13.335357 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:13.335110 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b18109-7d498769dc-sgr4m" Sep 09 02:45:13.336986 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:13.336795 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b18109-55d84bf57d-cm2wr" event={"ID":"75d2b491-13b2-443a-8a19-4e8e7b4f35e9","Type":"ContainerStarted","Data":"ab1d42e851fc8249195bfe71be737027882089db74cf90b0713cf9e2b57bdfbe"} Sep 09 02:45:13.336986 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:13.336835 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b18109-55d84bf57d-cm2wr" event={"ID":"75d2b491-13b2-443a-8a19-4e8e7b4f35e9","Type":"ContainerStarted","Data":"ced0b4445e32bdac8e96d00d783a95013617aa75c823894345cdf89f64857975"} Sep 09 02:45:13.336986 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:13.336869 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b18109-55d84bf57d-cm2wr" Sep 09 02:45:13.352617 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:13.352569 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-b18109-7d498769dc-sgr4m" podStartSLOduration=1.352556723 podStartE2EDuration="1.352556723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:45:13.351351311 +0000 UTC m=+906.715685775" watchObservedRunningTime="2026-09-09 02:45:13.352556723 +0000 UTC m=+906.716891187" Sep 09 02:45:13.373346 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:13.373298 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b18109-55d84bf57d-cm2wr" podStartSLOduration=1.373284073 podStartE2EDuration="1.373284073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:45:13.371538498 +0000 UTC m=+906.735872966" watchObservedRunningTime="2026-09-09 02:45:13.373284073 +0000 UTC m=+906.737618536" Sep 09 02:45:16.416598 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:16.416562 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-79a2702e-b995f748f-gdsz2"] Sep 09 02:45:16.420137 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:16.420121 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-79a2702e-b995f748f-gdsz2" Sep 09 02:45:16.422870 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:16.422846 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-79a2702e\"" Sep 09 02:45:16.422954 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:16.422896 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-79a2702e-dockercfg-q647b\"" Sep 09 02:45:16.438310 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:16.438281 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-79a2702e-b995f748f-gdsz2"] Sep 09 02:45:16.461096 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:16.461070 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-79a2702e-6b47975fbb-w6hnt"] Sep 09 02:45:16.464456 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:16.464441 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-79a2702e-6b47975fbb-w6hnt" Sep 09 02:45:16.475358 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:16.475339 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-79a2702e-6b47975fbb-w6hnt"] Sep 09 02:45:16.613829 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:16.613788 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcdwp\" (UniqueName: \"kubernetes.io/projected/536af2f7-73dc-403d-994e-9f968f8a5f0d-kube-api-access-zcdwp\") pod \"payload-processing-e2e-ait-79a2702e-b995f748f-gdsz2\" (UID: \"536af2f7-73dc-403d-994e-9f968f8a5f0d\") " pod="openshift-ingress/payload-processing-e2e-ait-79a2702e-b995f748f-gdsz2" Sep 09 02:45:16.614014 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:16.613845 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/536af2f7-73dc-403d-994e-9f968f8a5f0d-plugins-config-volume\") pod \"payload-processing-e2e-ait-79a2702e-b995f748f-gdsz2\" (UID: \"536af2f7-73dc-403d-994e-9f968f8a5f0d\") " pod="openshift-ingress/payload-processing-e2e-ait-79a2702e-b995f748f-gdsz2" Sep 09 02:45:16.614014 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:16.613926 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lghgx\" (UniqueName: \"kubernetes.io/projected/9600e019-4472-4e90-8ada-e4aa09c6a179-kube-api-access-lghgx\") pod \"payload-pre-processing-e2e-ait-79a2702e-6b47975fbb-w6hnt\" (UID: \"9600e019-4472-4e90-8ada-e4aa09c6a179\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-79a2702e-6b47975fbb-w6hnt" Sep 09 02:45:16.614105 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:16.614016 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9600e019-4472-4e90-8ada-e4aa09c6a179-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-79a2702e-6b47975fbb-w6hnt\" (UID: \"9600e019-4472-4e90-8ada-e4aa09c6a179\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-79a2702e-6b47975fbb-w6hnt" Sep 09 02:45:16.715578 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:16.715490 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9600e019-4472-4e90-8ada-e4aa09c6a179-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-79a2702e-6b47975fbb-w6hnt\" (UID: \"9600e019-4472-4e90-8ada-e4aa09c6a179\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-79a2702e-6b47975fbb-w6hnt" Sep 09 02:45:16.715740 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:16.715609 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcdwp\" (UniqueName: \"kubernetes.io/projected/536af2f7-73dc-403d-994e-9f968f8a5f0d-kube-api-access-zcdwp\") pod \"payload-processing-e2e-ait-79a2702e-b995f748f-gdsz2\" (UID: \"536af2f7-73dc-403d-994e-9f968f8a5f0d\") " pod="openshift-ingress/payload-processing-e2e-ait-79a2702e-b995f748f-gdsz2" Sep 09 02:45:16.715740 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:16.715647 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/536af2f7-73dc-403d-994e-9f968f8a5f0d-plugins-config-volume\") pod \"payload-processing-e2e-ait-79a2702e-b995f748f-gdsz2\" (UID: \"536af2f7-73dc-403d-994e-9f968f8a5f0d\") " pod="openshift-ingress/payload-processing-e2e-ait-79a2702e-b995f748f-gdsz2" Sep 09 02:45:16.715740 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:16.715678 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lghgx\" (UniqueName: \"kubernetes.io/projected/9600e019-4472-4e90-8ada-e4aa09c6a179-kube-api-access-lghgx\") pod \"payload-pre-processing-e2e-ait-79a2702e-6b47975fbb-w6hnt\" (UID: \"9600e019-4472-4e90-8ada-e4aa09c6a179\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-79a2702e-6b47975fbb-w6hnt" Sep 09 02:45:16.716373 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:16.716352 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9600e019-4472-4e90-8ada-e4aa09c6a179-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-79a2702e-6b47975fbb-w6hnt\" (UID: \"9600e019-4472-4e90-8ada-e4aa09c6a179\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-79a2702e-6b47975fbb-w6hnt" Sep 09 02:45:16.716638 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:16.716618 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/536af2f7-73dc-403d-994e-9f968f8a5f0d-plugins-config-volume\") pod \"payload-processing-e2e-ait-79a2702e-b995f748f-gdsz2\" (UID: \"536af2f7-73dc-403d-994e-9f968f8a5f0d\") " pod="openshift-ingress/payload-processing-e2e-ait-79a2702e-b995f748f-gdsz2" Sep 09 02:45:16.724441 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:16.724410 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lghgx\" (UniqueName: \"kubernetes.io/projected/9600e019-4472-4e90-8ada-e4aa09c6a179-kube-api-access-lghgx\") pod \"payload-pre-processing-e2e-ait-79a2702e-6b47975fbb-w6hnt\" (UID: \"9600e019-4472-4e90-8ada-e4aa09c6a179\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-79a2702e-6b47975fbb-w6hnt" Sep 09 02:45:16.724556 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:16.724493 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcdwp\" (UniqueName: \"kubernetes.io/projected/536af2f7-73dc-403d-994e-9f968f8a5f0d-kube-api-access-zcdwp\") pod \"payload-processing-e2e-ait-79a2702e-b995f748f-gdsz2\" (UID: \"536af2f7-73dc-403d-994e-9f968f8a5f0d\") " pod="openshift-ingress/payload-processing-e2e-ait-79a2702e-b995f748f-gdsz2" Sep 09 02:45:16.731286 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:16.731264 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-79a2702e-b995f748f-gdsz2" Sep 09 02:45:16.775003 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:16.774953 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-79a2702e-6b47975fbb-w6hnt" Sep 09 02:45:17.018516 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:17.018487 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-79a2702e-b995f748f-gdsz2"] Sep 09 02:45:17.065679 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:17.065656 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-79a2702e-6b47975fbb-w6hnt"] Sep 09 02:45:17.068150 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:45:17.068127 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9600e019_4472_4e90_8ada_e4aa09c6a179.slice/crio-559f6626e98fd7f6d83f445d42dafc5c5d75fbcf7875697fc9e45d29276ac248 WatchSource:0}: Error finding container 559f6626e98fd7f6d83f445d42dafc5c5d75fbcf7875697fc9e45d29276ac248: Status 404 returned error can't find the container with id 559f6626e98fd7f6d83f445d42dafc5c5d75fbcf7875697fc9e45d29276ac248 Sep 09 02:45:17.346572 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:17.346546 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-07b2d4-7df964488b-hblrk"] Sep 09 02:45:17.350243 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:17.350222 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-07b2d4-7df964488b-hblrk" Sep 09 02:45:17.354494 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:17.354466 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-07b2d4\"" Sep 09 02:45:17.364470 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:17.364439 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-79a2702e-6b47975fbb-w6hnt" event={"ID":"9600e019-4472-4e90-8ada-e4aa09c6a179","Type":"ContainerStarted","Data":"2e0c1d8169c884f7294d86f59871e6edf49fa6cbc92981b06e427009502ad72a"} Sep 09 02:45:17.364583 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:17.364495 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-79a2702e-6b47975fbb-w6hnt" event={"ID":"9600e019-4472-4e90-8ada-e4aa09c6a179","Type":"ContainerStarted","Data":"559f6626e98fd7f6d83f445d42dafc5c5d75fbcf7875697fc9e45d29276ac248"} Sep 09 02:45:17.364650 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:17.364612 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-79a2702e-6b47975fbb-w6hnt" Sep 09 02:45:17.366581 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:17.366558 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-79a2702e-b995f748f-gdsz2" event={"ID":"536af2f7-73dc-403d-994e-9f968f8a5f0d","Type":"ContainerStarted","Data":"c14939eda697434b51aefefd97492c55109a3eeb15bcbe6fa1ad54196306f186"} Sep 09 02:45:17.366673 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:17.366601 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-79a2702e-b995f748f-gdsz2" event={"ID":"536af2f7-73dc-403d-994e-9f968f8a5f0d","Type":"ContainerStarted","Data":"f31ad10588eb4fde67d9ca5a6a2893310f0bf96670de12df224a954117b4e0c9"} Sep 09 02:45:17.366733 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:17.366720 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-79a2702e-b995f748f-gdsz2" Sep 09 02:45:17.367760 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:17.367739 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-07b2d4-7df964488b-hblrk"] Sep 09 02:45:17.383350 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:17.382108 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-07b2d4-55957f7c59-t4k5w"] Sep 09 02:45:17.388775 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:17.388696 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-07b2d4-55957f7c59-t4k5w" Sep 09 02:45:17.395694 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:17.394476 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-07b2d4-dockercfg-26rzn\"" Sep 09 02:45:17.397022 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:17.396978 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-79a2702e-6b47975fbb-w6hnt" podStartSLOduration=1.396945316 podStartE2EDuration="1.396945316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:45:17.39455112 +0000 UTC m=+910.758885586" watchObservedRunningTime="2026-09-09 02:45:17.396945316 +0000 UTC m=+910.761279786" Sep 09 02:45:17.402820 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:17.401948 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-07b2d4-55957f7c59-t4k5w"] Sep 09 02:45:17.419149 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:17.419112 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-79a2702e-b995f748f-gdsz2" podStartSLOduration=1.41909943 podStartE2EDuration="1.41909943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:45:17.418576638 +0000 UTC m=+910.782911103" watchObservedRunningTime="2026-09-09 02:45:17.41909943 +0000 UTC m=+910.783433978" Sep 09 02:45:17.423183 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:17.423157 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8e4df0b-58ad-4e84-b89a-fabc65693900-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-07b2d4-7df964488b-hblrk\" (UID: \"d8e4df0b-58ad-4e84-b89a-fabc65693900\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-07b2d4-7df964488b-hblrk" Sep 09 02:45:17.423431 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:17.423401 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9grp\" (UniqueName: \"kubernetes.io/projected/bc91db92-1311-42c9-aa74-fffad72f6ccc-kube-api-access-c9grp\") pod \"payload-processing-e2e-autores-07b2d4-55957f7c59-t4k5w\" (UID: \"bc91db92-1311-42c9-aa74-fffad72f6ccc\") " pod="openshift-ingress/payload-processing-e2e-autores-07b2d4-55957f7c59-t4k5w" Sep 09 02:45:17.423509 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:17.423449 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gktgr\" (UniqueName: \"kubernetes.io/projected/d8e4df0b-58ad-4e84-b89a-fabc65693900-kube-api-access-gktgr\") pod \"payload-pre-processing-e2e-autores-07b2d4-7df964488b-hblrk\" (UID: \"d8e4df0b-58ad-4e84-b89a-fabc65693900\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-07b2d4-7df964488b-hblrk" Sep 09 02:45:17.423572 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:17.423526 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc91db92-1311-42c9-aa74-fffad72f6ccc-plugins-config-volume\") pod \"payload-processing-e2e-autores-07b2d4-55957f7c59-t4k5w\" (UID: \"bc91db92-1311-42c9-aa74-fffad72f6ccc\") " pod="openshift-ingress/payload-processing-e2e-autores-07b2d4-55957f7c59-t4k5w" Sep 09 02:45:17.524144 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:17.524109 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8e4df0b-58ad-4e84-b89a-fabc65693900-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-07b2d4-7df964488b-hblrk\" (UID: \"d8e4df0b-58ad-4e84-b89a-fabc65693900\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-07b2d4-7df964488b-hblrk" Sep 09 02:45:17.524312 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:17.524166 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9grp\" (UniqueName: \"kubernetes.io/projected/bc91db92-1311-42c9-aa74-fffad72f6ccc-kube-api-access-c9grp\") pod \"payload-processing-e2e-autores-07b2d4-55957f7c59-t4k5w\" (UID: \"bc91db92-1311-42c9-aa74-fffad72f6ccc\") " pod="openshift-ingress/payload-processing-e2e-autores-07b2d4-55957f7c59-t4k5w" Sep 09 02:45:17.524312 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:17.524183 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gktgr\" (UniqueName: \"kubernetes.io/projected/d8e4df0b-58ad-4e84-b89a-fabc65693900-kube-api-access-gktgr\") pod \"payload-pre-processing-e2e-autores-07b2d4-7df964488b-hblrk\" (UID: \"d8e4df0b-58ad-4e84-b89a-fabc65693900\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-07b2d4-7df964488b-hblrk" Sep 09 02:45:17.524312 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:17.524209 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc91db92-1311-42c9-aa74-fffad72f6ccc-plugins-config-volume\") pod \"payload-processing-e2e-autores-07b2d4-55957f7c59-t4k5w\" (UID: \"bc91db92-1311-42c9-aa74-fffad72f6ccc\") " pod="openshift-ingress/payload-processing-e2e-autores-07b2d4-55957f7c59-t4k5w" Sep 09 02:45:17.524867 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:17.524850 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc91db92-1311-42c9-aa74-fffad72f6ccc-plugins-config-volume\") pod \"payload-processing-e2e-autores-07b2d4-55957f7c59-t4k5w\" (UID: \"bc91db92-1311-42c9-aa74-fffad72f6ccc\") " pod="openshift-ingress/payload-processing-e2e-autores-07b2d4-55957f7c59-t4k5w" Sep 09 02:45:17.524911 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:17.524883 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8e4df0b-58ad-4e84-b89a-fabc65693900-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-07b2d4-7df964488b-hblrk\" (UID: \"d8e4df0b-58ad-4e84-b89a-fabc65693900\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-07b2d4-7df964488b-hblrk" Sep 09 02:45:17.533573 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:17.533544 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9grp\" (UniqueName: \"kubernetes.io/projected/bc91db92-1311-42c9-aa74-fffad72f6ccc-kube-api-access-c9grp\") pod \"payload-processing-e2e-autores-07b2d4-55957f7c59-t4k5w\" (UID: \"bc91db92-1311-42c9-aa74-fffad72f6ccc\") " pod="openshift-ingress/payload-processing-e2e-autores-07b2d4-55957f7c59-t4k5w" Sep 09 02:45:17.533841 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:17.533820 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gktgr\" (UniqueName: \"kubernetes.io/projected/d8e4df0b-58ad-4e84-b89a-fabc65693900-kube-api-access-gktgr\") pod \"payload-pre-processing-e2e-autores-07b2d4-7df964488b-hblrk\" (UID: \"d8e4df0b-58ad-4e84-b89a-fabc65693900\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-07b2d4-7df964488b-hblrk" Sep 09 02:45:17.667120 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:17.667042 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-07b2d4-7df964488b-hblrk" Sep 09 02:45:17.715732 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:17.715700 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-07b2d4-55957f7c59-t4k5w" Sep 09 02:45:17.952927 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:17.952905 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-07b2d4-7df964488b-hblrk"] Sep 09 02:45:17.954621 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:45:17.954591 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8e4df0b_58ad_4e84_b89a_fabc65693900.slice/crio-8b12eeb30149d770e1d2e9371618afb9103d92b39e52f5065fc27e9fd3b63b38 WatchSource:0}: Error finding container 8b12eeb30149d770e1d2e9371618afb9103d92b39e52f5065fc27e9fd3b63b38: Status 404 returned error can't find the container with id 8b12eeb30149d770e1d2e9371618afb9103d92b39e52f5065fc27e9fd3b63b38 Sep 09 02:45:18.013042 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:18.012874 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-07b2d4-55957f7c59-t4k5w"] Sep 09 02:45:18.016876 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:45:18.016850 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc91db92_1311_42c9_aa74_fffad72f6ccc.slice/crio-03d133edd38f571fc18bdfd27126467fa6b2aedf4978fbb1635fdd1b6051d6b5 WatchSource:0}: Error finding container 03d133edd38f571fc18bdfd27126467fa6b2aedf4978fbb1635fdd1b6051d6b5: Status 404 returned error can't find the container with id 03d133edd38f571fc18bdfd27126467fa6b2aedf4978fbb1635fdd1b6051d6b5 Sep 09 02:45:18.373896 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:18.373851 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-07b2d4-55957f7c59-t4k5w" event={"ID":"bc91db92-1311-42c9-aa74-fffad72f6ccc","Type":"ContainerStarted","Data":"8c047580a3e58c81bc89dbaa15e90486758bfd6167b46b885e015f574be175ac"} Sep 09 02:45:18.374102 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:18.373927 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-07b2d4-55957f7c59-t4k5w" event={"ID":"bc91db92-1311-42c9-aa74-fffad72f6ccc","Type":"ContainerStarted","Data":"03d133edd38f571fc18bdfd27126467fa6b2aedf4978fbb1635fdd1b6051d6b5"} Sep 09 02:45:18.374102 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:18.374069 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-07b2d4-55957f7c59-t4k5w" Sep 09 02:45:18.375641 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:18.375608 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-07b2d4-7df964488b-hblrk" event={"ID":"d8e4df0b-58ad-4e84-b89a-fabc65693900","Type":"ContainerStarted","Data":"031d5eb7867afddc975359b6f11314fced050895b440b12046ff7b63ec59e49c"} Sep 09 02:45:18.375746 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:18.375664 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-07b2d4-7df964488b-hblrk" event={"ID":"d8e4df0b-58ad-4e84-b89a-fabc65693900","Type":"ContainerStarted","Data":"8b12eeb30149d770e1d2e9371618afb9103d92b39e52f5065fc27e9fd3b63b38"} Sep 09 02:45:18.375746 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:18.375712 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-07b2d4-7df964488b-hblrk" Sep 09 02:45:18.395014 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:18.394946 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-07b2d4-55957f7c59-t4k5w" podStartSLOduration=1.394928813 podStartE2EDuration="1.394928813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:45:18.393321258 +0000 UTC m=+911.757655722" watchObservedRunningTime="2026-09-09 02:45:18.394928813 +0000 UTC m=+911.759263277" Sep 09 02:45:18.413036 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:18.412319 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-07b2d4-7df964488b-hblrk" podStartSLOduration=1.412299835 podStartE2EDuration="1.412299835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:45:18.41010117 +0000 UTC m=+911.774435635" watchObservedRunningTime="2026-09-09 02:45:18.412299835 +0000 UTC m=+911.776634300" Sep 09 02:45:24.343768 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:24.343731 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b18109-7d498769dc-sgr4m" Sep 09 02:45:24.344152 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:24.343813 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b18109-55d84bf57d-cm2wr" Sep 09 02:45:28.377568 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:28.377534 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-79a2702e-b995f748f-gdsz2" Sep 09 02:45:28.378095 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:28.377627 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-79a2702e-6b47975fbb-w6hnt" Sep 09 02:45:29.383371 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:29.383340 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-07b2d4-55957f7c59-t4k5w" Sep 09 02:45:29.383824 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:29.383507 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-07b2d4-7df964488b-hblrk" Sep 09 02:45:42.019982 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:42.019921 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6b11a2-8cfdf467c-wlprd"] Sep 09 02:45:42.033722 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:42.033686 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6b11a2-8cfdf467c-wlprd" Sep 09 02:45:42.036590 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:42.036558 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6b11a2-8cfdf467c-wlprd"] Sep 09 02:45:42.037196 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:42.037171 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-6b11a2\"" Sep 09 02:45:42.037452 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:42.037436 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-6b11a2-dockercfg-hhk6s\"" Sep 09 02:45:42.144756 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:42.144722 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp2sb\" (UniqueName: \"kubernetes.io/projected/f45fbd28-69e2-4cc5-be16-a30e218fabaf-kube-api-access-mp2sb\") pod \"payload-processing-e2e-inf-b-6b11a2-8cfdf467c-wlprd\" (UID: \"f45fbd28-69e2-4cc5-be16-a30e218fabaf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6b11a2-8cfdf467c-wlprd" Sep 09 02:45:42.144921 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:42.144853 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f45fbd28-69e2-4cc5-be16-a30e218fabaf-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6b11a2-8cfdf467c-wlprd\" (UID: \"f45fbd28-69e2-4cc5-be16-a30e218fabaf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6b11a2-8cfdf467c-wlprd" Sep 09 02:45:42.246400 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:42.246350 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f45fbd28-69e2-4cc5-be16-a30e218fabaf-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6b11a2-8cfdf467c-wlprd\" (UID: \"f45fbd28-69e2-4cc5-be16-a30e218fabaf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6b11a2-8cfdf467c-wlprd" Sep 09 02:45:42.246621 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:42.246437 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp2sb\" (UniqueName: \"kubernetes.io/projected/f45fbd28-69e2-4cc5-be16-a30e218fabaf-kube-api-access-mp2sb\") pod \"payload-processing-e2e-inf-b-6b11a2-8cfdf467c-wlprd\" (UID: \"f45fbd28-69e2-4cc5-be16-a30e218fabaf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6b11a2-8cfdf467c-wlprd" Sep 09 02:45:42.247196 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:42.247174 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f45fbd28-69e2-4cc5-be16-a30e218fabaf-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6b11a2-8cfdf467c-wlprd\" (UID: \"f45fbd28-69e2-4cc5-be16-a30e218fabaf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6b11a2-8cfdf467c-wlprd" Sep 09 02:45:42.255353 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:42.255331 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp2sb\" (UniqueName: \"kubernetes.io/projected/f45fbd28-69e2-4cc5-be16-a30e218fabaf-kube-api-access-mp2sb\") pod \"payload-processing-e2e-inf-b-6b11a2-8cfdf467c-wlprd\" (UID: \"f45fbd28-69e2-4cc5-be16-a30e218fabaf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6b11a2-8cfdf467c-wlprd" Sep 09 02:45:42.344887 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:42.344855 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6b11a2-8cfdf467c-wlprd" Sep 09 02:45:42.627457 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:42.627433 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6b11a2-8cfdf467c-wlprd"] Sep 09 02:45:42.628825 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:45:42.628799 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf45fbd28_69e2_4cc5_be16_a30e218fabaf.slice/crio-9ff5b2ce340b1810ac173717b89b9dc150e283b72b596ec4251d33819f4df7b8 WatchSource:0}: Error finding container 9ff5b2ce340b1810ac173717b89b9dc150e283b72b596ec4251d33819f4df7b8: Status 404 returned error can't find the container with id 9ff5b2ce340b1810ac173717b89b9dc150e283b72b596ec4251d33819f4df7b8 Sep 09 02:45:43.528924 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:43.528886 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6b11a2-8cfdf467c-wlprd" event={"ID":"f45fbd28-69e2-4cc5-be16-a30e218fabaf","Type":"ContainerStarted","Data":"e84c3e3deb00af1abec2eebde914cef4651001ad320031f60e5b1cf796fac9ca"} Sep 09 02:45:43.529333 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:43.528953 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-6b11a2-8cfdf467c-wlprd" Sep 09 02:45:43.529333 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:43.528999 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6b11a2-8cfdf467c-wlprd" event={"ID":"f45fbd28-69e2-4cc5-be16-a30e218fabaf","Type":"ContainerStarted","Data":"9ff5b2ce340b1810ac173717b89b9dc150e283b72b596ec4251d33819f4df7b8"} Sep 09 02:45:43.548128 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:43.548077 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-6b11a2-8cfdf467c-wlprd" podStartSLOduration=1.54806175 podStartE2EDuration="1.54806175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:45:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:45:43.547213534 +0000 UTC m=+936.911548011" watchObservedRunningTime="2026-09-09 02:45:43.54806175 +0000 UTC m=+936.912396213" Sep 09 02:45:54.536867 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:54.536837 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-6b11a2-8cfdf467c-wlprd" Sep 09 02:45:56.476433 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:56.476390 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-79a2702e-b995f748f-gdsz2"] Sep 09 02:45:56.476839 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:56.476661 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-79a2702e-b995f748f-gdsz2" podUID="536af2f7-73dc-403d-994e-9f968f8a5f0d" containerName="payload-processing" containerID="cri-o://c14939eda697434b51aefefd97492c55109a3eeb15bcbe6fa1ad54196306f186" gracePeriod=30 Sep 09 02:45:56.484268 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:56.484237 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-79a2702e-6b47975fbb-w6hnt"] Sep 09 02:45:56.484883 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:56.484466 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-79a2702e-6b47975fbb-w6hnt" podUID="9600e019-4472-4e90-8ada-e4aa09c6a179" containerName="payload-pre-processing" containerID="cri-o://2e0c1d8169c884f7294d86f59871e6edf49fa6cbc92981b06e427009502ad72a" gracePeriod=30 Sep 09 02:45:56.615808 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:56.615765 2577 generic.go:356] "Generic (PLEG): container finished" podID="536af2f7-73dc-403d-994e-9f968f8a5f0d" containerID="c14939eda697434b51aefefd97492c55109a3eeb15bcbe6fa1ad54196306f186" exitCode=0 Sep 09 02:45:56.615947 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:56.615926 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-79a2702e-b995f748f-gdsz2" event={"ID":"536af2f7-73dc-403d-994e-9f968f8a5f0d","Type":"ContainerDied","Data":"c14939eda697434b51aefefd97492c55109a3eeb15bcbe6fa1ad54196306f186"} Sep 09 02:45:56.617991 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:56.617935 2577 generic.go:356] "Generic (PLEG): container finished" podID="9600e019-4472-4e90-8ada-e4aa09c6a179" containerID="2e0c1d8169c884f7294d86f59871e6edf49fa6cbc92981b06e427009502ad72a" exitCode=0 Sep 09 02:45:56.618101 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:56.618001 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-79a2702e-6b47975fbb-w6hnt" event={"ID":"9600e019-4472-4e90-8ada-e4aa09c6a179","Type":"ContainerDied","Data":"2e0c1d8169c884f7294d86f59871e6edf49fa6cbc92981b06e427009502ad72a"} Sep 09 02:45:56.748775 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:56.748750 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-79a2702e-b995f748f-gdsz2" Sep 09 02:45:56.793392 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:56.793361 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-79a2702e-6b47975fbb-w6hnt" Sep 09 02:45:56.871648 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:56.871619 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lghgx\" (UniqueName: \"kubernetes.io/projected/9600e019-4472-4e90-8ada-e4aa09c6a179-kube-api-access-lghgx\") pod \"9600e019-4472-4e90-8ada-e4aa09c6a179\" (UID: \"9600e019-4472-4e90-8ada-e4aa09c6a179\") " Sep 09 02:45:56.871830 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:56.871664 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9600e019-4472-4e90-8ada-e4aa09c6a179-plugins-config-volume\") pod \"9600e019-4472-4e90-8ada-e4aa09c6a179\" (UID: \"9600e019-4472-4e90-8ada-e4aa09c6a179\") " Sep 09 02:45:56.871830 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:56.871726 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zcdwp\" (UniqueName: \"kubernetes.io/projected/536af2f7-73dc-403d-994e-9f968f8a5f0d-kube-api-access-zcdwp\") pod \"536af2f7-73dc-403d-994e-9f968f8a5f0d\" (UID: \"536af2f7-73dc-403d-994e-9f968f8a5f0d\") " Sep 09 02:45:56.871830 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:56.871783 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/536af2f7-73dc-403d-994e-9f968f8a5f0d-plugins-config-volume\") pod \"536af2f7-73dc-403d-994e-9f968f8a5f0d\" (UID: \"536af2f7-73dc-403d-994e-9f968f8a5f0d\") " Sep 09 02:45:56.872177 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:56.872149 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9600e019-4472-4e90-8ada-e4aa09c6a179-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9600e019-4472-4e90-8ada-e4aa09c6a179" (UID: "9600e019-4472-4e90-8ada-e4aa09c6a179"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:45:56.872290 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:56.872237 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/536af2f7-73dc-403d-994e-9f968f8a5f0d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "536af2f7-73dc-403d-994e-9f968f8a5f0d" (UID: "536af2f7-73dc-403d-994e-9f968f8a5f0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:45:56.873980 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:56.873941 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9600e019-4472-4e90-8ada-e4aa09c6a179-kube-api-access-lghgx" (OuterVolumeSpecName: "kube-api-access-lghgx") pod "9600e019-4472-4e90-8ada-e4aa09c6a179" (UID: "9600e019-4472-4e90-8ada-e4aa09c6a179"). InnerVolumeSpecName "kube-api-access-lghgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:45:56.874059 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:56.873997 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/536af2f7-73dc-403d-994e-9f968f8a5f0d-kube-api-access-zcdwp" (OuterVolumeSpecName: "kube-api-access-zcdwp") pod "536af2f7-73dc-403d-994e-9f968f8a5f0d" (UID: "536af2f7-73dc-403d-994e-9f968f8a5f0d"). InnerVolumeSpecName "kube-api-access-zcdwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:45:56.973416 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:56.973380 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/536af2f7-73dc-403d-994e-9f968f8a5f0d-plugins-config-volume\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:45:56.973416 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:56.973411 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lghgx\" (UniqueName: \"kubernetes.io/projected/9600e019-4472-4e90-8ada-e4aa09c6a179-kube-api-access-lghgx\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:45:56.973416 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:56.973420 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9600e019-4472-4e90-8ada-e4aa09c6a179-plugins-config-volume\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:45:56.973659 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:56.973428 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcdwp\" (UniqueName: \"kubernetes.io/projected/536af2f7-73dc-403d-994e-9f968f8a5f0d-kube-api-access-zcdwp\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:45:57.625535 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:57.625506 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-79a2702e-6b47975fbb-w6hnt" Sep 09 02:45:57.626027 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:57.625528 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-79a2702e-6b47975fbb-w6hnt" event={"ID":"9600e019-4472-4e90-8ada-e4aa09c6a179","Type":"ContainerDied","Data":"559f6626e98fd7f6d83f445d42dafc5c5d75fbcf7875697fc9e45d29276ac248"} Sep 09 02:45:57.626027 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:57.625623 2577 scope.go:117] "RemoveContainer" containerID="2e0c1d8169c884f7294d86f59871e6edf49fa6cbc92981b06e427009502ad72a" Sep 09 02:45:57.627207 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:57.627137 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-79a2702e-b995f748f-gdsz2" event={"ID":"536af2f7-73dc-403d-994e-9f968f8a5f0d","Type":"ContainerDied","Data":"f31ad10588eb4fde67d9ca5a6a2893310f0bf96670de12df224a954117b4e0c9"} Sep 09 02:45:57.627756 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:57.627459 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-79a2702e-b995f748f-gdsz2" Sep 09 02:45:57.635397 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:57.635378 2577 scope.go:117] "RemoveContainer" containerID="c14939eda697434b51aefefd97492c55109a3eeb15bcbe6fa1ad54196306f186" Sep 09 02:45:57.648260 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:57.648236 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-79a2702e-6b47975fbb-w6hnt"] Sep 09 02:45:57.657921 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:57.657900 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-79a2702e-6b47975fbb-w6hnt"] Sep 09 02:45:57.668678 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:57.668656 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-79a2702e-b995f748f-gdsz2"] Sep 09 02:45:57.677198 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:57.677179 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-79a2702e-b995f748f-gdsz2"] Sep 09 02:45:59.166529 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:59.166483 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="536af2f7-73dc-403d-994e-9f968f8a5f0d" path="/var/lib/kubelet/pods/536af2f7-73dc-403d-994e-9f968f8a5f0d/volumes" Sep 09 02:45:59.167083 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:45:59.167053 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9600e019-4472-4e90-8ada-e4aa09c6a179" path="/var/lib/kubelet/pods/9600e019-4472-4e90-8ada-e4aa09c6a179/volumes" Sep 09 02:46:01.542239 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:01.542206 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b0df6c69-d7f779758-bb9ng"] Sep 09 02:46:01.542674 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:01.542568 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9600e019-4472-4e90-8ada-e4aa09c6a179" containerName="payload-pre-processing" Sep 09 02:46:01.542674 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:01.542579 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9600e019-4472-4e90-8ada-e4aa09c6a179" containerName="payload-pre-processing" Sep 09 02:46:01.542674 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:01.542588 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="536af2f7-73dc-403d-994e-9f968f8a5f0d" containerName="payload-processing" Sep 09 02:46:01.542674 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:01.542594 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="536af2f7-73dc-403d-994e-9f968f8a5f0d" containerName="payload-processing" Sep 09 02:46:01.542674 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:01.542648 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="536af2f7-73dc-403d-994e-9f968f8a5f0d" containerName="payload-processing" Sep 09 02:46:01.542674 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:01.542658 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9600e019-4472-4e90-8ada-e4aa09c6a179" containerName="payload-pre-processing" Sep 09 02:46:01.547240 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:01.547219 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b0df6c69-d7f779758-bb9ng" Sep 09 02:46:01.550008 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:01.549990 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-b0df6c69\"" Sep 09 02:46:01.559025 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:01.559001 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b0df6c69-d7f779758-bb9ng"] Sep 09 02:46:01.574996 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:01.574952 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b0df6c69-6c7dbbb77d-65kml"] Sep 09 02:46:01.578395 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:01.578380 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b0df6c69-6c7dbbb77d-65kml" Sep 09 02:46:01.581589 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:01.581569 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-b0df6c69-dockercfg-pmrdv\"" Sep 09 02:46:01.593978 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:01.593927 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b0df6c69-6c7dbbb77d-65kml"] Sep 09 02:46:01.717861 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:01.717826 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x42p9\" (UniqueName: \"kubernetes.io/projected/eedbe68f-0cfe-4fd7-b46d-bb8eda41f60d-kube-api-access-x42p9\") pod \"payload-processing-e2e-migrate-b0df6c69-6c7dbbb77d-65kml\" (UID: \"eedbe68f-0cfe-4fd7-b46d-bb8eda41f60d\") " pod="openshift-ingress/payload-processing-e2e-migrate-b0df6c69-6c7dbbb77d-65kml" Sep 09 02:46:01.718062 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:01.717880 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eedbe68f-0cfe-4fd7-b46d-bb8eda41f60d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b0df6c69-6c7dbbb77d-65kml\" (UID: \"eedbe68f-0cfe-4fd7-b46d-bb8eda41f60d\") " pod="openshift-ingress/payload-processing-e2e-migrate-b0df6c69-6c7dbbb77d-65kml" Sep 09 02:46:01.718062 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:01.718025 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5446b30d-310e-44f9-869d-4b9a2315ec01-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b0df6c69-d7f779758-bb9ng\" (UID: \"5446b30d-310e-44f9-869d-4b9a2315ec01\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b0df6c69-d7f779758-bb9ng" Sep 09 02:46:01.718062 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:01.718057 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf5lw\" (UniqueName: \"kubernetes.io/projected/5446b30d-310e-44f9-869d-4b9a2315ec01-kube-api-access-zf5lw\") pod \"payload-pre-processing-e2e-migrate-b0df6c69-d7f779758-bb9ng\" (UID: \"5446b30d-310e-44f9-869d-4b9a2315ec01\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b0df6c69-d7f779758-bb9ng" Sep 09 02:46:01.818702 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:01.818612 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5446b30d-310e-44f9-869d-4b9a2315ec01-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b0df6c69-d7f779758-bb9ng\" (UID: \"5446b30d-310e-44f9-869d-4b9a2315ec01\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b0df6c69-d7f779758-bb9ng" Sep 09 02:46:01.818702 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:01.818658 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf5lw\" (UniqueName: \"kubernetes.io/projected/5446b30d-310e-44f9-869d-4b9a2315ec01-kube-api-access-zf5lw\") pod \"payload-pre-processing-e2e-migrate-b0df6c69-d7f779758-bb9ng\" (UID: \"5446b30d-310e-44f9-869d-4b9a2315ec01\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b0df6c69-d7f779758-bb9ng" Sep 09 02:46:01.818929 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:01.818712 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x42p9\" (UniqueName: \"kubernetes.io/projected/eedbe68f-0cfe-4fd7-b46d-bb8eda41f60d-kube-api-access-x42p9\") pod \"payload-processing-e2e-migrate-b0df6c69-6c7dbbb77d-65kml\" (UID: \"eedbe68f-0cfe-4fd7-b46d-bb8eda41f60d\") " pod="openshift-ingress/payload-processing-e2e-migrate-b0df6c69-6c7dbbb77d-65kml" Sep 09 02:46:01.818929 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:01.818751 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eedbe68f-0cfe-4fd7-b46d-bb8eda41f60d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b0df6c69-6c7dbbb77d-65kml\" (UID: \"eedbe68f-0cfe-4fd7-b46d-bb8eda41f60d\") " pod="openshift-ingress/payload-processing-e2e-migrate-b0df6c69-6c7dbbb77d-65kml" Sep 09 02:46:01.819374 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:01.819347 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5446b30d-310e-44f9-869d-4b9a2315ec01-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b0df6c69-d7f779758-bb9ng\" (UID: \"5446b30d-310e-44f9-869d-4b9a2315ec01\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b0df6c69-d7f779758-bb9ng" Sep 09 02:46:01.819488 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:01.819465 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eedbe68f-0cfe-4fd7-b46d-bb8eda41f60d-plugins-config-volume\") pod \"payload-processing-e2e-migrate-b0df6c69-6c7dbbb77d-65kml\" (UID: \"eedbe68f-0cfe-4fd7-b46d-bb8eda41f60d\") " pod="openshift-ingress/payload-processing-e2e-migrate-b0df6c69-6c7dbbb77d-65kml" Sep 09 02:46:01.829423 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:01.829396 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf5lw\" (UniqueName: \"kubernetes.io/projected/5446b30d-310e-44f9-869d-4b9a2315ec01-kube-api-access-zf5lw\") pod \"payload-pre-processing-e2e-migrate-b0df6c69-d7f779758-bb9ng\" (UID: \"5446b30d-310e-44f9-869d-4b9a2315ec01\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b0df6c69-d7f779758-bb9ng" Sep 09 02:46:01.829531 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:01.829400 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x42p9\" (UniqueName: \"kubernetes.io/projected/eedbe68f-0cfe-4fd7-b46d-bb8eda41f60d-kube-api-access-x42p9\") pod \"payload-processing-e2e-migrate-b0df6c69-6c7dbbb77d-65kml\" (UID: \"eedbe68f-0cfe-4fd7-b46d-bb8eda41f60d\") " pod="openshift-ingress/payload-processing-e2e-migrate-b0df6c69-6c7dbbb77d-65kml" Sep 09 02:46:01.859383 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:01.859354 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b0df6c69-d7f779758-bb9ng" Sep 09 02:46:01.888864 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:01.888837 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b0df6c69-6c7dbbb77d-65kml" Sep 09 02:46:02.136432 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:02.136404 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b0df6c69-d7f779758-bb9ng"] Sep 09 02:46:02.137708 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:46:02.137681 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5446b30d_310e_44f9_869d_4b9a2315ec01.slice/crio-22e9867afcbaea61721fb0bc067715de72d5cf761eac9818f105240d9c997107 WatchSource:0}: Error finding container 22e9867afcbaea61721fb0bc067715de72d5cf761eac9818f105240d9c997107: Status 404 returned error can't find the container with id 22e9867afcbaea61721fb0bc067715de72d5cf761eac9818f105240d9c997107 Sep 09 02:46:02.179349 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:02.179324 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b0df6c69-6c7dbbb77d-65kml"] Sep 09 02:46:02.189137 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:46:02.189111 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeedbe68f_0cfe_4fd7_b46d_bb8eda41f60d.slice/crio-3c568eb0a40df5237d6052fc8320af6ca3eb368e680ec5729e97d33ef0fd3eba WatchSource:0}: Error finding container 3c568eb0a40df5237d6052fc8320af6ca3eb368e680ec5729e97d33ef0fd3eba: Status 404 returned error can't find the container with id 3c568eb0a40df5237d6052fc8320af6ca3eb368e680ec5729e97d33ef0fd3eba Sep 09 02:46:02.660588 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:02.660548 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b0df6c69-d7f779758-bb9ng" event={"ID":"5446b30d-310e-44f9-869d-4b9a2315ec01","Type":"ContainerStarted","Data":"258a9e521404db96dc381f92af0489ad77c256686db2caa7a5c393408bce31f4"} Sep 09 02:46:02.661071 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:02.660613 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b0df6c69-d7f779758-bb9ng" event={"ID":"5446b30d-310e-44f9-869d-4b9a2315ec01","Type":"ContainerStarted","Data":"22e9867afcbaea61721fb0bc067715de72d5cf761eac9818f105240d9c997107"} Sep 09 02:46:02.661071 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:02.660642 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b0df6c69-d7f779758-bb9ng" Sep 09 02:46:02.662330 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:02.662307 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b0df6c69-6c7dbbb77d-65kml" event={"ID":"eedbe68f-0cfe-4fd7-b46d-bb8eda41f60d","Type":"ContainerStarted","Data":"a3b7c7fc38c839f68424676a09f782db8e366c3670a1f262530e6c89b6187443"} Sep 09 02:46:02.662460 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:02.662352 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b0df6c69-6c7dbbb77d-65kml" event={"ID":"eedbe68f-0cfe-4fd7-b46d-bb8eda41f60d","Type":"ContainerStarted","Data":"3c568eb0a40df5237d6052fc8320af6ca3eb368e680ec5729e97d33ef0fd3eba"} Sep 09 02:46:02.662460 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:02.662422 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-b0df6c69-6c7dbbb77d-65kml" Sep 09 02:46:02.681890 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:02.681837 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b0df6c69-d7f779758-bb9ng" podStartSLOduration=1.681823093 podStartE2EDuration="1.681823093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:46:02.678715313 +0000 UTC m=+956.043049776" watchObservedRunningTime="2026-09-09 02:46:02.681823093 +0000 UTC m=+956.046157554" Sep 09 02:46:13.670586 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:13.670551 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-b0df6c69-6c7dbbb77d-65kml" Sep 09 02:46:13.671007 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:13.670721 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b0df6c69-d7f779758-bb9ng" Sep 09 02:46:13.689980 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:13.689915 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-b0df6c69-6c7dbbb77d-65kml" podStartSLOduration=12.689898967 podStartE2EDuration="12.689898967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:46:02.699116046 +0000 UTC m=+956.063450530" watchObservedRunningTime="2026-09-09 02:46:13.689898967 +0000 UTC m=+967.054233429" Sep 09 02:46:31.691142 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:31.690857 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b0df6c69-6c7dbbb77d-65kml"] Sep 09 02:46:31.697024 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:31.696981 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-b0df6c69-6c7dbbb77d-65kml" podUID="eedbe68f-0cfe-4fd7-b46d-bb8eda41f60d" containerName="payload-processing" containerID="cri-o://a3b7c7fc38c839f68424676a09f782db8e366c3670a1f262530e6c89b6187443" gracePeriod=30 Sep 09 02:46:31.698758 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:31.698733 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b0df6c69-d7f779758-bb9ng"] Sep 09 02:46:31.699017 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:31.698988 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b0df6c69-d7f779758-bb9ng" podUID="5446b30d-310e-44f9-869d-4b9a2315ec01" containerName="payload-pre-processing" containerID="cri-o://258a9e521404db96dc381f92af0489ad77c256686db2caa7a5c393408bce31f4" gracePeriod=30 Sep 09 02:46:31.830789 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:31.830753 2577 generic.go:356] "Generic (PLEG): container finished" podID="eedbe68f-0cfe-4fd7-b46d-bb8eda41f60d" containerID="a3b7c7fc38c839f68424676a09f782db8e366c3670a1f262530e6c89b6187443" exitCode=0 Sep 09 02:46:31.830919 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:31.830816 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b0df6c69-6c7dbbb77d-65kml" event={"ID":"eedbe68f-0cfe-4fd7-b46d-bb8eda41f60d","Type":"ContainerDied","Data":"a3b7c7fc38c839f68424676a09f782db8e366c3670a1f262530e6c89b6187443"} Sep 09 02:46:31.832700 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:31.832647 2577 generic.go:356] "Generic (PLEG): container finished" podID="5446b30d-310e-44f9-869d-4b9a2315ec01" containerID="258a9e521404db96dc381f92af0489ad77c256686db2caa7a5c393408bce31f4" exitCode=0 Sep 09 02:46:31.832822 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:31.832723 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b0df6c69-d7f779758-bb9ng" event={"ID":"5446b30d-310e-44f9-869d-4b9a2315ec01","Type":"ContainerDied","Data":"258a9e521404db96dc381f92af0489ad77c256686db2caa7a5c393408bce31f4"} Sep 09 02:46:31.979760 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:31.979723 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b0df6c69-6c7dbbb77d-65kml" Sep 09 02:46:31.983709 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:31.983682 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b0df6c69-d7f779758-bb9ng" Sep 09 02:46:32.073299 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:32.073272 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x42p9\" (UniqueName: \"kubernetes.io/projected/eedbe68f-0cfe-4fd7-b46d-bb8eda41f60d-kube-api-access-x42p9\") pod \"eedbe68f-0cfe-4fd7-b46d-bb8eda41f60d\" (UID: \"eedbe68f-0cfe-4fd7-b46d-bb8eda41f60d\") " Sep 09 02:46:32.073464 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:32.073311 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eedbe68f-0cfe-4fd7-b46d-bb8eda41f60d-plugins-config-volume\") pod \"eedbe68f-0cfe-4fd7-b46d-bb8eda41f60d\" (UID: \"eedbe68f-0cfe-4fd7-b46d-bb8eda41f60d\") " Sep 09 02:46:32.073464 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:32.073351 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zf5lw\" (UniqueName: \"kubernetes.io/projected/5446b30d-310e-44f9-869d-4b9a2315ec01-kube-api-access-zf5lw\") pod \"5446b30d-310e-44f9-869d-4b9a2315ec01\" (UID: \"5446b30d-310e-44f9-869d-4b9a2315ec01\") " Sep 09 02:46:32.073464 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:32.073377 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5446b30d-310e-44f9-869d-4b9a2315ec01-plugins-config-volume\") pod \"5446b30d-310e-44f9-869d-4b9a2315ec01\" (UID: \"5446b30d-310e-44f9-869d-4b9a2315ec01\") " Sep 09 02:46:32.073678 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:32.073650 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eedbe68f-0cfe-4fd7-b46d-bb8eda41f60d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eedbe68f-0cfe-4fd7-b46d-bb8eda41f60d" (UID: "eedbe68f-0cfe-4fd7-b46d-bb8eda41f60d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:46:32.073753 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:32.073728 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5446b30d-310e-44f9-869d-4b9a2315ec01-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5446b30d-310e-44f9-869d-4b9a2315ec01" (UID: "5446b30d-310e-44f9-869d-4b9a2315ec01"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:46:32.075729 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:32.075703 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5446b30d-310e-44f9-869d-4b9a2315ec01-kube-api-access-zf5lw" (OuterVolumeSpecName: "kube-api-access-zf5lw") pod "5446b30d-310e-44f9-869d-4b9a2315ec01" (UID: "5446b30d-310e-44f9-869d-4b9a2315ec01"). InnerVolumeSpecName "kube-api-access-zf5lw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:46:32.075729 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:32.075710 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eedbe68f-0cfe-4fd7-b46d-bb8eda41f60d-kube-api-access-x42p9" (OuterVolumeSpecName: "kube-api-access-x42p9") pod "eedbe68f-0cfe-4fd7-b46d-bb8eda41f60d" (UID: "eedbe68f-0cfe-4fd7-b46d-bb8eda41f60d"). InnerVolumeSpecName "kube-api-access-x42p9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:46:32.174361 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:32.174324 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf5lw\" (UniqueName: \"kubernetes.io/projected/5446b30d-310e-44f9-869d-4b9a2315ec01-kube-api-access-zf5lw\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:46:32.174361 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:32.174353 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5446b30d-310e-44f9-869d-4b9a2315ec01-plugins-config-volume\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:46:32.174361 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:32.174362 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x42p9\" (UniqueName: \"kubernetes.io/projected/eedbe68f-0cfe-4fd7-b46d-bb8eda41f60d-kube-api-access-x42p9\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:46:32.174361 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:32.174370 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eedbe68f-0cfe-4fd7-b46d-bb8eda41f60d-plugins-config-volume\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:46:32.839712 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:32.839686 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-b0df6c69-6c7dbbb77d-65kml" Sep 09 02:46:32.840258 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:32.839724 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-b0df6c69-6c7dbbb77d-65kml" event={"ID":"eedbe68f-0cfe-4fd7-b46d-bb8eda41f60d","Type":"ContainerDied","Data":"3c568eb0a40df5237d6052fc8320af6ca3eb368e680ec5729e97d33ef0fd3eba"} Sep 09 02:46:32.840258 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:32.839791 2577 scope.go:117] "RemoveContainer" containerID="a3b7c7fc38c839f68424676a09f782db8e366c3670a1f262530e6c89b6187443" Sep 09 02:46:32.841291 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:32.841259 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b0df6c69-d7f779758-bb9ng" event={"ID":"5446b30d-310e-44f9-869d-4b9a2315ec01","Type":"ContainerDied","Data":"22e9867afcbaea61721fb0bc067715de72d5cf761eac9818f105240d9c997107"} Sep 09 02:46:32.841361 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:32.841316 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b0df6c69-d7f779758-bb9ng" Sep 09 02:46:32.852855 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:32.852794 2577 scope.go:117] "RemoveContainer" containerID="258a9e521404db96dc381f92af0489ad77c256686db2caa7a5c393408bce31f4" Sep 09 02:46:32.876474 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:32.876450 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b0df6c69-d7f779758-bb9ng"] Sep 09 02:46:32.880662 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:32.880636 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b0df6c69-d7f779758-bb9ng"] Sep 09 02:46:32.894340 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:32.894306 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b0df6c69-6c7dbbb77d-65kml"] Sep 09 02:46:32.899277 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:32.899253 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-b0df6c69-6c7dbbb77d-65kml"] Sep 09 02:46:33.164852 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:33.164816 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5446b30d-310e-44f9-869d-4b9a2315ec01" path="/var/lib/kubelet/pods/5446b30d-310e-44f9-869d-4b9a2315ec01/volumes" Sep 09 02:46:33.165186 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:33.165172 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eedbe68f-0cfe-4fd7-b46d-bb8eda41f60d" path="/var/lib/kubelet/pods/eedbe68f-0cfe-4fd7-b46d-bb8eda41f60d/volumes" Sep 09 02:46:36.892869 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:36.892827 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7e17db81-574fc94c6f-v5wqw"] Sep 09 02:46:36.893812 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:36.893787 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5446b30d-310e-44f9-869d-4b9a2315ec01" containerName="payload-pre-processing" Sep 09 02:46:36.893812 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:36.893814 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5446b30d-310e-44f9-869d-4b9a2315ec01" containerName="payload-pre-processing" Sep 09 02:46:36.894011 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:36.893864 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eedbe68f-0cfe-4fd7-b46d-bb8eda41f60d" containerName="payload-processing" Sep 09 02:46:36.894011 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:36.893874 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="eedbe68f-0cfe-4fd7-b46d-bb8eda41f60d" containerName="payload-processing" Sep 09 02:46:36.894162 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:36.894069 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5446b30d-310e-44f9-869d-4b9a2315ec01" containerName="payload-pre-processing" Sep 09 02:46:36.894162 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:36.894085 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="eedbe68f-0cfe-4fd7-b46d-bb8eda41f60d" containerName="payload-processing" Sep 09 02:46:36.899714 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:36.898153 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7e17db81-574fc94c6f-v5wqw" Sep 09 02:46:36.901648 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:36.901623 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-7e17db81\"" Sep 09 02:46:36.901767 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:36.901683 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-7e17db81-dockercfg-bknll\"" Sep 09 02:46:36.906753 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:36.906733 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7e17db81-574fc94c6f-v5wqw"] Sep 09 02:46:36.927560 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:36.927524 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7e17db81-5b4c9f6874-9lk74"] Sep 09 02:46:36.930875 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:36.930856 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e17db81-5b4c9f6874-9lk74" Sep 09 02:46:36.941672 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:36.941651 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7e17db81-5b4c9f6874-9lk74"] Sep 09 02:46:37.012050 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:37.012017 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b794a0ba-93c6-42b5-8967-939e9cb110e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7e17db81-5b4c9f6874-9lk74\" (UID: \"b794a0ba-93c6-42b5-8967-939e9cb110e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7e17db81-5b4c9f6874-9lk74" Sep 09 02:46:37.012208 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:37.012058 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f619240-5bb6-48b6-a80b-b60f8e356489-plugins-config-volume\") pod \"payload-processing-e2e-ait-7e17db81-574fc94c6f-v5wqw\" (UID: \"7f619240-5bb6-48b6-a80b-b60f8e356489\") " pod="openshift-ingress/payload-processing-e2e-ait-7e17db81-574fc94c6f-v5wqw" Sep 09 02:46:37.012208 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:37.012139 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmsbx\" (UniqueName: \"kubernetes.io/projected/b794a0ba-93c6-42b5-8967-939e9cb110e6-kube-api-access-fmsbx\") pod \"payload-pre-processing-e2e-ait-7e17db81-5b4c9f6874-9lk74\" (UID: \"b794a0ba-93c6-42b5-8967-939e9cb110e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7e17db81-5b4c9f6874-9lk74" Sep 09 02:46:37.012298 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:37.012244 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpxgp\" (UniqueName: \"kubernetes.io/projected/7f619240-5bb6-48b6-a80b-b60f8e356489-kube-api-access-mpxgp\") pod \"payload-processing-e2e-ait-7e17db81-574fc94c6f-v5wqw\" (UID: \"7f619240-5bb6-48b6-a80b-b60f8e356489\") " pod="openshift-ingress/payload-processing-e2e-ait-7e17db81-574fc94c6f-v5wqw" Sep 09 02:46:37.112887 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:37.112851 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b794a0ba-93c6-42b5-8967-939e9cb110e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7e17db81-5b4c9f6874-9lk74\" (UID: \"b794a0ba-93c6-42b5-8967-939e9cb110e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7e17db81-5b4c9f6874-9lk74" Sep 09 02:46:37.112887 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:37.112888 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f619240-5bb6-48b6-a80b-b60f8e356489-plugins-config-volume\") pod \"payload-processing-e2e-ait-7e17db81-574fc94c6f-v5wqw\" (UID: \"7f619240-5bb6-48b6-a80b-b60f8e356489\") " pod="openshift-ingress/payload-processing-e2e-ait-7e17db81-574fc94c6f-v5wqw" Sep 09 02:46:37.113107 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:37.112904 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmsbx\" (UniqueName: \"kubernetes.io/projected/b794a0ba-93c6-42b5-8967-939e9cb110e6-kube-api-access-fmsbx\") pod \"payload-pre-processing-e2e-ait-7e17db81-5b4c9f6874-9lk74\" (UID: \"b794a0ba-93c6-42b5-8967-939e9cb110e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7e17db81-5b4c9f6874-9lk74" Sep 09 02:46:37.113107 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:37.112936 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpxgp\" (UniqueName: \"kubernetes.io/projected/7f619240-5bb6-48b6-a80b-b60f8e356489-kube-api-access-mpxgp\") pod \"payload-processing-e2e-ait-7e17db81-574fc94c6f-v5wqw\" (UID: \"7f619240-5bb6-48b6-a80b-b60f8e356489\") " pod="openshift-ingress/payload-processing-e2e-ait-7e17db81-574fc94c6f-v5wqw" Sep 09 02:46:37.113633 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:37.113614 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f619240-5bb6-48b6-a80b-b60f8e356489-plugins-config-volume\") pod \"payload-processing-e2e-ait-7e17db81-574fc94c6f-v5wqw\" (UID: \"7f619240-5bb6-48b6-a80b-b60f8e356489\") " pod="openshift-ingress/payload-processing-e2e-ait-7e17db81-574fc94c6f-v5wqw" Sep 09 02:46:37.113679 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:37.113620 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b794a0ba-93c6-42b5-8967-939e9cb110e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-7e17db81-5b4c9f6874-9lk74\" (UID: \"b794a0ba-93c6-42b5-8967-939e9cb110e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7e17db81-5b4c9f6874-9lk74" Sep 09 02:46:37.121593 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:37.121557 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpxgp\" (UniqueName: \"kubernetes.io/projected/7f619240-5bb6-48b6-a80b-b60f8e356489-kube-api-access-mpxgp\") pod \"payload-processing-e2e-ait-7e17db81-574fc94c6f-v5wqw\" (UID: \"7f619240-5bb6-48b6-a80b-b60f8e356489\") " pod="openshift-ingress/payload-processing-e2e-ait-7e17db81-574fc94c6f-v5wqw" Sep 09 02:46:37.121734 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:37.121706 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmsbx\" (UniqueName: \"kubernetes.io/projected/b794a0ba-93c6-42b5-8967-939e9cb110e6-kube-api-access-fmsbx\") pod \"payload-pre-processing-e2e-ait-7e17db81-5b4c9f6874-9lk74\" (UID: \"b794a0ba-93c6-42b5-8967-939e9cb110e6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-7e17db81-5b4c9f6874-9lk74" Sep 09 02:46:37.210624 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:37.210542 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7e17db81-574fc94c6f-v5wqw" Sep 09 02:46:37.240394 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:37.240367 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e17db81-5b4c9f6874-9lk74" Sep 09 02:46:37.494340 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:37.494313 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7e17db81-574fc94c6f-v5wqw"] Sep 09 02:46:37.495163 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:46:37.495137 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f619240_5bb6_48b6_a80b_b60f8e356489.slice/crio-dcc6a2d4139d3ee6c3238045a55fc4b4bc8ffd508f44aa3e601f84ead212b9cd WatchSource:0}: Error finding container dcc6a2d4139d3ee6c3238045a55fc4b4bc8ffd508f44aa3e601f84ead212b9cd: Status 404 returned error can't find the container with id dcc6a2d4139d3ee6c3238045a55fc4b4bc8ffd508f44aa3e601f84ead212b9cd Sep 09 02:46:37.528135 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:37.528111 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7e17db81-5b4c9f6874-9lk74"] Sep 09 02:46:37.531013 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:46:37.530990 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb794a0ba_93c6_42b5_8967_939e9cb110e6.slice/crio-bb21578a9a7fffa843fa523939a58ef8f57686ada788c8da71c38b4209c31699 WatchSource:0}: Error finding container bb21578a9a7fffa843fa523939a58ef8f57686ada788c8da71c38b4209c31699: Status 404 returned error can't find the container with id bb21578a9a7fffa843fa523939a58ef8f57686ada788c8da71c38b4209c31699 Sep 09 02:46:37.875252 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:37.875218 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e17db81-5b4c9f6874-9lk74" event={"ID":"b794a0ba-93c6-42b5-8967-939e9cb110e6","Type":"ContainerStarted","Data":"c2d7523738145e2f08dbf6960fb0d7c7ed42148a7f1d2624bb4d7f2c6833d2c9"} Sep 09 02:46:37.875431 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:37.875280 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e17db81-5b4c9f6874-9lk74" event={"ID":"b794a0ba-93c6-42b5-8967-939e9cb110e6","Type":"ContainerStarted","Data":"bb21578a9a7fffa843fa523939a58ef8f57686ada788c8da71c38b4209c31699"} Sep 09 02:46:37.875431 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:37.875393 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e17db81-5b4c9f6874-9lk74" Sep 09 02:46:37.876782 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:37.876754 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7e17db81-574fc94c6f-v5wqw" event={"ID":"7f619240-5bb6-48b6-a80b-b60f8e356489","Type":"ContainerStarted","Data":"2098b386f57489cf3673bb01c4f129e3c8d94440ac2afd0a312ae544b1874007"} Sep 09 02:46:37.876897 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:37.876804 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7e17db81-574fc94c6f-v5wqw" event={"ID":"7f619240-5bb6-48b6-a80b-b60f8e356489","Type":"ContainerStarted","Data":"dcc6a2d4139d3ee6c3238045a55fc4b4bc8ffd508f44aa3e601f84ead212b9cd"} Sep 09 02:46:37.876942 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:37.876887 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-7e17db81-574fc94c6f-v5wqw" Sep 09 02:46:37.894536 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:37.894485 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e17db81-5b4c9f6874-9lk74" podStartSLOduration=1.894470472 podStartE2EDuration="1.894470472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:46:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:46:37.891513171 +0000 UTC m=+991.255847635" watchObservedRunningTime="2026-09-09 02:46:37.894470472 +0000 UTC m=+991.258804934" Sep 09 02:46:37.909033 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:37.908997 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-7e17db81-574fc94c6f-v5wqw" podStartSLOduration=1.908950438 podStartE2EDuration="1.908950438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:46:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:46:37.907666628 +0000 UTC m=+991.272001096" watchObservedRunningTime="2026-09-09 02:46:37.908950438 +0000 UTC m=+991.273284936" Sep 09 02:46:48.885436 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:48.885402 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-7e17db81-574fc94c6f-v5wqw" Sep 09 02:46:48.885866 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:46:48.885517 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e17db81-5b4c9f6874-9lk74" Sep 09 02:47:06.965649 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:06.965608 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7e17db81-574fc94c6f-v5wqw"] Sep 09 02:47:06.966187 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:06.965830 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-7e17db81-574fc94c6f-v5wqw" podUID="7f619240-5bb6-48b6-a80b-b60f8e356489" containerName="payload-processing" containerID="cri-o://2098b386f57489cf3673bb01c4f129e3c8d94440ac2afd0a312ae544b1874007" gracePeriod=30 Sep 09 02:47:06.975088 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:06.975057 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7e17db81-5b4c9f6874-9lk74"] Sep 09 02:47:06.975472 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:06.975442 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e17db81-5b4c9f6874-9lk74" podUID="b794a0ba-93c6-42b5-8967-939e9cb110e6" containerName="payload-pre-processing" containerID="cri-o://c2d7523738145e2f08dbf6960fb0d7c7ed42148a7f1d2624bb4d7f2c6833d2c9" gracePeriod=30 Sep 09 02:47:07.234482 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:07.234459 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7e17db81-574fc94c6f-v5wqw" Sep 09 02:47:07.256846 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:07.256824 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e17db81-5b4c9f6874-9lk74" Sep 09 02:47:07.381711 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:07.381680 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mpxgp\" (UniqueName: \"kubernetes.io/projected/7f619240-5bb6-48b6-a80b-b60f8e356489-kube-api-access-mpxgp\") pod \"7f619240-5bb6-48b6-a80b-b60f8e356489\" (UID: \"7f619240-5bb6-48b6-a80b-b60f8e356489\") " Sep 09 02:47:07.381889 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:07.381788 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f619240-5bb6-48b6-a80b-b60f8e356489-plugins-config-volume\") pod \"7f619240-5bb6-48b6-a80b-b60f8e356489\" (UID: \"7f619240-5bb6-48b6-a80b-b60f8e356489\") " Sep 09 02:47:07.381889 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:07.381859 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b794a0ba-93c6-42b5-8967-939e9cb110e6-plugins-config-volume\") pod \"b794a0ba-93c6-42b5-8967-939e9cb110e6\" (UID: \"b794a0ba-93c6-42b5-8967-939e9cb110e6\") " Sep 09 02:47:07.381889 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:07.381876 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmsbx\" (UniqueName: \"kubernetes.io/projected/b794a0ba-93c6-42b5-8967-939e9cb110e6-kube-api-access-fmsbx\") pod \"b794a0ba-93c6-42b5-8967-939e9cb110e6\" (UID: \"b794a0ba-93c6-42b5-8967-939e9cb110e6\") " Sep 09 02:47:07.382267 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:07.382238 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b794a0ba-93c6-42b5-8967-939e9cb110e6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b794a0ba-93c6-42b5-8967-939e9cb110e6" (UID: "b794a0ba-93c6-42b5-8967-939e9cb110e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:47:07.382398 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:07.382267 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f619240-5bb6-48b6-a80b-b60f8e356489-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7f619240-5bb6-48b6-a80b-b60f8e356489" (UID: "7f619240-5bb6-48b6-a80b-b60f8e356489"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:47:07.384270 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:07.384245 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f619240-5bb6-48b6-a80b-b60f8e356489-kube-api-access-mpxgp" (OuterVolumeSpecName: "kube-api-access-mpxgp") pod "7f619240-5bb6-48b6-a80b-b60f8e356489" (UID: "7f619240-5bb6-48b6-a80b-b60f8e356489"). InnerVolumeSpecName "kube-api-access-mpxgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:47:07.384372 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:07.384267 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b794a0ba-93c6-42b5-8967-939e9cb110e6-kube-api-access-fmsbx" (OuterVolumeSpecName: "kube-api-access-fmsbx") pod "b794a0ba-93c6-42b5-8967-939e9cb110e6" (UID: "b794a0ba-93c6-42b5-8967-939e9cb110e6"). InnerVolumeSpecName "kube-api-access-fmsbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:47:07.482513 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:07.482439 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b794a0ba-93c6-42b5-8967-939e9cb110e6-plugins-config-volume\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:47:07.482513 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:07.482463 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmsbx\" (UniqueName: \"kubernetes.io/projected/b794a0ba-93c6-42b5-8967-939e9cb110e6-kube-api-access-fmsbx\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:47:07.482513 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:07.482474 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpxgp\" (UniqueName: \"kubernetes.io/projected/7f619240-5bb6-48b6-a80b-b60f8e356489-kube-api-access-mpxgp\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:47:07.482513 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:07.482483 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f619240-5bb6-48b6-a80b-b60f8e356489-plugins-config-volume\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:47:08.062408 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:08.062370 2577 generic.go:356] "Generic (PLEG): container finished" podID="b794a0ba-93c6-42b5-8967-939e9cb110e6" containerID="c2d7523738145e2f08dbf6960fb0d7c7ed42148a7f1d2624bb4d7f2c6833d2c9" exitCode=0 Sep 09 02:47:08.062876 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:08.062424 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e17db81-5b4c9f6874-9lk74" Sep 09 02:47:08.062876 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:08.062453 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e17db81-5b4c9f6874-9lk74" event={"ID":"b794a0ba-93c6-42b5-8967-939e9cb110e6","Type":"ContainerDied","Data":"c2d7523738145e2f08dbf6960fb0d7c7ed42148a7f1d2624bb4d7f2c6833d2c9"} Sep 09 02:47:08.062876 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:08.062502 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-7e17db81-5b4c9f6874-9lk74" event={"ID":"b794a0ba-93c6-42b5-8967-939e9cb110e6","Type":"ContainerDied","Data":"bb21578a9a7fffa843fa523939a58ef8f57686ada788c8da71c38b4209c31699"} Sep 09 02:47:08.062876 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:08.062524 2577 scope.go:117] "RemoveContainer" containerID="c2d7523738145e2f08dbf6960fb0d7c7ed42148a7f1d2624bb4d7f2c6833d2c9" Sep 09 02:47:08.064243 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:08.064221 2577 generic.go:356] "Generic (PLEG): container finished" podID="7f619240-5bb6-48b6-a80b-b60f8e356489" containerID="2098b386f57489cf3673bb01c4f129e3c8d94440ac2afd0a312ae544b1874007" exitCode=0 Sep 09 02:47:08.064350 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:08.064254 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7e17db81-574fc94c6f-v5wqw" event={"ID":"7f619240-5bb6-48b6-a80b-b60f8e356489","Type":"ContainerDied","Data":"2098b386f57489cf3673bb01c4f129e3c8d94440ac2afd0a312ae544b1874007"} Sep 09 02:47:08.064350 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:08.064283 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7e17db81-574fc94c6f-v5wqw" Sep 09 02:47:08.064435 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:08.064285 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7e17db81-574fc94c6f-v5wqw" event={"ID":"7f619240-5bb6-48b6-a80b-b60f8e356489","Type":"ContainerDied","Data":"dcc6a2d4139d3ee6c3238045a55fc4b4bc8ffd508f44aa3e601f84ead212b9cd"} Sep 09 02:47:08.071797 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:08.071776 2577 scope.go:117] "RemoveContainer" containerID="c2d7523738145e2f08dbf6960fb0d7c7ed42148a7f1d2624bb4d7f2c6833d2c9" Sep 09 02:47:08.072203 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:47:08.072183 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2d7523738145e2f08dbf6960fb0d7c7ed42148a7f1d2624bb4d7f2c6833d2c9\": container with ID starting with c2d7523738145e2f08dbf6960fb0d7c7ed42148a7f1d2624bb4d7f2c6833d2c9 not found: ID does not exist" containerID="c2d7523738145e2f08dbf6960fb0d7c7ed42148a7f1d2624bb4d7f2c6833d2c9" Sep 09 02:47:08.072302 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:08.072210 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2d7523738145e2f08dbf6960fb0d7c7ed42148a7f1d2624bb4d7f2c6833d2c9"} err="failed to get container status \"c2d7523738145e2f08dbf6960fb0d7c7ed42148a7f1d2624bb4d7f2c6833d2c9\": rpc error: code = NotFound desc = could not find container \"c2d7523738145e2f08dbf6960fb0d7c7ed42148a7f1d2624bb4d7f2c6833d2c9\": container with ID starting with c2d7523738145e2f08dbf6960fb0d7c7ed42148a7f1d2624bb4d7f2c6833d2c9 not found: ID does not exist" Sep 09 02:47:08.072302 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:08.072227 2577 scope.go:117] "RemoveContainer" containerID="2098b386f57489cf3673bb01c4f129e3c8d94440ac2afd0a312ae544b1874007" Sep 09 02:47:08.080241 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:08.080224 2577 scope.go:117] "RemoveContainer" containerID="2098b386f57489cf3673bb01c4f129e3c8d94440ac2afd0a312ae544b1874007" Sep 09 02:47:08.080513 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:47:08.080495 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2098b386f57489cf3673bb01c4f129e3c8d94440ac2afd0a312ae544b1874007\": container with ID starting with 2098b386f57489cf3673bb01c4f129e3c8d94440ac2afd0a312ae544b1874007 not found: ID does not exist" containerID="2098b386f57489cf3673bb01c4f129e3c8d94440ac2afd0a312ae544b1874007" Sep 09 02:47:08.080553 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:08.080521 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2098b386f57489cf3673bb01c4f129e3c8d94440ac2afd0a312ae544b1874007"} err="failed to get container status \"2098b386f57489cf3673bb01c4f129e3c8d94440ac2afd0a312ae544b1874007\": rpc error: code = NotFound desc = could not find container \"2098b386f57489cf3673bb01c4f129e3c8d94440ac2afd0a312ae544b1874007\": container with ID starting with 2098b386f57489cf3673bb01c4f129e3c8d94440ac2afd0a312ae544b1874007 not found: ID does not exist" Sep 09 02:47:08.088472 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:08.088450 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7e17db81-574fc94c6f-v5wqw"] Sep 09 02:47:08.093504 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:08.093486 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7e17db81-574fc94c6f-v5wqw"] Sep 09 02:47:08.103525 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:08.103504 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7e17db81-5b4c9f6874-9lk74"] Sep 09 02:47:08.108050 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:08.108032 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-7e17db81-5b4c9f6874-9lk74"] Sep 09 02:47:09.173840 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:09.173812 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f619240-5bb6-48b6-a80b-b60f8e356489" path="/var/lib/kubelet/pods/7f619240-5bb6-48b6-a80b-b60f8e356489/volumes" Sep 09 02:47:09.174287 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:09.174227 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b794a0ba-93c6-42b5-8967-939e9cb110e6" path="/var/lib/kubelet/pods/b794a0ba-93c6-42b5-8967-939e9cb110e6/volumes" Sep 09 02:47:12.577024 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:12.576994 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-07b2d4-7lj9q"] Sep 09 02:47:12.577393 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:12.577348 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f619240-5bb6-48b6-a80b-b60f8e356489" containerName="payload-processing" Sep 09 02:47:12.577393 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:12.577358 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f619240-5bb6-48b6-a80b-b60f8e356489" containerName="payload-processing" Sep 09 02:47:12.577393 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:12.577370 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b794a0ba-93c6-42b5-8967-939e9cb110e6" containerName="payload-pre-processing" Sep 09 02:47:12.577393 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:12.577375 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b794a0ba-93c6-42b5-8967-939e9cb110e6" containerName="payload-pre-processing" Sep 09 02:47:12.577536 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:12.577437 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b794a0ba-93c6-42b5-8967-939e9cb110e6" containerName="payload-pre-processing" Sep 09 02:47:12.577536 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:12.577450 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f619240-5bb6-48b6-a80b-b60f8e356489" containerName="payload-processing" Sep 09 02:47:12.582187 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:12.582169 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-07b2d4-7lj9q" Sep 09 02:47:12.584720 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:12.584698 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbr52\"" Sep 09 02:47:12.584816 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:12.584718 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 02:47:12.594790 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:12.594765 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-07b2d4-7lj9q"] Sep 09 02:47:12.731425 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:12.731378 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b9a3e84-ad43-4d90-8f71-582bff305cc5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-07b2d4-7lj9q\" (UID: \"3b9a3e84-ad43-4d90-8f71-582bff305cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-07b2d4-7lj9q" Sep 09 02:47:12.832868 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:12.832786 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b9a3e84-ad43-4d90-8f71-582bff305cc5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-07b2d4-7lj9q\" (UID: \"3b9a3e84-ad43-4d90-8f71-582bff305cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-07b2d4-7lj9q" Sep 09 02:47:12.833366 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:12.833347 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b9a3e84-ad43-4d90-8f71-582bff305cc5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-07b2d4-7lj9q\" (UID: \"3b9a3e84-ad43-4d90-8f71-582bff305cc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-07b2d4-7lj9q" Sep 09 02:47:12.894460 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:12.894427 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-07b2d4-7lj9q" Sep 09 02:47:13.179385 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:13.179356 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-07b2d4-7lj9q"] Sep 09 02:47:13.180289 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:47:13.180264 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b9a3e84_ad43_4d90_8f71_582bff305cc5.slice/crio-0733c2d256276440565a526fbc8ac91b72324abd15dd6014b7958d497516faf3 WatchSource:0}: Error finding container 0733c2d256276440565a526fbc8ac91b72324abd15dd6014b7958d497516faf3: Status 404 returned error can't find the container with id 0733c2d256276440565a526fbc8ac91b72324abd15dd6014b7958d497516faf3 Sep 09 02:47:14.104753 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:14.104717 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-07b2d4-7lj9q" event={"ID":"3b9a3e84-ad43-4d90-8f71-582bff305cc5","Type":"ContainerStarted","Data":"afaa9821366ee56940caf154c4f392520467ea78ee7cbe11bb22db9fb32b4474"} Sep 09 02:47:14.105098 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:14.104768 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-07b2d4-7lj9q" event={"ID":"3b9a3e84-ad43-4d90-8f71-582bff305cc5","Type":"ContainerStarted","Data":"0733c2d256276440565a526fbc8ac91b72324abd15dd6014b7958d497516faf3"} Sep 09 02:47:14.314360 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:14.314318 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1816a546-9d655c8b-66bx4"] Sep 09 02:47:14.319196 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:14.319172 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1816a546-9d655c8b-66bx4" Sep 09 02:47:14.321919 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:14.321889 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-1816a546\"" Sep 09 02:47:14.322079 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:14.321921 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-1816a546-dockercfg-mbrng\"" Sep 09 02:47:14.328752 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:14.328726 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1816a546-9d655c8b-66bx4"] Sep 09 02:47:14.349464 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:14.349408 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1816a546-6f474d47fc-jm574"] Sep 09 02:47:14.352745 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:14.352730 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1816a546-6f474d47fc-jm574" Sep 09 02:47:14.364414 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:14.364392 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1816a546-6f474d47fc-jm574"] Sep 09 02:47:14.445675 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:14.445643 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91d9eb1c-a11b-4a67-b364-5107164c309a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-1816a546-9d655c8b-66bx4\" (UID: \"91d9eb1c-a11b-4a67-b364-5107164c309a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1816a546-9d655c8b-66bx4" Sep 09 02:47:14.445848 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:14.445699 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt7rr\" (UniqueName: \"kubernetes.io/projected/91d9eb1c-a11b-4a67-b364-5107164c309a-kube-api-access-zt7rr\") pod \"payload-pre-processing-e2e-derive-1816a546-9d655c8b-66bx4\" (UID: \"91d9eb1c-a11b-4a67-b364-5107164c309a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1816a546-9d655c8b-66bx4" Sep 09 02:47:14.546877 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:14.546845 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt7rr\" (UniqueName: \"kubernetes.io/projected/91d9eb1c-a11b-4a67-b364-5107164c309a-kube-api-access-zt7rr\") pod \"payload-pre-processing-e2e-derive-1816a546-9d655c8b-66bx4\" (UID: \"91d9eb1c-a11b-4a67-b364-5107164c309a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1816a546-9d655c8b-66bx4" Sep 09 02:47:14.547070 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:14.546919 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tchv\" (UniqueName: \"kubernetes.io/projected/976bf3d7-56f0-47a7-a3b6-c510a53d7fb7-kube-api-access-6tchv\") pod \"payload-processing-e2e-derive-1816a546-6f474d47fc-jm574\" (UID: \"976bf3d7-56f0-47a7-a3b6-c510a53d7fb7\") " pod="openshift-ingress/payload-processing-e2e-derive-1816a546-6f474d47fc-jm574" Sep 09 02:47:14.547070 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:14.546950 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91d9eb1c-a11b-4a67-b364-5107164c309a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-1816a546-9d655c8b-66bx4\" (UID: \"91d9eb1c-a11b-4a67-b364-5107164c309a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1816a546-9d655c8b-66bx4" Sep 09 02:47:14.547070 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:14.546986 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/976bf3d7-56f0-47a7-a3b6-c510a53d7fb7-plugins-config-volume\") pod \"payload-processing-e2e-derive-1816a546-6f474d47fc-jm574\" (UID: \"976bf3d7-56f0-47a7-a3b6-c510a53d7fb7\") " pod="openshift-ingress/payload-processing-e2e-derive-1816a546-6f474d47fc-jm574" Sep 09 02:47:14.547640 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:14.547615 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91d9eb1c-a11b-4a67-b364-5107164c309a-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-1816a546-9d655c8b-66bx4\" (UID: \"91d9eb1c-a11b-4a67-b364-5107164c309a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1816a546-9d655c8b-66bx4" Sep 09 02:47:14.555866 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:14.555843 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt7rr\" (UniqueName: \"kubernetes.io/projected/91d9eb1c-a11b-4a67-b364-5107164c309a-kube-api-access-zt7rr\") pod \"payload-pre-processing-e2e-derive-1816a546-9d655c8b-66bx4\" (UID: \"91d9eb1c-a11b-4a67-b364-5107164c309a\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1816a546-9d655c8b-66bx4" Sep 09 02:47:14.631385 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:14.631313 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1816a546-9d655c8b-66bx4" Sep 09 02:47:14.647663 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:14.647640 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tchv\" (UniqueName: \"kubernetes.io/projected/976bf3d7-56f0-47a7-a3b6-c510a53d7fb7-kube-api-access-6tchv\") pod \"payload-processing-e2e-derive-1816a546-6f474d47fc-jm574\" (UID: \"976bf3d7-56f0-47a7-a3b6-c510a53d7fb7\") " pod="openshift-ingress/payload-processing-e2e-derive-1816a546-6f474d47fc-jm574" Sep 09 02:47:14.647774 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:14.647694 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/976bf3d7-56f0-47a7-a3b6-c510a53d7fb7-plugins-config-volume\") pod \"payload-processing-e2e-derive-1816a546-6f474d47fc-jm574\" (UID: \"976bf3d7-56f0-47a7-a3b6-c510a53d7fb7\") " pod="openshift-ingress/payload-processing-e2e-derive-1816a546-6f474d47fc-jm574" Sep 09 02:47:14.648363 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:14.648345 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/976bf3d7-56f0-47a7-a3b6-c510a53d7fb7-plugins-config-volume\") pod \"payload-processing-e2e-derive-1816a546-6f474d47fc-jm574\" (UID: \"976bf3d7-56f0-47a7-a3b6-c510a53d7fb7\") " pod="openshift-ingress/payload-processing-e2e-derive-1816a546-6f474d47fc-jm574" Sep 09 02:47:14.657519 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:14.657490 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tchv\" (UniqueName: \"kubernetes.io/projected/976bf3d7-56f0-47a7-a3b6-c510a53d7fb7-kube-api-access-6tchv\") pod \"payload-processing-e2e-derive-1816a546-6f474d47fc-jm574\" (UID: \"976bf3d7-56f0-47a7-a3b6-c510a53d7fb7\") " pod="openshift-ingress/payload-processing-e2e-derive-1816a546-6f474d47fc-jm574" Sep 09 02:47:14.663309 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:14.663285 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1816a546-6f474d47fc-jm574" Sep 09 02:47:14.920034 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:14.918267 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1816a546-9d655c8b-66bx4"] Sep 09 02:47:14.947760 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:14.947735 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1816a546-6f474d47fc-jm574"] Sep 09 02:47:14.957669 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:47:14.957646 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod976bf3d7_56f0_47a7_a3b6_c510a53d7fb7.slice/crio-eb96f5b77a6bd0762a0a29268b2587d7263ad38b7996ec5a52189bcd7b383592 WatchSource:0}: Error finding container eb96f5b77a6bd0762a0a29268b2587d7263ad38b7996ec5a52189bcd7b383592: Status 404 returned error can't find the container with id eb96f5b77a6bd0762a0a29268b2587d7263ad38b7996ec5a52189bcd7b383592 Sep 09 02:47:15.113345 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:15.113304 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1816a546-6f474d47fc-jm574" event={"ID":"976bf3d7-56f0-47a7-a3b6-c510a53d7fb7","Type":"ContainerStarted","Data":"5a99c371b5615296fd9223e597ac3c652da7772ee21b4e1e03e882f8b4b294ee"} Sep 09 02:47:15.113779 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:15.113371 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-1816a546-6f474d47fc-jm574" Sep 09 02:47:15.113779 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:15.113400 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1816a546-6f474d47fc-jm574" event={"ID":"976bf3d7-56f0-47a7-a3b6-c510a53d7fb7","Type":"ContainerStarted","Data":"eb96f5b77a6bd0762a0a29268b2587d7263ad38b7996ec5a52189bcd7b383592"} Sep 09 02:47:15.115076 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:15.115051 2577 generic.go:356] "Generic (PLEG): container finished" podID="3b9a3e84-ad43-4d90-8f71-582bff305cc5" containerID="afaa9821366ee56940caf154c4f392520467ea78ee7cbe11bb22db9fb32b4474" exitCode=0 Sep 09 02:47:15.115218 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:15.115140 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-07b2d4-7lj9q" event={"ID":"3b9a3e84-ad43-4d90-8f71-582bff305cc5","Type":"ContainerDied","Data":"afaa9821366ee56940caf154c4f392520467ea78ee7cbe11bb22db9fb32b4474"} Sep 09 02:47:15.118820 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:15.118791 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1816a546-9d655c8b-66bx4" event={"ID":"91d9eb1c-a11b-4a67-b364-5107164c309a","Type":"ContainerStarted","Data":"9172d5274f83b04ac90ca024d5733e383a323e4aac33072b28378746cddce386"} Sep 09 02:47:15.118951 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:15.118838 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1816a546-9d655c8b-66bx4" event={"ID":"91d9eb1c-a11b-4a67-b364-5107164c309a","Type":"ContainerStarted","Data":"fedbad701a9805b4da6f6e92ba149c74e36c0d2441113f898c79cccb1f3cfa37"} Sep 09 02:47:15.118951 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:15.118867 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-1816a546-9d655c8b-66bx4" Sep 09 02:47:15.134932 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:15.133631 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-1816a546-6f474d47fc-jm574" podStartSLOduration=1.133610532 podStartE2EDuration="1.133610532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:47:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:47:15.132158461 +0000 UTC m=+1028.496492924" watchObservedRunningTime="2026-09-09 02:47:15.133610532 +0000 UTC m=+1028.497944997" Sep 09 02:47:15.150321 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:15.150267 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-1816a546-9d655c8b-66bx4" podStartSLOduration=1.150247772 podStartE2EDuration="1.150247772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:47:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:47:15.147424495 +0000 UTC m=+1028.511758981" watchObservedRunningTime="2026-09-09 02:47:15.150247772 +0000 UTC m=+1028.514582235" Sep 09 02:47:15.258726 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:15.258704 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-07b2d4-7lj9q" Sep 09 02:47:15.454547 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:15.454471 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b9a3e84-ad43-4d90-8f71-582bff305cc5-maas-api-service-ca\") pod \"3b9a3e84-ad43-4d90-8f71-582bff305cc5\" (UID: \"3b9a3e84-ad43-4d90-8f71-582bff305cc5\") " Sep 09 02:47:15.454777 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:15.454752 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b9a3e84-ad43-4d90-8f71-582bff305cc5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3b9a3e84-ad43-4d90-8f71-582bff305cc5" (UID: "3b9a3e84-ad43-4d90-8f71-582bff305cc5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:47:15.555399 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:15.555370 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b9a3e84-ad43-4d90-8f71-582bff305cc5-maas-api-service-ca\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:47:16.127524 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:16.127485 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-07b2d4-7lj9q" event={"ID":"3b9a3e84-ad43-4d90-8f71-582bff305cc5","Type":"ContainerDied","Data":"0733c2d256276440565a526fbc8ac91b72324abd15dd6014b7958d497516faf3"} Sep 09 02:47:16.127985 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:16.127937 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0733c2d256276440565a526fbc8ac91b72324abd15dd6014b7958d497516faf3" Sep 09 02:47:16.127985 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:16.127512 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-07b2d4-7lj9q" Sep 09 02:47:23.609683 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:23.609649 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-07b2d4-55957f7c59-t4k5w"] Sep 09 02:47:23.610165 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:23.609900 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-07b2d4-55957f7c59-t4k5w" podUID="bc91db92-1311-42c9-aa74-fffad72f6ccc" containerName="payload-processing" containerID="cri-o://8c047580a3e58c81bc89dbaa15e90486758bfd6167b46b885e015f574be175ac" gracePeriod=30 Sep 09 02:47:23.619406 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:23.619376 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-07b2d4-7df964488b-hblrk"] Sep 09 02:47:23.620940 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:23.620899 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-07b2d4-7df964488b-hblrk" podUID="d8e4df0b-58ad-4e84-b89a-fabc65693900" containerName="payload-pre-processing" containerID="cri-o://031d5eb7867afddc975359b6f11314fced050895b440b12046ff7b63ec59e49c" gracePeriod=30 Sep 09 02:47:23.781246 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:23.781191 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-07b2d4-7lj9q"] Sep 09 02:47:23.785126 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:23.785077 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-07b2d4-7lj9q"] Sep 09 02:47:23.870754 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:23.870699 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-07b2d4-55957f7c59-t4k5w" Sep 09 02:47:23.909656 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:23.909628 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-07b2d4-7df964488b-hblrk" Sep 09 02:47:23.928858 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:23.928829 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8e4df0b-58ad-4e84-b89a-fabc65693900-plugins-config-volume\") pod \"d8e4df0b-58ad-4e84-b89a-fabc65693900\" (UID: \"d8e4df0b-58ad-4e84-b89a-fabc65693900\") " Sep 09 02:47:23.929027 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:23.928904 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc91db92-1311-42c9-aa74-fffad72f6ccc-plugins-config-volume\") pod \"bc91db92-1311-42c9-aa74-fffad72f6ccc\" (UID: \"bc91db92-1311-42c9-aa74-fffad72f6ccc\") " Sep 09 02:47:23.929027 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:23.928985 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gktgr\" (UniqueName: \"kubernetes.io/projected/d8e4df0b-58ad-4e84-b89a-fabc65693900-kube-api-access-gktgr\") pod \"d8e4df0b-58ad-4e84-b89a-fabc65693900\" (UID: \"d8e4df0b-58ad-4e84-b89a-fabc65693900\") " Sep 09 02:47:23.929158 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:23.929038 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9grp\" (UniqueName: \"kubernetes.io/projected/bc91db92-1311-42c9-aa74-fffad72f6ccc-kube-api-access-c9grp\") pod \"bc91db92-1311-42c9-aa74-fffad72f6ccc\" (UID: \"bc91db92-1311-42c9-aa74-fffad72f6ccc\") " Sep 09 02:47:23.929273 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:23.929246 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8e4df0b-58ad-4e84-b89a-fabc65693900-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d8e4df0b-58ad-4e84-b89a-fabc65693900" (UID: "d8e4df0b-58ad-4e84-b89a-fabc65693900"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:47:23.929613 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:23.929575 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc91db92-1311-42c9-aa74-fffad72f6ccc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bc91db92-1311-42c9-aa74-fffad72f6ccc" (UID: "bc91db92-1311-42c9-aa74-fffad72f6ccc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:47:23.932003 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:23.931936 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8e4df0b-58ad-4e84-b89a-fabc65693900-kube-api-access-gktgr" (OuterVolumeSpecName: "kube-api-access-gktgr") pod "d8e4df0b-58ad-4e84-b89a-fabc65693900" (UID: "d8e4df0b-58ad-4e84-b89a-fabc65693900"). InnerVolumeSpecName "kube-api-access-gktgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:47:23.932003 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:23.931945 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc91db92-1311-42c9-aa74-fffad72f6ccc-kube-api-access-c9grp" (OuterVolumeSpecName: "kube-api-access-c9grp") pod "bc91db92-1311-42c9-aa74-fffad72f6ccc" (UID: "bc91db92-1311-42c9-aa74-fffad72f6ccc"). InnerVolumeSpecName "kube-api-access-c9grp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:47:24.029762 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:24.029726 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc91db92-1311-42c9-aa74-fffad72f6ccc-plugins-config-volume\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:47:24.029762 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:24.029757 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gktgr\" (UniqueName: \"kubernetes.io/projected/d8e4df0b-58ad-4e84-b89a-fabc65693900-kube-api-access-gktgr\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:47:24.029762 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:24.029768 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9grp\" (UniqueName: \"kubernetes.io/projected/bc91db92-1311-42c9-aa74-fffad72f6ccc-kube-api-access-c9grp\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:47:24.030037 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:24.029777 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8e4df0b-58ad-4e84-b89a-fabc65693900-plugins-config-volume\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:47:24.174771 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:24.174677 2577 generic.go:356] "Generic (PLEG): container finished" podID="d8e4df0b-58ad-4e84-b89a-fabc65693900" containerID="031d5eb7867afddc975359b6f11314fced050895b440b12046ff7b63ec59e49c" exitCode=0 Sep 09 02:47:24.174771 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:24.174726 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-07b2d4-7df964488b-hblrk" Sep 09 02:47:24.175036 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:24.174772 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-07b2d4-7df964488b-hblrk" event={"ID":"d8e4df0b-58ad-4e84-b89a-fabc65693900","Type":"ContainerDied","Data":"031d5eb7867afddc975359b6f11314fced050895b440b12046ff7b63ec59e49c"} Sep 09 02:47:24.175036 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:24.174827 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-07b2d4-7df964488b-hblrk" event={"ID":"d8e4df0b-58ad-4e84-b89a-fabc65693900","Type":"ContainerDied","Data":"8b12eeb30149d770e1d2e9371618afb9103d92b39e52f5065fc27e9fd3b63b38"} Sep 09 02:47:24.175036 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:24.174851 2577 scope.go:117] "RemoveContainer" containerID="031d5eb7867afddc975359b6f11314fced050895b440b12046ff7b63ec59e49c" Sep 09 02:47:24.176294 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:24.176261 2577 generic.go:356] "Generic (PLEG): container finished" podID="bc91db92-1311-42c9-aa74-fffad72f6ccc" containerID="8c047580a3e58c81bc89dbaa15e90486758bfd6167b46b885e015f574be175ac" exitCode=0 Sep 09 02:47:24.176404 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:24.176295 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-07b2d4-55957f7c59-t4k5w" Sep 09 02:47:24.176404 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:24.176317 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-07b2d4-55957f7c59-t4k5w" event={"ID":"bc91db92-1311-42c9-aa74-fffad72f6ccc","Type":"ContainerDied","Data":"8c047580a3e58c81bc89dbaa15e90486758bfd6167b46b885e015f574be175ac"} Sep 09 02:47:24.176404 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:24.176369 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-07b2d4-55957f7c59-t4k5w" event={"ID":"bc91db92-1311-42c9-aa74-fffad72f6ccc","Type":"ContainerDied","Data":"03d133edd38f571fc18bdfd27126467fa6b2aedf4978fbb1635fdd1b6051d6b5"} Sep 09 02:47:24.184489 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:24.184464 2577 scope.go:117] "RemoveContainer" containerID="031d5eb7867afddc975359b6f11314fced050895b440b12046ff7b63ec59e49c" Sep 09 02:47:24.184808 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:47:24.184785 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"031d5eb7867afddc975359b6f11314fced050895b440b12046ff7b63ec59e49c\": container with ID starting with 031d5eb7867afddc975359b6f11314fced050895b440b12046ff7b63ec59e49c not found: ID does not exist" containerID="031d5eb7867afddc975359b6f11314fced050895b440b12046ff7b63ec59e49c" Sep 09 02:47:24.184894 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:24.184816 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"031d5eb7867afddc975359b6f11314fced050895b440b12046ff7b63ec59e49c"} err="failed to get container status \"031d5eb7867afddc975359b6f11314fced050895b440b12046ff7b63ec59e49c\": rpc error: code = NotFound desc = could not find container \"031d5eb7867afddc975359b6f11314fced050895b440b12046ff7b63ec59e49c\": container with ID starting with 031d5eb7867afddc975359b6f11314fced050895b440b12046ff7b63ec59e49c not found: ID does not exist" Sep 09 02:47:24.184894 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:24.184833 2577 scope.go:117] "RemoveContainer" containerID="8c047580a3e58c81bc89dbaa15e90486758bfd6167b46b885e015f574be175ac" Sep 09 02:47:24.194326 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:24.194307 2577 scope.go:117] "RemoveContainer" containerID="8c047580a3e58c81bc89dbaa15e90486758bfd6167b46b885e015f574be175ac" Sep 09 02:47:24.194621 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:47:24.194603 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c047580a3e58c81bc89dbaa15e90486758bfd6167b46b885e015f574be175ac\": container with ID starting with 8c047580a3e58c81bc89dbaa15e90486758bfd6167b46b885e015f574be175ac not found: ID does not exist" containerID="8c047580a3e58c81bc89dbaa15e90486758bfd6167b46b885e015f574be175ac" Sep 09 02:47:24.194686 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:24.194625 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c047580a3e58c81bc89dbaa15e90486758bfd6167b46b885e015f574be175ac"} err="failed to get container status \"8c047580a3e58c81bc89dbaa15e90486758bfd6167b46b885e015f574be175ac\": rpc error: code = NotFound desc = could not find container \"8c047580a3e58c81bc89dbaa15e90486758bfd6167b46b885e015f574be175ac\": container with ID starting with 8c047580a3e58c81bc89dbaa15e90486758bfd6167b46b885e015f574be175ac not found: ID does not exist" Sep 09 02:47:24.200976 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:24.200937 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-07b2d4-7df964488b-hblrk"] Sep 09 02:47:24.207282 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:24.207263 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-07b2d4-7df964488b-hblrk"] Sep 09 02:47:24.218822 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:24.218797 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-07b2d4-55957f7c59-t4k5w"] Sep 09 02:47:24.223753 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:24.223734 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-07b2d4-55957f7c59-t4k5w"] Sep 09 02:47:25.165910 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:25.165871 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b9a3e84-ad43-4d90-8f71-582bff305cc5" path="/var/lib/kubelet/pods/3b9a3e84-ad43-4d90-8f71-582bff305cc5/volumes" Sep 09 02:47:25.166294 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:25.166234 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc91db92-1311-42c9-aa74-fffad72f6ccc" path="/var/lib/kubelet/pods/bc91db92-1311-42c9-aa74-fffad72f6ccc/volumes" Sep 09 02:47:25.166529 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:25.166516 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8e4df0b-58ad-4e84-b89a-fabc65693900" path="/var/lib/kubelet/pods/d8e4df0b-58ad-4e84-b89a-fabc65693900/volumes" Sep 09 02:47:26.129504 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:26.129475 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-1816a546-6f474d47fc-jm574" Sep 09 02:47:26.129856 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:26.129836 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-1816a546-9d655c8b-66bx4" Sep 09 02:47:41.808662 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:41.808618 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b52336-76795b4f45-rq9vd"] Sep 09 02:47:41.809274 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:41.809184 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8e4df0b-58ad-4e84-b89a-fabc65693900" containerName="payload-pre-processing" Sep 09 02:47:41.809274 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:41.809205 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8e4df0b-58ad-4e84-b89a-fabc65693900" containerName="payload-pre-processing" Sep 09 02:47:41.809274 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:41.809236 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc91db92-1311-42c9-aa74-fffad72f6ccc" containerName="payload-processing" Sep 09 02:47:41.809274 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:41.809245 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc91db92-1311-42c9-aa74-fffad72f6ccc" containerName="payload-processing" Sep 09 02:47:41.809274 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:41.809257 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b9a3e84-ad43-4d90-8f71-582bff305cc5" containerName="revoke-keys" Sep 09 02:47:41.809274 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:41.809264 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b9a3e84-ad43-4d90-8f71-582bff305cc5" containerName="revoke-keys" Sep 09 02:47:41.809582 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:41.809347 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc91db92-1311-42c9-aa74-fffad72f6ccc" containerName="payload-processing" Sep 09 02:47:41.809582 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:41.809360 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b9a3e84-ad43-4d90-8f71-582bff305cc5" containerName="revoke-keys" Sep 09 02:47:41.809582 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:41.809373 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8e4df0b-58ad-4e84-b89a-fabc65693900" containerName="payload-pre-processing" Sep 09 02:47:41.812603 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:41.812583 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b52336-76795b4f45-rq9vd" Sep 09 02:47:41.815284 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:41.815266 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-b52336\"" Sep 09 02:47:41.815417 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:41.815393 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-b52336-dockercfg-z7t2c\"" Sep 09 02:47:41.823725 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:41.823704 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b52336-76795b4f45-rq9vd"] Sep 09 02:47:41.849627 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:41.849598 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b52336-556bb79b9d-zlmhw"] Sep 09 02:47:41.853008 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:41.852993 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b52336-556bb79b9d-zlmhw" Sep 09 02:47:41.862388 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:41.862366 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b52336-556bb79b9d-zlmhw"] Sep 09 02:47:41.881559 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:41.881530 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ccaf914-1106-45be-94fb-01462d2eb9d5-plugins-config-volume\") pod \"payload-processing-e2e-explicit-b52336-76795b4f45-rq9vd\" (UID: \"8ccaf914-1106-45be-94fb-01462d2eb9d5\") " pod="openshift-ingress/payload-processing-e2e-explicit-b52336-76795b4f45-rq9vd" Sep 09 02:47:41.881828 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:41.881714 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjrw6\" (UniqueName: \"kubernetes.io/projected/8ccaf914-1106-45be-94fb-01462d2eb9d5-kube-api-access-rjrw6\") pod \"payload-processing-e2e-explicit-b52336-76795b4f45-rq9vd\" (UID: \"8ccaf914-1106-45be-94fb-01462d2eb9d5\") " pod="openshift-ingress/payload-processing-e2e-explicit-b52336-76795b4f45-rq9vd" Sep 09 02:47:41.982572 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:41.982535 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x97p\" (UniqueName: \"kubernetes.io/projected/3ac2aecd-d411-4d96-a19e-904369e694e6-kube-api-access-9x97p\") pod \"payload-pre-processing-e2e-explicit-b52336-556bb79b9d-zlmhw\" (UID: \"3ac2aecd-d411-4d96-a19e-904369e694e6\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b52336-556bb79b9d-zlmhw" Sep 09 02:47:41.982740 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:41.982583 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ac2aecd-d411-4d96-a19e-904369e694e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-b52336-556bb79b9d-zlmhw\" (UID: \"3ac2aecd-d411-4d96-a19e-904369e694e6\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b52336-556bb79b9d-zlmhw" Sep 09 02:47:41.982740 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:41.982617 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjrw6\" (UniqueName: \"kubernetes.io/projected/8ccaf914-1106-45be-94fb-01462d2eb9d5-kube-api-access-rjrw6\") pod \"payload-processing-e2e-explicit-b52336-76795b4f45-rq9vd\" (UID: \"8ccaf914-1106-45be-94fb-01462d2eb9d5\") " pod="openshift-ingress/payload-processing-e2e-explicit-b52336-76795b4f45-rq9vd" Sep 09 02:47:41.982740 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:41.982655 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ccaf914-1106-45be-94fb-01462d2eb9d5-plugins-config-volume\") pod \"payload-processing-e2e-explicit-b52336-76795b4f45-rq9vd\" (UID: \"8ccaf914-1106-45be-94fb-01462d2eb9d5\") " pod="openshift-ingress/payload-processing-e2e-explicit-b52336-76795b4f45-rq9vd" Sep 09 02:47:41.983318 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:41.983300 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ccaf914-1106-45be-94fb-01462d2eb9d5-plugins-config-volume\") pod \"payload-processing-e2e-explicit-b52336-76795b4f45-rq9vd\" (UID: \"8ccaf914-1106-45be-94fb-01462d2eb9d5\") " pod="openshift-ingress/payload-processing-e2e-explicit-b52336-76795b4f45-rq9vd" Sep 09 02:47:41.991484 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:41.991459 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjrw6\" (UniqueName: \"kubernetes.io/projected/8ccaf914-1106-45be-94fb-01462d2eb9d5-kube-api-access-rjrw6\") pod \"payload-processing-e2e-explicit-b52336-76795b4f45-rq9vd\" (UID: \"8ccaf914-1106-45be-94fb-01462d2eb9d5\") " pod="openshift-ingress/payload-processing-e2e-explicit-b52336-76795b4f45-rq9vd" Sep 09 02:47:42.083284 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:42.083208 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x97p\" (UniqueName: \"kubernetes.io/projected/3ac2aecd-d411-4d96-a19e-904369e694e6-kube-api-access-9x97p\") pod \"payload-pre-processing-e2e-explicit-b52336-556bb79b9d-zlmhw\" (UID: \"3ac2aecd-d411-4d96-a19e-904369e694e6\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b52336-556bb79b9d-zlmhw" Sep 09 02:47:42.083284 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:42.083251 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ac2aecd-d411-4d96-a19e-904369e694e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-b52336-556bb79b9d-zlmhw\" (UID: \"3ac2aecd-d411-4d96-a19e-904369e694e6\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b52336-556bb79b9d-zlmhw" Sep 09 02:47:42.083978 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:42.083935 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ac2aecd-d411-4d96-a19e-904369e694e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-b52336-556bb79b9d-zlmhw\" (UID: \"3ac2aecd-d411-4d96-a19e-904369e694e6\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b52336-556bb79b9d-zlmhw" Sep 09 02:47:42.091137 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:42.091104 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x97p\" (UniqueName: \"kubernetes.io/projected/3ac2aecd-d411-4d96-a19e-904369e694e6-kube-api-access-9x97p\") pod \"payload-pre-processing-e2e-explicit-b52336-556bb79b9d-zlmhw\" (UID: \"3ac2aecd-d411-4d96-a19e-904369e694e6\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b52336-556bb79b9d-zlmhw" Sep 09 02:47:42.123669 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:42.123640 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b52336-76795b4f45-rq9vd" Sep 09 02:47:42.163843 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:42.163810 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b52336-556bb79b9d-zlmhw" Sep 09 02:47:42.397130 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:42.397099 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b52336-76795b4f45-rq9vd"] Sep 09 02:47:42.398369 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:47:42.398347 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ccaf914_1106_45be_94fb_01462d2eb9d5.slice/crio-c511fd6915cd0a1bd53981df1352a7a290d631f2aa127864a6d0ba2e8ec00c25 WatchSource:0}: Error finding container c511fd6915cd0a1bd53981df1352a7a290d631f2aa127864a6d0ba2e8ec00c25: Status 404 returned error can't find the container with id c511fd6915cd0a1bd53981df1352a7a290d631f2aa127864a6d0ba2e8ec00c25 Sep 09 02:47:42.448678 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:42.448652 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b52336-556bb79b9d-zlmhw"] Sep 09 02:47:42.450150 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:47:42.450127 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac2aecd_d411_4d96_a19e_904369e694e6.slice/crio-f5d160aeb51f4cc8549b8dbc42d061d9e3b39ab2b1d14b6e20b8b4a3d43ba973 WatchSource:0}: Error finding container f5d160aeb51f4cc8549b8dbc42d061d9e3b39ab2b1d14b6e20b8b4a3d43ba973: Status 404 returned error can't find the container with id f5d160aeb51f4cc8549b8dbc42d061d9e3b39ab2b1d14b6e20b8b4a3d43ba973 Sep 09 02:47:43.290740 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:43.290704 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b52336-76795b4f45-rq9vd" event={"ID":"8ccaf914-1106-45be-94fb-01462d2eb9d5","Type":"ContainerStarted","Data":"972700d5ee1c276b064fdac5326e0df3bab6b34944d58f8cd6c9dbf9cbb8d4d7"} Sep 09 02:47:43.291240 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:43.290769 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b52336-76795b4f45-rq9vd" event={"ID":"8ccaf914-1106-45be-94fb-01462d2eb9d5","Type":"ContainerStarted","Data":"c511fd6915cd0a1bd53981df1352a7a290d631f2aa127864a6d0ba2e8ec00c25"} Sep 09 02:47:43.291240 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:43.290826 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-b52336-76795b4f45-rq9vd" Sep 09 02:47:43.292749 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:43.292720 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b52336-556bb79b9d-zlmhw" event={"ID":"3ac2aecd-d411-4d96-a19e-904369e694e6","Type":"ContainerStarted","Data":"23b675299cbec0e7c99ff04f6abcc2b0851e570f0d81ecb33f0b356b45994701"} Sep 09 02:47:43.292885 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:43.292774 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b52336-556bb79b9d-zlmhw" Sep 09 02:47:43.292885 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:43.292797 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b52336-556bb79b9d-zlmhw" event={"ID":"3ac2aecd-d411-4d96-a19e-904369e694e6","Type":"ContainerStarted","Data":"f5d160aeb51f4cc8549b8dbc42d061d9e3b39ab2b1d14b6e20b8b4a3d43ba973"} Sep 09 02:47:43.308150 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:43.308108 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-b52336-76795b4f45-rq9vd" podStartSLOduration=2.30809511 podStartE2EDuration="2.30809511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:47:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:47:43.307119252 +0000 UTC m=+1056.671453716" watchObservedRunningTime="2026-09-09 02:47:43.30809511 +0000 UTC m=+1056.672429574" Sep 09 02:47:43.327042 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:43.327013 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1816a546-6f474d47fc-jm574"] Sep 09 02:47:43.327593 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:43.327318 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-1816a546-6f474d47fc-jm574" podUID="976bf3d7-56f0-47a7-a3b6-c510a53d7fb7" containerName="payload-processing" containerID="cri-o://5a99c371b5615296fd9223e597ac3c652da7772ee21b4e1e03e882f8b4b294ee" gracePeriod=30 Sep 09 02:47:43.330319 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:43.330268 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b52336-556bb79b9d-zlmhw" podStartSLOduration=2.330250617 podStartE2EDuration="2.330250617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:47:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:47:43.326271782 +0000 UTC m=+1056.690606246" watchObservedRunningTime="2026-09-09 02:47:43.330250617 +0000 UTC m=+1056.694585080" Sep 09 02:47:43.339838 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:43.339747 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1816a546-9d655c8b-66bx4"] Sep 09 02:47:43.339969 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:43.339936 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-1816a546-9d655c8b-66bx4" podUID="91d9eb1c-a11b-4a67-b364-5107164c309a" containerName="payload-pre-processing" containerID="cri-o://9172d5274f83b04ac90ca024d5733e383a323e4aac33072b28378746cddce386" gracePeriod=30 Sep 09 02:47:43.594891 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:43.594861 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1816a546-6f474d47fc-jm574" Sep 09 02:47:43.613685 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:43.613655 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1816a546-9d655c8b-66bx4" Sep 09 02:47:43.702080 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:43.702049 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6tchv\" (UniqueName: \"kubernetes.io/projected/976bf3d7-56f0-47a7-a3b6-c510a53d7fb7-kube-api-access-6tchv\") pod \"976bf3d7-56f0-47a7-a3b6-c510a53d7fb7\" (UID: \"976bf3d7-56f0-47a7-a3b6-c510a53d7fb7\") " Sep 09 02:47:43.702263 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:43.702098 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91d9eb1c-a11b-4a67-b364-5107164c309a-plugins-config-volume\") pod \"91d9eb1c-a11b-4a67-b364-5107164c309a\" (UID: \"91d9eb1c-a11b-4a67-b364-5107164c309a\") " Sep 09 02:47:43.702263 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:43.702128 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zt7rr\" (UniqueName: \"kubernetes.io/projected/91d9eb1c-a11b-4a67-b364-5107164c309a-kube-api-access-zt7rr\") pod \"91d9eb1c-a11b-4a67-b364-5107164c309a\" (UID: \"91d9eb1c-a11b-4a67-b364-5107164c309a\") " Sep 09 02:47:43.702263 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:43.702165 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/976bf3d7-56f0-47a7-a3b6-c510a53d7fb7-plugins-config-volume\") pod \"976bf3d7-56f0-47a7-a3b6-c510a53d7fb7\" (UID: \"976bf3d7-56f0-47a7-a3b6-c510a53d7fb7\") " Sep 09 02:47:43.702478 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:43.702450 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91d9eb1c-a11b-4a67-b364-5107164c309a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "91d9eb1c-a11b-4a67-b364-5107164c309a" (UID: "91d9eb1c-a11b-4a67-b364-5107164c309a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:47:43.702606 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:43.702581 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/976bf3d7-56f0-47a7-a3b6-c510a53d7fb7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "976bf3d7-56f0-47a7-a3b6-c510a53d7fb7" (UID: "976bf3d7-56f0-47a7-a3b6-c510a53d7fb7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:47:43.704329 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:43.704308 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91d9eb1c-a11b-4a67-b364-5107164c309a-kube-api-access-zt7rr" (OuterVolumeSpecName: "kube-api-access-zt7rr") pod "91d9eb1c-a11b-4a67-b364-5107164c309a" (UID: "91d9eb1c-a11b-4a67-b364-5107164c309a"). InnerVolumeSpecName "kube-api-access-zt7rr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:47:43.704404 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:43.704339 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/976bf3d7-56f0-47a7-a3b6-c510a53d7fb7-kube-api-access-6tchv" (OuterVolumeSpecName: "kube-api-access-6tchv") pod "976bf3d7-56f0-47a7-a3b6-c510a53d7fb7" (UID: "976bf3d7-56f0-47a7-a3b6-c510a53d7fb7"). InnerVolumeSpecName "kube-api-access-6tchv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:47:43.803172 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:43.803136 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tchv\" (UniqueName: \"kubernetes.io/projected/976bf3d7-56f0-47a7-a3b6-c510a53d7fb7-kube-api-access-6tchv\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:47:43.803172 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:43.803167 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91d9eb1c-a11b-4a67-b364-5107164c309a-plugins-config-volume\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:47:43.803172 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:43.803178 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt7rr\" (UniqueName: \"kubernetes.io/projected/91d9eb1c-a11b-4a67-b364-5107164c309a-kube-api-access-zt7rr\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:47:43.803401 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:43.803189 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/976bf3d7-56f0-47a7-a3b6-c510a53d7fb7-plugins-config-volume\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:47:44.299798 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:44.299762 2577 generic.go:356] "Generic (PLEG): container finished" podID="91d9eb1c-a11b-4a67-b364-5107164c309a" containerID="9172d5274f83b04ac90ca024d5733e383a323e4aac33072b28378746cddce386" exitCode=0 Sep 09 02:47:44.300249 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:44.299826 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1816a546-9d655c8b-66bx4" Sep 09 02:47:44.300249 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:44.299841 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1816a546-9d655c8b-66bx4" event={"ID":"91d9eb1c-a11b-4a67-b364-5107164c309a","Type":"ContainerDied","Data":"9172d5274f83b04ac90ca024d5733e383a323e4aac33072b28378746cddce386"} Sep 09 02:47:44.300249 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:44.299888 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1816a546-9d655c8b-66bx4" event={"ID":"91d9eb1c-a11b-4a67-b364-5107164c309a","Type":"ContainerDied","Data":"fedbad701a9805b4da6f6e92ba149c74e36c0d2441113f898c79cccb1f3cfa37"} Sep 09 02:47:44.300249 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:44.299909 2577 scope.go:117] "RemoveContainer" containerID="9172d5274f83b04ac90ca024d5733e383a323e4aac33072b28378746cddce386" Sep 09 02:47:44.301305 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:44.301279 2577 generic.go:356] "Generic (PLEG): container finished" podID="976bf3d7-56f0-47a7-a3b6-c510a53d7fb7" containerID="5a99c371b5615296fd9223e597ac3c652da7772ee21b4e1e03e882f8b4b294ee" exitCode=0 Sep 09 02:47:44.301426 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:44.301348 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1816a546-6f474d47fc-jm574" Sep 09 02:47:44.301426 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:44.301386 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1816a546-6f474d47fc-jm574" event={"ID":"976bf3d7-56f0-47a7-a3b6-c510a53d7fb7","Type":"ContainerDied","Data":"5a99c371b5615296fd9223e597ac3c652da7772ee21b4e1e03e882f8b4b294ee"} Sep 09 02:47:44.301608 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:44.301592 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1816a546-6f474d47fc-jm574" event={"ID":"976bf3d7-56f0-47a7-a3b6-c510a53d7fb7","Type":"ContainerDied","Data":"eb96f5b77a6bd0762a0a29268b2587d7263ad38b7996ec5a52189bcd7b383592"} Sep 09 02:47:44.311329 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:44.311312 2577 scope.go:117] "RemoveContainer" containerID="9172d5274f83b04ac90ca024d5733e383a323e4aac33072b28378746cddce386" Sep 09 02:47:44.311646 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:47:44.311625 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9172d5274f83b04ac90ca024d5733e383a323e4aac33072b28378746cddce386\": container with ID starting with 9172d5274f83b04ac90ca024d5733e383a323e4aac33072b28378746cddce386 not found: ID does not exist" containerID="9172d5274f83b04ac90ca024d5733e383a323e4aac33072b28378746cddce386" Sep 09 02:47:44.311695 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:44.311654 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9172d5274f83b04ac90ca024d5733e383a323e4aac33072b28378746cddce386"} err="failed to get container status \"9172d5274f83b04ac90ca024d5733e383a323e4aac33072b28378746cddce386\": rpc error: code = NotFound desc = could not find container \"9172d5274f83b04ac90ca024d5733e383a323e4aac33072b28378746cddce386\": container with ID starting with 9172d5274f83b04ac90ca024d5733e383a323e4aac33072b28378746cddce386 not found: ID does not exist" Sep 09 02:47:44.311695 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:44.311670 2577 scope.go:117] "RemoveContainer" containerID="5a99c371b5615296fd9223e597ac3c652da7772ee21b4e1e03e882f8b4b294ee" Sep 09 02:47:44.319839 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:44.319820 2577 scope.go:117] "RemoveContainer" containerID="5a99c371b5615296fd9223e597ac3c652da7772ee21b4e1e03e882f8b4b294ee" Sep 09 02:47:44.320157 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:47:44.320128 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a99c371b5615296fd9223e597ac3c652da7772ee21b4e1e03e882f8b4b294ee\": container with ID starting with 5a99c371b5615296fd9223e597ac3c652da7772ee21b4e1e03e882f8b4b294ee not found: ID does not exist" containerID="5a99c371b5615296fd9223e597ac3c652da7772ee21b4e1e03e882f8b4b294ee" Sep 09 02:47:44.320273 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:44.320168 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a99c371b5615296fd9223e597ac3c652da7772ee21b4e1e03e882f8b4b294ee"} err="failed to get container status \"5a99c371b5615296fd9223e597ac3c652da7772ee21b4e1e03e882f8b4b294ee\": rpc error: code = NotFound desc = could not find container \"5a99c371b5615296fd9223e597ac3c652da7772ee21b4e1e03e882f8b4b294ee\": container with ID starting with 5a99c371b5615296fd9223e597ac3c652da7772ee21b4e1e03e882f8b4b294ee not found: ID does not exist" Sep 09 02:47:44.323306 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:44.323284 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1816a546-9d655c8b-66bx4"] Sep 09 02:47:44.328054 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:44.328031 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1816a546-9d655c8b-66bx4"] Sep 09 02:47:44.349184 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:44.349158 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1816a546-6f474d47fc-jm574"] Sep 09 02:47:44.351419 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:44.351401 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1816a546-6f474d47fc-jm574"] Sep 09 02:47:45.164832 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:45.164804 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91d9eb1c-a11b-4a67-b364-5107164c309a" path="/var/lib/kubelet/pods/91d9eb1c-a11b-4a67-b364-5107164c309a/volumes" Sep 09 02:47:45.165223 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:45.165207 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="976bf3d7-56f0-47a7-a3b6-c510a53d7fb7" path="/var/lib/kubelet/pods/976bf3d7-56f0-47a7-a3b6-c510a53d7fb7/volumes" Sep 09 02:47:54.304150 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:54.304113 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-b52336-76795b4f45-rq9vd" Sep 09 02:47:54.304623 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:47:54.304297 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b52336-556bb79b9d-zlmhw" Sep 09 02:48:38.607186 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:38.607152 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b52336-7g7h9"] Sep 09 02:48:38.607764 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:38.607686 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="976bf3d7-56f0-47a7-a3b6-c510a53d7fb7" containerName="payload-processing" Sep 09 02:48:38.607764 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:38.607703 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="976bf3d7-56f0-47a7-a3b6-c510a53d7fb7" containerName="payload-processing" Sep 09 02:48:38.607764 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:38.607734 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91d9eb1c-a11b-4a67-b364-5107164c309a" containerName="payload-pre-processing" Sep 09 02:48:38.607764 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:38.607743 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="91d9eb1c-a11b-4a67-b364-5107164c309a" containerName="payload-pre-processing" Sep 09 02:48:38.608016 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:38.607829 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="91d9eb1c-a11b-4a67-b364-5107164c309a" containerName="payload-pre-processing" Sep 09 02:48:38.608016 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:38.607843 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="976bf3d7-56f0-47a7-a3b6-c510a53d7fb7" containerName="payload-processing" Sep 09 02:48:38.611000 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:38.610981 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b52336-7g7h9" Sep 09 02:48:38.613525 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:38.613504 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbr52\"" Sep 09 02:48:38.613657 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:38.613525 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 02:48:38.621396 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:38.621374 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b52336-7g7h9"] Sep 09 02:48:38.654373 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:38.654348 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ad8f9dd-ca4e-44b0-bf18-86c008722ffa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-b52336-7g7h9\" (UID: \"7ad8f9dd-ca4e-44b0-bf18-86c008722ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b52336-7g7h9" Sep 09 02:48:38.755588 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:38.755557 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ad8f9dd-ca4e-44b0-bf18-86c008722ffa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-b52336-7g7h9\" (UID: \"7ad8f9dd-ca4e-44b0-bf18-86c008722ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b52336-7g7h9" Sep 09 02:48:38.756244 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:38.756223 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ad8f9dd-ca4e-44b0-bf18-86c008722ffa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-b52336-7g7h9\" (UID: \"7ad8f9dd-ca4e-44b0-bf18-86c008722ffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b52336-7g7h9" Sep 09 02:48:38.921536 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:38.921455 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b52336-7g7h9" Sep 09 02:48:39.193850 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:39.193737 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b52336-7g7h9"] Sep 09 02:48:39.194235 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:48:39.194209 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ad8f9dd_ca4e_44b0_bf18_86c008722ffa.slice/crio-de4bff413be3123219ae1b6aa5d2e47bda3a9ee1e698419ba4638a9fe205cce3 WatchSource:0}: Error finding container de4bff413be3123219ae1b6aa5d2e47bda3a9ee1e698419ba4638a9fe205cce3: Status 404 returned error can't find the container with id de4bff413be3123219ae1b6aa5d2e47bda3a9ee1e698419ba4638a9fe205cce3 Sep 09 02:48:39.609921 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:39.609888 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b52336-7g7h9" event={"ID":"7ad8f9dd-ca4e-44b0-bf18-86c008722ffa","Type":"ContainerStarted","Data":"4eba8d3231de22675d5c03d934f938d75c8fcc8201cd66cf15103deff5fc825c"} Sep 09 02:48:39.610291 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:39.609994 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b52336-7g7h9" event={"ID":"7ad8f9dd-ca4e-44b0-bf18-86c008722ffa","Type":"ContainerStarted","Data":"de4bff413be3123219ae1b6aa5d2e47bda3a9ee1e698419ba4638a9fe205cce3"} Sep 09 02:48:39.626880 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:39.626837 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b52336-7g7h9" podStartSLOduration=1.626823243 podStartE2EDuration="1.626823243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:48:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:48:39.625180101 +0000 UTC m=+1112.989514625" watchObservedRunningTime="2026-09-09 02:48:39.626823243 +0000 UTC m=+1112.991157739" Sep 09 02:48:40.617412 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:40.617379 2577 generic.go:356] "Generic (PLEG): container finished" podID="7ad8f9dd-ca4e-44b0-bf18-86c008722ffa" containerID="4eba8d3231de22675d5c03d934f938d75c8fcc8201cd66cf15103deff5fc825c" exitCode=0 Sep 09 02:48:40.617860 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:40.617436 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b52336-7g7h9" event={"ID":"7ad8f9dd-ca4e-44b0-bf18-86c008722ffa","Type":"ContainerDied","Data":"4eba8d3231de22675d5c03d934f938d75c8fcc8201cd66cf15103deff5fc825c"} Sep 09 02:48:41.756838 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:41.756816 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b52336-7g7h9" Sep 09 02:48:41.882832 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:41.882754 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ad8f9dd-ca4e-44b0-bf18-86c008722ffa-maas-api-service-ca\") pod \"7ad8f9dd-ca4e-44b0-bf18-86c008722ffa\" (UID: \"7ad8f9dd-ca4e-44b0-bf18-86c008722ffa\") " Sep 09 02:48:41.883112 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:41.883088 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ad8f9dd-ca4e-44b0-bf18-86c008722ffa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7ad8f9dd-ca4e-44b0-bf18-86c008722ffa" (UID: "7ad8f9dd-ca4e-44b0-bf18-86c008722ffa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:48:41.984192 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:41.984159 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ad8f9dd-ca4e-44b0-bf18-86c008722ffa-maas-api-service-ca\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:48:42.629840 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:42.629812 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b52336-7g7h9" Sep 09 02:48:42.630050 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:42.629858 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b52336-7g7h9" event={"ID":"7ad8f9dd-ca4e-44b0-bf18-86c008722ffa","Type":"ContainerDied","Data":"de4bff413be3123219ae1b6aa5d2e47bda3a9ee1e698419ba4638a9fe205cce3"} Sep 09 02:48:42.630050 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:42.629904 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de4bff413be3123219ae1b6aa5d2e47bda3a9ee1e698419ba4638a9fe205cce3" Sep 09 02:48:49.266558 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:49.266522 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b52336-76795b4f45-rq9vd"] Sep 09 02:48:49.267123 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:49.266789 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-b52336-76795b4f45-rq9vd" podUID="8ccaf914-1106-45be-94fb-01462d2eb9d5" containerName="payload-processing" containerID="cri-o://972700d5ee1c276b064fdac5326e0df3bab6b34944d58f8cd6c9dbf9cbb8d4d7" gracePeriod=30 Sep 09 02:48:49.286411 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:49.286382 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b52336-556bb79b9d-zlmhw"] Sep 09 02:48:49.286952 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:49.286923 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b52336-556bb79b9d-zlmhw" podUID="3ac2aecd-d411-4d96-a19e-904369e694e6" containerName="payload-pre-processing" containerID="cri-o://23b675299cbec0e7c99ff04f6abcc2b0851e570f0d81ecb33f0b356b45994701" gracePeriod=30 Sep 09 02:48:49.436309 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:49.436263 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b52336-7g7h9"] Sep 09 02:48:49.437451 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:49.437414 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b52336-7g7h9"] Sep 09 02:48:49.536050 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:49.536027 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b52336-76795b4f45-rq9vd" Sep 09 02:48:49.568367 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:49.568341 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b52336-556bb79b9d-zlmhw" Sep 09 02:48:49.648152 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:49.648126 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9x97p\" (UniqueName: \"kubernetes.io/projected/3ac2aecd-d411-4d96-a19e-904369e694e6-kube-api-access-9x97p\") pod \"3ac2aecd-d411-4d96-a19e-904369e694e6\" (UID: \"3ac2aecd-d411-4d96-a19e-904369e694e6\") " Sep 09 02:48:49.648330 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:49.648161 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ac2aecd-d411-4d96-a19e-904369e694e6-plugins-config-volume\") pod \"3ac2aecd-d411-4d96-a19e-904369e694e6\" (UID: \"3ac2aecd-d411-4d96-a19e-904369e694e6\") " Sep 09 02:48:49.648330 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:49.648254 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjrw6\" (UniqueName: \"kubernetes.io/projected/8ccaf914-1106-45be-94fb-01462d2eb9d5-kube-api-access-rjrw6\") pod \"8ccaf914-1106-45be-94fb-01462d2eb9d5\" (UID: \"8ccaf914-1106-45be-94fb-01462d2eb9d5\") " Sep 09 02:48:49.648330 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:49.648296 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ccaf914-1106-45be-94fb-01462d2eb9d5-plugins-config-volume\") pod \"8ccaf914-1106-45be-94fb-01462d2eb9d5\" (UID: \"8ccaf914-1106-45be-94fb-01462d2eb9d5\") " Sep 09 02:48:49.648677 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:49.648643 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ac2aecd-d411-4d96-a19e-904369e694e6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3ac2aecd-d411-4d96-a19e-904369e694e6" (UID: "3ac2aecd-d411-4d96-a19e-904369e694e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:48:49.648789 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:49.648689 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ccaf914-1106-45be-94fb-01462d2eb9d5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8ccaf914-1106-45be-94fb-01462d2eb9d5" (UID: "8ccaf914-1106-45be-94fb-01462d2eb9d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:48:49.650425 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:49.650402 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ac2aecd-d411-4d96-a19e-904369e694e6-kube-api-access-9x97p" (OuterVolumeSpecName: "kube-api-access-9x97p") pod "3ac2aecd-d411-4d96-a19e-904369e694e6" (UID: "3ac2aecd-d411-4d96-a19e-904369e694e6"). InnerVolumeSpecName "kube-api-access-9x97p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:48:49.650498 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:49.650452 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ccaf914-1106-45be-94fb-01462d2eb9d5-kube-api-access-rjrw6" (OuterVolumeSpecName: "kube-api-access-rjrw6") pod "8ccaf914-1106-45be-94fb-01462d2eb9d5" (UID: "8ccaf914-1106-45be-94fb-01462d2eb9d5"). InnerVolumeSpecName "kube-api-access-rjrw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:48:49.669797 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:49.669765 2577 generic.go:356] "Generic (PLEG): container finished" podID="8ccaf914-1106-45be-94fb-01462d2eb9d5" containerID="972700d5ee1c276b064fdac5326e0df3bab6b34944d58f8cd6c9dbf9cbb8d4d7" exitCode=0 Sep 09 02:48:49.669981 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:49.669818 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b52336-76795b4f45-rq9vd" Sep 09 02:48:49.669981 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:49.669853 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b52336-76795b4f45-rq9vd" event={"ID":"8ccaf914-1106-45be-94fb-01462d2eb9d5","Type":"ContainerDied","Data":"972700d5ee1c276b064fdac5326e0df3bab6b34944d58f8cd6c9dbf9cbb8d4d7"} Sep 09 02:48:49.669981 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:49.669907 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b52336-76795b4f45-rq9vd" event={"ID":"8ccaf914-1106-45be-94fb-01462d2eb9d5","Type":"ContainerDied","Data":"c511fd6915cd0a1bd53981df1352a7a290d631f2aa127864a6d0ba2e8ec00c25"} Sep 09 02:48:49.669981 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:49.669931 2577 scope.go:117] "RemoveContainer" containerID="972700d5ee1c276b064fdac5326e0df3bab6b34944d58f8cd6c9dbf9cbb8d4d7" Sep 09 02:48:49.671432 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:49.671404 2577 generic.go:356] "Generic (PLEG): container finished" podID="3ac2aecd-d411-4d96-a19e-904369e694e6" containerID="23b675299cbec0e7c99ff04f6abcc2b0851e570f0d81ecb33f0b356b45994701" exitCode=0 Sep 09 02:48:49.671539 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:49.671451 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b52336-556bb79b9d-zlmhw" event={"ID":"3ac2aecd-d411-4d96-a19e-904369e694e6","Type":"ContainerDied","Data":"23b675299cbec0e7c99ff04f6abcc2b0851e570f0d81ecb33f0b356b45994701"} Sep 09 02:48:49.671539 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:49.671479 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b52336-556bb79b9d-zlmhw" Sep 09 02:48:49.671539 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:49.671493 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b52336-556bb79b9d-zlmhw" event={"ID":"3ac2aecd-d411-4d96-a19e-904369e694e6","Type":"ContainerDied","Data":"f5d160aeb51f4cc8549b8dbc42d061d9e3b39ab2b1d14b6e20b8b4a3d43ba973"} Sep 09 02:48:49.680288 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:49.680268 2577 scope.go:117] "RemoveContainer" containerID="972700d5ee1c276b064fdac5326e0df3bab6b34944d58f8cd6c9dbf9cbb8d4d7" Sep 09 02:48:49.680607 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:48:49.680587 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"972700d5ee1c276b064fdac5326e0df3bab6b34944d58f8cd6c9dbf9cbb8d4d7\": container with ID starting with 972700d5ee1c276b064fdac5326e0df3bab6b34944d58f8cd6c9dbf9cbb8d4d7 not found: ID does not exist" containerID="972700d5ee1c276b064fdac5326e0df3bab6b34944d58f8cd6c9dbf9cbb8d4d7" Sep 09 02:48:49.680690 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:49.680620 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"972700d5ee1c276b064fdac5326e0df3bab6b34944d58f8cd6c9dbf9cbb8d4d7"} err="failed to get container status \"972700d5ee1c276b064fdac5326e0df3bab6b34944d58f8cd6c9dbf9cbb8d4d7\": rpc error: code = NotFound desc = could not find container \"972700d5ee1c276b064fdac5326e0df3bab6b34944d58f8cd6c9dbf9cbb8d4d7\": container with ID starting with 972700d5ee1c276b064fdac5326e0df3bab6b34944d58f8cd6c9dbf9cbb8d4d7 not found: ID does not exist" Sep 09 02:48:49.680690 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:49.680648 2577 scope.go:117] "RemoveContainer" containerID="23b675299cbec0e7c99ff04f6abcc2b0851e570f0d81ecb33f0b356b45994701" Sep 09 02:48:49.691196 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:49.691168 2577 scope.go:117] "RemoveContainer" containerID="23b675299cbec0e7c99ff04f6abcc2b0851e570f0d81ecb33f0b356b45994701" Sep 09 02:48:49.691558 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:48:49.691538 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23b675299cbec0e7c99ff04f6abcc2b0851e570f0d81ecb33f0b356b45994701\": container with ID starting with 23b675299cbec0e7c99ff04f6abcc2b0851e570f0d81ecb33f0b356b45994701 not found: ID does not exist" containerID="23b675299cbec0e7c99ff04f6abcc2b0851e570f0d81ecb33f0b356b45994701" Sep 09 02:48:49.691643 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:49.691566 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23b675299cbec0e7c99ff04f6abcc2b0851e570f0d81ecb33f0b356b45994701"} err="failed to get container status \"23b675299cbec0e7c99ff04f6abcc2b0851e570f0d81ecb33f0b356b45994701\": rpc error: code = NotFound desc = could not find container \"23b675299cbec0e7c99ff04f6abcc2b0851e570f0d81ecb33f0b356b45994701\": container with ID starting with 23b675299cbec0e7c99ff04f6abcc2b0851e570f0d81ecb33f0b356b45994701 not found: ID does not exist" Sep 09 02:48:49.696475 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:49.696453 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b52336-76795b4f45-rq9vd"] Sep 09 02:48:49.702015 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:49.701995 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b52336-76795b4f45-rq9vd"] Sep 09 02:48:49.712832 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:49.712809 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b52336-556bb79b9d-zlmhw"] Sep 09 02:48:49.716349 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:49.716322 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b52336-556bb79b9d-zlmhw"] Sep 09 02:48:49.749557 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:49.749533 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x97p\" (UniqueName: \"kubernetes.io/projected/3ac2aecd-d411-4d96-a19e-904369e694e6-kube-api-access-9x97p\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:48:49.749557 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:49.749555 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ac2aecd-d411-4d96-a19e-904369e694e6-plugins-config-volume\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:48:49.749694 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:49.749566 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjrw6\" (UniqueName: \"kubernetes.io/projected/8ccaf914-1106-45be-94fb-01462d2eb9d5-kube-api-access-rjrw6\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:48:49.749694 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:49.749574 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ccaf914-1106-45be-94fb-01462d2eb9d5-plugins-config-volume\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:48:51.164556 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:51.164524 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ac2aecd-d411-4d96-a19e-904369e694e6" path="/var/lib/kubelet/pods/3ac2aecd-d411-4d96-a19e-904369e694e6/volumes" Sep 09 02:48:51.164897 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:51.164851 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ad8f9dd-ca4e-44b0-bf18-86c008722ffa" path="/var/lib/kubelet/pods/7ad8f9dd-ca4e-44b0-bf18-86c008722ffa/volumes" Sep 09 02:48:51.165172 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:48:51.165158 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ccaf914-1106-45be-94fb-01462d2eb9d5" path="/var/lib/kubelet/pods/8ccaf914-1106-45be-94fb-01462d2eb9d5/volumes" Sep 09 02:49:25.063818 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:25.063783 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-81f1254e-796f9bb66b-9bb74"] Sep 09 02:49:25.064287 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:25.064159 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ad8f9dd-ca4e-44b0-bf18-86c008722ffa" containerName="revoke-keys" Sep 09 02:49:25.064287 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:25.064171 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ad8f9dd-ca4e-44b0-bf18-86c008722ffa" containerName="revoke-keys" Sep 09 02:49:25.064287 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:25.064185 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ccaf914-1106-45be-94fb-01462d2eb9d5" containerName="payload-processing" Sep 09 02:49:25.064287 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:25.064191 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ccaf914-1106-45be-94fb-01462d2eb9d5" containerName="payload-processing" Sep 09 02:49:25.064287 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:25.064203 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ac2aecd-d411-4d96-a19e-904369e694e6" containerName="payload-pre-processing" Sep 09 02:49:25.064287 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:25.064208 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ac2aecd-d411-4d96-a19e-904369e694e6" containerName="payload-pre-processing" Sep 09 02:49:25.064287 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:25.064258 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ac2aecd-d411-4d96-a19e-904369e694e6" containerName="payload-pre-processing" Sep 09 02:49:25.064287 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:25.064267 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ccaf914-1106-45be-94fb-01462d2eb9d5" containerName="payload-processing" Sep 09 02:49:25.064287 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:25.064274 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ad8f9dd-ca4e-44b0-bf18-86c008722ffa" containerName="revoke-keys" Sep 09 02:49:25.068479 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:25.068457 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-81f1254e-796f9bb66b-9bb74" Sep 09 02:49:25.071227 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:25.071208 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-81f1254e\"" Sep 09 02:49:25.071341 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:25.071221 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-81f1254e-dockercfg-rpvpg\"" Sep 09 02:49:25.079719 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:25.079699 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-81f1254e-796f9bb66b-9bb74"] Sep 09 02:49:25.081743 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:25.081725 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-81f1254e-5b7fd6b9df-b9wcr"] Sep 09 02:49:25.084874 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:25.084859 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-81f1254e-5b7fd6b9df-b9wcr" Sep 09 02:49:25.097048 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:25.097025 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-81f1254e-5b7fd6b9df-b9wcr"] Sep 09 02:49:25.147742 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:25.147715 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbb1c560-f8b3-4e5e-b4df-b88d665f36a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-81f1254e-796f9bb66b-9bb74\" (UID: \"fbb1c560-f8b3-4e5e-b4df-b88d665f36a7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-81f1254e-796f9bb66b-9bb74" Sep 09 02:49:25.147887 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:25.147768 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbvr2\" (UniqueName: \"kubernetes.io/projected/42bb72c6-06bc-479b-9277-91e6429806be-kube-api-access-lbvr2\") pod \"payload-processing-e2e-mt-81f1254e-5b7fd6b9df-b9wcr\" (UID: \"42bb72c6-06bc-479b-9277-91e6429806be\") " pod="openshift-ingress/payload-processing-e2e-mt-81f1254e-5b7fd6b9df-b9wcr" Sep 09 02:49:25.147887 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:25.147790 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndjv7\" (UniqueName: \"kubernetes.io/projected/fbb1c560-f8b3-4e5e-b4df-b88d665f36a7-kube-api-access-ndjv7\") pod \"payload-pre-processing-e2e-mt-81f1254e-796f9bb66b-9bb74\" (UID: \"fbb1c560-f8b3-4e5e-b4df-b88d665f36a7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-81f1254e-796f9bb66b-9bb74" Sep 09 02:49:25.147887 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:25.147847 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42bb72c6-06bc-479b-9277-91e6429806be-plugins-config-volume\") pod \"payload-processing-e2e-mt-81f1254e-5b7fd6b9df-b9wcr\" (UID: \"42bb72c6-06bc-479b-9277-91e6429806be\") " pod="openshift-ingress/payload-processing-e2e-mt-81f1254e-5b7fd6b9df-b9wcr" Sep 09 02:49:25.248806 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:25.248771 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbvr2\" (UniqueName: \"kubernetes.io/projected/42bb72c6-06bc-479b-9277-91e6429806be-kube-api-access-lbvr2\") pod \"payload-processing-e2e-mt-81f1254e-5b7fd6b9df-b9wcr\" (UID: \"42bb72c6-06bc-479b-9277-91e6429806be\") " pod="openshift-ingress/payload-processing-e2e-mt-81f1254e-5b7fd6b9df-b9wcr" Sep 09 02:49:25.249004 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:25.248825 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndjv7\" (UniqueName: \"kubernetes.io/projected/fbb1c560-f8b3-4e5e-b4df-b88d665f36a7-kube-api-access-ndjv7\") pod \"payload-pre-processing-e2e-mt-81f1254e-796f9bb66b-9bb74\" (UID: \"fbb1c560-f8b3-4e5e-b4df-b88d665f36a7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-81f1254e-796f9bb66b-9bb74" Sep 09 02:49:25.249004 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:25.248862 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42bb72c6-06bc-479b-9277-91e6429806be-plugins-config-volume\") pod \"payload-processing-e2e-mt-81f1254e-5b7fd6b9df-b9wcr\" (UID: \"42bb72c6-06bc-479b-9277-91e6429806be\") " pod="openshift-ingress/payload-processing-e2e-mt-81f1254e-5b7fd6b9df-b9wcr" Sep 09 02:49:25.249004 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:25.248920 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbb1c560-f8b3-4e5e-b4df-b88d665f36a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-81f1254e-796f9bb66b-9bb74\" (UID: \"fbb1c560-f8b3-4e5e-b4df-b88d665f36a7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-81f1254e-796f9bb66b-9bb74" Sep 09 02:49:25.249629 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:25.249608 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbb1c560-f8b3-4e5e-b4df-b88d665f36a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-81f1254e-796f9bb66b-9bb74\" (UID: \"fbb1c560-f8b3-4e5e-b4df-b88d665f36a7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-81f1254e-796f9bb66b-9bb74" Sep 09 02:49:25.249679 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:25.249653 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42bb72c6-06bc-479b-9277-91e6429806be-plugins-config-volume\") pod \"payload-processing-e2e-mt-81f1254e-5b7fd6b9df-b9wcr\" (UID: \"42bb72c6-06bc-479b-9277-91e6429806be\") " pod="openshift-ingress/payload-processing-e2e-mt-81f1254e-5b7fd6b9df-b9wcr" Sep 09 02:49:25.257346 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:25.257323 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbvr2\" (UniqueName: \"kubernetes.io/projected/42bb72c6-06bc-479b-9277-91e6429806be-kube-api-access-lbvr2\") pod \"payload-processing-e2e-mt-81f1254e-5b7fd6b9df-b9wcr\" (UID: \"42bb72c6-06bc-479b-9277-91e6429806be\") " pod="openshift-ingress/payload-processing-e2e-mt-81f1254e-5b7fd6b9df-b9wcr" Sep 09 02:49:25.257448 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:25.257346 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndjv7\" (UniqueName: \"kubernetes.io/projected/fbb1c560-f8b3-4e5e-b4df-b88d665f36a7-kube-api-access-ndjv7\") pod \"payload-pre-processing-e2e-mt-81f1254e-796f9bb66b-9bb74\" (UID: \"fbb1c560-f8b3-4e5e-b4df-b88d665f36a7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-81f1254e-796f9bb66b-9bb74" Sep 09 02:49:25.379897 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:25.379819 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-81f1254e-796f9bb66b-9bb74" Sep 09 02:49:25.399006 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:25.398977 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-81f1254e-5b7fd6b9df-b9wcr" Sep 09 02:49:25.660165 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:25.660135 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-81f1254e-796f9bb66b-9bb74"] Sep 09 02:49:25.661229 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:49:25.661200 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbb1c560_f8b3_4e5e_b4df_b88d665f36a7.slice/crio-076fe21cf2fd6161cf3985027e7a0df37656b6afb5cb8867bba3f1e5690c60dc WatchSource:0}: Error finding container 076fe21cf2fd6161cf3985027e7a0df37656b6afb5cb8867bba3f1e5690c60dc: Status 404 returned error can't find the container with id 076fe21cf2fd6161cf3985027e7a0df37656b6afb5cb8867bba3f1e5690c60dc Sep 09 02:49:25.680592 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:25.680569 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-81f1254e-5b7fd6b9df-b9wcr"] Sep 09 02:49:25.681356 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:49:25.681325 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42bb72c6_06bc_479b_9277_91e6429806be.slice/crio-f12568ab5a0ffe27ee2614de7a3017c2422047bb2b66c8d3aea2c53546a2e6b4 WatchSource:0}: Error finding container f12568ab5a0ffe27ee2614de7a3017c2422047bb2b66c8d3aea2c53546a2e6b4: Status 404 returned error can't find the container with id f12568ab5a0ffe27ee2614de7a3017c2422047bb2b66c8d3aea2c53546a2e6b4 Sep 09 02:49:25.872253 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:25.872201 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-81f1254e-5b7fd6b9df-b9wcr" event={"ID":"42bb72c6-06bc-479b-9277-91e6429806be","Type":"ContainerStarted","Data":"89d9c356e78d76eb7a64acc4a8c0fc3e9b5ea39503a02c54b526c63a4cb73b2e"} Sep 09 02:49:25.872438 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:25.872274 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-81f1254e-5b7fd6b9df-b9wcr" Sep 09 02:49:25.872438 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:25.872302 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-81f1254e-5b7fd6b9df-b9wcr" event={"ID":"42bb72c6-06bc-479b-9277-91e6429806be","Type":"ContainerStarted","Data":"f12568ab5a0ffe27ee2614de7a3017c2422047bb2b66c8d3aea2c53546a2e6b4"} Sep 09 02:49:25.873881 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:25.873849 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-81f1254e-796f9bb66b-9bb74" event={"ID":"fbb1c560-f8b3-4e5e-b4df-b88d665f36a7","Type":"ContainerStarted","Data":"a151412594853044553fc2a173227e388552e52eed3691292eae4195e6de1752"} Sep 09 02:49:25.874003 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:25.873895 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-81f1254e-796f9bb66b-9bb74" event={"ID":"fbb1c560-f8b3-4e5e-b4df-b88d665f36a7","Type":"ContainerStarted","Data":"076fe21cf2fd6161cf3985027e7a0df37656b6afb5cb8867bba3f1e5690c60dc"} Sep 09 02:49:25.874003 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:25.873924 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-81f1254e-796f9bb66b-9bb74" Sep 09 02:49:25.889901 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:25.889847 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-81f1254e-5b7fd6b9df-b9wcr" podStartSLOduration=0.889828625 podStartE2EDuration="889.828625ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:49:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:49:25.888213199 +0000 UTC m=+1159.252547665" watchObservedRunningTime="2026-09-09 02:49:25.889828625 +0000 UTC m=+1159.254163090" Sep 09 02:49:25.907142 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:25.907088 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-81f1254e-796f9bb66b-9bb74" podStartSLOduration=0.90706973 podStartE2EDuration="907.06973ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:49:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:49:25.903360468 +0000 UTC m=+1159.267694930" watchObservedRunningTime="2026-09-09 02:49:25.90706973 +0000 UTC m=+1159.271404195" Sep 09 02:49:36.881028 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:36.880920 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-81f1254e-796f9bb66b-9bb74" Sep 09 02:49:36.881028 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:36.881020 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-81f1254e-5b7fd6b9df-b9wcr" Sep 09 02:49:47.307587 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:47.307547 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b18109-7d498769dc-sgr4m"] Sep 09 02:49:47.307990 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:47.307803 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-b18109-7d498769dc-sgr4m" podUID="8d288425-ef73-4564-b67c-1e7d7c544efc" containerName="payload-processing" containerID="cri-o://714b58baa6d18efb4260ae2d5c85521e57848f5d34a76256b285569abac32251" gracePeriod=30 Sep 09 02:49:47.317568 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:47.317421 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b18109-55d84bf57d-cm2wr"] Sep 09 02:49:47.317751 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:47.317702 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b18109-55d84bf57d-cm2wr" podUID="75d2b491-13b2-443a-8a19-4e8e7b4f35e9" containerName="payload-pre-processing" containerID="cri-o://ab1d42e851fc8249195bfe71be737027882089db74cf90b0713cf9e2b57bdfbe" gracePeriod=30 Sep 09 02:49:47.602003 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:47.601978 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b18109-7d498769dc-sgr4m" Sep 09 02:49:47.605589 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:47.605567 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b18109-55d84bf57d-cm2wr" Sep 09 02:49:47.737828 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:47.737795 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75d2b491-13b2-443a-8a19-4e8e7b4f35e9-plugins-config-volume\") pod \"75d2b491-13b2-443a-8a19-4e8e7b4f35e9\" (UID: \"75d2b491-13b2-443a-8a19-4e8e7b4f35e9\") " Sep 09 02:49:47.738039 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:47.737869 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9djqb\" (UniqueName: \"kubernetes.io/projected/75d2b491-13b2-443a-8a19-4e8e7b4f35e9-kube-api-access-9djqb\") pod \"75d2b491-13b2-443a-8a19-4e8e7b4f35e9\" (UID: \"75d2b491-13b2-443a-8a19-4e8e7b4f35e9\") " Sep 09 02:49:47.738039 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:47.737935 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7b57\" (UniqueName: \"kubernetes.io/projected/8d288425-ef73-4564-b67c-1e7d7c544efc-kube-api-access-d7b57\") pod \"8d288425-ef73-4564-b67c-1e7d7c544efc\" (UID: \"8d288425-ef73-4564-b67c-1e7d7c544efc\") " Sep 09 02:49:47.738039 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:47.738036 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d288425-ef73-4564-b67c-1e7d7c544efc-plugins-config-volume\") pod \"8d288425-ef73-4564-b67c-1e7d7c544efc\" (UID: \"8d288425-ef73-4564-b67c-1e7d7c544efc\") " Sep 09 02:49:47.738360 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:47.738334 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75d2b491-13b2-443a-8a19-4e8e7b4f35e9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "75d2b491-13b2-443a-8a19-4e8e7b4f35e9" (UID: "75d2b491-13b2-443a-8a19-4e8e7b4f35e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:49:47.738446 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:47.738378 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d288425-ef73-4564-b67c-1e7d7c544efc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8d288425-ef73-4564-b67c-1e7d7c544efc" (UID: "8d288425-ef73-4564-b67c-1e7d7c544efc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:49:47.740152 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:47.740127 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d288425-ef73-4564-b67c-1e7d7c544efc-kube-api-access-d7b57" (OuterVolumeSpecName: "kube-api-access-d7b57") pod "8d288425-ef73-4564-b67c-1e7d7c544efc" (UID: "8d288425-ef73-4564-b67c-1e7d7c544efc"). InnerVolumeSpecName "kube-api-access-d7b57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:49:47.740228 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:47.740208 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75d2b491-13b2-443a-8a19-4e8e7b4f35e9-kube-api-access-9djqb" (OuterVolumeSpecName: "kube-api-access-9djqb") pod "75d2b491-13b2-443a-8a19-4e8e7b4f35e9" (UID: "75d2b491-13b2-443a-8a19-4e8e7b4f35e9"). InnerVolumeSpecName "kube-api-access-9djqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:49:47.839122 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:47.839042 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d288425-ef73-4564-b67c-1e7d7c544efc-plugins-config-volume\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:49:47.839122 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:47.839071 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75d2b491-13b2-443a-8a19-4e8e7b4f35e9-plugins-config-volume\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:49:47.839122 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:47.839081 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9djqb\" (UniqueName: \"kubernetes.io/projected/75d2b491-13b2-443a-8a19-4e8e7b4f35e9-kube-api-access-9djqb\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:49:47.839122 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:47.839089 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7b57\" (UniqueName: \"kubernetes.io/projected/8d288425-ef73-4564-b67c-1e7d7c544efc-kube-api-access-d7b57\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:49:47.995385 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:47.995349 2577 generic.go:356] "Generic (PLEG): container finished" podID="75d2b491-13b2-443a-8a19-4e8e7b4f35e9" containerID="ab1d42e851fc8249195bfe71be737027882089db74cf90b0713cf9e2b57bdfbe" exitCode=0 Sep 09 02:49:47.995598 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:47.995404 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b18109-55d84bf57d-cm2wr" Sep 09 02:49:47.995598 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:47.995447 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b18109-55d84bf57d-cm2wr" event={"ID":"75d2b491-13b2-443a-8a19-4e8e7b4f35e9","Type":"ContainerDied","Data":"ab1d42e851fc8249195bfe71be737027882089db74cf90b0713cf9e2b57bdfbe"} Sep 09 02:49:47.995598 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:47.995505 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b18109-55d84bf57d-cm2wr" event={"ID":"75d2b491-13b2-443a-8a19-4e8e7b4f35e9","Type":"ContainerDied","Data":"ced0b4445e32bdac8e96d00d783a95013617aa75c823894345cdf89f64857975"} Sep 09 02:49:47.995598 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:47.995536 2577 scope.go:117] "RemoveContainer" containerID="ab1d42e851fc8249195bfe71be737027882089db74cf90b0713cf9e2b57bdfbe" Sep 09 02:49:47.997028 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:47.997000 2577 generic.go:356] "Generic (PLEG): container finished" podID="8d288425-ef73-4564-b67c-1e7d7c544efc" containerID="714b58baa6d18efb4260ae2d5c85521e57848f5d34a76256b285569abac32251" exitCode=0 Sep 09 02:49:47.997151 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:47.997060 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b18109-7d498769dc-sgr4m" Sep 09 02:49:47.997151 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:47.997076 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b18109-7d498769dc-sgr4m" event={"ID":"8d288425-ef73-4564-b67c-1e7d7c544efc","Type":"ContainerDied","Data":"714b58baa6d18efb4260ae2d5c85521e57848f5d34a76256b285569abac32251"} Sep 09 02:49:47.997151 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:47.997117 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b18109-7d498769dc-sgr4m" event={"ID":"8d288425-ef73-4564-b67c-1e7d7c544efc","Type":"ContainerDied","Data":"5de5714c99c2ae833ebc75a0f71143182285ca3bc19528adabf0dcb5e2052b97"} Sep 09 02:49:48.006341 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:48.006322 2577 scope.go:117] "RemoveContainer" containerID="ab1d42e851fc8249195bfe71be737027882089db74cf90b0713cf9e2b57bdfbe" Sep 09 02:49:48.006688 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:49:48.006662 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab1d42e851fc8249195bfe71be737027882089db74cf90b0713cf9e2b57bdfbe\": container with ID starting with ab1d42e851fc8249195bfe71be737027882089db74cf90b0713cf9e2b57bdfbe not found: ID does not exist" containerID="ab1d42e851fc8249195bfe71be737027882089db74cf90b0713cf9e2b57bdfbe" Sep 09 02:49:48.006751 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:48.006700 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab1d42e851fc8249195bfe71be737027882089db74cf90b0713cf9e2b57bdfbe"} err="failed to get container status \"ab1d42e851fc8249195bfe71be737027882089db74cf90b0713cf9e2b57bdfbe\": rpc error: code = NotFound desc = could not find container \"ab1d42e851fc8249195bfe71be737027882089db74cf90b0713cf9e2b57bdfbe\": container with ID starting with ab1d42e851fc8249195bfe71be737027882089db74cf90b0713cf9e2b57bdfbe not found: ID does not exist" Sep 09 02:49:48.006751 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:48.006728 2577 scope.go:117] "RemoveContainer" containerID="714b58baa6d18efb4260ae2d5c85521e57848f5d34a76256b285569abac32251" Sep 09 02:49:48.015546 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:48.015529 2577 scope.go:117] "RemoveContainer" containerID="714b58baa6d18efb4260ae2d5c85521e57848f5d34a76256b285569abac32251" Sep 09 02:49:48.015823 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:49:48.015807 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"714b58baa6d18efb4260ae2d5c85521e57848f5d34a76256b285569abac32251\": container with ID starting with 714b58baa6d18efb4260ae2d5c85521e57848f5d34a76256b285569abac32251 not found: ID does not exist" containerID="714b58baa6d18efb4260ae2d5c85521e57848f5d34a76256b285569abac32251" Sep 09 02:49:48.015863 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:48.015833 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"714b58baa6d18efb4260ae2d5c85521e57848f5d34a76256b285569abac32251"} err="failed to get container status \"714b58baa6d18efb4260ae2d5c85521e57848f5d34a76256b285569abac32251\": rpc error: code = NotFound desc = could not find container \"714b58baa6d18efb4260ae2d5c85521e57848f5d34a76256b285569abac32251\": container with ID starting with 714b58baa6d18efb4260ae2d5c85521e57848f5d34a76256b285569abac32251 not found: ID does not exist" Sep 09 02:49:48.022343 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:48.022323 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b18109-55d84bf57d-cm2wr"] Sep 09 02:49:48.029215 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:48.029191 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b18109-55d84bf57d-cm2wr"] Sep 09 02:49:48.039657 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:48.039635 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b18109-7d498769dc-sgr4m"] Sep 09 02:49:48.046503 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:48.046484 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b18109-7d498769dc-sgr4m"] Sep 09 02:49:49.165107 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:49.165078 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75d2b491-13b2-443a-8a19-4e8e7b4f35e9" path="/var/lib/kubelet/pods/75d2b491-13b2-443a-8a19-4e8e7b4f35e9/volumes" Sep 09 02:49:49.165479 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:49.165464 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d288425-ef73-4564-b67c-1e7d7c544efc" path="/var/lib/kubelet/pods/8d288425-ef73-4564-b67c-1e7d7c544efc/volumes" Sep 09 02:49:55.829298 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:55.829257 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81f1254e-62d49"] Sep 09 02:49:55.829797 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:55.829779 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75d2b491-13b2-443a-8a19-4e8e7b4f35e9" containerName="payload-pre-processing" Sep 09 02:49:55.829886 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:55.829801 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="75d2b491-13b2-443a-8a19-4e8e7b4f35e9" containerName="payload-pre-processing" Sep 09 02:49:55.829886 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:55.829820 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d288425-ef73-4564-b67c-1e7d7c544efc" containerName="payload-processing" Sep 09 02:49:55.829886 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:55.829829 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d288425-ef73-4564-b67c-1e7d7c544efc" containerName="payload-processing" Sep 09 02:49:55.830089 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:55.829913 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d288425-ef73-4564-b67c-1e7d7c544efc" containerName="payload-processing" Sep 09 02:49:55.830089 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:55.829927 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="75d2b491-13b2-443a-8a19-4e8e7b4f35e9" containerName="payload-pre-processing" Sep 09 02:49:55.834394 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:55.834373 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81f1254e-62d49" Sep 09 02:49:55.837136 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:55.837113 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 02:49:55.837244 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:55.837155 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbr52\"" Sep 09 02:49:55.842596 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:55.841993 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81f1254e-62d49"] Sep 09 02:49:55.909426 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:55.909389 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eec439b-6ef3-47d9-b21f-b2167d3bcc2a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-81f1254e-62d49\" (UID: \"9eec439b-6ef3-47d9-b21f-b2167d3bcc2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81f1254e-62d49" Sep 09 02:49:56.010259 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:56.010213 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eec439b-6ef3-47d9-b21f-b2167d3bcc2a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-81f1254e-62d49\" (UID: \"9eec439b-6ef3-47d9-b21f-b2167d3bcc2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81f1254e-62d49" Sep 09 02:49:56.010779 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:56.010760 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eec439b-6ef3-47d9-b21f-b2167d3bcc2a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-81f1254e-62d49\" (UID: \"9eec439b-6ef3-47d9-b21f-b2167d3bcc2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81f1254e-62d49" Sep 09 02:49:56.147051 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:56.146949 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81f1254e-62d49" Sep 09 02:49:56.423115 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:56.423092 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81f1254e-62d49"] Sep 09 02:49:56.424919 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:49:56.424898 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eec439b_6ef3_47d9_b21f_b2167d3bcc2a.slice/crio-db4f6b0c4d6bdbcd84ab75662d60d61f25553e7a45afd320ad26ba4241f323c0 WatchSource:0}: Error finding container db4f6b0c4d6bdbcd84ab75662d60d61f25553e7a45afd320ad26ba4241f323c0: Status 404 returned error can't find the container with id db4f6b0c4d6bdbcd84ab75662d60d61f25553e7a45afd320ad26ba4241f323c0 Sep 09 02:49:57.049652 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:57.049620 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81f1254e-62d49" event={"ID":"9eec439b-6ef3-47d9-b21f-b2167d3bcc2a","Type":"ContainerStarted","Data":"4197e411c87e1cdf3fad6eda65e850addfd922b96c1cf410792397c96e6e7299"} Sep 09 02:49:57.050051 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:57.049669 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81f1254e-62d49" event={"ID":"9eec439b-6ef3-47d9-b21f-b2167d3bcc2a","Type":"ContainerStarted","Data":"db4f6b0c4d6bdbcd84ab75662d60d61f25553e7a45afd320ad26ba4241f323c0"} Sep 09 02:49:57.067747 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:57.067700 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81f1254e-62d49" podStartSLOduration=2.067686057 podStartE2EDuration="2.067686057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:49:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:49:57.065672614 +0000 UTC m=+1190.430007077" watchObservedRunningTime="2026-09-09 02:49:57.067686057 +0000 UTC m=+1190.432020519" Sep 09 02:49:58.056339 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:58.056305 2577 generic.go:356] "Generic (PLEG): container finished" podID="9eec439b-6ef3-47d9-b21f-b2167d3bcc2a" containerID="4197e411c87e1cdf3fad6eda65e850addfd922b96c1cf410792397c96e6e7299" exitCode=0 Sep 09 02:49:58.056716 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:58.056385 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81f1254e-62d49" event={"ID":"9eec439b-6ef3-47d9-b21f-b2167d3bcc2a","Type":"ContainerDied","Data":"4197e411c87e1cdf3fad6eda65e850addfd922b96c1cf410792397c96e6e7299"} Sep 09 02:49:59.203392 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:59.203362 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81f1254e-62d49" Sep 09 02:49:59.338816 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:59.338719 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eec439b-6ef3-47d9-b21f-b2167d3bcc2a-maas-api-service-ca\") pod \"9eec439b-6ef3-47d9-b21f-b2167d3bcc2a\" (UID: \"9eec439b-6ef3-47d9-b21f-b2167d3bcc2a\") " Sep 09 02:49:59.339115 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:59.339090 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eec439b-6ef3-47d9-b21f-b2167d3bcc2a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9eec439b-6ef3-47d9-b21f-b2167d3bcc2a" (UID: "9eec439b-6ef3-47d9-b21f-b2167d3bcc2a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:49:59.439690 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:49:59.439641 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eec439b-6ef3-47d9-b21f-b2167d3bcc2a-maas-api-service-ca\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:50:00.068079 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:00.068051 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81f1254e-62d49" Sep 09 02:50:00.068234 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:00.068090 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81f1254e-62d49" event={"ID":"9eec439b-6ef3-47d9-b21f-b2167d3bcc2a","Type":"ContainerDied","Data":"db4f6b0c4d6bdbcd84ab75662d60d61f25553e7a45afd320ad26ba4241f323c0"} Sep 09 02:50:00.068234 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:00.068136 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db4f6b0c4d6bdbcd84ab75662d60d61f25553e7a45afd320ad26ba4241f323c0" Sep 09 02:50:05.478161 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:05.478126 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6b11a2-8cfdf467c-wlprd"] Sep 09 02:50:05.478615 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:05.478407 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-6b11a2-8cfdf467c-wlprd" podUID="f45fbd28-69e2-4cc5-be16-a30e218fabaf" containerName="payload-processing" containerID="cri-o://e84c3e3deb00af1abec2eebde914cef4651001ad320031f60e5b1cf796fac9ca" gracePeriod=30 Sep 09 02:50:05.741543 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:05.741485 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6b11a2-8cfdf467c-wlprd" Sep 09 02:50:05.902501 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:05.902465 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mp2sb\" (UniqueName: \"kubernetes.io/projected/f45fbd28-69e2-4cc5-be16-a30e218fabaf-kube-api-access-mp2sb\") pod \"f45fbd28-69e2-4cc5-be16-a30e218fabaf\" (UID: \"f45fbd28-69e2-4cc5-be16-a30e218fabaf\") " Sep 09 02:50:05.902649 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:05.902555 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f45fbd28-69e2-4cc5-be16-a30e218fabaf-plugins-config-volume\") pod \"f45fbd28-69e2-4cc5-be16-a30e218fabaf\" (UID: \"f45fbd28-69e2-4cc5-be16-a30e218fabaf\") " Sep 09 02:50:05.903045 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:05.903016 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f45fbd28-69e2-4cc5-be16-a30e218fabaf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f45fbd28-69e2-4cc5-be16-a30e218fabaf" (UID: "f45fbd28-69e2-4cc5-be16-a30e218fabaf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:50:05.904744 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:05.904713 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f45fbd28-69e2-4cc5-be16-a30e218fabaf-kube-api-access-mp2sb" (OuterVolumeSpecName: "kube-api-access-mp2sb") pod "f45fbd28-69e2-4cc5-be16-a30e218fabaf" (UID: "f45fbd28-69e2-4cc5-be16-a30e218fabaf"). InnerVolumeSpecName "kube-api-access-mp2sb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:50:06.003660 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:06.003580 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp2sb\" (UniqueName: \"kubernetes.io/projected/f45fbd28-69e2-4cc5-be16-a30e218fabaf-kube-api-access-mp2sb\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:50:06.003660 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:06.003614 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f45fbd28-69e2-4cc5-be16-a30e218fabaf-plugins-config-volume\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:50:06.103002 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:06.102945 2577 generic.go:356] "Generic (PLEG): container finished" podID="f45fbd28-69e2-4cc5-be16-a30e218fabaf" containerID="e84c3e3deb00af1abec2eebde914cef4651001ad320031f60e5b1cf796fac9ca" exitCode=0 Sep 09 02:50:06.103002 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:06.103007 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6b11a2-8cfdf467c-wlprd" Sep 09 02:50:06.103192 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:06.103000 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6b11a2-8cfdf467c-wlprd" event={"ID":"f45fbd28-69e2-4cc5-be16-a30e218fabaf","Type":"ContainerDied","Data":"e84c3e3deb00af1abec2eebde914cef4651001ad320031f60e5b1cf796fac9ca"} Sep 09 02:50:06.103192 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:06.103129 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6b11a2-8cfdf467c-wlprd" event={"ID":"f45fbd28-69e2-4cc5-be16-a30e218fabaf","Type":"ContainerDied","Data":"9ff5b2ce340b1810ac173717b89b9dc150e283b72b596ec4251d33819f4df7b8"} Sep 09 02:50:06.103192 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:06.103152 2577 scope.go:117] "RemoveContainer" containerID="e84c3e3deb00af1abec2eebde914cef4651001ad320031f60e5b1cf796fac9ca" Sep 09 02:50:06.112671 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:06.112655 2577 scope.go:117] "RemoveContainer" containerID="e84c3e3deb00af1abec2eebde914cef4651001ad320031f60e5b1cf796fac9ca" Sep 09 02:50:06.112937 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:50:06.112919 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e84c3e3deb00af1abec2eebde914cef4651001ad320031f60e5b1cf796fac9ca\": container with ID starting with e84c3e3deb00af1abec2eebde914cef4651001ad320031f60e5b1cf796fac9ca not found: ID does not exist" containerID="e84c3e3deb00af1abec2eebde914cef4651001ad320031f60e5b1cf796fac9ca" Sep 09 02:50:06.113028 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:06.112949 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e84c3e3deb00af1abec2eebde914cef4651001ad320031f60e5b1cf796fac9ca"} err="failed to get container status \"e84c3e3deb00af1abec2eebde914cef4651001ad320031f60e5b1cf796fac9ca\": rpc error: code = NotFound desc = could not find container \"e84c3e3deb00af1abec2eebde914cef4651001ad320031f60e5b1cf796fac9ca\": container with ID starting with e84c3e3deb00af1abec2eebde914cef4651001ad320031f60e5b1cf796fac9ca not found: ID does not exist" Sep 09 02:50:06.126221 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:06.126190 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6b11a2-8cfdf467c-wlprd"] Sep 09 02:50:06.130180 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:06.130161 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6b11a2-8cfdf467c-wlprd"] Sep 09 02:50:06.728120 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:06.728088 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-81f1254e-5b7fd6b9df-b9wcr"] Sep 09 02:50:06.728490 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:06.728349 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-81f1254e-5b7fd6b9df-b9wcr" podUID="42bb72c6-06bc-479b-9277-91e6429806be" containerName="payload-processing" containerID="cri-o://89d9c356e78d76eb7a64acc4a8c0fc3e9b5ea39503a02c54b526c63a4cb73b2e" gracePeriod=30 Sep 09 02:50:06.736154 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:06.735758 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-81f1254e-796f9bb66b-9bb74"] Sep 09 02:50:06.736154 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:06.735975 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-81f1254e-796f9bb66b-9bb74" podUID="fbb1c560-f8b3-4e5e-b4df-b88d665f36a7" containerName="payload-pre-processing" containerID="cri-o://a151412594853044553fc2a173227e388552e52eed3691292eae4195e6de1752" gracePeriod=30 Sep 09 02:50:06.880104 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:06.880058 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-81f1254e-5b7fd6b9df-b9wcr" podUID="42bb72c6-06bc-479b-9277-91e6429806be" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.62:9004: connect: connection refused" Sep 09 02:50:06.880267 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:06.880066 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-81f1254e-796f9bb66b-9bb74" podUID="fbb1c560-f8b3-4e5e-b4df-b88d665f36a7" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.61:9004: connect: connection refused" Sep 09 02:50:06.908800 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:06.908772 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81f1254e-62d49"] Sep 09 02:50:06.913314 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:06.913291 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-81f1254e-62d49"] Sep 09 02:50:07.008334 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:07.008311 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-81f1254e-5b7fd6b9df-b9wcr" Sep 09 02:50:07.011228 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:07.011206 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-81f1254e-796f9bb66b-9bb74" Sep 09 02:50:07.110504 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:07.110468 2577 generic.go:356] "Generic (PLEG): container finished" podID="42bb72c6-06bc-479b-9277-91e6429806be" containerID="89d9c356e78d76eb7a64acc4a8c0fc3e9b5ea39503a02c54b526c63a4cb73b2e" exitCode=0 Sep 09 02:50:07.110675 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:07.110524 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-81f1254e-5b7fd6b9df-b9wcr" Sep 09 02:50:07.110675 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:07.110548 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-81f1254e-5b7fd6b9df-b9wcr" event={"ID":"42bb72c6-06bc-479b-9277-91e6429806be","Type":"ContainerDied","Data":"89d9c356e78d76eb7a64acc4a8c0fc3e9b5ea39503a02c54b526c63a4cb73b2e"} Sep 09 02:50:07.110675 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:07.110621 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-81f1254e-5b7fd6b9df-b9wcr" event={"ID":"42bb72c6-06bc-479b-9277-91e6429806be","Type":"ContainerDied","Data":"f12568ab5a0ffe27ee2614de7a3017c2422047bb2b66c8d3aea2c53546a2e6b4"} Sep 09 02:50:07.110675 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:07.110647 2577 scope.go:117] "RemoveContainer" containerID="89d9c356e78d76eb7a64acc4a8c0fc3e9b5ea39503a02c54b526c63a4cb73b2e" Sep 09 02:50:07.111924 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:07.111903 2577 generic.go:356] "Generic (PLEG): container finished" podID="fbb1c560-f8b3-4e5e-b4df-b88d665f36a7" containerID="a151412594853044553fc2a173227e388552e52eed3691292eae4195e6de1752" exitCode=0 Sep 09 02:50:07.112049 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:07.111986 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-81f1254e-796f9bb66b-9bb74" Sep 09 02:50:07.112049 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:07.111990 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-81f1254e-796f9bb66b-9bb74" event={"ID":"fbb1c560-f8b3-4e5e-b4df-b88d665f36a7","Type":"ContainerDied","Data":"a151412594853044553fc2a173227e388552e52eed3691292eae4195e6de1752"} Sep 09 02:50:07.112049 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:07.112034 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-81f1254e-796f9bb66b-9bb74" event={"ID":"fbb1c560-f8b3-4e5e-b4df-b88d665f36a7","Type":"ContainerDied","Data":"076fe21cf2fd6161cf3985027e7a0df37656b6afb5cb8867bba3f1e5690c60dc"} Sep 09 02:50:07.113128 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:07.113108 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndjv7\" (UniqueName: \"kubernetes.io/projected/fbb1c560-f8b3-4e5e-b4df-b88d665f36a7-kube-api-access-ndjv7\") pod \"fbb1c560-f8b3-4e5e-b4df-b88d665f36a7\" (UID: \"fbb1c560-f8b3-4e5e-b4df-b88d665f36a7\") " Sep 09 02:50:07.113211 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:07.113146 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42bb72c6-06bc-479b-9277-91e6429806be-plugins-config-volume\") pod \"42bb72c6-06bc-479b-9277-91e6429806be\" (UID: \"42bb72c6-06bc-479b-9277-91e6429806be\") " Sep 09 02:50:07.113254 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:07.113215 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbb1c560-f8b3-4e5e-b4df-b88d665f36a7-plugins-config-volume\") pod \"fbb1c560-f8b3-4e5e-b4df-b88d665f36a7\" (UID: \"fbb1c560-f8b3-4e5e-b4df-b88d665f36a7\") " Sep 09 02:50:07.113355 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:07.113297 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbvr2\" (UniqueName: \"kubernetes.io/projected/42bb72c6-06bc-479b-9277-91e6429806be-kube-api-access-lbvr2\") pod \"42bb72c6-06bc-479b-9277-91e6429806be\" (UID: \"42bb72c6-06bc-479b-9277-91e6429806be\") " Sep 09 02:50:07.113515 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:07.113490 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42bb72c6-06bc-479b-9277-91e6429806be-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "42bb72c6-06bc-479b-9277-91e6429806be" (UID: "42bb72c6-06bc-479b-9277-91e6429806be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:50:07.113601 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:07.113564 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbb1c560-f8b3-4e5e-b4df-b88d665f36a7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fbb1c560-f8b3-4e5e-b4df-b88d665f36a7" (UID: "fbb1c560-f8b3-4e5e-b4df-b88d665f36a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:50:07.113601 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:07.113584 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42bb72c6-06bc-479b-9277-91e6429806be-plugins-config-volume\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:50:07.115309 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:07.115285 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb1c560-f8b3-4e5e-b4df-b88d665f36a7-kube-api-access-ndjv7" (OuterVolumeSpecName: "kube-api-access-ndjv7") pod "fbb1c560-f8b3-4e5e-b4df-b88d665f36a7" (UID: "fbb1c560-f8b3-4e5e-b4df-b88d665f36a7"). InnerVolumeSpecName "kube-api-access-ndjv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:50:07.115550 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:07.115525 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42bb72c6-06bc-479b-9277-91e6429806be-kube-api-access-lbvr2" (OuterVolumeSpecName: "kube-api-access-lbvr2") pod "42bb72c6-06bc-479b-9277-91e6429806be" (UID: "42bb72c6-06bc-479b-9277-91e6429806be"). InnerVolumeSpecName "kube-api-access-lbvr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:50:07.119793 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:07.119776 2577 scope.go:117] "RemoveContainer" containerID="89d9c356e78d76eb7a64acc4a8c0fc3e9b5ea39503a02c54b526c63a4cb73b2e" Sep 09 02:50:07.120097 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:50:07.120074 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89d9c356e78d76eb7a64acc4a8c0fc3e9b5ea39503a02c54b526c63a4cb73b2e\": container with ID starting with 89d9c356e78d76eb7a64acc4a8c0fc3e9b5ea39503a02c54b526c63a4cb73b2e not found: ID does not exist" containerID="89d9c356e78d76eb7a64acc4a8c0fc3e9b5ea39503a02c54b526c63a4cb73b2e" Sep 09 02:50:07.120156 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:07.120104 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89d9c356e78d76eb7a64acc4a8c0fc3e9b5ea39503a02c54b526c63a4cb73b2e"} err="failed to get container status \"89d9c356e78d76eb7a64acc4a8c0fc3e9b5ea39503a02c54b526c63a4cb73b2e\": rpc error: code = NotFound desc = could not find container \"89d9c356e78d76eb7a64acc4a8c0fc3e9b5ea39503a02c54b526c63a4cb73b2e\": container with ID starting with 89d9c356e78d76eb7a64acc4a8c0fc3e9b5ea39503a02c54b526c63a4cb73b2e not found: ID does not exist" Sep 09 02:50:07.120156 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:07.120119 2577 scope.go:117] "RemoveContainer" containerID="a151412594853044553fc2a173227e388552e52eed3691292eae4195e6de1752" Sep 09 02:50:07.127904 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:07.127887 2577 scope.go:117] "RemoveContainer" containerID="a151412594853044553fc2a173227e388552e52eed3691292eae4195e6de1752" Sep 09 02:50:07.128193 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:50:07.128172 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a151412594853044553fc2a173227e388552e52eed3691292eae4195e6de1752\": container with ID starting with a151412594853044553fc2a173227e388552e52eed3691292eae4195e6de1752 not found: ID does not exist" containerID="a151412594853044553fc2a173227e388552e52eed3691292eae4195e6de1752" Sep 09 02:50:07.128246 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:07.128200 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a151412594853044553fc2a173227e388552e52eed3691292eae4195e6de1752"} err="failed to get container status \"a151412594853044553fc2a173227e388552e52eed3691292eae4195e6de1752\": rpc error: code = NotFound desc = could not find container \"a151412594853044553fc2a173227e388552e52eed3691292eae4195e6de1752\": container with ID starting with a151412594853044553fc2a173227e388552e52eed3691292eae4195e6de1752 not found: ID does not exist" Sep 09 02:50:07.165650 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:07.165624 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9eec439b-6ef3-47d9-b21f-b2167d3bcc2a" path="/var/lib/kubelet/pods/9eec439b-6ef3-47d9-b21f-b2167d3bcc2a/volumes" Sep 09 02:50:07.166120 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:07.166094 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f45fbd28-69e2-4cc5-be16-a30e218fabaf" path="/var/lib/kubelet/pods/f45fbd28-69e2-4cc5-be16-a30e218fabaf/volumes" Sep 09 02:50:07.214201 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:07.214181 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbvr2\" (UniqueName: \"kubernetes.io/projected/42bb72c6-06bc-479b-9277-91e6429806be-kube-api-access-lbvr2\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:50:07.214201 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:07.214202 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndjv7\" (UniqueName: \"kubernetes.io/projected/fbb1c560-f8b3-4e5e-b4df-b88d665f36a7-kube-api-access-ndjv7\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:50:07.214351 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:07.214212 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbb1c560-f8b3-4e5e-b4df-b88d665f36a7-plugins-config-volume\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:50:07.428461 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:07.428427 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-81f1254e-5b7fd6b9df-b9wcr"] Sep 09 02:50:07.434678 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:07.434648 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-81f1254e-5b7fd6b9df-b9wcr"] Sep 09 02:50:07.444712 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:07.444686 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-81f1254e-796f9bb66b-9bb74"] Sep 09 02:50:07.448274 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:07.448255 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-81f1254e-796f9bb66b-9bb74"] Sep 09 02:50:09.164859 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:09.164830 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42bb72c6-06bc-479b-9277-91e6429806be" path="/var/lib/kubelet/pods/42bb72c6-06bc-479b-9277-91e6429806be/volumes" Sep 09 02:50:09.165333 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:09.165192 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbb1c560-f8b3-4e5e-b4df-b88d665f36a7" path="/var/lib/kubelet/pods/fbb1c560-f8b3-4e5e-b4df-b88d665f36a7/volumes" Sep 09 02:50:18.356337 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:18.356304 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-56dbc0-59594db65d-9smp5"] Sep 09 02:50:18.356752 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:18.356657 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9eec439b-6ef3-47d9-b21f-b2167d3bcc2a" containerName="revoke-keys" Sep 09 02:50:18.356752 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:18.356668 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9eec439b-6ef3-47d9-b21f-b2167d3bcc2a" containerName="revoke-keys" Sep 09 02:50:18.356752 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:18.356680 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbb1c560-f8b3-4e5e-b4df-b88d665f36a7" containerName="payload-pre-processing" Sep 09 02:50:18.356752 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:18.356686 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbb1c560-f8b3-4e5e-b4df-b88d665f36a7" containerName="payload-pre-processing" Sep 09 02:50:18.356752 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:18.356701 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42bb72c6-06bc-479b-9277-91e6429806be" containerName="payload-processing" Sep 09 02:50:18.356752 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:18.356707 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="42bb72c6-06bc-479b-9277-91e6429806be" containerName="payload-processing" Sep 09 02:50:18.356752 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:18.356716 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f45fbd28-69e2-4cc5-be16-a30e218fabaf" containerName="payload-processing" Sep 09 02:50:18.356752 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:18.356721 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f45fbd28-69e2-4cc5-be16-a30e218fabaf" containerName="payload-processing" Sep 09 02:50:18.357081 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:18.356780 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="42bb72c6-06bc-479b-9277-91e6429806be" containerName="payload-processing" Sep 09 02:50:18.357081 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:18.356797 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9eec439b-6ef3-47d9-b21f-b2167d3bcc2a" containerName="revoke-keys" Sep 09 02:50:18.357081 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:18.356808 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f45fbd28-69e2-4cc5-be16-a30e218fabaf" containerName="payload-processing" Sep 09 02:50:18.357081 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:18.356817 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbb1c560-f8b3-4e5e-b4df-b88d665f36a7" containerName="payload-pre-processing" Sep 09 02:50:18.361430 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:18.361410 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-56dbc0-59594db65d-9smp5" Sep 09 02:50:18.366528 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:18.366509 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-56dbc0\"" Sep 09 02:50:18.366888 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:18.366868 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-56dbc0-dockercfg-vln5p\"" Sep 09 02:50:18.371773 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:18.371748 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-56dbc0-59594db65d-9smp5"] Sep 09 02:50:18.382484 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:18.382464 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-56dbc0-759764479f-6z97x"] Sep 09 02:50:18.386313 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:18.386292 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-56dbc0-759764479f-6z97x" Sep 09 02:50:18.399488 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:18.399466 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-56dbc0-759764479f-6z97x"] Sep 09 02:50:18.408712 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:18.408691 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdjvb\" (UniqueName: \"kubernetes.io/projected/3e293447-dee0-47d1-8868-99d0502d3dd8-kube-api-access-wdjvb\") pod \"payload-pre-processing-e2e-shared-a-w3-56dbc0-59594db65d-9smp5\" (UID: \"3e293447-dee0-47d1-8868-99d0502d3dd8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-56dbc0-59594db65d-9smp5" Sep 09 02:50:18.408818 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:18.408732 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e293447-dee0-47d1-8868-99d0502d3dd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-56dbc0-59594db65d-9smp5\" (UID: \"3e293447-dee0-47d1-8868-99d0502d3dd8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-56dbc0-59594db65d-9smp5" Sep 09 02:50:18.408818 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:18.408759 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d5fa8f3-486e-477f-b44f-f91fbb4729bd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-56dbc0-759764479f-6z97x\" (UID: \"5d5fa8f3-486e-477f-b44f-f91fbb4729bd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-56dbc0-759764479f-6z97x" Sep 09 02:50:18.408899 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:18.408876 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52tsj\" (UniqueName: \"kubernetes.io/projected/5d5fa8f3-486e-477f-b44f-f91fbb4729bd-kube-api-access-52tsj\") pod \"payload-processing-e2e-shared-a-w3-56dbc0-759764479f-6z97x\" (UID: \"5d5fa8f3-486e-477f-b44f-f91fbb4729bd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-56dbc0-759764479f-6z97x" Sep 09 02:50:18.510253 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:18.510213 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdjvb\" (UniqueName: \"kubernetes.io/projected/3e293447-dee0-47d1-8868-99d0502d3dd8-kube-api-access-wdjvb\") pod \"payload-pre-processing-e2e-shared-a-w3-56dbc0-59594db65d-9smp5\" (UID: \"3e293447-dee0-47d1-8868-99d0502d3dd8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-56dbc0-59594db65d-9smp5" Sep 09 02:50:18.510460 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:18.510266 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e293447-dee0-47d1-8868-99d0502d3dd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-56dbc0-59594db65d-9smp5\" (UID: \"3e293447-dee0-47d1-8868-99d0502d3dd8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-56dbc0-59594db65d-9smp5" Sep 09 02:50:18.510460 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:18.510295 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d5fa8f3-486e-477f-b44f-f91fbb4729bd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-56dbc0-759764479f-6z97x\" (UID: \"5d5fa8f3-486e-477f-b44f-f91fbb4729bd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-56dbc0-759764479f-6z97x" Sep 09 02:50:18.510460 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:18.510351 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52tsj\" (UniqueName: \"kubernetes.io/projected/5d5fa8f3-486e-477f-b44f-f91fbb4729bd-kube-api-access-52tsj\") pod \"payload-processing-e2e-shared-a-w3-56dbc0-759764479f-6z97x\" (UID: \"5d5fa8f3-486e-477f-b44f-f91fbb4729bd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-56dbc0-759764479f-6z97x" Sep 09 02:50:18.510952 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:18.510929 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e293447-dee0-47d1-8868-99d0502d3dd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-56dbc0-59594db65d-9smp5\" (UID: \"3e293447-dee0-47d1-8868-99d0502d3dd8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-56dbc0-59594db65d-9smp5" Sep 09 02:50:18.511037 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:18.511022 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d5fa8f3-486e-477f-b44f-f91fbb4729bd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-56dbc0-759764479f-6z97x\" (UID: \"5d5fa8f3-486e-477f-b44f-f91fbb4729bd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-56dbc0-759764479f-6z97x" Sep 09 02:50:18.518695 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:18.518672 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52tsj\" (UniqueName: \"kubernetes.io/projected/5d5fa8f3-486e-477f-b44f-f91fbb4729bd-kube-api-access-52tsj\") pod \"payload-processing-e2e-shared-a-w3-56dbc0-759764479f-6z97x\" (UID: \"5d5fa8f3-486e-477f-b44f-f91fbb4729bd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-56dbc0-759764479f-6z97x" Sep 09 02:50:18.518809 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:18.518751 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdjvb\" (UniqueName: \"kubernetes.io/projected/3e293447-dee0-47d1-8868-99d0502d3dd8-kube-api-access-wdjvb\") pod \"payload-pre-processing-e2e-shared-a-w3-56dbc0-59594db65d-9smp5\" (UID: \"3e293447-dee0-47d1-8868-99d0502d3dd8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-56dbc0-59594db65d-9smp5" Sep 09 02:50:18.673021 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:18.672908 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-56dbc0-59594db65d-9smp5" Sep 09 02:50:18.698264 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:18.698236 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-56dbc0-759764479f-6z97x" Sep 09 02:50:18.950651 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:18.950560 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-56dbc0-59594db65d-9smp5"] Sep 09 02:50:18.953518 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:50:18.953492 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e293447_dee0_47d1_8868_99d0502d3dd8.slice/crio-2e42d89ff43158993a21feac95179d95a18e5ab5d44d71d031f0b676d72ecc0a WatchSource:0}: Error finding container 2e42d89ff43158993a21feac95179d95a18e5ab5d44d71d031f0b676d72ecc0a: Status 404 returned error can't find the container with id 2e42d89ff43158993a21feac95179d95a18e5ab5d44d71d031f0b676d72ecc0a Sep 09 02:50:18.983471 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:18.983450 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-56dbc0-759764479f-6z97x"] Sep 09 02:50:18.990245 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:50:18.990209 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d5fa8f3_486e_477f_b44f_f91fbb4729bd.slice/crio-0aa93eb5f97112ff35c421d2a130bac43d33c2a795e9f27540106b29203a2973 WatchSource:0}: Error finding container 0aa93eb5f97112ff35c421d2a130bac43d33c2a795e9f27540106b29203a2973: Status 404 returned error can't find the container with id 0aa93eb5f97112ff35c421d2a130bac43d33c2a795e9f27540106b29203a2973 Sep 09 02:50:19.179212 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:19.179175 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-56dbc0-59594db65d-9smp5" event={"ID":"3e293447-dee0-47d1-8868-99d0502d3dd8","Type":"ContainerStarted","Data":"45553ecbc1963bf970329b5acca05e240e2ac8f4496d7d7a60ba002ae4ae4f4b"} Sep 09 02:50:19.179401 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:19.179232 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-56dbc0-59594db65d-9smp5" event={"ID":"3e293447-dee0-47d1-8868-99d0502d3dd8","Type":"ContainerStarted","Data":"2e42d89ff43158993a21feac95179d95a18e5ab5d44d71d031f0b676d72ecc0a"} Sep 09 02:50:19.179401 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:19.179276 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-56dbc0-59594db65d-9smp5" Sep 09 02:50:19.180945 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:19.180918 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-56dbc0-759764479f-6z97x" event={"ID":"5d5fa8f3-486e-477f-b44f-f91fbb4729bd","Type":"ContainerStarted","Data":"e5a9bb08d4b5395908a1a6153b7420141b8af4d588b2e231443ecfb883762eae"} Sep 09 02:50:19.181089 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:19.181003 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-56dbc0-759764479f-6z97x" event={"ID":"5d5fa8f3-486e-477f-b44f-f91fbb4729bd","Type":"ContainerStarted","Data":"0aa93eb5f97112ff35c421d2a130bac43d33c2a795e9f27540106b29203a2973"} Sep 09 02:50:19.181089 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:19.181039 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-56dbc0-759764479f-6z97x" Sep 09 02:50:19.198328 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:19.198271 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-56dbc0-59594db65d-9smp5" podStartSLOduration=1.197954029 podStartE2EDuration="1.197954029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:50:19.196944367 +0000 UTC m=+1212.561278832" watchObservedRunningTime="2026-09-09 02:50:19.197954029 +0000 UTC m=+1212.562288683" Sep 09 02:50:19.220849 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:19.220752 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-56dbc0-759764479f-6z97x" podStartSLOduration=1.220736972 podStartE2EDuration="1.220736972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:50:19.220244638 +0000 UTC m=+1212.584579102" watchObservedRunningTime="2026-09-09 02:50:19.220736972 +0000 UTC m=+1212.585071435" Sep 09 02:50:30.188814 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:30.188779 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-56dbc0-759764479f-6z97x" Sep 09 02:50:30.189328 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:30.188875 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-56dbc0-59594db65d-9smp5" Sep 09 02:50:33.611172 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:33.611136 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d838aa-6898647b87-dwfj4"] Sep 09 02:50:33.614621 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:33.614604 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d838aa-6898647b87-dwfj4" Sep 09 02:50:33.618009 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:33.617987 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-d838aa\"" Sep 09 02:50:33.618094 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:33.617991 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-d838aa-dockercfg-s9d85\"" Sep 09 02:50:33.628598 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:33.628576 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d838aa-6898647b87-dwfj4"] Sep 09 02:50:33.647935 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:33.647913 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d838aa-8698746cd6-wkqmc"] Sep 09 02:50:33.651146 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:33.651131 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d838aa-8698746cd6-wkqmc" Sep 09 02:50:33.660153 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:33.660123 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d838aa-8698746cd6-wkqmc"] Sep 09 02:50:33.741777 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:33.741743 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a8826d2-acbe-4142-9e21-2079a3aa7ce9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-d838aa-6898647b87-dwfj4\" (UID: \"3a8826d2-acbe-4142-9e21-2079a3aa7ce9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d838aa-6898647b87-dwfj4" Sep 09 02:50:33.741989 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:33.741792 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3818b516-1d11-47ec-816d-8851bb3035da-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-d838aa-8698746cd6-wkqmc\" (UID: \"3818b516-1d11-47ec-816d-8851bb3035da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d838aa-8698746cd6-wkqmc" Sep 09 02:50:33.741989 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:33.741845 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ls54\" (UniqueName: \"kubernetes.io/projected/3a8826d2-acbe-4142-9e21-2079a3aa7ce9-kube-api-access-4ls54\") pod \"payload-processing-e2e-shared-a-w2-d838aa-6898647b87-dwfj4\" (UID: \"3a8826d2-acbe-4142-9e21-2079a3aa7ce9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d838aa-6898647b87-dwfj4" Sep 09 02:50:33.741989 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:33.741861 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59nh2\" (UniqueName: \"kubernetes.io/projected/3818b516-1d11-47ec-816d-8851bb3035da-kube-api-access-59nh2\") pod \"payload-pre-processing-e2e-shared-a-w2-d838aa-8698746cd6-wkqmc\" (UID: \"3818b516-1d11-47ec-816d-8851bb3035da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d838aa-8698746cd6-wkqmc" Sep 09 02:50:33.842824 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:33.842791 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3818b516-1d11-47ec-816d-8851bb3035da-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-d838aa-8698746cd6-wkqmc\" (UID: \"3818b516-1d11-47ec-816d-8851bb3035da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d838aa-8698746cd6-wkqmc" Sep 09 02:50:33.843037 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:33.842834 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ls54\" (UniqueName: \"kubernetes.io/projected/3a8826d2-acbe-4142-9e21-2079a3aa7ce9-kube-api-access-4ls54\") pod \"payload-processing-e2e-shared-a-w2-d838aa-6898647b87-dwfj4\" (UID: \"3a8826d2-acbe-4142-9e21-2079a3aa7ce9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d838aa-6898647b87-dwfj4" Sep 09 02:50:33.843037 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:33.842861 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59nh2\" (UniqueName: \"kubernetes.io/projected/3818b516-1d11-47ec-816d-8851bb3035da-kube-api-access-59nh2\") pod \"payload-pre-processing-e2e-shared-a-w2-d838aa-8698746cd6-wkqmc\" (UID: \"3818b516-1d11-47ec-816d-8851bb3035da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d838aa-8698746cd6-wkqmc" Sep 09 02:50:33.843037 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:33.842919 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a8826d2-acbe-4142-9e21-2079a3aa7ce9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-d838aa-6898647b87-dwfj4\" (UID: \"3a8826d2-acbe-4142-9e21-2079a3aa7ce9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d838aa-6898647b87-dwfj4" Sep 09 02:50:33.843720 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:33.843698 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3818b516-1d11-47ec-816d-8851bb3035da-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-d838aa-8698746cd6-wkqmc\" (UID: \"3818b516-1d11-47ec-816d-8851bb3035da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d838aa-8698746cd6-wkqmc" Sep 09 02:50:33.843763 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:33.843698 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a8826d2-acbe-4142-9e21-2079a3aa7ce9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-d838aa-6898647b87-dwfj4\" (UID: \"3a8826d2-acbe-4142-9e21-2079a3aa7ce9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d838aa-6898647b87-dwfj4" Sep 09 02:50:33.851174 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:33.851146 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59nh2\" (UniqueName: \"kubernetes.io/projected/3818b516-1d11-47ec-816d-8851bb3035da-kube-api-access-59nh2\") pod \"payload-pre-processing-e2e-shared-a-w2-d838aa-8698746cd6-wkqmc\" (UID: \"3818b516-1d11-47ec-816d-8851bb3035da\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d838aa-8698746cd6-wkqmc" Sep 09 02:50:33.851303 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:33.851257 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ls54\" (UniqueName: \"kubernetes.io/projected/3a8826d2-acbe-4142-9e21-2079a3aa7ce9-kube-api-access-4ls54\") pod \"payload-processing-e2e-shared-a-w2-d838aa-6898647b87-dwfj4\" (UID: \"3a8826d2-acbe-4142-9e21-2079a3aa7ce9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d838aa-6898647b87-dwfj4" Sep 09 02:50:33.925869 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:33.925794 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d838aa-6898647b87-dwfj4" Sep 09 02:50:33.962365 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:33.962329 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d838aa-8698746cd6-wkqmc" Sep 09 02:50:34.202852 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:34.202828 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d838aa-6898647b87-dwfj4"] Sep 09 02:50:34.204577 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:50:34.204551 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a8826d2_acbe_4142_9e21_2079a3aa7ce9.slice/crio-05d7819eb23b4688933804088f3b72249431ceecf5df39bee9d5ece4e684c907 WatchSource:0}: Error finding container 05d7819eb23b4688933804088f3b72249431ceecf5df39bee9d5ece4e684c907: Status 404 returned error can't find the container with id 05d7819eb23b4688933804088f3b72249431ceecf5df39bee9d5ece4e684c907 Sep 09 02:50:34.251259 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:34.251238 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d838aa-8698746cd6-wkqmc"] Sep 09 02:50:34.252946 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:50:34.252925 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3818b516_1d11_47ec_816d_8851bb3035da.slice/crio-aa166e803b9429a50efb434a17ad148aadc0281fd2b59175ae5c89d8f94c7381 WatchSource:0}: Error finding container aa166e803b9429a50efb434a17ad148aadc0281fd2b59175ae5c89d8f94c7381: Status 404 returned error can't find the container with id aa166e803b9429a50efb434a17ad148aadc0281fd2b59175ae5c89d8f94c7381 Sep 09 02:50:34.274834 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:34.274810 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d838aa-6898647b87-dwfj4" event={"ID":"3a8826d2-acbe-4142-9e21-2079a3aa7ce9","Type":"ContainerStarted","Data":"adce36075512f5a5a96306707bb5d3010da07a3fc7372711f5987342f59bbd12"} Sep 09 02:50:34.274931 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:34.274860 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d838aa-6898647b87-dwfj4" event={"ID":"3a8826d2-acbe-4142-9e21-2079a3aa7ce9","Type":"ContainerStarted","Data":"05d7819eb23b4688933804088f3b72249431ceecf5df39bee9d5ece4e684c907"} Sep 09 02:50:34.276069 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:34.276048 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d838aa-8698746cd6-wkqmc" event={"ID":"3818b516-1d11-47ec-816d-8851bb3035da","Type":"ContainerStarted","Data":"aa166e803b9429a50efb434a17ad148aadc0281fd2b59175ae5c89d8f94c7381"} Sep 09 02:50:35.283343 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:35.283294 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d838aa-8698746cd6-wkqmc" event={"ID":"3818b516-1d11-47ec-816d-8851bb3035da","Type":"ContainerStarted","Data":"3f9575f15c07f100a5eab6966a3374f7213e845ab79adaffefb3f7bb77b58dd5"} Sep 09 02:50:35.283761 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:35.283382 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d838aa-8698746cd6-wkqmc" Sep 09 02:50:35.283761 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:35.283594 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d838aa-6898647b87-dwfj4" Sep 09 02:50:35.304076 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:35.304025 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d838aa-8698746cd6-wkqmc" podStartSLOduration=2.3040067300000002 podStartE2EDuration="2.30400673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:50:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:50:35.300106082 +0000 UTC m=+1228.664440546" watchObservedRunningTime="2026-09-09 02:50:35.30400673 +0000 UTC m=+1228.668341195" Sep 09 02:50:35.319785 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:35.319728 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d838aa-6898647b87-dwfj4" podStartSLOduration=2.319708345 podStartE2EDuration="2.319708345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:50:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:50:35.318511496 +0000 UTC m=+1228.682845960" watchObservedRunningTime="2026-09-09 02:50:35.319708345 +0000 UTC m=+1228.684042810" Sep 09 02:50:46.292902 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:46.292871 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d838aa-8698746cd6-wkqmc" Sep 09 02:50:46.293377 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:46.292972 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d838aa-6898647b87-dwfj4" Sep 09 02:50:46.609699 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:46.609621 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c173ad-677d666774-kcl26"] Sep 09 02:50:46.613170 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:46.613148 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c173ad-677d666774-kcl26" Sep 09 02:50:46.615685 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:46.615667 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-c173ad-dockercfg-ngtx5\"" Sep 09 02:50:46.615685 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:46.615685 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-c173ad\"" Sep 09 02:50:46.622612 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:46.622584 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c173ad-677d666774-kcl26"] Sep 09 02:50:46.756862 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:46.756823 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggmcb\" (UniqueName: \"kubernetes.io/projected/651f46c0-a79e-4a89-b24a-42286db15ad1-kube-api-access-ggmcb\") pod \"payload-pre-processing-e2e-shared-b-w3-c173ad-677d666774-kcl26\" (UID: \"651f46c0-a79e-4a89-b24a-42286db15ad1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c173ad-677d666774-kcl26" Sep 09 02:50:46.756862 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:46.756859 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/651f46c0-a79e-4a89-b24a-42286db15ad1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-c173ad-677d666774-kcl26\" (UID: \"651f46c0-a79e-4a89-b24a-42286db15ad1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c173ad-677d666774-kcl26" Sep 09 02:50:46.858146 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:46.858110 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggmcb\" (UniqueName: \"kubernetes.io/projected/651f46c0-a79e-4a89-b24a-42286db15ad1-kube-api-access-ggmcb\") pod \"payload-pre-processing-e2e-shared-b-w3-c173ad-677d666774-kcl26\" (UID: \"651f46c0-a79e-4a89-b24a-42286db15ad1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c173ad-677d666774-kcl26" Sep 09 02:50:46.858146 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:46.858152 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/651f46c0-a79e-4a89-b24a-42286db15ad1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-c173ad-677d666774-kcl26\" (UID: \"651f46c0-a79e-4a89-b24a-42286db15ad1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c173ad-677d666774-kcl26" Sep 09 02:50:46.859013 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:46.858994 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/651f46c0-a79e-4a89-b24a-42286db15ad1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-c173ad-677d666774-kcl26\" (UID: \"651f46c0-a79e-4a89-b24a-42286db15ad1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c173ad-677d666774-kcl26" Sep 09 02:50:46.871050 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:46.870991 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggmcb\" (UniqueName: \"kubernetes.io/projected/651f46c0-a79e-4a89-b24a-42286db15ad1-kube-api-access-ggmcb\") pod \"payload-pre-processing-e2e-shared-b-w3-c173ad-677d666774-kcl26\" (UID: \"651f46c0-a79e-4a89-b24a-42286db15ad1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c173ad-677d666774-kcl26" Sep 09 02:50:46.925437 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:46.925396 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c173ad-677d666774-kcl26" Sep 09 02:50:47.201153 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:47.201124 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c173ad-677d666774-kcl26"] Sep 09 02:50:47.202228 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:50:47.202202 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod651f46c0_a79e_4a89_b24a_42286db15ad1.slice/crio-a25b3f9c082f5536ec6357f8fca44be48a919c3102b05c8cb5c0fff5d0023361 WatchSource:0}: Error finding container a25b3f9c082f5536ec6357f8fca44be48a919c3102b05c8cb5c0fff5d0023361: Status 404 returned error can't find the container with id a25b3f9c082f5536ec6357f8fca44be48a919c3102b05c8cb5c0fff5d0023361 Sep 09 02:50:47.359582 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:47.359534 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c173ad-677d666774-kcl26" event={"ID":"651f46c0-a79e-4a89-b24a-42286db15ad1","Type":"ContainerStarted","Data":"b5f2a526a2a8f2fa4ea8227666e01c07a249fbd4d9bcade4c1fd5a2983710704"} Sep 09 02:50:47.360012 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:47.359607 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c173ad-677d666774-kcl26" event={"ID":"651f46c0-a79e-4a89-b24a-42286db15ad1","Type":"ContainerStarted","Data":"a25b3f9c082f5536ec6357f8fca44be48a919c3102b05c8cb5c0fff5d0023361"} Sep 09 02:50:47.360012 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:47.359631 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c173ad-677d666774-kcl26" Sep 09 02:50:47.376634 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:47.376584 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c173ad-677d666774-kcl26" podStartSLOduration=1.376565827 podStartE2EDuration="1.376565827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:50:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:50:47.374294574 +0000 UTC m=+1240.738629038" watchObservedRunningTime="2026-09-09 02:50:47.376565827 +0000 UTC m=+1240.740900290" Sep 09 02:50:58.366621 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:50:58.366590 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c173ad-677d666774-kcl26" Sep 09 02:51:02.681119 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:02.681084 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f5aebb-589bb94786-2lghl"] Sep 09 02:51:02.685771 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:02.685745 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f5aebb-589bb94786-2lghl" Sep 09 02:51:02.688453 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:02.688413 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-f5aebb\"" Sep 09 02:51:02.688604 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:02.688465 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-f5aebb-dockercfg-k84zl\"" Sep 09 02:51:02.696486 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:02.696457 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f5aebb-589bb94786-2lghl"] Sep 09 02:51:02.707091 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:02.707057 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f5aebb-85db48d8f-679bs"] Sep 09 02:51:02.710531 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:02.710496 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f5aebb-85db48d8f-679bs" Sep 09 02:51:02.721454 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:02.721421 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f5aebb-85db48d8f-679bs"] Sep 09 02:51:02.794172 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:02.794116 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m52sp\" (UniqueName: \"kubernetes.io/projected/1dd3e673-b63e-465d-889b-25fca919f4d2-kube-api-access-m52sp\") pod \"payload-processing-e2e-shared-b-w2-f5aebb-85db48d8f-679bs\" (UID: \"1dd3e673-b63e-465d-889b-25fca919f4d2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f5aebb-85db48d8f-679bs" Sep 09 02:51:02.794387 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:02.794194 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dd3e673-b63e-465d-889b-25fca919f4d2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-f5aebb-85db48d8f-679bs\" (UID: \"1dd3e673-b63e-465d-889b-25fca919f4d2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f5aebb-85db48d8f-679bs" Sep 09 02:51:02.794387 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:02.794226 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9784127c-8779-4c1a-aae0-b811d4e76751-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-f5aebb-589bb94786-2lghl\" (UID: \"9784127c-8779-4c1a-aae0-b811d4e76751\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f5aebb-589bb94786-2lghl" Sep 09 02:51:02.794387 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:02.794339 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28nrf\" (UniqueName: \"kubernetes.io/projected/9784127c-8779-4c1a-aae0-b811d4e76751-kube-api-access-28nrf\") pod \"payload-pre-processing-e2e-shared-b-w2-f5aebb-589bb94786-2lghl\" (UID: \"9784127c-8779-4c1a-aae0-b811d4e76751\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f5aebb-589bb94786-2lghl" Sep 09 02:51:02.895606 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:02.895559 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m52sp\" (UniqueName: \"kubernetes.io/projected/1dd3e673-b63e-465d-889b-25fca919f4d2-kube-api-access-m52sp\") pod \"payload-processing-e2e-shared-b-w2-f5aebb-85db48d8f-679bs\" (UID: \"1dd3e673-b63e-465d-889b-25fca919f4d2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f5aebb-85db48d8f-679bs" Sep 09 02:51:02.895793 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:02.895632 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dd3e673-b63e-465d-889b-25fca919f4d2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-f5aebb-85db48d8f-679bs\" (UID: \"1dd3e673-b63e-465d-889b-25fca919f4d2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f5aebb-85db48d8f-679bs" Sep 09 02:51:02.895793 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:02.895660 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9784127c-8779-4c1a-aae0-b811d4e76751-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-f5aebb-589bb94786-2lghl\" (UID: \"9784127c-8779-4c1a-aae0-b811d4e76751\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f5aebb-589bb94786-2lghl" Sep 09 02:51:02.895793 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:02.895722 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28nrf\" (UniqueName: \"kubernetes.io/projected/9784127c-8779-4c1a-aae0-b811d4e76751-kube-api-access-28nrf\") pod \"payload-pre-processing-e2e-shared-b-w2-f5aebb-589bb94786-2lghl\" (UID: \"9784127c-8779-4c1a-aae0-b811d4e76751\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f5aebb-589bb94786-2lghl" Sep 09 02:51:02.896475 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:02.896454 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dd3e673-b63e-465d-889b-25fca919f4d2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-f5aebb-85db48d8f-679bs\" (UID: \"1dd3e673-b63e-465d-889b-25fca919f4d2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f5aebb-85db48d8f-679bs" Sep 09 02:51:02.896545 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:02.896460 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9784127c-8779-4c1a-aae0-b811d4e76751-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-f5aebb-589bb94786-2lghl\" (UID: \"9784127c-8779-4c1a-aae0-b811d4e76751\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f5aebb-589bb94786-2lghl" Sep 09 02:51:02.905228 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:02.905201 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28nrf\" (UniqueName: \"kubernetes.io/projected/9784127c-8779-4c1a-aae0-b811d4e76751-kube-api-access-28nrf\") pod \"payload-pre-processing-e2e-shared-b-w2-f5aebb-589bb94786-2lghl\" (UID: \"9784127c-8779-4c1a-aae0-b811d4e76751\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f5aebb-589bb94786-2lghl" Sep 09 02:51:02.905376 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:02.905358 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m52sp\" (UniqueName: \"kubernetes.io/projected/1dd3e673-b63e-465d-889b-25fca919f4d2-kube-api-access-m52sp\") pod \"payload-processing-e2e-shared-b-w2-f5aebb-85db48d8f-679bs\" (UID: \"1dd3e673-b63e-465d-889b-25fca919f4d2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f5aebb-85db48d8f-679bs" Sep 09 02:51:02.998158 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:02.998076 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f5aebb-589bb94786-2lghl" Sep 09 02:51:03.022522 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:03.022492 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f5aebb-85db48d8f-679bs" Sep 09 02:51:03.279703 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:03.279673 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f5aebb-589bb94786-2lghl"] Sep 09 02:51:03.280988 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:51:03.280943 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9784127c_8779_4c1a_aae0_b811d4e76751.slice/crio-a0df75e6df9c07dee9e70e332575e5c24955d044ea3ae6dae3c3c6446b2c0781 WatchSource:0}: Error finding container a0df75e6df9c07dee9e70e332575e5c24955d044ea3ae6dae3c3c6446b2c0781: Status 404 returned error can't find the container with id a0df75e6df9c07dee9e70e332575e5c24955d044ea3ae6dae3c3c6446b2c0781 Sep 09 02:51:03.309937 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:03.309915 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f5aebb-85db48d8f-679bs"] Sep 09 02:51:03.316090 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:51:03.316062 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dd3e673_b63e_465d_889b_25fca919f4d2.slice/crio-8b113a52042d234c86588f5fa8eacc7ab2cf52ff64dcc952263b3985f14287a7 WatchSource:0}: Error finding container 8b113a52042d234c86588f5fa8eacc7ab2cf52ff64dcc952263b3985f14287a7: Status 404 returned error can't find the container with id 8b113a52042d234c86588f5fa8eacc7ab2cf52ff64dcc952263b3985f14287a7 Sep 09 02:51:03.450723 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:03.450688 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f5aebb-85db48d8f-679bs" event={"ID":"1dd3e673-b63e-465d-889b-25fca919f4d2","Type":"ContainerStarted","Data":"903553863a47e69dc1b1d4215604e24b1970f9da516273971b4f6d23f75f30c6"} Sep 09 02:51:03.450891 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:03.450749 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f5aebb-85db48d8f-679bs" event={"ID":"1dd3e673-b63e-465d-889b-25fca919f4d2","Type":"ContainerStarted","Data":"8b113a52042d234c86588f5fa8eacc7ab2cf52ff64dcc952263b3985f14287a7"} Sep 09 02:51:03.450891 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:03.450876 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f5aebb-85db48d8f-679bs" Sep 09 02:51:03.452864 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:03.452840 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f5aebb-589bb94786-2lghl" event={"ID":"9784127c-8779-4c1a-aae0-b811d4e76751","Type":"ContainerStarted","Data":"e0dbfa41fb647df3c609f59f05598a4cc525153f9fe43be9611998bdd798bc0c"} Sep 09 02:51:03.453036 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:03.452882 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f5aebb-589bb94786-2lghl" event={"ID":"9784127c-8779-4c1a-aae0-b811d4e76751","Type":"ContainerStarted","Data":"a0df75e6df9c07dee9e70e332575e5c24955d044ea3ae6dae3c3c6446b2c0781"} Sep 09 02:51:03.453036 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:03.452902 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f5aebb-589bb94786-2lghl" Sep 09 02:51:03.471212 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:03.471166 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f5aebb-85db48d8f-679bs" podStartSLOduration=1.471150629 podStartE2EDuration="1.471150629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:51:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:51:03.469062807 +0000 UTC m=+1256.833397269" watchObservedRunningTime="2026-09-09 02:51:03.471150629 +0000 UTC m=+1256.835485092" Sep 09 02:51:03.484050 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:03.484004 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f5aebb-589bb94786-2lghl" podStartSLOduration=1.483989194 podStartE2EDuration="1.483989194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:51:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:51:03.483528322 +0000 UTC m=+1256.847862782" watchObservedRunningTime="2026-09-09 02:51:03.483989194 +0000 UTC m=+1256.848323667" Sep 09 02:51:14.461493 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:14.461448 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f5aebb-85db48d8f-679bs" Sep 09 02:51:14.463929 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:14.461643 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f5aebb-589bb94786-2lghl" Sep 09 02:51:43.933105 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:43.933069 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ad1d02af-5db6b58bcd-nlzqt"] Sep 09 02:51:43.936576 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:43.936553 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ad1d02af-5db6b58bcd-nlzqt" Sep 09 02:51:43.939225 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:43.939208 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ad1d02af-dockercfg-lb8p5\"" Sep 09 02:51:43.939307 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:43.939227 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ad1d02af\"" Sep 09 02:51:43.946937 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:43.946915 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ad1d02af-5db6b58bcd-nlzqt"] Sep 09 02:51:43.961521 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:43.961501 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ad1d02af-5c4d9df57-lblgs"] Sep 09 02:51:43.965201 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:43.965158 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ad1d02af-5c4d9df57-lblgs" Sep 09 02:51:43.974933 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:43.974910 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ad1d02af-5c4d9df57-lblgs"] Sep 09 02:51:44.059173 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:44.059142 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcmp6\" (UniqueName: \"kubernetes.io/projected/976dec6b-3271-4032-a27c-d1ad6a9c767a-kube-api-access-pcmp6\") pod \"payload-pre-processing-e2e-mt-ad1d02af-5db6b58bcd-nlzqt\" (UID: \"976dec6b-3271-4032-a27c-d1ad6a9c767a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ad1d02af-5db6b58bcd-nlzqt" Sep 09 02:51:44.059327 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:44.059183 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/976dec6b-3271-4032-a27c-d1ad6a9c767a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ad1d02af-5db6b58bcd-nlzqt\" (UID: \"976dec6b-3271-4032-a27c-d1ad6a9c767a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ad1d02af-5db6b58bcd-nlzqt" Sep 09 02:51:44.059327 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:44.059203 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmqc9\" (UniqueName: \"kubernetes.io/projected/98430462-f8cf-43ba-b60f-7affc4bee624-kube-api-access-tmqc9\") pod \"payload-processing-e2e-mt-ad1d02af-5c4d9df57-lblgs\" (UID: \"98430462-f8cf-43ba-b60f-7affc4bee624\") " pod="openshift-ingress/payload-processing-e2e-mt-ad1d02af-5c4d9df57-lblgs" Sep 09 02:51:44.059327 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:44.059316 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98430462-f8cf-43ba-b60f-7affc4bee624-plugins-config-volume\") pod \"payload-processing-e2e-mt-ad1d02af-5c4d9df57-lblgs\" (UID: \"98430462-f8cf-43ba-b60f-7affc4bee624\") " pod="openshift-ingress/payload-processing-e2e-mt-ad1d02af-5c4d9df57-lblgs" Sep 09 02:51:44.160492 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:44.160452 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcmp6\" (UniqueName: \"kubernetes.io/projected/976dec6b-3271-4032-a27c-d1ad6a9c767a-kube-api-access-pcmp6\") pod \"payload-pre-processing-e2e-mt-ad1d02af-5db6b58bcd-nlzqt\" (UID: \"976dec6b-3271-4032-a27c-d1ad6a9c767a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ad1d02af-5db6b58bcd-nlzqt" Sep 09 02:51:44.160700 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:44.160534 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/976dec6b-3271-4032-a27c-d1ad6a9c767a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ad1d02af-5db6b58bcd-nlzqt\" (UID: \"976dec6b-3271-4032-a27c-d1ad6a9c767a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ad1d02af-5db6b58bcd-nlzqt" Sep 09 02:51:44.160700 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:44.160561 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmqc9\" (UniqueName: \"kubernetes.io/projected/98430462-f8cf-43ba-b60f-7affc4bee624-kube-api-access-tmqc9\") pod \"payload-processing-e2e-mt-ad1d02af-5c4d9df57-lblgs\" (UID: \"98430462-f8cf-43ba-b60f-7affc4bee624\") " pod="openshift-ingress/payload-processing-e2e-mt-ad1d02af-5c4d9df57-lblgs" Sep 09 02:51:44.160700 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:44.160642 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98430462-f8cf-43ba-b60f-7affc4bee624-plugins-config-volume\") pod \"payload-processing-e2e-mt-ad1d02af-5c4d9df57-lblgs\" (UID: \"98430462-f8cf-43ba-b60f-7affc4bee624\") " pod="openshift-ingress/payload-processing-e2e-mt-ad1d02af-5c4d9df57-lblgs" Sep 09 02:51:44.161496 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:44.161469 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/976dec6b-3271-4032-a27c-d1ad6a9c767a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ad1d02af-5db6b58bcd-nlzqt\" (UID: \"976dec6b-3271-4032-a27c-d1ad6a9c767a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ad1d02af-5db6b58bcd-nlzqt" Sep 09 02:51:44.161587 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:44.161473 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98430462-f8cf-43ba-b60f-7affc4bee624-plugins-config-volume\") pod \"payload-processing-e2e-mt-ad1d02af-5c4d9df57-lblgs\" (UID: \"98430462-f8cf-43ba-b60f-7affc4bee624\") " pod="openshift-ingress/payload-processing-e2e-mt-ad1d02af-5c4d9df57-lblgs" Sep 09 02:51:44.168770 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:44.168747 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcmp6\" (UniqueName: \"kubernetes.io/projected/976dec6b-3271-4032-a27c-d1ad6a9c767a-kube-api-access-pcmp6\") pod \"payload-pre-processing-e2e-mt-ad1d02af-5db6b58bcd-nlzqt\" (UID: \"976dec6b-3271-4032-a27c-d1ad6a9c767a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ad1d02af-5db6b58bcd-nlzqt" Sep 09 02:51:44.168918 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:44.168900 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmqc9\" (UniqueName: \"kubernetes.io/projected/98430462-f8cf-43ba-b60f-7affc4bee624-kube-api-access-tmqc9\") pod \"payload-processing-e2e-mt-ad1d02af-5c4d9df57-lblgs\" (UID: \"98430462-f8cf-43ba-b60f-7affc4bee624\") " pod="openshift-ingress/payload-processing-e2e-mt-ad1d02af-5c4d9df57-lblgs" Sep 09 02:51:44.246953 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:44.246876 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ad1d02af-5db6b58bcd-nlzqt" Sep 09 02:51:44.276039 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:44.276012 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ad1d02af-5c4d9df57-lblgs" Sep 09 02:51:44.523926 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:44.523904 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ad1d02af-5db6b58bcd-nlzqt"] Sep 09 02:51:44.524762 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:51:44.524742 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod976dec6b_3271_4032_a27c_d1ad6a9c767a.slice/crio-6c8769d9a49cb20f38bf145c2b7393a710a3637b7311960e745f4f01aa7e5553 WatchSource:0}: Error finding container 6c8769d9a49cb20f38bf145c2b7393a710a3637b7311960e745f4f01aa7e5553: Status 404 returned error can't find the container with id 6c8769d9a49cb20f38bf145c2b7393a710a3637b7311960e745f4f01aa7e5553 Sep 09 02:51:44.562395 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:44.562371 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ad1d02af-5c4d9df57-lblgs"] Sep 09 02:51:44.563693 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:51:44.563669 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98430462_f8cf_43ba_b60f_7affc4bee624.slice/crio-347d74bcacb0718e90e99e6798079a76872d25bf8ab249bc46dac9cd327ff7db WatchSource:0}: Error finding container 347d74bcacb0718e90e99e6798079a76872d25bf8ab249bc46dac9cd327ff7db: Status 404 returned error can't find the container with id 347d74bcacb0718e90e99e6798079a76872d25bf8ab249bc46dac9cd327ff7db Sep 09 02:51:44.692888 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:44.692844 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ad1d02af-5c4d9df57-lblgs" event={"ID":"98430462-f8cf-43ba-b60f-7affc4bee624","Type":"ContainerStarted","Data":"24c3041a6688630200b6534f6c43cb8f29ed926a54ce49c289ca3874f773a810"} Sep 09 02:51:44.693174 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:44.692917 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ad1d02af-5c4d9df57-lblgs" event={"ID":"98430462-f8cf-43ba-b60f-7affc4bee624","Type":"ContainerStarted","Data":"347d74bcacb0718e90e99e6798079a76872d25bf8ab249bc46dac9cd327ff7db"} Sep 09 02:51:44.693174 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:44.692947 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-ad1d02af-5c4d9df57-lblgs" Sep 09 02:51:44.694981 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:44.694925 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ad1d02af-5db6b58bcd-nlzqt" event={"ID":"976dec6b-3271-4032-a27c-d1ad6a9c767a","Type":"ContainerStarted","Data":"fda4385386fb66d99c83f91c8c3a615e5f1f007181391c842378e0dedc476986"} Sep 09 02:51:44.695097 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:44.695008 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ad1d02af-5db6b58bcd-nlzqt" event={"ID":"976dec6b-3271-4032-a27c-d1ad6a9c767a","Type":"ContainerStarted","Data":"6c8769d9a49cb20f38bf145c2b7393a710a3637b7311960e745f4f01aa7e5553"} Sep 09 02:51:44.695097 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:44.695040 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ad1d02af-5db6b58bcd-nlzqt" Sep 09 02:51:44.709347 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:44.709286 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-ad1d02af-5c4d9df57-lblgs" podStartSLOduration=1.709271398 podStartE2EDuration="1.709271398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:51:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:51:44.708387419 +0000 UTC m=+1298.072721882" watchObservedRunningTime="2026-09-09 02:51:44.709271398 +0000 UTC m=+1298.073605881" Sep 09 02:51:44.728050 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:44.727984 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-ad1d02af-5db6b58bcd-nlzqt" podStartSLOduration=1.7279515399999998 podStartE2EDuration="1.72795154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:51:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:51:44.724427294 +0000 UTC m=+1298.088761758" watchObservedRunningTime="2026-09-09 02:51:44.72795154 +0000 UTC m=+1298.092286003" Sep 09 02:51:55.704566 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:55.704534 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-ad1d02af-5c4d9df57-lblgs" Sep 09 02:51:55.705099 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:51:55.704606 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ad1d02af-5db6b58bcd-nlzqt" Sep 09 02:53:12.342601 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:53:12.342565 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ad1d02af-5c4d9df57-lblgs"] Sep 09 02:53:12.343056 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:53:12.342819 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-ad1d02af-5c4d9df57-lblgs" podUID="98430462-f8cf-43ba-b60f-7affc4bee624" containerName="payload-processing" containerID="cri-o://24c3041a6688630200b6534f6c43cb8f29ed926a54ce49c289ca3874f773a810" gracePeriod=30 Sep 09 02:53:12.351307 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:53:12.351266 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ad1d02af-5db6b58bcd-nlzqt"] Sep 09 02:53:12.351500 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:53:12.351481 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-ad1d02af-5db6b58bcd-nlzqt" podUID="976dec6b-3271-4032-a27c-d1ad6a9c767a" containerName="payload-pre-processing" containerID="cri-o://fda4385386fb66d99c83f91c8c3a615e5f1f007181391c842378e0dedc476986" gracePeriod=30 Sep 09 02:53:12.620191 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:53:12.620171 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ad1d02af-5c4d9df57-lblgs" Sep 09 02:53:12.633263 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:53:12.633243 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ad1d02af-5db6b58bcd-nlzqt" Sep 09 02:53:12.716521 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:53:12.716488 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pcmp6\" (UniqueName: \"kubernetes.io/projected/976dec6b-3271-4032-a27c-d1ad6a9c767a-kube-api-access-pcmp6\") pod \"976dec6b-3271-4032-a27c-d1ad6a9c767a\" (UID: \"976dec6b-3271-4032-a27c-d1ad6a9c767a\") " Sep 09 02:53:12.716724 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:53:12.716540 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98430462-f8cf-43ba-b60f-7affc4bee624-plugins-config-volume\") pod \"98430462-f8cf-43ba-b60f-7affc4bee624\" (UID: \"98430462-f8cf-43ba-b60f-7affc4bee624\") " Sep 09 02:53:12.716724 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:53:12.716566 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/976dec6b-3271-4032-a27c-d1ad6a9c767a-plugins-config-volume\") pod \"976dec6b-3271-4032-a27c-d1ad6a9c767a\" (UID: \"976dec6b-3271-4032-a27c-d1ad6a9c767a\") " Sep 09 02:53:12.716724 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:53:12.716593 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmqc9\" (UniqueName: \"kubernetes.io/projected/98430462-f8cf-43ba-b60f-7affc4bee624-kube-api-access-tmqc9\") pod \"98430462-f8cf-43ba-b60f-7affc4bee624\" (UID: \"98430462-f8cf-43ba-b60f-7affc4bee624\") " Sep 09 02:53:12.716988 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:53:12.716945 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/976dec6b-3271-4032-a27c-d1ad6a9c767a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "976dec6b-3271-4032-a27c-d1ad6a9c767a" (UID: "976dec6b-3271-4032-a27c-d1ad6a9c767a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:53:12.717077 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:53:12.716986 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98430462-f8cf-43ba-b60f-7affc4bee624-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "98430462-f8cf-43ba-b60f-7affc4bee624" (UID: "98430462-f8cf-43ba-b60f-7affc4bee624"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:53:12.718854 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:53:12.718826 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/976dec6b-3271-4032-a27c-d1ad6a9c767a-kube-api-access-pcmp6" (OuterVolumeSpecName: "kube-api-access-pcmp6") pod "976dec6b-3271-4032-a27c-d1ad6a9c767a" (UID: "976dec6b-3271-4032-a27c-d1ad6a9c767a"). InnerVolumeSpecName "kube-api-access-pcmp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:53:12.719009 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:53:12.718857 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98430462-f8cf-43ba-b60f-7affc4bee624-kube-api-access-tmqc9" (OuterVolumeSpecName: "kube-api-access-tmqc9") pod "98430462-f8cf-43ba-b60f-7affc4bee624" (UID: "98430462-f8cf-43ba-b60f-7affc4bee624"). InnerVolumeSpecName "kube-api-access-tmqc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:53:12.817759 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:53:12.817726 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98430462-f8cf-43ba-b60f-7affc4bee624-plugins-config-volume\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:53:12.817759 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:53:12.817754 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/976dec6b-3271-4032-a27c-d1ad6a9c767a-plugins-config-volume\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:53:12.817759 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:53:12.817763 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmqc9\" (UniqueName: \"kubernetes.io/projected/98430462-f8cf-43ba-b60f-7affc4bee624-kube-api-access-tmqc9\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:53:12.818028 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:53:12.817772 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcmp6\" (UniqueName: \"kubernetes.io/projected/976dec6b-3271-4032-a27c-d1ad6a9c767a-kube-api-access-pcmp6\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:53:13.214139 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:53:13.214104 2577 generic.go:356] "Generic (PLEG): container finished" podID="98430462-f8cf-43ba-b60f-7affc4bee624" containerID="24c3041a6688630200b6534f6c43cb8f29ed926a54ce49c289ca3874f773a810" exitCode=0 Sep 09 02:53:13.214315 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:53:13.214152 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ad1d02af-5c4d9df57-lblgs" Sep 09 02:53:13.214315 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:53:13.214196 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ad1d02af-5c4d9df57-lblgs" event={"ID":"98430462-f8cf-43ba-b60f-7affc4bee624","Type":"ContainerDied","Data":"24c3041a6688630200b6534f6c43cb8f29ed926a54ce49c289ca3874f773a810"} Sep 09 02:53:13.214315 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:53:13.214257 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ad1d02af-5c4d9df57-lblgs" event={"ID":"98430462-f8cf-43ba-b60f-7affc4bee624","Type":"ContainerDied","Data":"347d74bcacb0718e90e99e6798079a76872d25bf8ab249bc46dac9cd327ff7db"} Sep 09 02:53:13.214315 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:53:13.214286 2577 scope.go:117] "RemoveContainer" containerID="24c3041a6688630200b6534f6c43cb8f29ed926a54ce49c289ca3874f773a810" Sep 09 02:53:13.215754 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:53:13.215729 2577 generic.go:356] "Generic (PLEG): container finished" podID="976dec6b-3271-4032-a27c-d1ad6a9c767a" containerID="fda4385386fb66d99c83f91c8c3a615e5f1f007181391c842378e0dedc476986" exitCode=0 Sep 09 02:53:13.215862 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:53:13.215835 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ad1d02af-5db6b58bcd-nlzqt" event={"ID":"976dec6b-3271-4032-a27c-d1ad6a9c767a","Type":"ContainerDied","Data":"fda4385386fb66d99c83f91c8c3a615e5f1f007181391c842378e0dedc476986"} Sep 09 02:53:13.215922 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:53:13.215877 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ad1d02af-5db6b58bcd-nlzqt" event={"ID":"976dec6b-3271-4032-a27c-d1ad6a9c767a","Type":"ContainerDied","Data":"6c8769d9a49cb20f38bf145c2b7393a710a3637b7311960e745f4f01aa7e5553"} Sep 09 02:53:13.215922 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:53:13.215840 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ad1d02af-5db6b58bcd-nlzqt" Sep 09 02:53:13.224675 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:53:13.224652 2577 scope.go:117] "RemoveContainer" containerID="24c3041a6688630200b6534f6c43cb8f29ed926a54ce49c289ca3874f773a810" Sep 09 02:53:13.230408 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:53:13.230385 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24c3041a6688630200b6534f6c43cb8f29ed926a54ce49c289ca3874f773a810\": container with ID starting with 24c3041a6688630200b6534f6c43cb8f29ed926a54ce49c289ca3874f773a810 not found: ID does not exist" containerID="24c3041a6688630200b6534f6c43cb8f29ed926a54ce49c289ca3874f773a810" Sep 09 02:53:13.230527 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:53:13.230417 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24c3041a6688630200b6534f6c43cb8f29ed926a54ce49c289ca3874f773a810"} err="failed to get container status \"24c3041a6688630200b6534f6c43cb8f29ed926a54ce49c289ca3874f773a810\": rpc error: code = NotFound desc = could not find container \"24c3041a6688630200b6534f6c43cb8f29ed926a54ce49c289ca3874f773a810\": container with ID starting with 24c3041a6688630200b6534f6c43cb8f29ed926a54ce49c289ca3874f773a810 not found: ID does not exist" Sep 09 02:53:13.230527 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:53:13.230444 2577 scope.go:117] "RemoveContainer" containerID="fda4385386fb66d99c83f91c8c3a615e5f1f007181391c842378e0dedc476986" Sep 09 02:53:13.237051 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:53:13.237007 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ad1d02af-5c4d9df57-lblgs"] Sep 09 02:53:13.240857 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:53:13.240834 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ad1d02af-5c4d9df57-lblgs"] Sep 09 02:53:13.241705 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:53:13.241690 2577 scope.go:117] "RemoveContainer" containerID="fda4385386fb66d99c83f91c8c3a615e5f1f007181391c842378e0dedc476986" Sep 09 02:53:13.242025 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:53:13.242004 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fda4385386fb66d99c83f91c8c3a615e5f1f007181391c842378e0dedc476986\": container with ID starting with fda4385386fb66d99c83f91c8c3a615e5f1f007181391c842378e0dedc476986 not found: ID does not exist" containerID="fda4385386fb66d99c83f91c8c3a615e5f1f007181391c842378e0dedc476986" Sep 09 02:53:13.242096 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:53:13.242036 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fda4385386fb66d99c83f91c8c3a615e5f1f007181391c842378e0dedc476986"} err="failed to get container status \"fda4385386fb66d99c83f91c8c3a615e5f1f007181391c842378e0dedc476986\": rpc error: code = NotFound desc = could not find container \"fda4385386fb66d99c83f91c8c3a615e5f1f007181391c842378e0dedc476986\": container with ID starting with fda4385386fb66d99c83f91c8c3a615e5f1f007181391c842378e0dedc476986 not found: ID does not exist" Sep 09 02:53:13.249574 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:53:13.249556 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ad1d02af-5db6b58bcd-nlzqt"] Sep 09 02:53:13.253487 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:53:13.253468 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ad1d02af-5db6b58bcd-nlzqt"] Sep 09 02:53:15.165200 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:53:15.165169 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="976dec6b-3271-4032-a27c-d1ad6a9c767a" path="/var/lib/kubelet/pods/976dec6b-3271-4032-a27c-d1ad6a9c767a/volumes" Sep 09 02:53:15.165564 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:53:15.165502 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98430462-f8cf-43ba-b60f-7affc4bee624" path="/var/lib/kubelet/pods/98430462-f8cf-43ba-b60f-7affc4bee624/volumes" Sep 09 02:54:07.326705 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:54:07.326629 2577 scope.go:117] "RemoveContainer" containerID="afaa9821366ee56940caf154c4f392520467ea78ee7cbe11bb22db9fb32b4474" Sep 09 02:55:07.355896 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:07.355864 2577 scope.go:117] "RemoveContainer" containerID="4eba8d3231de22675d5c03d934f938d75c8fcc8201cd66cf15103deff5fc825c" Sep 09 02:55:09.048993 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:09.048937 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2b9f0d-7cdcdcd766-zfbxg"] Sep 09 02:55:09.049372 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:09.049345 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="976dec6b-3271-4032-a27c-d1ad6a9c767a" containerName="payload-pre-processing" Sep 09 02:55:09.049372 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:09.049357 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="976dec6b-3271-4032-a27c-d1ad6a9c767a" containerName="payload-pre-processing" Sep 09 02:55:09.049372 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:09.049369 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98430462-f8cf-43ba-b60f-7affc4bee624" containerName="payload-processing" Sep 09 02:55:09.049372 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:09.049375 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="98430462-f8cf-43ba-b60f-7affc4bee624" containerName="payload-processing" Sep 09 02:55:09.049503 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:09.049436 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="976dec6b-3271-4032-a27c-d1ad6a9c767a" containerName="payload-pre-processing" Sep 09 02:55:09.049503 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:09.049446 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="98430462-f8cf-43ba-b60f-7affc4bee624" containerName="payload-processing" Sep 09 02:55:09.053896 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:09.053879 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2b9f0d-7cdcdcd766-zfbxg" Sep 09 02:55:09.056858 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:09.056829 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-2b9f0d\"" Sep 09 02:55:09.057029 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:09.056831 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-2b9f0d-dockercfg-mwfkq\"" Sep 09 02:55:09.063989 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:09.063947 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2b9f0d-7cdcdcd766-zfbxg"] Sep 09 02:55:09.081569 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:09.081538 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2b9f0d-786bf654d5-s4kj5"] Sep 09 02:55:09.084921 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:09.084905 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2b9f0d-786bf654d5-s4kj5" Sep 09 02:55:09.093420 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:09.093387 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2b9f0d-786bf654d5-s4kj5"] Sep 09 02:55:09.135569 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:09.135131 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5ncs\" (UniqueName: \"kubernetes.io/projected/fbc0dc84-9c56-4945-b183-9584c5460f4c-kube-api-access-b5ncs\") pod \"payload-pre-processing-e2e-api-a-2b9f0d-7cdcdcd766-zfbxg\" (UID: \"fbc0dc84-9c56-4945-b183-9584c5460f4c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2b9f0d-7cdcdcd766-zfbxg" Sep 09 02:55:09.135569 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:09.135237 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q97bx\" (UniqueName: \"kubernetes.io/projected/f0009f07-dc0b-4d12-a57d-b8d47015052b-kube-api-access-q97bx\") pod \"payload-processing-e2e-api-a-2b9f0d-786bf654d5-s4kj5\" (UID: \"f0009f07-dc0b-4d12-a57d-b8d47015052b\") " pod="openshift-ingress/payload-processing-e2e-api-a-2b9f0d-786bf654d5-s4kj5" Sep 09 02:55:09.135569 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:09.135381 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbc0dc84-9c56-4945-b183-9584c5460f4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2b9f0d-7cdcdcd766-zfbxg\" (UID: \"fbc0dc84-9c56-4945-b183-9584c5460f4c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2b9f0d-7cdcdcd766-zfbxg" Sep 09 02:55:09.135569 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:09.135409 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0009f07-dc0b-4d12-a57d-b8d47015052b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2b9f0d-786bf654d5-s4kj5\" (UID: \"f0009f07-dc0b-4d12-a57d-b8d47015052b\") " pod="openshift-ingress/payload-processing-e2e-api-a-2b9f0d-786bf654d5-s4kj5" Sep 09 02:55:09.236523 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:09.236490 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q97bx\" (UniqueName: \"kubernetes.io/projected/f0009f07-dc0b-4d12-a57d-b8d47015052b-kube-api-access-q97bx\") pod \"payload-processing-e2e-api-a-2b9f0d-786bf654d5-s4kj5\" (UID: \"f0009f07-dc0b-4d12-a57d-b8d47015052b\") " pod="openshift-ingress/payload-processing-e2e-api-a-2b9f0d-786bf654d5-s4kj5" Sep 09 02:55:09.236713 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:09.236535 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbc0dc84-9c56-4945-b183-9584c5460f4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2b9f0d-7cdcdcd766-zfbxg\" (UID: \"fbc0dc84-9c56-4945-b183-9584c5460f4c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2b9f0d-7cdcdcd766-zfbxg" Sep 09 02:55:09.236713 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:09.236551 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0009f07-dc0b-4d12-a57d-b8d47015052b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2b9f0d-786bf654d5-s4kj5\" (UID: \"f0009f07-dc0b-4d12-a57d-b8d47015052b\") " pod="openshift-ingress/payload-processing-e2e-api-a-2b9f0d-786bf654d5-s4kj5" Sep 09 02:55:09.236713 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:09.236616 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5ncs\" (UniqueName: \"kubernetes.io/projected/fbc0dc84-9c56-4945-b183-9584c5460f4c-kube-api-access-b5ncs\") pod \"payload-pre-processing-e2e-api-a-2b9f0d-7cdcdcd766-zfbxg\" (UID: \"fbc0dc84-9c56-4945-b183-9584c5460f4c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2b9f0d-7cdcdcd766-zfbxg" Sep 09 02:55:09.237346 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:09.237326 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0009f07-dc0b-4d12-a57d-b8d47015052b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2b9f0d-786bf654d5-s4kj5\" (UID: \"f0009f07-dc0b-4d12-a57d-b8d47015052b\") " pod="openshift-ingress/payload-processing-e2e-api-a-2b9f0d-786bf654d5-s4kj5" Sep 09 02:55:09.237394 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:09.237326 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbc0dc84-9c56-4945-b183-9584c5460f4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2b9f0d-7cdcdcd766-zfbxg\" (UID: \"fbc0dc84-9c56-4945-b183-9584c5460f4c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2b9f0d-7cdcdcd766-zfbxg" Sep 09 02:55:09.244465 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:09.244434 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q97bx\" (UniqueName: \"kubernetes.io/projected/f0009f07-dc0b-4d12-a57d-b8d47015052b-kube-api-access-q97bx\") pod \"payload-processing-e2e-api-a-2b9f0d-786bf654d5-s4kj5\" (UID: \"f0009f07-dc0b-4d12-a57d-b8d47015052b\") " pod="openshift-ingress/payload-processing-e2e-api-a-2b9f0d-786bf654d5-s4kj5" Sep 09 02:55:09.244577 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:09.244484 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5ncs\" (UniqueName: \"kubernetes.io/projected/fbc0dc84-9c56-4945-b183-9584c5460f4c-kube-api-access-b5ncs\") pod \"payload-pre-processing-e2e-api-a-2b9f0d-7cdcdcd766-zfbxg\" (UID: \"fbc0dc84-9c56-4945-b183-9584c5460f4c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2b9f0d-7cdcdcd766-zfbxg" Sep 09 02:55:09.365390 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:09.365300 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2b9f0d-7cdcdcd766-zfbxg" Sep 09 02:55:09.397867 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:09.397831 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2b9f0d-786bf654d5-s4kj5" Sep 09 02:55:09.642007 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:09.641983 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2b9f0d-7cdcdcd766-zfbxg"] Sep 09 02:55:09.645982 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:55:09.645936 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbc0dc84_9c56_4945_b183_9584c5460f4c.slice/crio-a6c3f60a68f75ecd27b96cc4d046c748d66c7dbcd2c7d76fbcb386f7dfdc5fb4 WatchSource:0}: Error finding container a6c3f60a68f75ecd27b96cc4d046c748d66c7dbcd2c7d76fbcb386f7dfdc5fb4: Status 404 returned error can't find the container with id a6c3f60a68f75ecd27b96cc4d046c748d66c7dbcd2c7d76fbcb386f7dfdc5fb4 Sep 09 02:55:09.678817 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:09.678797 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2b9f0d-786bf654d5-s4kj5"] Sep 09 02:55:09.683187 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:55:09.683150 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0009f07_dc0b_4d12_a57d_b8d47015052b.slice/crio-622617c6346781989d9c74bde817382df241cfdc1969f4a65a2a57359fe9e6de WatchSource:0}: Error finding container 622617c6346781989d9c74bde817382df241cfdc1969f4a65a2a57359fe9e6de: Status 404 returned error can't find the container with id 622617c6346781989d9c74bde817382df241cfdc1969f4a65a2a57359fe9e6de Sep 09 02:55:09.891085 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:09.890976 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2b9f0d-7cdcdcd766-zfbxg" event={"ID":"fbc0dc84-9c56-4945-b183-9584c5460f4c","Type":"ContainerStarted","Data":"94da6ead87a0b6c7a4371e7dae939ec9df85ec402c641339758c3a79c7386b5b"} Sep 09 02:55:09.891085 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:09.891048 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2b9f0d-7cdcdcd766-zfbxg" event={"ID":"fbc0dc84-9c56-4945-b183-9584c5460f4c","Type":"ContainerStarted","Data":"a6c3f60a68f75ecd27b96cc4d046c748d66c7dbcd2c7d76fbcb386f7dfdc5fb4"} Sep 09 02:55:09.891386 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:09.891363 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2b9f0d-7cdcdcd766-zfbxg" Sep 09 02:55:09.893218 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:09.893187 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2b9f0d-786bf654d5-s4kj5" event={"ID":"f0009f07-dc0b-4d12-a57d-b8d47015052b","Type":"ContainerStarted","Data":"4059d9c75c9a51a7b2a1ea1716be94e360dc7f820c99b8016585a87365260a53"} Sep 09 02:55:09.893372 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:09.893239 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2b9f0d-786bf654d5-s4kj5" event={"ID":"f0009f07-dc0b-4d12-a57d-b8d47015052b","Type":"ContainerStarted","Data":"622617c6346781989d9c74bde817382df241cfdc1969f4a65a2a57359fe9e6de"} Sep 09 02:55:09.893372 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:09.893308 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-2b9f0d-786bf654d5-s4kj5" Sep 09 02:55:09.910037 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:09.909979 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2b9f0d-7cdcdcd766-zfbxg" podStartSLOduration=0.909941201 podStartE2EDuration="909.941201ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:55:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:55:09.907108592 +0000 UTC m=+1503.271443055" watchObservedRunningTime="2026-09-09 02:55:09.909941201 +0000 UTC m=+1503.274275670" Sep 09 02:55:09.925031 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:09.924968 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-2b9f0d-786bf654d5-s4kj5" podStartSLOduration=0.924937245 podStartE2EDuration="924.937245ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:55:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:55:09.922596646 +0000 UTC m=+1503.286931111" watchObservedRunningTime="2026-09-09 02:55:09.924937245 +0000 UTC m=+1503.289271708" Sep 09 02:55:20.902450 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:20.902417 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2b9f0d-7cdcdcd766-zfbxg" Sep 09 02:55:20.902833 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:20.902492 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-2b9f0d-786bf654d5-s4kj5" Sep 09 02:55:38.316669 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:38.316587 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4bd452-5fc8b9d9f4-mrw7j"] Sep 09 02:55:38.319825 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:38.319806 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4bd452-5fc8b9d9f4-mrw7j" Sep 09 02:55:38.322253 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:38.322232 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-4bd452-dockercfg-ml4j4\"" Sep 09 02:55:38.322354 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:38.322232 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-4bd452\"" Sep 09 02:55:38.331074 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:38.331047 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4bd452-5fc8b9d9f4-mrw7j"] Sep 09 02:55:38.339948 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:38.339927 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4bd452-67fc6fdccf-9k2h5"] Sep 09 02:55:38.345576 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:38.345557 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4bd452-67fc6fdccf-9k2h5" Sep 09 02:55:38.359391 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:38.359370 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4bd452-67fc6fdccf-9k2h5"] Sep 09 02:55:38.410349 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:38.410316 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10755e29-715e-4164-954e-01a8df1f49f6-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4bd452-67fc6fdccf-9k2h5\" (UID: \"10755e29-715e-4164-954e-01a8df1f49f6\") " pod="openshift-ingress/payload-processing-e2e-api-b-4bd452-67fc6fdccf-9k2h5" Sep 09 02:55:38.410511 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:38.410356 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k6n2\" (UniqueName: \"kubernetes.io/projected/10755e29-715e-4164-954e-01a8df1f49f6-kube-api-access-9k6n2\") pod \"payload-processing-e2e-api-b-4bd452-67fc6fdccf-9k2h5\" (UID: \"10755e29-715e-4164-954e-01a8df1f49f6\") " pod="openshift-ingress/payload-processing-e2e-api-b-4bd452-67fc6fdccf-9k2h5" Sep 09 02:55:38.410511 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:38.410413 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6hfw\" (UniqueName: \"kubernetes.io/projected/4420f778-2b7a-4020-984d-c46b438cb7da-kube-api-access-q6hfw\") pod \"payload-pre-processing-e2e-api-b-4bd452-5fc8b9d9f4-mrw7j\" (UID: \"4420f778-2b7a-4020-984d-c46b438cb7da\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4bd452-5fc8b9d9f4-mrw7j" Sep 09 02:55:38.410511 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:38.410466 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4420f778-2b7a-4020-984d-c46b438cb7da-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4bd452-5fc8b9d9f4-mrw7j\" (UID: \"4420f778-2b7a-4020-984d-c46b438cb7da\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4bd452-5fc8b9d9f4-mrw7j" Sep 09 02:55:38.511714 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:38.511668 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4420f778-2b7a-4020-984d-c46b438cb7da-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4bd452-5fc8b9d9f4-mrw7j\" (UID: \"4420f778-2b7a-4020-984d-c46b438cb7da\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4bd452-5fc8b9d9f4-mrw7j" Sep 09 02:55:38.511935 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:38.511783 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10755e29-715e-4164-954e-01a8df1f49f6-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4bd452-67fc6fdccf-9k2h5\" (UID: \"10755e29-715e-4164-954e-01a8df1f49f6\") " pod="openshift-ingress/payload-processing-e2e-api-b-4bd452-67fc6fdccf-9k2h5" Sep 09 02:55:38.511935 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:38.511817 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k6n2\" (UniqueName: \"kubernetes.io/projected/10755e29-715e-4164-954e-01a8df1f49f6-kube-api-access-9k6n2\") pod \"payload-processing-e2e-api-b-4bd452-67fc6fdccf-9k2h5\" (UID: \"10755e29-715e-4164-954e-01a8df1f49f6\") " pod="openshift-ingress/payload-processing-e2e-api-b-4bd452-67fc6fdccf-9k2h5" Sep 09 02:55:38.511935 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:38.511882 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6hfw\" (UniqueName: \"kubernetes.io/projected/4420f778-2b7a-4020-984d-c46b438cb7da-kube-api-access-q6hfw\") pod \"payload-pre-processing-e2e-api-b-4bd452-5fc8b9d9f4-mrw7j\" (UID: \"4420f778-2b7a-4020-984d-c46b438cb7da\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4bd452-5fc8b9d9f4-mrw7j" Sep 09 02:55:38.512698 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:38.512676 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4420f778-2b7a-4020-984d-c46b438cb7da-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4bd452-5fc8b9d9f4-mrw7j\" (UID: \"4420f778-2b7a-4020-984d-c46b438cb7da\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4bd452-5fc8b9d9f4-mrw7j" Sep 09 02:55:38.512796 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:38.512749 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10755e29-715e-4164-954e-01a8df1f49f6-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4bd452-67fc6fdccf-9k2h5\" (UID: \"10755e29-715e-4164-954e-01a8df1f49f6\") " pod="openshift-ingress/payload-processing-e2e-api-b-4bd452-67fc6fdccf-9k2h5" Sep 09 02:55:38.520556 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:38.520532 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k6n2\" (UniqueName: \"kubernetes.io/projected/10755e29-715e-4164-954e-01a8df1f49f6-kube-api-access-9k6n2\") pod \"payload-processing-e2e-api-b-4bd452-67fc6fdccf-9k2h5\" (UID: \"10755e29-715e-4164-954e-01a8df1f49f6\") " pod="openshift-ingress/payload-processing-e2e-api-b-4bd452-67fc6fdccf-9k2h5" Sep 09 02:55:38.520647 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:38.520524 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6hfw\" (UniqueName: \"kubernetes.io/projected/4420f778-2b7a-4020-984d-c46b438cb7da-kube-api-access-q6hfw\") pod \"payload-pre-processing-e2e-api-b-4bd452-5fc8b9d9f4-mrw7j\" (UID: \"4420f778-2b7a-4020-984d-c46b438cb7da\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4bd452-5fc8b9d9f4-mrw7j" Sep 09 02:55:38.632826 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:38.632748 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4bd452-5fc8b9d9f4-mrw7j" Sep 09 02:55:38.655229 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:38.655205 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4bd452-67fc6fdccf-9k2h5" Sep 09 02:55:38.910114 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:38.910083 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4bd452-5fc8b9d9f4-mrw7j"] Sep 09 02:55:38.911355 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:55:38.911327 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4420f778_2b7a_4020_984d_c46b438cb7da.slice/crio-aa51375e049c89a211fe089c5a8f9f5c26d2abea49266bc879ec8568db689ce1 WatchSource:0}: Error finding container aa51375e049c89a211fe089c5a8f9f5c26d2abea49266bc879ec8568db689ce1: Status 404 returned error can't find the container with id aa51375e049c89a211fe089c5a8f9f5c26d2abea49266bc879ec8568db689ce1 Sep 09 02:55:38.943860 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:38.943839 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4bd452-67fc6fdccf-9k2h5"] Sep 09 02:55:38.944826 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:55:38.944799 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10755e29_715e_4164_954e_01a8df1f49f6.slice/crio-cc596ae35a3bea2d0f7376346e31eedd4be20bc541480d604e785dbcdcb90811 WatchSource:0}: Error finding container cc596ae35a3bea2d0f7376346e31eedd4be20bc541480d604e785dbcdcb90811: Status 404 returned error can't find the container with id cc596ae35a3bea2d0f7376346e31eedd4be20bc541480d604e785dbcdcb90811 Sep 09 02:55:39.065326 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:39.065285 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4bd452-67fc6fdccf-9k2h5" event={"ID":"10755e29-715e-4164-954e-01a8df1f49f6","Type":"ContainerStarted","Data":"61b31e012c0d57034399d99a95c36d22e984c43dcfb2bc1943c63661c2388994"} Sep 09 02:55:39.065510 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:39.065355 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4bd452-67fc6fdccf-9k2h5" event={"ID":"10755e29-715e-4164-954e-01a8df1f49f6","Type":"ContainerStarted","Data":"cc596ae35a3bea2d0f7376346e31eedd4be20bc541480d604e785dbcdcb90811"} Sep 09 02:55:39.065510 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:39.065389 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-4bd452-67fc6fdccf-9k2h5" Sep 09 02:55:39.067244 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:39.067217 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4bd452-5fc8b9d9f4-mrw7j" event={"ID":"4420f778-2b7a-4020-984d-c46b438cb7da","Type":"ContainerStarted","Data":"7c35d08f92894d67bd54eb1e553e266f4bad34bc57e453d1ec242ef6299b9c97"} Sep 09 02:55:39.067376 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:39.067265 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4bd452-5fc8b9d9f4-mrw7j" event={"ID":"4420f778-2b7a-4020-984d-c46b438cb7da","Type":"ContainerStarted","Data":"aa51375e049c89a211fe089c5a8f9f5c26d2abea49266bc879ec8568db689ce1"} Sep 09 02:55:39.067376 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:39.067295 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4bd452-5fc8b9d9f4-mrw7j" Sep 09 02:55:39.085737 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:39.085686 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-4bd452-67fc6fdccf-9k2h5" podStartSLOduration=1.085670235 podStartE2EDuration="1.085670235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:55:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:55:39.084356379 +0000 UTC m=+1532.448690841" watchObservedRunningTime="2026-09-09 02:55:39.085670235 +0000 UTC m=+1532.450004699" Sep 09 02:55:39.103267 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:39.103203 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4bd452-5fc8b9d9f4-mrw7j" podStartSLOduration=1.103190217 podStartE2EDuration="1.103190217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:55:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:55:39.1003921 +0000 UTC m=+1532.464726564" watchObservedRunningTime="2026-09-09 02:55:39.103190217 +0000 UTC m=+1532.467524682" Sep 09 02:55:50.075133 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:50.075102 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-4bd452-67fc6fdccf-9k2h5" Sep 09 02:55:50.075564 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:55:50.075402 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4bd452-5fc8b9d9f4-mrw7j" Sep 09 02:56:07.377427 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:07.377396 2577 scope.go:117] "RemoveContainer" containerID="4197e411c87e1cdf3fad6eda65e850addfd922b96c1cf410792397c96e6e7299" Sep 09 02:56:21.183097 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:21.183065 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2b9f0d-786bf654d5-s4kj5"] Sep 09 02:56:21.187116 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:21.183292 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-2b9f0d-786bf654d5-s4kj5" podUID="f0009f07-dc0b-4d12-a57d-b8d47015052b" containerName="payload-processing" containerID="cri-o://4059d9c75c9a51a7b2a1ea1716be94e360dc7f820c99b8016585a87365260a53" gracePeriod=30 Sep 09 02:56:21.196196 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:21.196167 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2b9f0d-7cdcdcd766-zfbxg"] Sep 09 02:56:21.196495 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:21.196440 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2b9f0d-7cdcdcd766-zfbxg" podUID="fbc0dc84-9c56-4945-b183-9584c5460f4c" containerName="payload-pre-processing" containerID="cri-o://94da6ead87a0b6c7a4371e7dae939ec9df85ec402c641339758c3a79c7386b5b" gracePeriod=30 Sep 09 02:56:21.319552 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:21.319521 2577 generic.go:356] "Generic (PLEG): container finished" podID="f0009f07-dc0b-4d12-a57d-b8d47015052b" containerID="4059d9c75c9a51a7b2a1ea1716be94e360dc7f820c99b8016585a87365260a53" exitCode=0 Sep 09 02:56:21.319672 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:21.319596 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2b9f0d-786bf654d5-s4kj5" event={"ID":"f0009f07-dc0b-4d12-a57d-b8d47015052b","Type":"ContainerDied","Data":"4059d9c75c9a51a7b2a1ea1716be94e360dc7f820c99b8016585a87365260a53"} Sep 09 02:56:21.321726 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:21.321699 2577 generic.go:356] "Generic (PLEG): container finished" podID="fbc0dc84-9c56-4945-b183-9584c5460f4c" containerID="94da6ead87a0b6c7a4371e7dae939ec9df85ec402c641339758c3a79c7386b5b" exitCode=0 Sep 09 02:56:21.321828 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:21.321739 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2b9f0d-7cdcdcd766-zfbxg" event={"ID":"fbc0dc84-9c56-4945-b183-9584c5460f4c","Type":"ContainerDied","Data":"94da6ead87a0b6c7a4371e7dae939ec9df85ec402c641339758c3a79c7386b5b"} Sep 09 02:56:21.473610 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:21.473576 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2b9f0d-786bf654d5-s4kj5" Sep 09 02:56:21.476514 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:21.476492 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2b9f0d-7cdcdcd766-zfbxg" Sep 09 02:56:21.587939 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:21.587908 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0009f07-dc0b-4d12-a57d-b8d47015052b-plugins-config-volume\") pod \"f0009f07-dc0b-4d12-a57d-b8d47015052b\" (UID: \"f0009f07-dc0b-4d12-a57d-b8d47015052b\") " Sep 09 02:56:21.588145 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:21.587978 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbc0dc84-9c56-4945-b183-9584c5460f4c-plugins-config-volume\") pod \"fbc0dc84-9c56-4945-b183-9584c5460f4c\" (UID: \"fbc0dc84-9c56-4945-b183-9584c5460f4c\") " Sep 09 02:56:21.588145 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:21.588017 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q97bx\" (UniqueName: \"kubernetes.io/projected/f0009f07-dc0b-4d12-a57d-b8d47015052b-kube-api-access-q97bx\") pod \"f0009f07-dc0b-4d12-a57d-b8d47015052b\" (UID: \"f0009f07-dc0b-4d12-a57d-b8d47015052b\") " Sep 09 02:56:21.588145 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:21.588063 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5ncs\" (UniqueName: \"kubernetes.io/projected/fbc0dc84-9c56-4945-b183-9584c5460f4c-kube-api-access-b5ncs\") pod \"fbc0dc84-9c56-4945-b183-9584c5460f4c\" (UID: \"fbc0dc84-9c56-4945-b183-9584c5460f4c\") " Sep 09 02:56:21.588426 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:21.588384 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0009f07-dc0b-4d12-a57d-b8d47015052b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f0009f07-dc0b-4d12-a57d-b8d47015052b" (UID: "f0009f07-dc0b-4d12-a57d-b8d47015052b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:56:21.588556 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:21.588423 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbc0dc84-9c56-4945-b183-9584c5460f4c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fbc0dc84-9c56-4945-b183-9584c5460f4c" (UID: "fbc0dc84-9c56-4945-b183-9584c5460f4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:56:21.590337 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:21.590311 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0009f07-dc0b-4d12-a57d-b8d47015052b-kube-api-access-q97bx" (OuterVolumeSpecName: "kube-api-access-q97bx") pod "f0009f07-dc0b-4d12-a57d-b8d47015052b" (UID: "f0009f07-dc0b-4d12-a57d-b8d47015052b"). InnerVolumeSpecName "kube-api-access-q97bx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:56:21.590452 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:21.590429 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbc0dc84-9c56-4945-b183-9584c5460f4c-kube-api-access-b5ncs" (OuterVolumeSpecName: "kube-api-access-b5ncs") pod "fbc0dc84-9c56-4945-b183-9584c5460f4c" (UID: "fbc0dc84-9c56-4945-b183-9584c5460f4c"). InnerVolumeSpecName "kube-api-access-b5ncs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:56:21.689469 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:21.689431 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5ncs\" (UniqueName: \"kubernetes.io/projected/fbc0dc84-9c56-4945-b183-9584c5460f4c-kube-api-access-b5ncs\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:56:21.689469 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:21.689465 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0009f07-dc0b-4d12-a57d-b8d47015052b-plugins-config-volume\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:56:21.689672 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:21.689480 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbc0dc84-9c56-4945-b183-9584c5460f4c-plugins-config-volume\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:56:21.689672 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:21.689492 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q97bx\" (UniqueName: \"kubernetes.io/projected/f0009f07-dc0b-4d12-a57d-b8d47015052b-kube-api-access-q97bx\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:56:22.329166 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:22.329133 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2b9f0d-786bf654d5-s4kj5" Sep 09 02:56:22.329166 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:22.329157 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2b9f0d-786bf654d5-s4kj5" event={"ID":"f0009f07-dc0b-4d12-a57d-b8d47015052b","Type":"ContainerDied","Data":"622617c6346781989d9c74bde817382df241cfdc1969f4a65a2a57359fe9e6de"} Sep 09 02:56:22.329673 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:22.329252 2577 scope.go:117] "RemoveContainer" containerID="4059d9c75c9a51a7b2a1ea1716be94e360dc7f820c99b8016585a87365260a53" Sep 09 02:56:22.330700 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:22.330674 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2b9f0d-7cdcdcd766-zfbxg" event={"ID":"fbc0dc84-9c56-4945-b183-9584c5460f4c","Type":"ContainerDied","Data":"a6c3f60a68f75ecd27b96cc4d046c748d66c7dbcd2c7d76fbcb386f7dfdc5fb4"} Sep 09 02:56:22.330773 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:22.330696 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2b9f0d-7cdcdcd766-zfbxg" Sep 09 02:56:22.342698 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:22.342680 2577 scope.go:117] "RemoveContainer" containerID="94da6ead87a0b6c7a4371e7dae939ec9df85ec402c641339758c3a79c7386b5b" Sep 09 02:56:22.357837 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:22.357810 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2b9f0d-7cdcdcd766-zfbxg"] Sep 09 02:56:22.361940 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:22.361915 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2b9f0d-7cdcdcd766-zfbxg"] Sep 09 02:56:22.373389 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:22.373344 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2b9f0d-786bf654d5-s4kj5"] Sep 09 02:56:22.377049 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:22.377024 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2b9f0d-786bf654d5-s4kj5"] Sep 09 02:56:23.167176 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:23.167141 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0009f07-dc0b-4d12-a57d-b8d47015052b" path="/var/lib/kubelet/pods/f0009f07-dc0b-4d12-a57d-b8d47015052b/volumes" Sep 09 02:56:23.167501 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:23.167488 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbc0dc84-9c56-4945-b183-9584c5460f4c" path="/var/lib/kubelet/pods/fbc0dc84-9c56-4945-b183-9584c5460f4c/volumes" Sep 09 02:56:43.865195 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:43.865161 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4bd452-67fc6fdccf-9k2h5"] Sep 09 02:56:43.865694 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:43.865374 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-4bd452-67fc6fdccf-9k2h5" podUID="10755e29-715e-4164-954e-01a8df1f49f6" containerName="payload-processing" containerID="cri-o://61b31e012c0d57034399d99a95c36d22e984c43dcfb2bc1943c63661c2388994" gracePeriod=30 Sep 09 02:56:43.875906 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:43.875877 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4bd452-5fc8b9d9f4-mrw7j"] Sep 09 02:56:43.876199 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:43.876147 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4bd452-5fc8b9d9f4-mrw7j" podUID="4420f778-2b7a-4020-984d-c46b438cb7da" containerName="payload-pre-processing" containerID="cri-o://7c35d08f92894d67bd54eb1e553e266f4bad34bc57e453d1ec242ef6299b9c97" gracePeriod=30 Sep 09 02:56:44.146111 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:44.146083 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4bd452-67fc6fdccf-9k2h5" Sep 09 02:56:44.174419 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:44.174393 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4bd452-5fc8b9d9f4-mrw7j" Sep 09 02:56:44.182131 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:44.182104 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10755e29-715e-4164-954e-01a8df1f49f6-plugins-config-volume\") pod \"10755e29-715e-4164-954e-01a8df1f49f6\" (UID: \"10755e29-715e-4164-954e-01a8df1f49f6\") " Sep 09 02:56:44.182379 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:44.182193 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9k6n2\" (UniqueName: \"kubernetes.io/projected/10755e29-715e-4164-954e-01a8df1f49f6-kube-api-access-9k6n2\") pod \"10755e29-715e-4164-954e-01a8df1f49f6\" (UID: \"10755e29-715e-4164-954e-01a8df1f49f6\") " Sep 09 02:56:44.182860 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:44.182509 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10755e29-715e-4164-954e-01a8df1f49f6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "10755e29-715e-4164-954e-01a8df1f49f6" (UID: "10755e29-715e-4164-954e-01a8df1f49f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:56:44.187819 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:44.185501 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10755e29-715e-4164-954e-01a8df1f49f6-kube-api-access-9k6n2" (OuterVolumeSpecName: "kube-api-access-9k6n2") pod "10755e29-715e-4164-954e-01a8df1f49f6" (UID: "10755e29-715e-4164-954e-01a8df1f49f6"). InnerVolumeSpecName "kube-api-access-9k6n2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:56:44.283575 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:44.283546 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6hfw\" (UniqueName: \"kubernetes.io/projected/4420f778-2b7a-4020-984d-c46b438cb7da-kube-api-access-q6hfw\") pod \"4420f778-2b7a-4020-984d-c46b438cb7da\" (UID: \"4420f778-2b7a-4020-984d-c46b438cb7da\") " Sep 09 02:56:44.283749 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:44.283613 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4420f778-2b7a-4020-984d-c46b438cb7da-plugins-config-volume\") pod \"4420f778-2b7a-4020-984d-c46b438cb7da\" (UID: \"4420f778-2b7a-4020-984d-c46b438cb7da\") " Sep 09 02:56:44.283749 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:44.283745 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k6n2\" (UniqueName: \"kubernetes.io/projected/10755e29-715e-4164-954e-01a8df1f49f6-kube-api-access-9k6n2\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:56:44.283822 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:44.283755 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10755e29-715e-4164-954e-01a8df1f49f6-plugins-config-volume\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:56:44.283951 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:44.283930 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4420f778-2b7a-4020-984d-c46b438cb7da-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4420f778-2b7a-4020-984d-c46b438cb7da" (UID: "4420f778-2b7a-4020-984d-c46b438cb7da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:56:44.285808 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:44.285783 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4420f778-2b7a-4020-984d-c46b438cb7da-kube-api-access-q6hfw" (OuterVolumeSpecName: "kube-api-access-q6hfw") pod "4420f778-2b7a-4020-984d-c46b438cb7da" (UID: "4420f778-2b7a-4020-984d-c46b438cb7da"). InnerVolumeSpecName "kube-api-access-q6hfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:56:44.384454 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:44.384370 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4420f778-2b7a-4020-984d-c46b438cb7da-plugins-config-volume\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:56:44.384454 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:44.384396 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6hfw\" (UniqueName: \"kubernetes.io/projected/4420f778-2b7a-4020-984d-c46b438cb7da-kube-api-access-q6hfw\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:56:44.460020 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:44.459982 2577 generic.go:356] "Generic (PLEG): container finished" podID="4420f778-2b7a-4020-984d-c46b438cb7da" containerID="7c35d08f92894d67bd54eb1e553e266f4bad34bc57e453d1ec242ef6299b9c97" exitCode=0 Sep 09 02:56:44.460218 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:44.460032 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4bd452-5fc8b9d9f4-mrw7j" Sep 09 02:56:44.460218 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:44.460086 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4bd452-5fc8b9d9f4-mrw7j" event={"ID":"4420f778-2b7a-4020-984d-c46b438cb7da","Type":"ContainerDied","Data":"7c35d08f92894d67bd54eb1e553e266f4bad34bc57e453d1ec242ef6299b9c97"} Sep 09 02:56:44.460218 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:44.460171 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4bd452-5fc8b9d9f4-mrw7j" event={"ID":"4420f778-2b7a-4020-984d-c46b438cb7da","Type":"ContainerDied","Data":"aa51375e049c89a211fe089c5a8f9f5c26d2abea49266bc879ec8568db689ce1"} Sep 09 02:56:44.460411 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:44.460218 2577 scope.go:117] "RemoveContainer" containerID="7c35d08f92894d67bd54eb1e553e266f4bad34bc57e453d1ec242ef6299b9c97" Sep 09 02:56:44.461779 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:44.461755 2577 generic.go:356] "Generic (PLEG): container finished" podID="10755e29-715e-4164-954e-01a8df1f49f6" containerID="61b31e012c0d57034399d99a95c36d22e984c43dcfb2bc1943c63661c2388994" exitCode=0 Sep 09 02:56:44.461920 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:44.461802 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4bd452-67fc6fdccf-9k2h5" event={"ID":"10755e29-715e-4164-954e-01a8df1f49f6","Type":"ContainerDied","Data":"61b31e012c0d57034399d99a95c36d22e984c43dcfb2bc1943c63661c2388994"} Sep 09 02:56:44.461920 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:44.461808 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4bd452-67fc6fdccf-9k2h5" Sep 09 02:56:44.461920 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:44.461838 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4bd452-67fc6fdccf-9k2h5" event={"ID":"10755e29-715e-4164-954e-01a8df1f49f6","Type":"ContainerDied","Data":"cc596ae35a3bea2d0f7376346e31eedd4be20bc541480d604e785dbcdcb90811"} Sep 09 02:56:44.472419 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:44.472209 2577 scope.go:117] "RemoveContainer" containerID="7c35d08f92894d67bd54eb1e553e266f4bad34bc57e453d1ec242ef6299b9c97" Sep 09 02:56:44.472644 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:56:44.472621 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c35d08f92894d67bd54eb1e553e266f4bad34bc57e453d1ec242ef6299b9c97\": container with ID starting with 7c35d08f92894d67bd54eb1e553e266f4bad34bc57e453d1ec242ef6299b9c97 not found: ID does not exist" containerID="7c35d08f92894d67bd54eb1e553e266f4bad34bc57e453d1ec242ef6299b9c97" Sep 09 02:56:44.472732 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:44.472656 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c35d08f92894d67bd54eb1e553e266f4bad34bc57e453d1ec242ef6299b9c97"} err="failed to get container status \"7c35d08f92894d67bd54eb1e553e266f4bad34bc57e453d1ec242ef6299b9c97\": rpc error: code = NotFound desc = could not find container \"7c35d08f92894d67bd54eb1e553e266f4bad34bc57e453d1ec242ef6299b9c97\": container with ID starting with 7c35d08f92894d67bd54eb1e553e266f4bad34bc57e453d1ec242ef6299b9c97 not found: ID does not exist" Sep 09 02:56:44.472732 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:44.472678 2577 scope.go:117] "RemoveContainer" containerID="61b31e012c0d57034399d99a95c36d22e984c43dcfb2bc1943c63661c2388994" Sep 09 02:56:44.481990 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:44.481943 2577 scope.go:117] "RemoveContainer" containerID="61b31e012c0d57034399d99a95c36d22e984c43dcfb2bc1943c63661c2388994" Sep 09 02:56:44.482260 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:56:44.482241 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61b31e012c0d57034399d99a95c36d22e984c43dcfb2bc1943c63661c2388994\": container with ID starting with 61b31e012c0d57034399d99a95c36d22e984c43dcfb2bc1943c63661c2388994 not found: ID does not exist" containerID="61b31e012c0d57034399d99a95c36d22e984c43dcfb2bc1943c63661c2388994" Sep 09 02:56:44.482323 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:44.482275 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61b31e012c0d57034399d99a95c36d22e984c43dcfb2bc1943c63661c2388994"} err="failed to get container status \"61b31e012c0d57034399d99a95c36d22e984c43dcfb2bc1943c63661c2388994\": rpc error: code = NotFound desc = could not find container \"61b31e012c0d57034399d99a95c36d22e984c43dcfb2bc1943c63661c2388994\": container with ID starting with 61b31e012c0d57034399d99a95c36d22e984c43dcfb2bc1943c63661c2388994 not found: ID does not exist" Sep 09 02:56:44.488558 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:44.488529 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4bd452-67fc6fdccf-9k2h5"] Sep 09 02:56:44.493072 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:44.493051 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4bd452-67fc6fdccf-9k2h5"] Sep 09 02:56:44.503099 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:44.503078 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4bd452-5fc8b9d9f4-mrw7j"] Sep 09 02:56:44.506843 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:44.506823 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4bd452-5fc8b9d9f4-mrw7j"] Sep 09 02:56:45.166120 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:45.166039 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10755e29-715e-4164-954e-01a8df1f49f6" path="/var/lib/kubelet/pods/10755e29-715e-4164-954e-01a8df1f49f6/volumes" Sep 09 02:56:45.166540 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:45.166502 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4420f778-2b7a-4020-984d-c46b438cb7da" path="/var/lib/kubelet/pods/4420f778-2b7a-4020-984d-c46b438cb7da/volumes" Sep 09 02:56:45.830889 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:45.830851 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2085ee-86d8f69df8-kz7mw"] Sep 09 02:56:45.831340 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:45.831325 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10755e29-715e-4164-954e-01a8df1f49f6" containerName="payload-processing" Sep 09 02:56:45.831390 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:45.831343 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="10755e29-715e-4164-954e-01a8df1f49f6" containerName="payload-processing" Sep 09 02:56:45.831390 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:45.831356 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbc0dc84-9c56-4945-b183-9584c5460f4c" containerName="payload-pre-processing" Sep 09 02:56:45.831390 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:45.831361 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbc0dc84-9c56-4945-b183-9584c5460f4c" containerName="payload-pre-processing" Sep 09 02:56:45.831390 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:45.831369 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0009f07-dc0b-4d12-a57d-b8d47015052b" containerName="payload-processing" Sep 09 02:56:45.831390 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:45.831375 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0009f07-dc0b-4d12-a57d-b8d47015052b" containerName="payload-processing" Sep 09 02:56:45.831390 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:45.831386 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4420f778-2b7a-4020-984d-c46b438cb7da" containerName="payload-pre-processing" Sep 09 02:56:45.831390 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:45.831391 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4420f778-2b7a-4020-984d-c46b438cb7da" containerName="payload-pre-processing" Sep 09 02:56:45.831602 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:45.831453 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbc0dc84-9c56-4945-b183-9584c5460f4c" containerName="payload-pre-processing" Sep 09 02:56:45.831602 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:45.831460 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0009f07-dc0b-4d12-a57d-b8d47015052b" containerName="payload-processing" Sep 09 02:56:45.831602 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:45.831466 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="10755e29-715e-4164-954e-01a8df1f49f6" containerName="payload-processing" Sep 09 02:56:45.831602 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:45.831472 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4420f778-2b7a-4020-984d-c46b438cb7da" containerName="payload-pre-processing" Sep 09 02:56:45.836129 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:45.836107 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2085ee-86d8f69df8-kz7mw" Sep 09 02:56:45.838885 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:45.838862 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2085ee\"" Sep 09 02:56:45.839030 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:45.838868 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2085ee-dockercfg-ctbz5\"" Sep 09 02:56:45.845619 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:45.845596 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2085ee-86d8f69df8-kz7mw"] Sep 09 02:56:45.867564 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:45.867517 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2085ee-68b7dcc7f-xwct8"] Sep 09 02:56:45.872559 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:45.872533 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2085ee-68b7dcc7f-xwct8" Sep 09 02:56:45.879916 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:45.879886 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2085ee-68b7dcc7f-xwct8"] Sep 09 02:56:45.899657 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:45.899622 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a18c4ba4-d4ec-4478-982f-6172e0dd566d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2085ee-86d8f69df8-kz7mw\" (UID: \"a18c4ba4-d4ec-4478-982f-6172e0dd566d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2085ee-86d8f69df8-kz7mw" Sep 09 02:56:45.899657 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:45.899665 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bjgj\" (UniqueName: \"kubernetes.io/projected/a18c4ba4-d4ec-4478-982f-6172e0dd566d-kube-api-access-2bjgj\") pod \"payload-pre-processing-e2e-ipp-a-2085ee-86d8f69df8-kz7mw\" (UID: \"a18c4ba4-d4ec-4478-982f-6172e0dd566d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2085ee-86d8f69df8-kz7mw" Sep 09 02:56:45.899884 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:45.899747 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpxkc\" (UniqueName: \"kubernetes.io/projected/a1d99fd6-805c-42b5-aa61-2e64742a4b15-kube-api-access-xpxkc\") pod \"payload-processing-e2e-ipp-a-2085ee-68b7dcc7f-xwct8\" (UID: \"a1d99fd6-805c-42b5-aa61-2e64742a4b15\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2085ee-68b7dcc7f-xwct8" Sep 09 02:56:45.899884 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:45.899769 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1d99fd6-805c-42b5-aa61-2e64742a4b15-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2085ee-68b7dcc7f-xwct8\" (UID: \"a1d99fd6-805c-42b5-aa61-2e64742a4b15\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2085ee-68b7dcc7f-xwct8" Sep 09 02:56:46.001285 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:46.001247 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpxkc\" (UniqueName: \"kubernetes.io/projected/a1d99fd6-805c-42b5-aa61-2e64742a4b15-kube-api-access-xpxkc\") pod \"payload-processing-e2e-ipp-a-2085ee-68b7dcc7f-xwct8\" (UID: \"a1d99fd6-805c-42b5-aa61-2e64742a4b15\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2085ee-68b7dcc7f-xwct8" Sep 09 02:56:46.001285 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:46.001287 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1d99fd6-805c-42b5-aa61-2e64742a4b15-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2085ee-68b7dcc7f-xwct8\" (UID: \"a1d99fd6-805c-42b5-aa61-2e64742a4b15\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2085ee-68b7dcc7f-xwct8" Sep 09 02:56:46.001503 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:46.001330 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a18c4ba4-d4ec-4478-982f-6172e0dd566d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2085ee-86d8f69df8-kz7mw\" (UID: \"a18c4ba4-d4ec-4478-982f-6172e0dd566d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2085ee-86d8f69df8-kz7mw" Sep 09 02:56:46.001503 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:46.001349 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bjgj\" (UniqueName: \"kubernetes.io/projected/a18c4ba4-d4ec-4478-982f-6172e0dd566d-kube-api-access-2bjgj\") pod \"payload-pre-processing-e2e-ipp-a-2085ee-86d8f69df8-kz7mw\" (UID: \"a18c4ba4-d4ec-4478-982f-6172e0dd566d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2085ee-86d8f69df8-kz7mw" Sep 09 02:56:46.002025 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:46.002007 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a18c4ba4-d4ec-4478-982f-6172e0dd566d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2085ee-86d8f69df8-kz7mw\" (UID: \"a18c4ba4-d4ec-4478-982f-6172e0dd566d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2085ee-86d8f69df8-kz7mw" Sep 09 02:56:46.002129 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:46.002109 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1d99fd6-805c-42b5-aa61-2e64742a4b15-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2085ee-68b7dcc7f-xwct8\" (UID: \"a1d99fd6-805c-42b5-aa61-2e64742a4b15\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2085ee-68b7dcc7f-xwct8" Sep 09 02:56:46.012365 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:46.012342 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpxkc\" (UniqueName: \"kubernetes.io/projected/a1d99fd6-805c-42b5-aa61-2e64742a4b15-kube-api-access-xpxkc\") pod \"payload-processing-e2e-ipp-a-2085ee-68b7dcc7f-xwct8\" (UID: \"a1d99fd6-805c-42b5-aa61-2e64742a4b15\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2085ee-68b7dcc7f-xwct8" Sep 09 02:56:46.012609 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:46.012587 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bjgj\" (UniqueName: \"kubernetes.io/projected/a18c4ba4-d4ec-4478-982f-6172e0dd566d-kube-api-access-2bjgj\") pod \"payload-pre-processing-e2e-ipp-a-2085ee-86d8f69df8-kz7mw\" (UID: \"a18c4ba4-d4ec-4478-982f-6172e0dd566d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2085ee-86d8f69df8-kz7mw" Sep 09 02:56:46.148427 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:46.148339 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2085ee-86d8f69df8-kz7mw" Sep 09 02:56:46.183784 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:46.183756 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2085ee-68b7dcc7f-xwct8" Sep 09 02:56:46.425290 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:46.425259 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2085ee-86d8f69df8-kz7mw"] Sep 09 02:56:46.425921 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:56:46.425898 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda18c4ba4_d4ec_4478_982f_6172e0dd566d.slice/crio-2fd1feceff18b6af3d0f1060106599030324c70a6d2f4c0332334d525f23bc27 WatchSource:0}: Error finding container 2fd1feceff18b6af3d0f1060106599030324c70a6d2f4c0332334d525f23bc27: Status 404 returned error can't find the container with id 2fd1feceff18b6af3d0f1060106599030324c70a6d2f4c0332334d525f23bc27 Sep 09 02:56:46.464699 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:46.464673 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2085ee-68b7dcc7f-xwct8"] Sep 09 02:56:46.466737 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:56:46.466705 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1d99fd6_805c_42b5_aa61_2e64742a4b15.slice/crio-f9dcbc3d4220092155d12cff51fca7d2d0dc21ea88092d158382cf22920cd26e WatchSource:0}: Error finding container f9dcbc3d4220092155d12cff51fca7d2d0dc21ea88092d158382cf22920cd26e: Status 404 returned error can't find the container with id f9dcbc3d4220092155d12cff51fca7d2d0dc21ea88092d158382cf22920cd26e Sep 09 02:56:46.475562 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:46.475536 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2085ee-86d8f69df8-kz7mw" event={"ID":"a18c4ba4-d4ec-4478-982f-6172e0dd566d","Type":"ContainerStarted","Data":"2fd1feceff18b6af3d0f1060106599030324c70a6d2f4c0332334d525f23bc27"} Sep 09 02:56:46.476764 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:46.476740 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2085ee-68b7dcc7f-xwct8" event={"ID":"a1d99fd6-805c-42b5-aa61-2e64742a4b15","Type":"ContainerStarted","Data":"f9dcbc3d4220092155d12cff51fca7d2d0dc21ea88092d158382cf22920cd26e"} Sep 09 02:56:47.485626 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:47.485591 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2085ee-86d8f69df8-kz7mw" event={"ID":"a18c4ba4-d4ec-4478-982f-6172e0dd566d","Type":"ContainerStarted","Data":"9d3c41f31c9dee38486205e8f62d8e8e903b2af6714502f127b50f83d18db6b2"} Sep 09 02:56:47.486149 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:47.485679 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2085ee-86d8f69df8-kz7mw" Sep 09 02:56:47.487612 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:47.487586 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2085ee-68b7dcc7f-xwct8" event={"ID":"a1d99fd6-805c-42b5-aa61-2e64742a4b15","Type":"ContainerStarted","Data":"f8c64c582eaa5ece535c5d9076ee2bdbec1546e94f93b6d34e51b279a3f59ced"} Sep 09 02:56:47.487749 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:47.487661 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2085ee-68b7dcc7f-xwct8" Sep 09 02:56:47.503504 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:47.503454 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2085ee-86d8f69df8-kz7mw" podStartSLOduration=2.503436065 podStartE2EDuration="2.503436065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:56:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:56:47.500573002 +0000 UTC m=+1600.864907464" watchObservedRunningTime="2026-09-09 02:56:47.503436065 +0000 UTC m=+1600.867770529" Sep 09 02:56:47.517102 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:47.517050 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-2085ee-68b7dcc7f-xwct8" podStartSLOduration=2.5170383530000002 podStartE2EDuration="2.517038353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:56:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:56:47.515343137 +0000 UTC m=+1600.879677601" watchObservedRunningTime="2026-09-09 02:56:47.517038353 +0000 UTC m=+1600.881372816" Sep 09 02:56:50.919486 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:50.919449 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d838aa-smslk"] Sep 09 02:56:50.922685 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:50.922668 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d838aa-smslk" Sep 09 02:56:50.925146 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:50.925127 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 02:56:50.925247 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:50.925212 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbr52\"" Sep 09 02:56:50.929930 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:50.929905 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d838aa-smslk"] Sep 09 02:56:50.947822 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:50.947796 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/223c4b50-1717-4cb9-9c47-3914b8a1bb77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d838aa-smslk\" (UID: \"223c4b50-1717-4cb9-9c47-3914b8a1bb77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d838aa-smslk" Sep 09 02:56:51.049079 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:51.049042 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/223c4b50-1717-4cb9-9c47-3914b8a1bb77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d838aa-smslk\" (UID: \"223c4b50-1717-4cb9-9c47-3914b8a1bb77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d838aa-smslk" Sep 09 02:56:51.049679 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:51.049657 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/223c4b50-1717-4cb9-9c47-3914b8a1bb77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d838aa-smslk\" (UID: \"223c4b50-1717-4cb9-9c47-3914b8a1bb77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d838aa-smslk" Sep 09 02:56:51.235101 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:51.235024 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d838aa-smslk" Sep 09 02:56:51.517838 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:51.517692 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d838aa-smslk"] Sep 09 02:56:51.521501 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:56:51.521476 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod223c4b50_1717_4cb9_9c47_3914b8a1bb77.slice/crio-ee48469a01d11e1c502f0bddeb83bad86b37685d1cc3285af5d331547eb61268 WatchSource:0}: Error finding container ee48469a01d11e1c502f0bddeb83bad86b37685d1cc3285af5d331547eb61268: Status 404 returned error can't find the container with id ee48469a01d11e1c502f0bddeb83bad86b37685d1cc3285af5d331547eb61268 Sep 09 02:56:52.524760 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:52.524727 2577 generic.go:356] "Generic (PLEG): container finished" podID="223c4b50-1717-4cb9-9c47-3914b8a1bb77" containerID="1f953bc3e2a00d0cecfb9bc86c92bc3d61f3b6f477a9eb4c5c5232f30ec76234" exitCode=0 Sep 09 02:56:52.525159 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:52.524819 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d838aa-smslk" event={"ID":"223c4b50-1717-4cb9-9c47-3914b8a1bb77","Type":"ContainerDied","Data":"1f953bc3e2a00d0cecfb9bc86c92bc3d61f3b6f477a9eb4c5c5232f30ec76234"} Sep 09 02:56:52.525159 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:52.524878 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d838aa-smslk" event={"ID":"223c4b50-1717-4cb9-9c47-3914b8a1bb77","Type":"ContainerStarted","Data":"ee48469a01d11e1c502f0bddeb83bad86b37685d1cc3285af5d331547eb61268"} Sep 09 02:56:53.670700 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:53.670674 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d838aa-smslk" Sep 09 02:56:53.776899 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:53.776859 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/223c4b50-1717-4cb9-9c47-3914b8a1bb77-maas-api-service-ca\") pod \"223c4b50-1717-4cb9-9c47-3914b8a1bb77\" (UID: \"223c4b50-1717-4cb9-9c47-3914b8a1bb77\") " Sep 09 02:56:53.777273 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:53.777252 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/223c4b50-1717-4cb9-9c47-3914b8a1bb77-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "223c4b50-1717-4cb9-9c47-3914b8a1bb77" (UID: "223c4b50-1717-4cb9-9c47-3914b8a1bb77"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:56:53.878131 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:53.878040 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/223c4b50-1717-4cb9-9c47-3914b8a1bb77-maas-api-service-ca\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:56:54.536573 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:54.536540 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d838aa-smslk" Sep 09 02:56:54.536765 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:54.536540 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d838aa-smslk" event={"ID":"223c4b50-1717-4cb9-9c47-3914b8a1bb77","Type":"ContainerDied","Data":"ee48469a01d11e1c502f0bddeb83bad86b37685d1cc3285af5d331547eb61268"} Sep 09 02:56:54.536765 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:54.536674 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee48469a01d11e1c502f0bddeb83bad86b37685d1cc3285af5d331547eb61268" Sep 09 02:56:58.495648 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:58.495553 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2085ee-68b7dcc7f-xwct8" Sep 09 02:56:58.496056 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:56:58.495883 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2085ee-86d8f69df8-kz7mw" Sep 09 02:57:01.420123 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:01.420094 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d838aa-6898647b87-dwfj4"] Sep 09 02:57:01.420549 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:01.420286 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d838aa-6898647b87-dwfj4" podUID="3a8826d2-acbe-4142-9e21-2079a3aa7ce9" containerName="payload-processing" containerID="cri-o://adce36075512f5a5a96306707bb5d3010da07a3fc7372711f5987342f59bbd12" gracePeriod=30 Sep 09 02:57:01.430052 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:01.429151 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d838aa-8698746cd6-wkqmc"] Sep 09 02:57:01.430052 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:01.429384 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d838aa-8698746cd6-wkqmc" podUID="3818b516-1d11-47ec-816d-8851bb3035da" containerName="payload-pre-processing" containerID="cri-o://3f9575f15c07f100a5eab6966a3374f7213e845ab79adaffefb3f7bb77b58dd5" gracePeriod=30 Sep 09 02:57:01.578362 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:01.578326 2577 generic.go:356] "Generic (PLEG): container finished" podID="3a8826d2-acbe-4142-9e21-2079a3aa7ce9" containerID="adce36075512f5a5a96306707bb5d3010da07a3fc7372711f5987342f59bbd12" exitCode=0 Sep 09 02:57:01.578535 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:01.578395 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d838aa-6898647b87-dwfj4" event={"ID":"3a8826d2-acbe-4142-9e21-2079a3aa7ce9","Type":"ContainerDied","Data":"adce36075512f5a5a96306707bb5d3010da07a3fc7372711f5987342f59bbd12"} Sep 09 02:57:01.580219 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:01.580197 2577 generic.go:356] "Generic (PLEG): container finished" podID="3818b516-1d11-47ec-816d-8851bb3035da" containerID="3f9575f15c07f100a5eab6966a3374f7213e845ab79adaffefb3f7bb77b58dd5" exitCode=0 Sep 09 02:57:01.580332 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:01.580263 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d838aa-8698746cd6-wkqmc" event={"ID":"3818b516-1d11-47ec-816d-8851bb3035da","Type":"ContainerDied","Data":"3f9575f15c07f100a5eab6966a3374f7213e845ab79adaffefb3f7bb77b58dd5"} Sep 09 02:57:01.592270 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:01.592237 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d838aa-smslk"] Sep 09 02:57:01.596866 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:01.596833 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d838aa-smslk"] Sep 09 02:57:01.686591 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:01.686559 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d838aa-6898647b87-dwfj4" Sep 09 02:57:01.715230 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:01.715207 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d838aa-8698746cd6-wkqmc" Sep 09 02:57:01.751815 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:01.751786 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ls54\" (UniqueName: \"kubernetes.io/projected/3a8826d2-acbe-4142-9e21-2079a3aa7ce9-kube-api-access-4ls54\") pod \"3a8826d2-acbe-4142-9e21-2079a3aa7ce9\" (UID: \"3a8826d2-acbe-4142-9e21-2079a3aa7ce9\") " Sep 09 02:57:01.751998 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:01.751845 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a8826d2-acbe-4142-9e21-2079a3aa7ce9-plugins-config-volume\") pod \"3a8826d2-acbe-4142-9e21-2079a3aa7ce9\" (UID: \"3a8826d2-acbe-4142-9e21-2079a3aa7ce9\") " Sep 09 02:57:01.752176 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:01.752153 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a8826d2-acbe-4142-9e21-2079a3aa7ce9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3a8826d2-acbe-4142-9e21-2079a3aa7ce9" (UID: "3a8826d2-acbe-4142-9e21-2079a3aa7ce9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:57:01.754242 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:01.754136 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a8826d2-acbe-4142-9e21-2079a3aa7ce9-kube-api-access-4ls54" (OuterVolumeSpecName: "kube-api-access-4ls54") pod "3a8826d2-acbe-4142-9e21-2079a3aa7ce9" (UID: "3a8826d2-acbe-4142-9e21-2079a3aa7ce9"). InnerVolumeSpecName "kube-api-access-4ls54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:57:01.853413 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:01.853376 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59nh2\" (UniqueName: \"kubernetes.io/projected/3818b516-1d11-47ec-816d-8851bb3035da-kube-api-access-59nh2\") pod \"3818b516-1d11-47ec-816d-8851bb3035da\" (UID: \"3818b516-1d11-47ec-816d-8851bb3035da\") " Sep 09 02:57:01.853567 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:01.853448 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3818b516-1d11-47ec-816d-8851bb3035da-plugins-config-volume\") pod \"3818b516-1d11-47ec-816d-8851bb3035da\" (UID: \"3818b516-1d11-47ec-816d-8851bb3035da\") " Sep 09 02:57:01.853626 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:01.853588 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ls54\" (UniqueName: \"kubernetes.io/projected/3a8826d2-acbe-4142-9e21-2079a3aa7ce9-kube-api-access-4ls54\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:57:01.853626 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:01.853602 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a8826d2-acbe-4142-9e21-2079a3aa7ce9-plugins-config-volume\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:57:01.853817 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:01.853786 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3818b516-1d11-47ec-816d-8851bb3035da-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3818b516-1d11-47ec-816d-8851bb3035da" (UID: "3818b516-1d11-47ec-816d-8851bb3035da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:57:01.855619 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:01.855598 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3818b516-1d11-47ec-816d-8851bb3035da-kube-api-access-59nh2" (OuterVolumeSpecName: "kube-api-access-59nh2") pod "3818b516-1d11-47ec-816d-8851bb3035da" (UID: "3818b516-1d11-47ec-816d-8851bb3035da"). InnerVolumeSpecName "kube-api-access-59nh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:57:01.954878 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:01.954788 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59nh2\" (UniqueName: \"kubernetes.io/projected/3818b516-1d11-47ec-816d-8851bb3035da-kube-api-access-59nh2\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:57:01.954878 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:01.954818 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3818b516-1d11-47ec-816d-8851bb3035da-plugins-config-volume\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:57:02.587361 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:02.587326 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d838aa-6898647b87-dwfj4" Sep 09 02:57:02.587361 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:02.587335 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d838aa-6898647b87-dwfj4" event={"ID":"3a8826d2-acbe-4142-9e21-2079a3aa7ce9","Type":"ContainerDied","Data":"05d7819eb23b4688933804088f3b72249431ceecf5df39bee9d5ece4e684c907"} Sep 09 02:57:02.587896 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:02.587402 2577 scope.go:117] "RemoveContainer" containerID="adce36075512f5a5a96306707bb5d3010da07a3fc7372711f5987342f59bbd12" Sep 09 02:57:02.588947 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:02.588834 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d838aa-8698746cd6-wkqmc" Sep 09 02:57:02.588947 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:02.588844 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d838aa-8698746cd6-wkqmc" event={"ID":"3818b516-1d11-47ec-816d-8851bb3035da","Type":"ContainerDied","Data":"aa166e803b9429a50efb434a17ad148aadc0281fd2b59175ae5c89d8f94c7381"} Sep 09 02:57:02.598129 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:02.598111 2577 scope.go:117] "RemoveContainer" containerID="3f9575f15c07f100a5eab6966a3374f7213e845ab79adaffefb3f7bb77b58dd5" Sep 09 02:57:02.612571 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:02.612549 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d838aa-6898647b87-dwfj4"] Sep 09 02:57:02.616584 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:02.616563 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d838aa-6898647b87-dwfj4"] Sep 09 02:57:02.626577 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:02.626556 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d838aa-8698746cd6-wkqmc"] Sep 09 02:57:02.630816 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:02.630785 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d838aa-8698746cd6-wkqmc"] Sep 09 02:57:03.167868 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:03.167837 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="223c4b50-1717-4cb9-9c47-3914b8a1bb77" path="/var/lib/kubelet/pods/223c4b50-1717-4cb9-9c47-3914b8a1bb77/volumes" Sep 09 02:57:03.168216 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:03.168203 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3818b516-1d11-47ec-816d-8851bb3035da" path="/var/lib/kubelet/pods/3818b516-1d11-47ec-816d-8851bb3035da/volumes" Sep 09 02:57:03.168501 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:03.168490 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a8826d2-acbe-4142-9e21-2079a3aa7ce9" path="/var/lib/kubelet/pods/3a8826d2-acbe-4142-9e21-2079a3aa7ce9/volumes" Sep 09 02:57:15.034657 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:15.034624 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-621ac9-68fb78f5c4-gz4mt"] Sep 09 02:57:15.035214 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:15.035194 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a8826d2-acbe-4142-9e21-2079a3aa7ce9" containerName="payload-processing" Sep 09 02:57:15.035309 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:15.035243 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a8826d2-acbe-4142-9e21-2079a3aa7ce9" containerName="payload-processing" Sep 09 02:57:15.035309 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:15.035256 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="223c4b50-1717-4cb9-9c47-3914b8a1bb77" containerName="revoke-keys" Sep 09 02:57:15.035309 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:15.035264 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="223c4b50-1717-4cb9-9c47-3914b8a1bb77" containerName="revoke-keys" Sep 09 02:57:15.035460 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:15.035309 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3818b516-1d11-47ec-816d-8851bb3035da" containerName="payload-pre-processing" Sep 09 02:57:15.035460 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:15.035320 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3818b516-1d11-47ec-816d-8851bb3035da" containerName="payload-pre-processing" Sep 09 02:57:15.035460 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:15.035420 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3818b516-1d11-47ec-816d-8851bb3035da" containerName="payload-pre-processing" Sep 09 02:57:15.035460 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:15.035436 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a8826d2-acbe-4142-9e21-2079a3aa7ce9" containerName="payload-processing" Sep 09 02:57:15.035460 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:15.035446 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="223c4b50-1717-4cb9-9c47-3914b8a1bb77" containerName="revoke-keys" Sep 09 02:57:15.039685 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:15.039666 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-621ac9-68fb78f5c4-gz4mt" Sep 09 02:57:15.042492 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:15.042475 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-621ac9-dockercfg-5tff2\"" Sep 09 02:57:15.042690 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:15.042664 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-621ac9\"" Sep 09 02:57:15.043482 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:15.043461 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-621ac9-769b7bc69f-mchtb"] Sep 09 02:57:15.047145 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:15.047125 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-621ac9-769b7bc69f-mchtb" Sep 09 02:57:15.047254 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:15.047196 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-621ac9-68fb78f5c4-gz4mt"] Sep 09 02:57:15.059484 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:15.059459 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-621ac9-769b7bc69f-mchtb"] Sep 09 02:57:15.165376 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:15.165338 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49d7c237-7630-4da3-bf72-97ef3e3ffaef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-621ac9-769b7bc69f-mchtb\" (UID: \"49d7c237-7630-4da3-bf72-97ef3e3ffaef\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-621ac9-769b7bc69f-mchtb" Sep 09 02:57:15.165546 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:15.165388 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwtg4\" (UniqueName: \"kubernetes.io/projected/a0ac0b5b-fe97-4589-9601-4af081c2c668-kube-api-access-fwtg4\") pod \"payload-pre-processing-e2e-ipp-b-621ac9-68fb78f5c4-gz4mt\" (UID: \"a0ac0b5b-fe97-4589-9601-4af081c2c668\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-621ac9-68fb78f5c4-gz4mt" Sep 09 02:57:15.165546 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:15.165420 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0ac0b5b-fe97-4589-9601-4af081c2c668-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-621ac9-68fb78f5c4-gz4mt\" (UID: \"a0ac0b5b-fe97-4589-9601-4af081c2c668\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-621ac9-68fb78f5c4-gz4mt" Sep 09 02:57:15.165546 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:15.165513 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqrth\" (UniqueName: \"kubernetes.io/projected/49d7c237-7630-4da3-bf72-97ef3e3ffaef-kube-api-access-pqrth\") pod \"payload-processing-e2e-ipp-b-621ac9-769b7bc69f-mchtb\" (UID: \"49d7c237-7630-4da3-bf72-97ef3e3ffaef\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-621ac9-769b7bc69f-mchtb" Sep 09 02:57:15.266568 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:15.266534 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqrth\" (UniqueName: \"kubernetes.io/projected/49d7c237-7630-4da3-bf72-97ef3e3ffaef-kube-api-access-pqrth\") pod \"payload-processing-e2e-ipp-b-621ac9-769b7bc69f-mchtb\" (UID: \"49d7c237-7630-4da3-bf72-97ef3e3ffaef\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-621ac9-769b7bc69f-mchtb" Sep 09 02:57:15.266764 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:15.266587 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49d7c237-7630-4da3-bf72-97ef3e3ffaef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-621ac9-769b7bc69f-mchtb\" (UID: \"49d7c237-7630-4da3-bf72-97ef3e3ffaef\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-621ac9-769b7bc69f-mchtb" Sep 09 02:57:15.266764 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:15.266700 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwtg4\" (UniqueName: \"kubernetes.io/projected/a0ac0b5b-fe97-4589-9601-4af081c2c668-kube-api-access-fwtg4\") pod \"payload-pre-processing-e2e-ipp-b-621ac9-68fb78f5c4-gz4mt\" (UID: \"a0ac0b5b-fe97-4589-9601-4af081c2c668\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-621ac9-68fb78f5c4-gz4mt" Sep 09 02:57:15.266764 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:15.266747 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0ac0b5b-fe97-4589-9601-4af081c2c668-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-621ac9-68fb78f5c4-gz4mt\" (UID: \"a0ac0b5b-fe97-4589-9601-4af081c2c668\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-621ac9-68fb78f5c4-gz4mt" Sep 09 02:57:15.267271 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:15.267255 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49d7c237-7630-4da3-bf72-97ef3e3ffaef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-621ac9-769b7bc69f-mchtb\" (UID: \"49d7c237-7630-4da3-bf72-97ef3e3ffaef\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-621ac9-769b7bc69f-mchtb" Sep 09 02:57:15.267418 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:15.267402 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0ac0b5b-fe97-4589-9601-4af081c2c668-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-621ac9-68fb78f5c4-gz4mt\" (UID: \"a0ac0b5b-fe97-4589-9601-4af081c2c668\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-621ac9-68fb78f5c4-gz4mt" Sep 09 02:57:15.278384 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:15.278360 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwtg4\" (UniqueName: \"kubernetes.io/projected/a0ac0b5b-fe97-4589-9601-4af081c2c668-kube-api-access-fwtg4\") pod \"payload-pre-processing-e2e-ipp-b-621ac9-68fb78f5c4-gz4mt\" (UID: \"a0ac0b5b-fe97-4589-9601-4af081c2c668\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-621ac9-68fb78f5c4-gz4mt" Sep 09 02:57:15.278384 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:15.278361 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqrth\" (UniqueName: \"kubernetes.io/projected/49d7c237-7630-4da3-bf72-97ef3e3ffaef-kube-api-access-pqrth\") pod \"payload-processing-e2e-ipp-b-621ac9-769b7bc69f-mchtb\" (UID: \"49d7c237-7630-4da3-bf72-97ef3e3ffaef\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-621ac9-769b7bc69f-mchtb" Sep 09 02:57:15.351327 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:15.351260 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-621ac9-68fb78f5c4-gz4mt" Sep 09 02:57:15.361939 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:15.361914 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-621ac9-769b7bc69f-mchtb" Sep 09 02:57:15.630469 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:15.630443 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-621ac9-68fb78f5c4-gz4mt"] Sep 09 02:57:15.631572 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:57:15.631547 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0ac0b5b_fe97_4589_9601_4af081c2c668.slice/crio-42926e3ebe117231c559b8803be8b91f52a7af601d4164653dc7f4bec83fa7d0 WatchSource:0}: Error finding container 42926e3ebe117231c559b8803be8b91f52a7af601d4164653dc7f4bec83fa7d0: Status 404 returned error can't find the container with id 42926e3ebe117231c559b8803be8b91f52a7af601d4164653dc7f4bec83fa7d0 Sep 09 02:57:15.638781 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:15.637804 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-621ac9-769b7bc69f-mchtb"] Sep 09 02:57:15.640819 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:57:15.640791 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49d7c237_7630_4da3_bf72_97ef3e3ffaef.slice/crio-86ad87fa4b015ac4f7098ab965cec99ef0292e4d7ca7ea695b6aa9f8ebc691bd WatchSource:0}: Error finding container 86ad87fa4b015ac4f7098ab965cec99ef0292e4d7ca7ea695b6aa9f8ebc691bd: Status 404 returned error can't find the container with id 86ad87fa4b015ac4f7098ab965cec99ef0292e4d7ca7ea695b6aa9f8ebc691bd Sep 09 02:57:15.663649 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:15.663625 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-621ac9-68fb78f5c4-gz4mt" event={"ID":"a0ac0b5b-fe97-4589-9601-4af081c2c668","Type":"ContainerStarted","Data":"42926e3ebe117231c559b8803be8b91f52a7af601d4164653dc7f4bec83fa7d0"} Sep 09 02:57:15.664823 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:15.664801 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-621ac9-769b7bc69f-mchtb" event={"ID":"49d7c237-7630-4da3-bf72-97ef3e3ffaef","Type":"ContainerStarted","Data":"86ad87fa4b015ac4f7098ab965cec99ef0292e4d7ca7ea695b6aa9f8ebc691bd"} Sep 09 02:57:16.672977 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:16.672931 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-621ac9-68fb78f5c4-gz4mt" event={"ID":"a0ac0b5b-fe97-4589-9601-4af081c2c668","Type":"ContainerStarted","Data":"79e49dfcde2440afc5a838cf190b3cb99406ba6b8189d02854a7e755e25c3e93"} Sep 09 02:57:16.673535 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:16.673014 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-621ac9-68fb78f5c4-gz4mt" Sep 09 02:57:16.674629 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:16.674608 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-621ac9-769b7bc69f-mchtb" event={"ID":"49d7c237-7630-4da3-bf72-97ef3e3ffaef","Type":"ContainerStarted","Data":"08d4e8c9ccc8cbbda314a111767109d75b6fe69781dc532d9bdb68d711a36de9"} Sep 09 02:57:16.674743 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:16.674723 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-621ac9-769b7bc69f-mchtb" Sep 09 02:57:16.689933 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:16.689868 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-621ac9-68fb78f5c4-gz4mt" podStartSLOduration=1.689850603 podStartE2EDuration="1.689850603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:57:16.689095716 +0000 UTC m=+1630.053430181" watchObservedRunningTime="2026-09-09 02:57:16.689850603 +0000 UTC m=+1630.054185071" Sep 09 02:57:16.704909 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:16.704865 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-621ac9-769b7bc69f-mchtb" podStartSLOduration=1.7048484560000001 podStartE2EDuration="1.704848456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:57:16.704825534 +0000 UTC m=+1630.069160024" watchObservedRunningTime="2026-09-09 02:57:16.704848456 +0000 UTC m=+1630.069182940" Sep 09 02:57:19.464610 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:19.464575 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f5aebb-85db48d8f-679bs"] Sep 09 02:57:19.465045 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:19.464822 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f5aebb-85db48d8f-679bs" podUID="1dd3e673-b63e-465d-889b-25fca919f4d2" containerName="payload-processing" containerID="cri-o://903553863a47e69dc1b1d4215604e24b1970f9da516273971b4f6d23f75f30c6" gracePeriod=30 Sep 09 02:57:19.471164 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:19.471118 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f5aebb-589bb94786-2lghl"] Sep 09 02:57:19.471362 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:19.471338 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f5aebb-589bb94786-2lghl" podUID="9784127c-8779-4c1a-aae0-b811d4e76751" containerName="payload-pre-processing" containerID="cri-o://e0dbfa41fb647df3c609f59f05598a4cc525153f9fe43be9611998bdd798bc0c" gracePeriod=30 Sep 09 02:57:19.695752 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:19.695710 2577 generic.go:356] "Generic (PLEG): container finished" podID="1dd3e673-b63e-465d-889b-25fca919f4d2" containerID="903553863a47e69dc1b1d4215604e24b1970f9da516273971b4f6d23f75f30c6" exitCode=0 Sep 09 02:57:19.695915 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:19.695800 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f5aebb-85db48d8f-679bs" event={"ID":"1dd3e673-b63e-465d-889b-25fca919f4d2","Type":"ContainerDied","Data":"903553863a47e69dc1b1d4215604e24b1970f9da516273971b4f6d23f75f30c6"} Sep 09 02:57:19.698336 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:19.698295 2577 generic.go:356] "Generic (PLEG): container finished" podID="9784127c-8779-4c1a-aae0-b811d4e76751" containerID="e0dbfa41fb647df3c609f59f05598a4cc525153f9fe43be9611998bdd798bc0c" exitCode=0 Sep 09 02:57:19.698469 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:19.698383 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f5aebb-589bb94786-2lghl" event={"ID":"9784127c-8779-4c1a-aae0-b811d4e76751","Type":"ContainerDied","Data":"e0dbfa41fb647df3c609f59f05598a4cc525153f9fe43be9611998bdd798bc0c"} Sep 09 02:57:19.730738 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:19.730712 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f5aebb-85db48d8f-679bs" Sep 09 02:57:19.762813 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:19.762789 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f5aebb-589bb94786-2lghl" Sep 09 02:57:19.904901 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:19.904869 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9784127c-8779-4c1a-aae0-b811d4e76751-plugins-config-volume\") pod \"9784127c-8779-4c1a-aae0-b811d4e76751\" (UID: \"9784127c-8779-4c1a-aae0-b811d4e76751\") " Sep 09 02:57:19.905091 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:19.904918 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m52sp\" (UniqueName: \"kubernetes.io/projected/1dd3e673-b63e-465d-889b-25fca919f4d2-kube-api-access-m52sp\") pod \"1dd3e673-b63e-465d-889b-25fca919f4d2\" (UID: \"1dd3e673-b63e-465d-889b-25fca919f4d2\") " Sep 09 02:57:19.905091 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:19.905026 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dd3e673-b63e-465d-889b-25fca919f4d2-plugins-config-volume\") pod \"1dd3e673-b63e-465d-889b-25fca919f4d2\" (UID: \"1dd3e673-b63e-465d-889b-25fca919f4d2\") " Sep 09 02:57:19.905091 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:19.905046 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28nrf\" (UniqueName: \"kubernetes.io/projected/9784127c-8779-4c1a-aae0-b811d4e76751-kube-api-access-28nrf\") pod \"9784127c-8779-4c1a-aae0-b811d4e76751\" (UID: \"9784127c-8779-4c1a-aae0-b811d4e76751\") " Sep 09 02:57:19.905341 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:19.905308 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9784127c-8779-4c1a-aae0-b811d4e76751-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9784127c-8779-4c1a-aae0-b811d4e76751" (UID: "9784127c-8779-4c1a-aae0-b811d4e76751"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:57:19.905462 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:19.905434 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dd3e673-b63e-465d-889b-25fca919f4d2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1dd3e673-b63e-465d-889b-25fca919f4d2" (UID: "1dd3e673-b63e-465d-889b-25fca919f4d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:57:19.907352 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:19.907332 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9784127c-8779-4c1a-aae0-b811d4e76751-kube-api-access-28nrf" (OuterVolumeSpecName: "kube-api-access-28nrf") pod "9784127c-8779-4c1a-aae0-b811d4e76751" (UID: "9784127c-8779-4c1a-aae0-b811d4e76751"). InnerVolumeSpecName "kube-api-access-28nrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:57:19.907432 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:19.907357 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dd3e673-b63e-465d-889b-25fca919f4d2-kube-api-access-m52sp" (OuterVolumeSpecName: "kube-api-access-m52sp") pod "1dd3e673-b63e-465d-889b-25fca919f4d2" (UID: "1dd3e673-b63e-465d-889b-25fca919f4d2"). InnerVolumeSpecName "kube-api-access-m52sp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:57:20.006660 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:20.006564 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m52sp\" (UniqueName: \"kubernetes.io/projected/1dd3e673-b63e-465d-889b-25fca919f4d2-kube-api-access-m52sp\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:57:20.006660 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:20.006598 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dd3e673-b63e-465d-889b-25fca919f4d2-plugins-config-volume\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:57:20.006660 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:20.006611 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28nrf\" (UniqueName: \"kubernetes.io/projected/9784127c-8779-4c1a-aae0-b811d4e76751-kube-api-access-28nrf\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:57:20.006660 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:20.006623 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9784127c-8779-4c1a-aae0-b811d4e76751-plugins-config-volume\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:57:20.705518 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:20.705476 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f5aebb-85db48d8f-679bs" event={"ID":"1dd3e673-b63e-465d-889b-25fca919f4d2","Type":"ContainerDied","Data":"8b113a52042d234c86588f5fa8eacc7ab2cf52ff64dcc952263b3985f14287a7"} Sep 09 02:57:20.706012 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:20.705562 2577 scope.go:117] "RemoveContainer" containerID="903553863a47e69dc1b1d4215604e24b1970f9da516273971b4f6d23f75f30c6" Sep 09 02:57:20.706012 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:20.705501 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f5aebb-85db48d8f-679bs" Sep 09 02:57:20.707043 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:20.706993 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f5aebb-589bb94786-2lghl" event={"ID":"9784127c-8779-4c1a-aae0-b811d4e76751","Type":"ContainerDied","Data":"a0df75e6df9c07dee9e70e332575e5c24955d044ea3ae6dae3c3c6446b2c0781"} Sep 09 02:57:20.707043 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:20.707014 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f5aebb-589bb94786-2lghl" Sep 09 02:57:20.716058 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:20.716036 2577 scope.go:117] "RemoveContainer" containerID="e0dbfa41fb647df3c609f59f05598a4cc525153f9fe43be9611998bdd798bc0c" Sep 09 02:57:20.731212 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:20.731141 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f5aebb-85db48d8f-679bs"] Sep 09 02:57:20.734761 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:20.734737 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f5aebb-85db48d8f-679bs"] Sep 09 02:57:20.745335 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:20.745312 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f5aebb-589bb94786-2lghl"] Sep 09 02:57:20.747555 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:20.747535 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f5aebb-589bb94786-2lghl"] Sep 09 02:57:21.165848 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:21.165817 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dd3e673-b63e-465d-889b-25fca919f4d2" path="/var/lib/kubelet/pods/1dd3e673-b63e-465d-889b-25fca919f4d2/volumes" Sep 09 02:57:21.166207 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:21.166192 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9784127c-8779-4c1a-aae0-b811d4e76751" path="/var/lib/kubelet/pods/9784127c-8779-4c1a-aae0-b811d4e76751/volumes" Sep 09 02:57:27.683588 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:27.683556 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-621ac9-68fb78f5c4-gz4mt" Sep 09 02:57:27.684017 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:27.683714 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-621ac9-769b7bc69f-mchtb" Sep 09 02:57:43.581138 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:43.581110 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-qd7xc_04c796d2-3ea0-4a3c-9635-4a5d5ef41cbe/payload-processing/0.log" Sep 09 02:57:43.717358 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:57:43.717330 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-621ac9-769b7bc69f-mchtb_49d7c237-7630-4da3-bf72-97ef3e3ffaef/payload-processing/0.log" Sep 09 02:58:09.383022 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:09.382991 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-2085ee-68b7dcc7f-xwct8_a1d99fd6-805c-42b5-aa61-2e64742a4b15/payload-processing/0.log" Sep 09 02:58:09.521781 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:09.521751 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-qd7xc_04c796d2-3ea0-4a3c-9635-4a5d5ef41cbe/payload-processing/0.log" Sep 09 02:58:09.646414 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:09.646334 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-621ac9-769b7bc69f-mchtb_49d7c237-7630-4da3-bf72-97ef3e3ffaef/payload-processing/0.log" Sep 09 02:58:15.257509 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:15.257479 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b8247-76b9f9458b-7tk7b"] Sep 09 02:58:15.264249 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:15.264222 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9784127c-8779-4c1a-aae0-b811d4e76751" containerName="payload-pre-processing" Sep 09 02:58:15.264249 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:15.264245 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9784127c-8779-4c1a-aae0-b811d4e76751" containerName="payload-pre-processing" Sep 09 02:58:15.264445 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:15.264258 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dd3e673-b63e-465d-889b-25fca919f4d2" containerName="payload-processing" Sep 09 02:58:15.264445 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:15.264264 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dd3e673-b63e-465d-889b-25fca919f4d2" containerName="payload-processing" Sep 09 02:58:15.264524 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:15.264450 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dd3e673-b63e-465d-889b-25fca919f4d2" containerName="payload-processing" Sep 09 02:58:15.264524 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:15.264463 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9784127c-8779-4c1a-aae0-b811d4e76751" containerName="payload-pre-processing" Sep 09 02:58:15.267302 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:15.267284 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b8247-76b9f9458b-7tk7b" Sep 09 02:58:15.270301 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:15.270272 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-2b8247-dockercfg-97s5m\"" Sep 09 02:58:15.270423 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:15.270277 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-2b8247\"" Sep 09 02:58:15.272737 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:15.272713 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b8247-76b9f9458b-7tk7b"] Sep 09 02:58:15.291300 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:15.291276 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2b8247-9dc7f6545-pzpzt"] Sep 09 02:58:15.294766 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:15.294749 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2b8247-9dc7f6545-pzpzt" Sep 09 02:58:15.307702 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:15.307680 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2b8247-9dc7f6545-pzpzt"] Sep 09 02:58:15.382766 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:15.382736 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb4mk\" (UniqueName: \"kubernetes.io/projected/c4e61628-e599-43eb-b378-291fa9a8749d-kube-api-access-pb4mk\") pod \"payload-pre-processing-e2e-ipp-cleanup-2b8247-76b9f9458b-7tk7b\" (UID: \"c4e61628-e599-43eb-b378-291fa9a8749d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b8247-76b9f9458b-7tk7b" Sep 09 02:58:15.382934 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:15.382770 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74ef7607-3caa-4369-bf9d-5ce56bcc6a8e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-2b8247-9dc7f6545-pzpzt\" (UID: \"74ef7607-3caa-4369-bf9d-5ce56bcc6a8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2b8247-9dc7f6545-pzpzt" Sep 09 02:58:15.382934 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:15.382797 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z27rh\" (UniqueName: \"kubernetes.io/projected/74ef7607-3caa-4369-bf9d-5ce56bcc6a8e-kube-api-access-z27rh\") pod \"payload-processing-e2e-ipp-cleanup-2b8247-9dc7f6545-pzpzt\" (UID: \"74ef7607-3caa-4369-bf9d-5ce56bcc6a8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2b8247-9dc7f6545-pzpzt" Sep 09 02:58:15.382934 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:15.382881 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e61628-e599-43eb-b378-291fa9a8749d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-2b8247-76b9f9458b-7tk7b\" (UID: \"c4e61628-e599-43eb-b378-291fa9a8749d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b8247-76b9f9458b-7tk7b" Sep 09 02:58:15.483739 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:15.483707 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb4mk\" (UniqueName: \"kubernetes.io/projected/c4e61628-e599-43eb-b378-291fa9a8749d-kube-api-access-pb4mk\") pod \"payload-pre-processing-e2e-ipp-cleanup-2b8247-76b9f9458b-7tk7b\" (UID: \"c4e61628-e599-43eb-b378-291fa9a8749d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b8247-76b9f9458b-7tk7b" Sep 09 02:58:15.483917 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:15.483743 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74ef7607-3caa-4369-bf9d-5ce56bcc6a8e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-2b8247-9dc7f6545-pzpzt\" (UID: \"74ef7607-3caa-4369-bf9d-5ce56bcc6a8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2b8247-9dc7f6545-pzpzt" Sep 09 02:58:15.483917 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:15.483769 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z27rh\" (UniqueName: \"kubernetes.io/projected/74ef7607-3caa-4369-bf9d-5ce56bcc6a8e-kube-api-access-z27rh\") pod \"payload-processing-e2e-ipp-cleanup-2b8247-9dc7f6545-pzpzt\" (UID: \"74ef7607-3caa-4369-bf9d-5ce56bcc6a8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2b8247-9dc7f6545-pzpzt" Sep 09 02:58:15.483917 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:15.483789 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e61628-e599-43eb-b378-291fa9a8749d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-2b8247-76b9f9458b-7tk7b\" (UID: \"c4e61628-e599-43eb-b378-291fa9a8749d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b8247-76b9f9458b-7tk7b" Sep 09 02:58:15.484492 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:15.484472 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e61628-e599-43eb-b378-291fa9a8749d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-2b8247-76b9f9458b-7tk7b\" (UID: \"c4e61628-e599-43eb-b378-291fa9a8749d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b8247-76b9f9458b-7tk7b" Sep 09 02:58:15.484492 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:15.484485 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74ef7607-3caa-4369-bf9d-5ce56bcc6a8e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-2b8247-9dc7f6545-pzpzt\" (UID: \"74ef7607-3caa-4369-bf9d-5ce56bcc6a8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2b8247-9dc7f6545-pzpzt" Sep 09 02:58:15.492467 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:15.492435 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb4mk\" (UniqueName: \"kubernetes.io/projected/c4e61628-e599-43eb-b378-291fa9a8749d-kube-api-access-pb4mk\") pod \"payload-pre-processing-e2e-ipp-cleanup-2b8247-76b9f9458b-7tk7b\" (UID: \"c4e61628-e599-43eb-b378-291fa9a8749d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b8247-76b9f9458b-7tk7b" Sep 09 02:58:15.492579 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:15.492528 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z27rh\" (UniqueName: \"kubernetes.io/projected/74ef7607-3caa-4369-bf9d-5ce56bcc6a8e-kube-api-access-z27rh\") pod \"payload-processing-e2e-ipp-cleanup-2b8247-9dc7f6545-pzpzt\" (UID: \"74ef7607-3caa-4369-bf9d-5ce56bcc6a8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2b8247-9dc7f6545-pzpzt" Sep 09 02:58:15.579324 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:15.579301 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b8247-76b9f9458b-7tk7b" Sep 09 02:58:15.605807 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:15.605782 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2b8247-9dc7f6545-pzpzt" Sep 09 02:58:15.856443 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:15.856418 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b8247-76b9f9458b-7tk7b"] Sep 09 02:58:15.858543 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:58:15.858520 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4e61628_e599_43eb_b378_291fa9a8749d.slice/crio-691cc66a81f30436490aab5cff2d8ecddcb8c5f8cf2463fe909952cbd5203ca8 WatchSource:0}: Error finding container 691cc66a81f30436490aab5cff2d8ecddcb8c5f8cf2463fe909952cbd5203ca8: Status 404 returned error can't find the container with id 691cc66a81f30436490aab5cff2d8ecddcb8c5f8cf2463fe909952cbd5203ca8 Sep 09 02:58:15.892590 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:58:15.892339 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74ef7607_3caa_4369_bf9d_5ce56bcc6a8e.slice/crio-6706d85b18f06905eb95f45759ead62fe3d867538b8ee066c2b88cdfec36ce6d WatchSource:0}: Error finding container 6706d85b18f06905eb95f45759ead62fe3d867538b8ee066c2b88cdfec36ce6d: Status 404 returned error can't find the container with id 6706d85b18f06905eb95f45759ead62fe3d867538b8ee066c2b88cdfec36ce6d Sep 09 02:58:15.892854 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:15.892822 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2b8247-9dc7f6545-pzpzt"] Sep 09 02:58:16.019248 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:16.019215 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2b8247-9dc7f6545-pzpzt" event={"ID":"74ef7607-3caa-4369-bf9d-5ce56bcc6a8e","Type":"ContainerStarted","Data":"3b6b6e50f902ce9b84629c15862f7b9fcbc7c7c7f4ab62e23c3ff7b09236bca3"} Sep 09 02:58:16.019432 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:16.019275 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2b8247-9dc7f6545-pzpzt" event={"ID":"74ef7607-3caa-4369-bf9d-5ce56bcc6a8e","Type":"ContainerStarted","Data":"6706d85b18f06905eb95f45759ead62fe3d867538b8ee066c2b88cdfec36ce6d"} Sep 09 02:58:16.019432 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:16.019379 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2b8247-9dc7f6545-pzpzt" Sep 09 02:58:16.020903 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:16.020876 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b8247-76b9f9458b-7tk7b" event={"ID":"c4e61628-e599-43eb-b378-291fa9a8749d","Type":"ContainerStarted","Data":"4a57ed2cc9915cbab5aafa13ead1a8bd6688436d1db233f327d9768194e7b69b"} Sep 09 02:58:16.021052 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:16.020923 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b8247-76b9f9458b-7tk7b" event={"ID":"c4e61628-e599-43eb-b378-291fa9a8749d","Type":"ContainerStarted","Data":"691cc66a81f30436490aab5cff2d8ecddcb8c5f8cf2463fe909952cbd5203ca8"} Sep 09 02:58:16.021052 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:16.020951 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b8247-76b9f9458b-7tk7b" Sep 09 02:58:16.038096 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:16.038054 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2b8247-9dc7f6545-pzpzt" podStartSLOduration=1.038039003 podStartE2EDuration="1.038039003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:58:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:58:16.035139589 +0000 UTC m=+1689.399474053" watchObservedRunningTime="2026-09-09 02:58:16.038039003 +0000 UTC m=+1689.402373464" Sep 09 02:58:16.051732 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:16.051690 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b8247-76b9f9458b-7tk7b" podStartSLOduration=1.051678727 podStartE2EDuration="1.051678727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:58:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:58:16.051230367 +0000 UTC m=+1689.415564831" watchObservedRunningTime="2026-09-09 02:58:16.051678727 +0000 UTC m=+1689.416013188" Sep 09 02:58:27.028779 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:27.028749 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2b8247-9dc7f6545-pzpzt" Sep 09 02:58:27.029286 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:27.028976 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b8247-76b9f9458b-7tk7b" Sep 09 02:58:45.226509 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:45.226412 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2b8247-9dc7f6545-pzpzt"] Sep 09 02:58:45.226998 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:45.226702 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2b8247-9dc7f6545-pzpzt" podUID="74ef7607-3caa-4369-bf9d-5ce56bcc6a8e" containerName="payload-processing" containerID="cri-o://3b6b6e50f902ce9b84629c15862f7b9fcbc7c7c7f4ab62e23c3ff7b09236bca3" gracePeriod=30 Sep 09 02:58:45.234057 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:45.234026 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b8247-76b9f9458b-7tk7b"] Sep 09 02:58:45.234436 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:45.234410 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b8247-76b9f9458b-7tk7b" podUID="c4e61628-e599-43eb-b378-291fa9a8749d" containerName="payload-pre-processing" containerID="cri-o://4a57ed2cc9915cbab5aafa13ead1a8bd6688436d1db233f327d9768194e7b69b" gracePeriod=30 Sep 09 02:58:45.491771 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:45.491708 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2b8247-9dc7f6545-pzpzt" Sep 09 02:58:45.517764 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:45.517742 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b8247-76b9f9458b-7tk7b" Sep 09 02:58:45.545563 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:45.545529 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74ef7607-3caa-4369-bf9d-5ce56bcc6a8e-plugins-config-volume\") pod \"74ef7607-3caa-4369-bf9d-5ce56bcc6a8e\" (UID: \"74ef7607-3caa-4369-bf9d-5ce56bcc6a8e\") " Sep 09 02:58:45.545770 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:45.545611 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z27rh\" (UniqueName: \"kubernetes.io/projected/74ef7607-3caa-4369-bf9d-5ce56bcc6a8e-kube-api-access-z27rh\") pod \"74ef7607-3caa-4369-bf9d-5ce56bcc6a8e\" (UID: \"74ef7607-3caa-4369-bf9d-5ce56bcc6a8e\") " Sep 09 02:58:45.545990 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:45.545946 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74ef7607-3caa-4369-bf9d-5ce56bcc6a8e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "74ef7607-3caa-4369-bf9d-5ce56bcc6a8e" (UID: "74ef7607-3caa-4369-bf9d-5ce56bcc6a8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:58:45.547982 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:45.547926 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74ef7607-3caa-4369-bf9d-5ce56bcc6a8e-kube-api-access-z27rh" (OuterVolumeSpecName: "kube-api-access-z27rh") pod "74ef7607-3caa-4369-bf9d-5ce56bcc6a8e" (UID: "74ef7607-3caa-4369-bf9d-5ce56bcc6a8e"). InnerVolumeSpecName "kube-api-access-z27rh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:58:45.647083 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:45.647053 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pb4mk\" (UniqueName: \"kubernetes.io/projected/c4e61628-e599-43eb-b378-291fa9a8749d-kube-api-access-pb4mk\") pod \"c4e61628-e599-43eb-b378-291fa9a8749d\" (UID: \"c4e61628-e599-43eb-b378-291fa9a8749d\") " Sep 09 02:58:45.647272 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:45.647098 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e61628-e599-43eb-b378-291fa9a8749d-plugins-config-volume\") pod \"c4e61628-e599-43eb-b378-291fa9a8749d\" (UID: \"c4e61628-e599-43eb-b378-291fa9a8749d\") " Sep 09 02:58:45.647272 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:45.647256 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74ef7607-3caa-4369-bf9d-5ce56bcc6a8e-plugins-config-volume\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:58:45.647272 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:45.647266 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z27rh\" (UniqueName: \"kubernetes.io/projected/74ef7607-3caa-4369-bf9d-5ce56bcc6a8e-kube-api-access-z27rh\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:58:45.647501 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:45.647478 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4e61628-e599-43eb-b378-291fa9a8749d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c4e61628-e599-43eb-b378-291fa9a8749d" (UID: "c4e61628-e599-43eb-b378-291fa9a8749d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:58:45.649304 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:45.649282 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4e61628-e599-43eb-b378-291fa9a8749d-kube-api-access-pb4mk" (OuterVolumeSpecName: "kube-api-access-pb4mk") pod "c4e61628-e599-43eb-b378-291fa9a8749d" (UID: "c4e61628-e599-43eb-b378-291fa9a8749d"). InnerVolumeSpecName "kube-api-access-pb4mk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:58:45.747940 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:45.747865 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb4mk\" (UniqueName: \"kubernetes.io/projected/c4e61628-e599-43eb-b378-291fa9a8749d-kube-api-access-pb4mk\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:58:45.747940 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:45.747894 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e61628-e599-43eb-b378-291fa9a8749d-plugins-config-volume\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:58:46.194561 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:46.194518 2577 generic.go:356] "Generic (PLEG): container finished" podID="74ef7607-3caa-4369-bf9d-5ce56bcc6a8e" containerID="3b6b6e50f902ce9b84629c15862f7b9fcbc7c7c7f4ab62e23c3ff7b09236bca3" exitCode=0 Sep 09 02:58:46.194561 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:46.194565 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2b8247-9dc7f6545-pzpzt" Sep 09 02:58:46.194823 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:46.194607 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2b8247-9dc7f6545-pzpzt" event={"ID":"74ef7607-3caa-4369-bf9d-5ce56bcc6a8e","Type":"ContainerDied","Data":"3b6b6e50f902ce9b84629c15862f7b9fcbc7c7c7f4ab62e23c3ff7b09236bca3"} Sep 09 02:58:46.194823 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:46.194669 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2b8247-9dc7f6545-pzpzt" event={"ID":"74ef7607-3caa-4369-bf9d-5ce56bcc6a8e","Type":"ContainerDied","Data":"6706d85b18f06905eb95f45759ead62fe3d867538b8ee066c2b88cdfec36ce6d"} Sep 09 02:58:46.194823 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:46.194700 2577 scope.go:117] "RemoveContainer" containerID="3b6b6e50f902ce9b84629c15862f7b9fcbc7c7c7f4ab62e23c3ff7b09236bca3" Sep 09 02:58:46.196087 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:46.196065 2577 generic.go:356] "Generic (PLEG): container finished" podID="c4e61628-e599-43eb-b378-291fa9a8749d" containerID="4a57ed2cc9915cbab5aafa13ead1a8bd6688436d1db233f327d9768194e7b69b" exitCode=0 Sep 09 02:58:46.196213 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:46.196096 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b8247-76b9f9458b-7tk7b" event={"ID":"c4e61628-e599-43eb-b378-291fa9a8749d","Type":"ContainerDied","Data":"4a57ed2cc9915cbab5aafa13ead1a8bd6688436d1db233f327d9768194e7b69b"} Sep 09 02:58:46.196213 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:46.196119 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b8247-76b9f9458b-7tk7b" Sep 09 02:58:46.196213 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:46.196146 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b8247-76b9f9458b-7tk7b" event={"ID":"c4e61628-e599-43eb-b378-291fa9a8749d","Type":"ContainerDied","Data":"691cc66a81f30436490aab5cff2d8ecddcb8c5f8cf2463fe909952cbd5203ca8"} Sep 09 02:58:46.205357 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:46.205335 2577 scope.go:117] "RemoveContainer" containerID="3b6b6e50f902ce9b84629c15862f7b9fcbc7c7c7f4ab62e23c3ff7b09236bca3" Sep 09 02:58:46.205692 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:58:46.205665 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b6b6e50f902ce9b84629c15862f7b9fcbc7c7c7f4ab62e23c3ff7b09236bca3\": container with ID starting with 3b6b6e50f902ce9b84629c15862f7b9fcbc7c7c7f4ab62e23c3ff7b09236bca3 not found: ID does not exist" containerID="3b6b6e50f902ce9b84629c15862f7b9fcbc7c7c7f4ab62e23c3ff7b09236bca3" Sep 09 02:58:46.205805 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:46.205705 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b6b6e50f902ce9b84629c15862f7b9fcbc7c7c7f4ab62e23c3ff7b09236bca3"} err="failed to get container status \"3b6b6e50f902ce9b84629c15862f7b9fcbc7c7c7f4ab62e23c3ff7b09236bca3\": rpc error: code = NotFound desc = could not find container \"3b6b6e50f902ce9b84629c15862f7b9fcbc7c7c7f4ab62e23c3ff7b09236bca3\": container with ID starting with 3b6b6e50f902ce9b84629c15862f7b9fcbc7c7c7f4ab62e23c3ff7b09236bca3 not found: ID does not exist" Sep 09 02:58:46.205805 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:46.205727 2577 scope.go:117] "RemoveContainer" containerID="4a57ed2cc9915cbab5aafa13ead1a8bd6688436d1db233f327d9768194e7b69b" Sep 09 02:58:46.214086 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:46.214068 2577 scope.go:117] "RemoveContainer" containerID="4a57ed2cc9915cbab5aafa13ead1a8bd6688436d1db233f327d9768194e7b69b" Sep 09 02:58:46.214337 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:58:46.214320 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a57ed2cc9915cbab5aafa13ead1a8bd6688436d1db233f327d9768194e7b69b\": container with ID starting with 4a57ed2cc9915cbab5aafa13ead1a8bd6688436d1db233f327d9768194e7b69b not found: ID does not exist" containerID="4a57ed2cc9915cbab5aafa13ead1a8bd6688436d1db233f327d9768194e7b69b" Sep 09 02:58:46.214378 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:46.214346 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a57ed2cc9915cbab5aafa13ead1a8bd6688436d1db233f327d9768194e7b69b"} err="failed to get container status \"4a57ed2cc9915cbab5aafa13ead1a8bd6688436d1db233f327d9768194e7b69b\": rpc error: code = NotFound desc = could not find container \"4a57ed2cc9915cbab5aafa13ead1a8bd6688436d1db233f327d9768194e7b69b\": container with ID starting with 4a57ed2cc9915cbab5aafa13ead1a8bd6688436d1db233f327d9768194e7b69b not found: ID does not exist" Sep 09 02:58:46.221046 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:46.221024 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b8247-76b9f9458b-7tk7b"] Sep 09 02:58:46.224724 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:46.224701 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2b8247-76b9f9458b-7tk7b"] Sep 09 02:58:46.235438 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:46.235420 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2b8247-9dc7f6545-pzpzt"] Sep 09 02:58:46.239488 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:46.239465 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2b8247-9dc7f6545-pzpzt"] Sep 09 02:58:47.166339 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:47.166306 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74ef7607-3caa-4369-bf9d-5ce56bcc6a8e" path="/var/lib/kubelet/pods/74ef7607-3caa-4369-bf9d-5ce56bcc6a8e/volumes" Sep 09 02:58:47.166655 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:47.166642 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4e61628-e599-43eb-b378-291fa9a8749d" path="/var/lib/kubelet/pods/c4e61628-e599-43eb-b378-291fa9a8749d/volumes" Sep 09 02:58:56.116604 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:56.116569 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2085ee-lhfqg"] Sep 09 02:58:56.117086 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:56.117070 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4e61628-e599-43eb-b378-291fa9a8749d" containerName="payload-pre-processing" Sep 09 02:58:56.117156 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:56.117088 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4e61628-e599-43eb-b378-291fa9a8749d" containerName="payload-pre-processing" Sep 09 02:58:56.117156 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:56.117103 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74ef7607-3caa-4369-bf9d-5ce56bcc6a8e" containerName="payload-processing" Sep 09 02:58:56.117156 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:56.117109 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="74ef7607-3caa-4369-bf9d-5ce56bcc6a8e" containerName="payload-processing" Sep 09 02:58:56.117268 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:56.117169 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="74ef7607-3caa-4369-bf9d-5ce56bcc6a8e" containerName="payload-processing" Sep 09 02:58:56.117268 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:56.117179 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4e61628-e599-43eb-b378-291fa9a8749d" containerName="payload-pre-processing" Sep 09 02:58:56.121822 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:56.121807 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2085ee-lhfqg" Sep 09 02:58:56.125512 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:56.125490 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 02:58:56.125757 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:56.125587 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbr52\"" Sep 09 02:58:56.127507 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:56.127485 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2085ee-lhfqg"] Sep 09 02:58:56.243214 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:56.243182 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a14ec798-16fb-46d0-a1f9-d8236101604c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2085ee-lhfqg\" (UID: \"a14ec798-16fb-46d0-a1f9-d8236101604c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2085ee-lhfqg" Sep 09 02:58:56.343713 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:56.343676 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a14ec798-16fb-46d0-a1f9-d8236101604c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2085ee-lhfqg\" (UID: \"a14ec798-16fb-46d0-a1f9-d8236101604c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2085ee-lhfqg" Sep 09 02:58:56.344293 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:56.344276 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a14ec798-16fb-46d0-a1f9-d8236101604c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2085ee-lhfqg\" (UID: \"a14ec798-16fb-46d0-a1f9-d8236101604c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2085ee-lhfqg" Sep 09 02:58:56.433874 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:56.433804 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2085ee-lhfqg" Sep 09 02:58:56.703691 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:56.703666 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2085ee-lhfqg"] Sep 09 02:58:56.707885 ip-10-0-136-89 kubenswrapper[2577]: W0909 02:58:56.707827 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda14ec798_16fb_46d0_a1f9_d8236101604c.slice/crio-ca5a5e9a9015a7ca4fb0f8914198156b9d15991d2b24345d81e98fb0378b3a78 WatchSource:0}: Error finding container ca5a5e9a9015a7ca4fb0f8914198156b9d15991d2b24345d81e98fb0378b3a78: Status 404 returned error can't find the container with id ca5a5e9a9015a7ca4fb0f8914198156b9d15991d2b24345d81e98fb0378b3a78 Sep 09 02:58:57.266841 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:57.266751 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2085ee-lhfqg" event={"ID":"a14ec798-16fb-46d0-a1f9-d8236101604c","Type":"ContainerStarted","Data":"a232447ff4eba4661f68c96b563254402fb4d388b22de365106ad5cd09e96727"} Sep 09 02:58:57.266841 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:57.266798 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2085ee-lhfqg" event={"ID":"a14ec798-16fb-46d0-a1f9-d8236101604c","Type":"ContainerStarted","Data":"ca5a5e9a9015a7ca4fb0f8914198156b9d15991d2b24345d81e98fb0378b3a78"} Sep 09 02:58:57.283708 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:57.283657 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2085ee-lhfqg" podStartSLOduration=1.28364191 podStartE2EDuration="1.28364191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:58:57.281715532 +0000 UTC m=+1730.646049997" watchObservedRunningTime="2026-09-09 02:58:57.28364191 +0000 UTC m=+1730.647976373" Sep 09 02:58:58.273894 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:58.273861 2577 generic.go:356] "Generic (PLEG): container finished" podID="a14ec798-16fb-46d0-a1f9-d8236101604c" containerID="a232447ff4eba4661f68c96b563254402fb4d388b22de365106ad5cd09e96727" exitCode=0 Sep 09 02:58:58.274365 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:58.273940 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2085ee-lhfqg" event={"ID":"a14ec798-16fb-46d0-a1f9-d8236101604c","Type":"ContainerDied","Data":"a232447ff4eba4661f68c96b563254402fb4d388b22de365106ad5cd09e96727"} Sep 09 02:58:59.421560 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:59.421533 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2085ee-lhfqg" Sep 09 02:58:59.572656 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:59.572566 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a14ec798-16fb-46d0-a1f9-d8236101604c-maas-api-service-ca\") pod \"a14ec798-16fb-46d0-a1f9-d8236101604c\" (UID: \"a14ec798-16fb-46d0-a1f9-d8236101604c\") " Sep 09 02:58:59.572941 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:59.572916 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a14ec798-16fb-46d0-a1f9-d8236101604c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a14ec798-16fb-46d0-a1f9-d8236101604c" (UID: "a14ec798-16fb-46d0-a1f9-d8236101604c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:58:59.673479 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:58:59.673434 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a14ec798-16fb-46d0-a1f9-d8236101604c-maas-api-service-ca\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:59:00.287672 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:59:00.287640 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2085ee-lhfqg" Sep 09 02:59:00.287672 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:59:00.287649 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2085ee-lhfqg" event={"ID":"a14ec798-16fb-46d0-a1f9-d8236101604c","Type":"ContainerDied","Data":"ca5a5e9a9015a7ca4fb0f8914198156b9d15991d2b24345d81e98fb0378b3a78"} Sep 09 02:59:00.287896 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:59:00.287706 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca5a5e9a9015a7ca4fb0f8914198156b9d15991d2b24345d81e98fb0378b3a78" Sep 09 02:59:11.643151 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:59:11.643120 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2085ee-68b7dcc7f-xwct8"] Sep 09 02:59:11.643542 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:59:11.643327 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-2085ee-68b7dcc7f-xwct8" podUID="a1d99fd6-805c-42b5-aa61-2e64742a4b15" containerName="payload-processing" containerID="cri-o://f8c64c582eaa5ece535c5d9076ee2bdbec1546e94f93b6d34e51b279a3f59ced" gracePeriod=30 Sep 09 02:59:11.650233 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:59:11.650203 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2085ee-86d8f69df8-kz7mw"] Sep 09 02:59:11.650454 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:59:11.650431 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2085ee-86d8f69df8-kz7mw" podUID="a18c4ba4-d4ec-4478-982f-6172e0dd566d" containerName="payload-pre-processing" containerID="cri-o://9d3c41f31c9dee38486205e8f62d8e8e903b2af6714502f127b50f83d18db6b2" gracePeriod=30 Sep 09 02:59:11.807584 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:59:11.807525 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2085ee-lhfqg"] Sep 09 02:59:11.811378 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:59:11.811348 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2085ee-lhfqg"] Sep 09 02:59:11.932246 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:59:11.932218 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2085ee-68b7dcc7f-xwct8" Sep 09 02:59:11.934757 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:59:11.934739 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2085ee-86d8f69df8-kz7mw" Sep 09 02:59:11.982160 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:59:11.982120 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bjgj\" (UniqueName: \"kubernetes.io/projected/a18c4ba4-d4ec-4478-982f-6172e0dd566d-kube-api-access-2bjgj\") pod \"a18c4ba4-d4ec-4478-982f-6172e0dd566d\" (UID: \"a18c4ba4-d4ec-4478-982f-6172e0dd566d\") " Sep 09 02:59:11.982355 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:59:11.982177 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a18c4ba4-d4ec-4478-982f-6172e0dd566d-plugins-config-volume\") pod \"a18c4ba4-d4ec-4478-982f-6172e0dd566d\" (UID: \"a18c4ba4-d4ec-4478-982f-6172e0dd566d\") " Sep 09 02:59:11.982355 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:59:11.982222 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xpxkc\" (UniqueName: \"kubernetes.io/projected/a1d99fd6-805c-42b5-aa61-2e64742a4b15-kube-api-access-xpxkc\") pod \"a1d99fd6-805c-42b5-aa61-2e64742a4b15\" (UID: \"a1d99fd6-805c-42b5-aa61-2e64742a4b15\") " Sep 09 02:59:11.982355 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:59:11.982261 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1d99fd6-805c-42b5-aa61-2e64742a4b15-plugins-config-volume\") pod \"a1d99fd6-805c-42b5-aa61-2e64742a4b15\" (UID: \"a1d99fd6-805c-42b5-aa61-2e64742a4b15\") " Sep 09 02:59:11.982781 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:59:11.982742 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1d99fd6-805c-42b5-aa61-2e64742a4b15-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a1d99fd6-805c-42b5-aa61-2e64742a4b15" (UID: "a1d99fd6-805c-42b5-aa61-2e64742a4b15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:59:11.982891 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:59:11.982751 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a18c4ba4-d4ec-4478-982f-6172e0dd566d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a18c4ba4-d4ec-4478-982f-6172e0dd566d" (UID: "a18c4ba4-d4ec-4478-982f-6172e0dd566d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 02:59:11.984612 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:59:11.984573 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1d99fd6-805c-42b5-aa61-2e64742a4b15-kube-api-access-xpxkc" (OuterVolumeSpecName: "kube-api-access-xpxkc") pod "a1d99fd6-805c-42b5-aa61-2e64742a4b15" (UID: "a1d99fd6-805c-42b5-aa61-2e64742a4b15"). InnerVolumeSpecName "kube-api-access-xpxkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:59:11.985045 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:59:11.985004 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a18c4ba4-d4ec-4478-982f-6172e0dd566d-kube-api-access-2bjgj" (OuterVolumeSpecName: "kube-api-access-2bjgj") pod "a18c4ba4-d4ec-4478-982f-6172e0dd566d" (UID: "a18c4ba4-d4ec-4478-982f-6172e0dd566d"). InnerVolumeSpecName "kube-api-access-2bjgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 02:59:12.083424 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:59:12.083383 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a18c4ba4-d4ec-4478-982f-6172e0dd566d-plugins-config-volume\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:59:12.083424 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:59:12.083418 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpxkc\" (UniqueName: \"kubernetes.io/projected/a1d99fd6-805c-42b5-aa61-2e64742a4b15-kube-api-access-xpxkc\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:59:12.083424 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:59:12.083427 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1d99fd6-805c-42b5-aa61-2e64742a4b15-plugins-config-volume\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:59:12.083424 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:59:12.083435 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bjgj\" (UniqueName: \"kubernetes.io/projected/a18c4ba4-d4ec-4478-982f-6172e0dd566d-kube-api-access-2bjgj\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 02:59:12.355085 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:59:12.355053 2577 generic.go:356] "Generic (PLEG): container finished" podID="a18c4ba4-d4ec-4478-982f-6172e0dd566d" containerID="9d3c41f31c9dee38486205e8f62d8e8e903b2af6714502f127b50f83d18db6b2" exitCode=0 Sep 09 02:59:12.355289 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:59:12.355097 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2085ee-86d8f69df8-kz7mw" Sep 09 02:59:12.355289 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:59:12.355138 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2085ee-86d8f69df8-kz7mw" event={"ID":"a18c4ba4-d4ec-4478-982f-6172e0dd566d","Type":"ContainerDied","Data":"9d3c41f31c9dee38486205e8f62d8e8e903b2af6714502f127b50f83d18db6b2"} Sep 09 02:59:12.355289 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:59:12.355194 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2085ee-86d8f69df8-kz7mw" event={"ID":"a18c4ba4-d4ec-4478-982f-6172e0dd566d","Type":"ContainerDied","Data":"2fd1feceff18b6af3d0f1060106599030324c70a6d2f4c0332334d525f23bc27"} Sep 09 02:59:12.355289 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:59:12.355224 2577 scope.go:117] "RemoveContainer" containerID="9d3c41f31c9dee38486205e8f62d8e8e903b2af6714502f127b50f83d18db6b2" Sep 09 02:59:12.356654 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:59:12.356628 2577 generic.go:356] "Generic (PLEG): container finished" podID="a1d99fd6-805c-42b5-aa61-2e64742a4b15" containerID="f8c64c582eaa5ece535c5d9076ee2bdbec1546e94f93b6d34e51b279a3f59ced" exitCode=0 Sep 09 02:59:12.356775 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:59:12.356660 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2085ee-68b7dcc7f-xwct8" event={"ID":"a1d99fd6-805c-42b5-aa61-2e64742a4b15","Type":"ContainerDied","Data":"f8c64c582eaa5ece535c5d9076ee2bdbec1546e94f93b6d34e51b279a3f59ced"} Sep 09 02:59:12.356775 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:59:12.356718 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2085ee-68b7dcc7f-xwct8" event={"ID":"a1d99fd6-805c-42b5-aa61-2e64742a4b15","Type":"ContainerDied","Data":"f9dcbc3d4220092155d12cff51fca7d2d0dc21ea88092d158382cf22920cd26e"} Sep 09 02:59:12.356775 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:59:12.356674 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2085ee-68b7dcc7f-xwct8" Sep 09 02:59:12.366578 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:59:12.365884 2577 scope.go:117] "RemoveContainer" containerID="9d3c41f31c9dee38486205e8f62d8e8e903b2af6714502f127b50f83d18db6b2" Sep 09 02:59:12.367322 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:59:12.367294 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d3c41f31c9dee38486205e8f62d8e8e903b2af6714502f127b50f83d18db6b2\": container with ID starting with 9d3c41f31c9dee38486205e8f62d8e8e903b2af6714502f127b50f83d18db6b2 not found: ID does not exist" containerID="9d3c41f31c9dee38486205e8f62d8e8e903b2af6714502f127b50f83d18db6b2" Sep 09 02:59:12.367429 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:59:12.367335 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d3c41f31c9dee38486205e8f62d8e8e903b2af6714502f127b50f83d18db6b2"} err="failed to get container status \"9d3c41f31c9dee38486205e8f62d8e8e903b2af6714502f127b50f83d18db6b2\": rpc error: code = NotFound desc = could not find container \"9d3c41f31c9dee38486205e8f62d8e8e903b2af6714502f127b50f83d18db6b2\": container with ID starting with 9d3c41f31c9dee38486205e8f62d8e8e903b2af6714502f127b50f83d18db6b2 not found: ID does not exist" Sep 09 02:59:12.367429 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:59:12.367357 2577 scope.go:117] "RemoveContainer" containerID="f8c64c582eaa5ece535c5d9076ee2bdbec1546e94f93b6d34e51b279a3f59ced" Sep 09 02:59:12.376454 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:59:12.376431 2577 scope.go:117] "RemoveContainer" containerID="f8c64c582eaa5ece535c5d9076ee2bdbec1546e94f93b6d34e51b279a3f59ced" Sep 09 02:59:12.376859 ip-10-0-136-89 kubenswrapper[2577]: E0909 02:59:12.376841 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8c64c582eaa5ece535c5d9076ee2bdbec1546e94f93b6d34e51b279a3f59ced\": container with ID starting with f8c64c582eaa5ece535c5d9076ee2bdbec1546e94f93b6d34e51b279a3f59ced not found: ID does not exist" containerID="f8c64c582eaa5ece535c5d9076ee2bdbec1546e94f93b6d34e51b279a3f59ced" Sep 09 02:59:12.376910 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:59:12.376869 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8c64c582eaa5ece535c5d9076ee2bdbec1546e94f93b6d34e51b279a3f59ced"} err="failed to get container status \"f8c64c582eaa5ece535c5d9076ee2bdbec1546e94f93b6d34e51b279a3f59ced\": rpc error: code = NotFound desc = could not find container \"f8c64c582eaa5ece535c5d9076ee2bdbec1546e94f93b6d34e51b279a3f59ced\": container with ID starting with f8c64c582eaa5ece535c5d9076ee2bdbec1546e94f93b6d34e51b279a3f59ced not found: ID does not exist" Sep 09 02:59:12.390141 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:59:12.390109 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2085ee-86d8f69df8-kz7mw"] Sep 09 02:59:12.395559 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:59:12.395532 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2085ee-86d8f69df8-kz7mw"] Sep 09 02:59:12.405646 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:59:12.405617 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2085ee-68b7dcc7f-xwct8"] Sep 09 02:59:12.411825 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:59:12.411798 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2085ee-68b7dcc7f-xwct8"] Sep 09 02:59:13.164765 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:59:13.164735 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a14ec798-16fb-46d0-a1f9-d8236101604c" path="/var/lib/kubelet/pods/a14ec798-16fb-46d0-a1f9-d8236101604c/volumes" Sep 09 02:59:13.165168 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:59:13.165093 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a18c4ba4-d4ec-4478-982f-6172e0dd566d" path="/var/lib/kubelet/pods/a18c4ba4-d4ec-4478-982f-6172e0dd566d/volumes" Sep 09 02:59:13.165388 ip-10-0-136-89 kubenswrapper[2577]: I0909 02:59:13.165376 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1d99fd6-805c-42b5-aa61-2e64742a4b15" path="/var/lib/kubelet/pods/a1d99fd6-805c-42b5-aa61-2e64742a4b15/volumes" Sep 09 03:00:00.145385 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:00.145293 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29815380-sd2qv"] Sep 09 03:00:00.145784 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:00.145736 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1d99fd6-805c-42b5-aa61-2e64742a4b15" containerName="payload-processing" Sep 09 03:00:00.145784 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:00.145749 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1d99fd6-805c-42b5-aa61-2e64742a4b15" containerName="payload-processing" Sep 09 03:00:00.145784 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:00.145761 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a14ec798-16fb-46d0-a1f9-d8236101604c" containerName="revoke-keys" Sep 09 03:00:00.145784 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:00.145767 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a14ec798-16fb-46d0-a1f9-d8236101604c" containerName="revoke-keys" Sep 09 03:00:00.145784 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:00.145773 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a18c4ba4-d4ec-4478-982f-6172e0dd566d" containerName="payload-pre-processing" Sep 09 03:00:00.145784 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:00.145779 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a18c4ba4-d4ec-4478-982f-6172e0dd566d" containerName="payload-pre-processing" Sep 09 03:00:00.146027 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:00.145837 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a18c4ba4-d4ec-4478-982f-6172e0dd566d" containerName="payload-pre-processing" Sep 09 03:00:00.146027 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:00.145845 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a14ec798-16fb-46d0-a1f9-d8236101604c" containerName="revoke-keys" Sep 09 03:00:00.146027 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:00.145853 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1d99fd6-805c-42b5-aa61-2e64742a4b15" containerName="payload-processing" Sep 09 03:00:00.148901 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:00.148878 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c173ad-29815380-wmzx2"] Sep 09 03:00:00.149050 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:00.149020 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815380-sd2qv" Sep 09 03:00:00.151991 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:00.151936 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbr52\"" Sep 09 03:00:00.152421 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:00.152389 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c173ad-29815380-wmzx2" Sep 09 03:00:00.162404 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:00.162273 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29815380-sd2qv"] Sep 09 03:00:00.165719 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:00.165693 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c173ad-29815380-wmzx2"] Sep 09 03:00:00.170451 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:00.166510 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815380-sd2qv" Sep 09 03:00:00.190521 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:00.190489 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c173ad-29815380-wmzx2" Sep 09 03:00:00.461626 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:00.461593 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29815380-sd2qv"] Sep 09 03:00:00.463411 ip-10-0-136-89 kubenswrapper[2577]: W0909 03:00:00.463384 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4fa990c_e418_4112_80c4_765746776796.slice/crio-f6e0757fc0b8974c12b4781e73ea11e3819469815ade9831dd13c0bf13d1577e WatchSource:0}: Error finding container f6e0757fc0b8974c12b4781e73ea11e3819469815ade9831dd13c0bf13d1577e: Status 404 returned error can't find the container with id f6e0757fc0b8974c12b4781e73ea11e3819469815ade9831dd13c0bf13d1577e Sep 09 03:00:00.487078 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:00.487052 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c173ad-29815380-wmzx2"] Sep 09 03:00:00.497158 ip-10-0-136-89 kubenswrapper[2577]: W0909 03:00:00.497127 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b10b3c5_c216_41b3_b962_e9bf3e61468a.slice/crio-40a2aa80def2a6bc65388197542b0debc9af55d697eb8ab3f1ae87800343a225 WatchSource:0}: Error finding container 40a2aa80def2a6bc65388197542b0debc9af55d697eb8ab3f1ae87800343a225: Status 404 returned error can't find the container with id 40a2aa80def2a6bc65388197542b0debc9af55d697eb8ab3f1ae87800343a225 Sep 09 03:00:00.585091 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:00.585053 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-621ac9-kj8nk"] Sep 09 03:00:00.589659 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:00.589632 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-621ac9-kj8nk" Sep 09 03:00:00.592473 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:00.592446 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 03:00:00.596892 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:00.596857 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-621ac9-kj8nk"] Sep 09 03:00:00.617419 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:00.617384 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a567ad9c-375a-4829-b86e-221f9d7c22b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-621ac9-kj8nk\" (UID: \"a567ad9c-375a-4829-b86e-221f9d7c22b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-621ac9-kj8nk" Sep 09 03:00:00.625435 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:00.625404 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815380-sd2qv" event={"ID":"f4fa990c-e418-4112-80c4-765746776796","Type":"ContainerStarted","Data":"f6e0757fc0b8974c12b4781e73ea11e3819469815ade9831dd13c0bf13d1577e"} Sep 09 03:00:00.626689 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:00.626656 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c173ad-29815380-wmzx2" event={"ID":"6b10b3c5-c216-41b3-b962-e9bf3e61468a","Type":"ContainerStarted","Data":"40a2aa80def2a6bc65388197542b0debc9af55d697eb8ab3f1ae87800343a225"} Sep 09 03:00:00.718148 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:00.718060 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a567ad9c-375a-4829-b86e-221f9d7c22b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-621ac9-kj8nk\" (UID: \"a567ad9c-375a-4829-b86e-221f9d7c22b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-621ac9-kj8nk" Sep 09 03:00:00.718752 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:00.718734 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a567ad9c-375a-4829-b86e-221f9d7c22b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-621ac9-kj8nk\" (UID: \"a567ad9c-375a-4829-b86e-221f9d7c22b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-621ac9-kj8nk" Sep 09 03:00:00.912552 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:00.912513 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-621ac9-kj8nk" Sep 09 03:00:01.182840 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:01.182812 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-621ac9-kj8nk"] Sep 09 03:00:01.184017 ip-10-0-136-89 kubenswrapper[2577]: W0909 03:00:01.183947 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda567ad9c_375a_4829_b86e_221f9d7c22b8.slice/crio-dd1f425b7665bd1ec7713d354da958873b76b50b1b674b952e6efc418ee00eb7 WatchSource:0}: Error finding container dd1f425b7665bd1ec7713d354da958873b76b50b1b674b952e6efc418ee00eb7: Status 404 returned error can't find the container with id dd1f425b7665bd1ec7713d354da958873b76b50b1b674b952e6efc418ee00eb7 Sep 09 03:00:01.513697 ip-10-0-136-89 kubenswrapper[2577]: E0909 03:00:01.513664 2577 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b10b3c5_c216_41b3_b962_e9bf3e61468a.slice/crio-35a68b61b3e01d069ec5376a107842ca9ef56f3fd996bb839f0d3c599ccd4837.scope\": RecentStats: unable to find data in memory cache]" Sep 09 03:00:01.635140 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:01.635114 2577 generic.go:356] "Generic (PLEG): container finished" podID="f4fa990c-e418-4112-80c4-765746776796" containerID="08faa6720e1ff7371c2fd9bdf5068a842608e94c889c889179ccab8b63d38d73" exitCode=0 Sep 09 03:00:01.635261 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:01.635194 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815380-sd2qv" event={"ID":"f4fa990c-e418-4112-80c4-765746776796","Type":"ContainerDied","Data":"08faa6720e1ff7371c2fd9bdf5068a842608e94c889c889179ccab8b63d38d73"} Sep 09 03:00:01.636840 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:01.636808 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-621ac9-kj8nk" event={"ID":"a567ad9c-375a-4829-b86e-221f9d7c22b8","Type":"ContainerStarted","Data":"c62fdca01e58dc9f1dd4ccfd84bdee6c82b4359473620be1029e87729198e8a5"} Sep 09 03:00:01.636997 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:01.636863 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-621ac9-kj8nk" event={"ID":"a567ad9c-375a-4829-b86e-221f9d7c22b8","Type":"ContainerStarted","Data":"dd1f425b7665bd1ec7713d354da958873b76b50b1b674b952e6efc418ee00eb7"} Sep 09 03:00:01.638404 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:01.638378 2577 generic.go:356] "Generic (PLEG): container finished" podID="6b10b3c5-c216-41b3-b962-e9bf3e61468a" containerID="35a68b61b3e01d069ec5376a107842ca9ef56f3fd996bb839f0d3c599ccd4837" exitCode=0 Sep 09 03:00:01.638503 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:01.638425 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c173ad-29815380-wmzx2" event={"ID":"6b10b3c5-c216-41b3-b962-e9bf3e61468a","Type":"ContainerDied","Data":"35a68b61b3e01d069ec5376a107842ca9ef56f3fd996bb839f0d3c599ccd4837"} Sep 09 03:00:01.681406 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:01.681335 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-621ac9-kj8nk" podStartSLOduration=1.6813199970000001 podStartE2EDuration="1.681319997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:00:01.679153052 +0000 UTC m=+1795.043487517" watchObservedRunningTime="2026-09-09 03:00:01.681319997 +0000 UTC m=+1795.045654501" Sep 09 03:00:02.652590 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:02.652558 2577 generic.go:356] "Generic (PLEG): container finished" podID="a567ad9c-375a-4829-b86e-221f9d7c22b8" containerID="c62fdca01e58dc9f1dd4ccfd84bdee6c82b4359473620be1029e87729198e8a5" exitCode=0 Sep 09 03:00:02.653039 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:02.652652 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-621ac9-kj8nk" event={"ID":"a567ad9c-375a-4829-b86e-221f9d7c22b8","Type":"ContainerDied","Data":"c62fdca01e58dc9f1dd4ccfd84bdee6c82b4359473620be1029e87729198e8a5"} Sep 09 03:00:02.804047 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:02.804022 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c173ad-29815380-wmzx2" Sep 09 03:00:02.840057 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:02.840029 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815380-sd2qv" Sep 09 03:00:03.660418 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:03.660390 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c173ad-29815380-wmzx2" Sep 09 03:00:03.660809 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:03.660431 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c173ad-29815380-wmzx2" event={"ID":"6b10b3c5-c216-41b3-b962-e9bf3e61468a","Type":"ContainerDied","Data":"40a2aa80def2a6bc65388197542b0debc9af55d697eb8ab3f1ae87800343a225"} Sep 09 03:00:03.660809 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:03.660476 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40a2aa80def2a6bc65388197542b0debc9af55d697eb8ab3f1ae87800343a225" Sep 09 03:00:03.661906 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:03.661869 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815380-sd2qv" event={"ID":"f4fa990c-e418-4112-80c4-765746776796","Type":"ContainerDied","Data":"f6e0757fc0b8974c12b4781e73ea11e3819469815ade9831dd13c0bf13d1577e"} Sep 09 03:00:03.661906 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:03.661891 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29815380-sd2qv" Sep 09 03:00:03.662137 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:03.661926 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6e0757fc0b8974c12b4781e73ea11e3819469815ade9831dd13c0bf13d1577e" Sep 09 03:00:03.795538 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:03.795514 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-621ac9-kj8nk" Sep 09 03:00:03.843010 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:03.842974 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a567ad9c-375a-4829-b86e-221f9d7c22b8-maas-api-service-ca\") pod \"a567ad9c-375a-4829-b86e-221f9d7c22b8\" (UID: \"a567ad9c-375a-4829-b86e-221f9d7c22b8\") " Sep 09 03:00:03.843301 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:03.843276 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a567ad9c-375a-4829-b86e-221f9d7c22b8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a567ad9c-375a-4829-b86e-221f9d7c22b8" (UID: "a567ad9c-375a-4829-b86e-221f9d7c22b8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 03:00:03.944419 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:03.944336 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a567ad9c-375a-4829-b86e-221f9d7c22b8-maas-api-service-ca\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 03:00:04.669330 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:04.669292 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-621ac9-kj8nk" event={"ID":"a567ad9c-375a-4829-b86e-221f9d7c22b8","Type":"ContainerDied","Data":"dd1f425b7665bd1ec7713d354da958873b76b50b1b674b952e6efc418ee00eb7"} Sep 09 03:00:04.669713 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:04.669352 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd1f425b7665bd1ec7713d354da958873b76b50b1b674b952e6efc418ee00eb7" Sep 09 03:00:04.669713 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:04.669302 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-621ac9-kj8nk" Sep 09 03:00:11.116703 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:11.116667 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-621ac9-769b7bc69f-mchtb"] Sep 09 03:00:11.117106 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:11.116911 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-621ac9-769b7bc69f-mchtb" podUID="49d7c237-7630-4da3-bf72-97ef3e3ffaef" containerName="payload-processing" containerID="cri-o://08d4e8c9ccc8cbbda314a111767109d75b6fe69781dc532d9bdb68d711a36de9" gracePeriod=30 Sep 09 03:00:11.128134 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:11.126711 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-621ac9-68fb78f5c4-gz4mt"] Sep 09 03:00:11.128134 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:11.127099 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-621ac9-68fb78f5c4-gz4mt" podUID="a0ac0b5b-fe97-4589-9601-4af081c2c668" containerName="payload-pre-processing" containerID="cri-o://79e49dfcde2440afc5a838cf190b3cb99406ba6b8189d02854a7e755e25c3e93" gracePeriod=30 Sep 09 03:00:11.336655 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:11.336616 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-621ac9-kj8nk"] Sep 09 03:00:11.351787 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:11.351738 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-621ac9-kj8nk"] Sep 09 03:00:11.416755 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:11.416731 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-621ac9-769b7bc69f-mchtb" Sep 09 03:00:11.419747 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:11.419728 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-621ac9-68fb78f5c4-gz4mt" Sep 09 03:00:11.508558 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:11.508528 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0ac0b5b-fe97-4589-9601-4af081c2c668-plugins-config-volume\") pod \"a0ac0b5b-fe97-4589-9601-4af081c2c668\" (UID: \"a0ac0b5b-fe97-4589-9601-4af081c2c668\") " Sep 09 03:00:11.508711 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:11.508578 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49d7c237-7630-4da3-bf72-97ef3e3ffaef-plugins-config-volume\") pod \"49d7c237-7630-4da3-bf72-97ef3e3ffaef\" (UID: \"49d7c237-7630-4da3-bf72-97ef3e3ffaef\") " Sep 09 03:00:11.508711 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:11.508616 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwtg4\" (UniqueName: \"kubernetes.io/projected/a0ac0b5b-fe97-4589-9601-4af081c2c668-kube-api-access-fwtg4\") pod \"a0ac0b5b-fe97-4589-9601-4af081c2c668\" (UID: \"a0ac0b5b-fe97-4589-9601-4af081c2c668\") " Sep 09 03:00:11.508711 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:11.508641 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqrth\" (UniqueName: \"kubernetes.io/projected/49d7c237-7630-4da3-bf72-97ef3e3ffaef-kube-api-access-pqrth\") pod \"49d7c237-7630-4da3-bf72-97ef3e3ffaef\" (UID: \"49d7c237-7630-4da3-bf72-97ef3e3ffaef\") " Sep 09 03:00:11.508933 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:11.508910 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0ac0b5b-fe97-4589-9601-4af081c2c668-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a0ac0b5b-fe97-4589-9601-4af081c2c668" (UID: "a0ac0b5b-fe97-4589-9601-4af081c2c668"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 03:00:11.509071 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:11.509034 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49d7c237-7630-4da3-bf72-97ef3e3ffaef-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "49d7c237-7630-4da3-bf72-97ef3e3ffaef" (UID: "49d7c237-7630-4da3-bf72-97ef3e3ffaef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 03:00:11.510886 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:11.510860 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0ac0b5b-fe97-4589-9601-4af081c2c668-kube-api-access-fwtg4" (OuterVolumeSpecName: "kube-api-access-fwtg4") pod "a0ac0b5b-fe97-4589-9601-4af081c2c668" (UID: "a0ac0b5b-fe97-4589-9601-4af081c2c668"). InnerVolumeSpecName "kube-api-access-fwtg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 03:00:11.510977 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:11.510894 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49d7c237-7630-4da3-bf72-97ef3e3ffaef-kube-api-access-pqrth" (OuterVolumeSpecName: "kube-api-access-pqrth") pod "49d7c237-7630-4da3-bf72-97ef3e3ffaef" (UID: "49d7c237-7630-4da3-bf72-97ef3e3ffaef"). InnerVolumeSpecName "kube-api-access-pqrth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 03:00:11.610189 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:11.610155 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49d7c237-7630-4da3-bf72-97ef3e3ffaef-plugins-config-volume\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 03:00:11.610189 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:11.610183 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwtg4\" (UniqueName: \"kubernetes.io/projected/a0ac0b5b-fe97-4589-9601-4af081c2c668-kube-api-access-fwtg4\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 03:00:11.610189 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:11.610192 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqrth\" (UniqueName: \"kubernetes.io/projected/49d7c237-7630-4da3-bf72-97ef3e3ffaef-kube-api-access-pqrth\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 03:00:11.610414 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:11.610200 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0ac0b5b-fe97-4589-9601-4af081c2c668-plugins-config-volume\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 03:00:11.710722 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:11.710639 2577 generic.go:356] "Generic (PLEG): container finished" podID="49d7c237-7630-4da3-bf72-97ef3e3ffaef" containerID="08d4e8c9ccc8cbbda314a111767109d75b6fe69781dc532d9bdb68d711a36de9" exitCode=0 Sep 09 03:00:11.710722 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:11.710681 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-621ac9-769b7bc69f-mchtb" Sep 09 03:00:11.710920 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:11.710723 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-621ac9-769b7bc69f-mchtb" event={"ID":"49d7c237-7630-4da3-bf72-97ef3e3ffaef","Type":"ContainerDied","Data":"08d4e8c9ccc8cbbda314a111767109d75b6fe69781dc532d9bdb68d711a36de9"} Sep 09 03:00:11.710920 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:11.710780 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-621ac9-769b7bc69f-mchtb" event={"ID":"49d7c237-7630-4da3-bf72-97ef3e3ffaef","Type":"ContainerDied","Data":"86ad87fa4b015ac4f7098ab965cec99ef0292e4d7ca7ea695b6aa9f8ebc691bd"} Sep 09 03:00:11.710920 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:11.710814 2577 scope.go:117] "RemoveContainer" containerID="08d4e8c9ccc8cbbda314a111767109d75b6fe69781dc532d9bdb68d711a36de9" Sep 09 03:00:11.712216 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:11.712188 2577 generic.go:356] "Generic (PLEG): container finished" podID="a0ac0b5b-fe97-4589-9601-4af081c2c668" containerID="79e49dfcde2440afc5a838cf190b3cb99406ba6b8189d02854a7e755e25c3e93" exitCode=0 Sep 09 03:00:11.712313 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:11.712225 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-621ac9-68fb78f5c4-gz4mt" Sep 09 03:00:11.712313 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:11.712235 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-621ac9-68fb78f5c4-gz4mt" event={"ID":"a0ac0b5b-fe97-4589-9601-4af081c2c668","Type":"ContainerDied","Data":"79e49dfcde2440afc5a838cf190b3cb99406ba6b8189d02854a7e755e25c3e93"} Sep 09 03:00:11.712313 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:11.712274 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-621ac9-68fb78f5c4-gz4mt" event={"ID":"a0ac0b5b-fe97-4589-9601-4af081c2c668","Type":"ContainerDied","Data":"42926e3ebe117231c559b8803be8b91f52a7af601d4164653dc7f4bec83fa7d0"} Sep 09 03:00:11.720551 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:11.720537 2577 scope.go:117] "RemoveContainer" containerID="08d4e8c9ccc8cbbda314a111767109d75b6fe69781dc532d9bdb68d711a36de9" Sep 09 03:00:11.720834 ip-10-0-136-89 kubenswrapper[2577]: E0909 03:00:11.720807 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08d4e8c9ccc8cbbda314a111767109d75b6fe69781dc532d9bdb68d711a36de9\": container with ID starting with 08d4e8c9ccc8cbbda314a111767109d75b6fe69781dc532d9bdb68d711a36de9 not found: ID does not exist" containerID="08d4e8c9ccc8cbbda314a111767109d75b6fe69781dc532d9bdb68d711a36de9" Sep 09 03:00:11.720925 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:11.720846 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08d4e8c9ccc8cbbda314a111767109d75b6fe69781dc532d9bdb68d711a36de9"} err="failed to get container status \"08d4e8c9ccc8cbbda314a111767109d75b6fe69781dc532d9bdb68d711a36de9\": rpc error: code = NotFound desc = could not find container \"08d4e8c9ccc8cbbda314a111767109d75b6fe69781dc532d9bdb68d711a36de9\": container with ID starting with 08d4e8c9ccc8cbbda314a111767109d75b6fe69781dc532d9bdb68d711a36de9 not found: ID does not exist" Sep 09 03:00:11.720925 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:11.720870 2577 scope.go:117] "RemoveContainer" containerID="79e49dfcde2440afc5a838cf190b3cb99406ba6b8189d02854a7e755e25c3e93" Sep 09 03:00:11.728985 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:11.728938 2577 scope.go:117] "RemoveContainer" containerID="79e49dfcde2440afc5a838cf190b3cb99406ba6b8189d02854a7e755e25c3e93" Sep 09 03:00:11.729229 ip-10-0-136-89 kubenswrapper[2577]: E0909 03:00:11.729210 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79e49dfcde2440afc5a838cf190b3cb99406ba6b8189d02854a7e755e25c3e93\": container with ID starting with 79e49dfcde2440afc5a838cf190b3cb99406ba6b8189d02854a7e755e25c3e93 not found: ID does not exist" containerID="79e49dfcde2440afc5a838cf190b3cb99406ba6b8189d02854a7e755e25c3e93" Sep 09 03:00:11.729295 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:11.729237 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79e49dfcde2440afc5a838cf190b3cb99406ba6b8189d02854a7e755e25c3e93"} err="failed to get container status \"79e49dfcde2440afc5a838cf190b3cb99406ba6b8189d02854a7e755e25c3e93\": rpc error: code = NotFound desc = could not find container \"79e49dfcde2440afc5a838cf190b3cb99406ba6b8189d02854a7e755e25c3e93\": container with ID starting with 79e49dfcde2440afc5a838cf190b3cb99406ba6b8189d02854a7e755e25c3e93 not found: ID does not exist" Sep 09 03:00:11.734683 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:11.734626 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-621ac9-769b7bc69f-mchtb"] Sep 09 03:00:11.737463 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:11.737444 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-621ac9-769b7bc69f-mchtb"] Sep 09 03:00:11.746720 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:11.746700 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-621ac9-68fb78f5c4-gz4mt"] Sep 09 03:00:11.749827 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:11.749810 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-621ac9-68fb78f5c4-gz4mt"] Sep 09 03:00:13.165141 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:13.165107 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49d7c237-7630-4da3-bf72-97ef3e3ffaef" path="/var/lib/kubelet/pods/49d7c237-7630-4da3-bf72-97ef3e3ffaef/volumes" Sep 09 03:00:13.165500 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:13.165440 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0ac0b5b-fe97-4589-9601-4af081c2c668" path="/var/lib/kubelet/pods/a0ac0b5b-fe97-4589-9601-4af081c2c668/volumes" Sep 09 03:00:13.165705 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:13.165693 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a567ad9c-375a-4829-b86e-221f9d7c22b8" path="/var/lib/kubelet/pods/a567ad9c-375a-4829-b86e-221f9d7c22b8/volumes" Sep 09 03:00:18.359460 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:18.359424 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-56dbc0-hwgtj"] Sep 09 03:00:18.359859 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:18.359785 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49d7c237-7630-4da3-bf72-97ef3e3ffaef" containerName="payload-processing" Sep 09 03:00:18.359859 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:18.359797 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="49d7c237-7630-4da3-bf72-97ef3e3ffaef" containerName="payload-processing" Sep 09 03:00:18.359859 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:18.359815 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a567ad9c-375a-4829-b86e-221f9d7c22b8" containerName="revoke-keys" Sep 09 03:00:18.359859 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:18.359822 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a567ad9c-375a-4829-b86e-221f9d7c22b8" containerName="revoke-keys" Sep 09 03:00:18.359859 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:18.359830 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0ac0b5b-fe97-4589-9601-4af081c2c668" containerName="payload-pre-processing" Sep 09 03:00:18.359859 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:18.359837 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0ac0b5b-fe97-4589-9601-4af081c2c668" containerName="payload-pre-processing" Sep 09 03:00:18.359859 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:18.359844 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b10b3c5-c216-41b3-b962-e9bf3e61468a" containerName="cleanup" Sep 09 03:00:18.359859 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:18.359850 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b10b3c5-c216-41b3-b962-e9bf3e61468a" containerName="cleanup" Sep 09 03:00:18.359859 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:18.359858 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4fa990c-e418-4112-80c4-765746776796" containerName="cleanup" Sep 09 03:00:18.359859 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:18.359865 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4fa990c-e418-4112-80c4-765746776796" containerName="cleanup" Sep 09 03:00:18.360195 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:18.359920 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4fa990c-e418-4112-80c4-765746776796" containerName="cleanup" Sep 09 03:00:18.360195 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:18.359928 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="49d7c237-7630-4da3-bf72-97ef3e3ffaef" containerName="payload-processing" Sep 09 03:00:18.360195 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:18.359934 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0ac0b5b-fe97-4589-9601-4af081c2c668" containerName="payload-pre-processing" Sep 09 03:00:18.360195 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:18.359942 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a567ad9c-375a-4829-b86e-221f9d7c22b8" containerName="revoke-keys" Sep 09 03:00:18.360195 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:18.359948 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b10b3c5-c216-41b3-b962-e9bf3e61468a" containerName="cleanup" Sep 09 03:00:18.364221 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:18.364202 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-56dbc0-hwgtj" Sep 09 03:00:18.366731 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:18.366711 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbr52\"" Sep 09 03:00:18.366849 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:18.366781 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 03:00:18.371206 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:18.371183 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-56dbc0-hwgtj"] Sep 09 03:00:18.465001 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:18.464938 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82610097-16c6-4132-9ce1-59a0abf17edd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-56dbc0-hwgtj\" (UID: \"82610097-16c6-4132-9ce1-59a0abf17edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-56dbc0-hwgtj" Sep 09 03:00:18.566434 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:18.566404 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82610097-16c6-4132-9ce1-59a0abf17edd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-56dbc0-hwgtj\" (UID: \"82610097-16c6-4132-9ce1-59a0abf17edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-56dbc0-hwgtj" Sep 09 03:00:18.567058 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:18.567037 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82610097-16c6-4132-9ce1-59a0abf17edd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-56dbc0-hwgtj\" (UID: \"82610097-16c6-4132-9ce1-59a0abf17edd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-56dbc0-hwgtj" Sep 09 03:00:18.675702 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:18.675623 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-56dbc0-hwgtj" Sep 09 03:00:18.945421 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:18.945394 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-56dbc0-hwgtj"] Sep 09 03:00:18.946682 ip-10-0-136-89 kubenswrapper[2577]: W0909 03:00:18.946646 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82610097_16c6_4132_9ce1_59a0abf17edd.slice/crio-02364bc9c0c5b0f2ed3f78cd5e1a03331f0d8456808bc078b5cc37e14b76ed9b WatchSource:0}: Error finding container 02364bc9c0c5b0f2ed3f78cd5e1a03331f0d8456808bc078b5cc37e14b76ed9b: Status 404 returned error can't find the container with id 02364bc9c0c5b0f2ed3f78cd5e1a03331f0d8456808bc078b5cc37e14b76ed9b Sep 09 03:00:19.761951 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:19.761912 2577 generic.go:356] "Generic (PLEG): container finished" podID="82610097-16c6-4132-9ce1-59a0abf17edd" containerID="9b847d836d711780156e5c59bba154b5b1fb46c61023c67122d3cb51e9a0a097" exitCode=0 Sep 09 03:00:19.762325 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:19.761992 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-56dbc0-hwgtj" event={"ID":"82610097-16c6-4132-9ce1-59a0abf17edd","Type":"ContainerDied","Data":"9b847d836d711780156e5c59bba154b5b1fb46c61023c67122d3cb51e9a0a097"} Sep 09 03:00:19.762325 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:19.762047 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-56dbc0-hwgtj" event={"ID":"82610097-16c6-4132-9ce1-59a0abf17edd","Type":"ContainerStarted","Data":"02364bc9c0c5b0f2ed3f78cd5e1a03331f0d8456808bc078b5cc37e14b76ed9b"} Sep 09 03:00:20.906035 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:20.906005 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-56dbc0-hwgtj" Sep 09 03:00:20.987018 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:20.986979 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82610097-16c6-4132-9ce1-59a0abf17edd-maas-api-service-ca\") pod \"82610097-16c6-4132-9ce1-59a0abf17edd\" (UID: \"82610097-16c6-4132-9ce1-59a0abf17edd\") " Sep 09 03:00:20.987321 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:20.987296 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82610097-16c6-4132-9ce1-59a0abf17edd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "82610097-16c6-4132-9ce1-59a0abf17edd" (UID: "82610097-16c6-4132-9ce1-59a0abf17edd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 03:00:21.088333 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:21.088264 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82610097-16c6-4132-9ce1-59a0abf17edd-maas-api-service-ca\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 03:00:21.773822 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:21.773792 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-56dbc0-hwgtj" Sep 09 03:00:21.774006 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:21.773789 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-56dbc0-hwgtj" event={"ID":"82610097-16c6-4132-9ce1-59a0abf17edd","Type":"ContainerDied","Data":"02364bc9c0c5b0f2ed3f78cd5e1a03331f0d8456808bc078b5cc37e14b76ed9b"} Sep 09 03:00:21.774006 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:21.773954 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02364bc9c0c5b0f2ed3f78cd5e1a03331f0d8456808bc078b5cc37e14b76ed9b" Sep 09 03:00:29.068304 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:29.068270 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-56dbc0-759764479f-6z97x"] Sep 09 03:00:29.068754 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:29.068501 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-56dbc0-759764479f-6z97x" podUID="5d5fa8f3-486e-477f-b44f-f91fbb4729bd" containerName="payload-processing" containerID="cri-o://e5a9bb08d4b5395908a1a6153b7420141b8af4d588b2e231443ecfb883762eae" gracePeriod=30 Sep 09 03:00:29.072498 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:29.072472 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-56dbc0-59594db65d-9smp5"] Sep 09 03:00:29.072746 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:29.072698 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-56dbc0-59594db65d-9smp5" podUID="3e293447-dee0-47d1-8868-99d0502d3dd8" containerName="payload-pre-processing" containerID="cri-o://45553ecbc1963bf970329b5acca05e240e2ac8f4496d7d7a60ba002ae4ae4f4b" gracePeriod=30 Sep 09 03:00:29.233337 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:29.233296 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-56dbc0-hwgtj"] Sep 09 03:00:29.235578 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:29.235547 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-56dbc0-hwgtj"] Sep 09 03:00:29.354360 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:29.354333 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-56dbc0-759764479f-6z97x" Sep 09 03:00:29.357100 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:29.357080 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-56dbc0-59594db65d-9smp5" Sep 09 03:00:29.462470 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:29.462430 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d5fa8f3-486e-477f-b44f-f91fbb4729bd-plugins-config-volume\") pod \"5d5fa8f3-486e-477f-b44f-f91fbb4729bd\" (UID: \"5d5fa8f3-486e-477f-b44f-f91fbb4729bd\") " Sep 09 03:00:29.462657 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:29.462479 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52tsj\" (UniqueName: \"kubernetes.io/projected/5d5fa8f3-486e-477f-b44f-f91fbb4729bd-kube-api-access-52tsj\") pod \"5d5fa8f3-486e-477f-b44f-f91fbb4729bd\" (UID: \"5d5fa8f3-486e-477f-b44f-f91fbb4729bd\") " Sep 09 03:00:29.462657 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:29.462506 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdjvb\" (UniqueName: \"kubernetes.io/projected/3e293447-dee0-47d1-8868-99d0502d3dd8-kube-api-access-wdjvb\") pod \"3e293447-dee0-47d1-8868-99d0502d3dd8\" (UID: \"3e293447-dee0-47d1-8868-99d0502d3dd8\") " Sep 09 03:00:29.462657 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:29.462537 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e293447-dee0-47d1-8868-99d0502d3dd8-plugins-config-volume\") pod \"3e293447-dee0-47d1-8868-99d0502d3dd8\" (UID: \"3e293447-dee0-47d1-8868-99d0502d3dd8\") " Sep 09 03:00:29.462932 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:29.462902 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d5fa8f3-486e-477f-b44f-f91fbb4729bd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5d5fa8f3-486e-477f-b44f-f91fbb4729bd" (UID: "5d5fa8f3-486e-477f-b44f-f91fbb4729bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 03:00:29.463039 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:29.462934 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e293447-dee0-47d1-8868-99d0502d3dd8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3e293447-dee0-47d1-8868-99d0502d3dd8" (UID: "3e293447-dee0-47d1-8868-99d0502d3dd8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 03:00:29.464757 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:29.464720 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e293447-dee0-47d1-8868-99d0502d3dd8-kube-api-access-wdjvb" (OuterVolumeSpecName: "kube-api-access-wdjvb") pod "3e293447-dee0-47d1-8868-99d0502d3dd8" (UID: "3e293447-dee0-47d1-8868-99d0502d3dd8"). InnerVolumeSpecName "kube-api-access-wdjvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 03:00:29.464851 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:29.464836 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d5fa8f3-486e-477f-b44f-f91fbb4729bd-kube-api-access-52tsj" (OuterVolumeSpecName: "kube-api-access-52tsj") pod "5d5fa8f3-486e-477f-b44f-f91fbb4729bd" (UID: "5d5fa8f3-486e-477f-b44f-f91fbb4729bd"). InnerVolumeSpecName "kube-api-access-52tsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 03:00:29.563515 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:29.563472 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d5fa8f3-486e-477f-b44f-f91fbb4729bd-plugins-config-volume\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 03:00:29.563515 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:29.563509 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52tsj\" (UniqueName: \"kubernetes.io/projected/5d5fa8f3-486e-477f-b44f-f91fbb4729bd-kube-api-access-52tsj\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 03:00:29.563515 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:29.563520 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdjvb\" (UniqueName: \"kubernetes.io/projected/3e293447-dee0-47d1-8868-99d0502d3dd8-kube-api-access-wdjvb\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 03:00:29.563760 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:29.563528 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e293447-dee0-47d1-8868-99d0502d3dd8-plugins-config-volume\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 03:00:29.820431 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:29.820395 2577 generic.go:356] "Generic (PLEG): container finished" podID="3e293447-dee0-47d1-8868-99d0502d3dd8" containerID="45553ecbc1963bf970329b5acca05e240e2ac8f4496d7d7a60ba002ae4ae4f4b" exitCode=0 Sep 09 03:00:29.820641 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:29.820451 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-56dbc0-59594db65d-9smp5" Sep 09 03:00:29.820641 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:29.820491 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-56dbc0-59594db65d-9smp5" event={"ID":"3e293447-dee0-47d1-8868-99d0502d3dd8","Type":"ContainerDied","Data":"45553ecbc1963bf970329b5acca05e240e2ac8f4496d7d7a60ba002ae4ae4f4b"} Sep 09 03:00:29.820641 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:29.820548 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-56dbc0-59594db65d-9smp5" event={"ID":"3e293447-dee0-47d1-8868-99d0502d3dd8","Type":"ContainerDied","Data":"2e42d89ff43158993a21feac95179d95a18e5ab5d44d71d031f0b676d72ecc0a"} Sep 09 03:00:29.820641 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:29.820572 2577 scope.go:117] "RemoveContainer" containerID="45553ecbc1963bf970329b5acca05e240e2ac8f4496d7d7a60ba002ae4ae4f4b" Sep 09 03:00:29.822090 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:29.822067 2577 generic.go:356] "Generic (PLEG): container finished" podID="5d5fa8f3-486e-477f-b44f-f91fbb4729bd" containerID="e5a9bb08d4b5395908a1a6153b7420141b8af4d588b2e231443ecfb883762eae" exitCode=0 Sep 09 03:00:29.822219 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:29.822145 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-56dbc0-759764479f-6z97x" event={"ID":"5d5fa8f3-486e-477f-b44f-f91fbb4729bd","Type":"ContainerDied","Data":"e5a9bb08d4b5395908a1a6153b7420141b8af4d588b2e231443ecfb883762eae"} Sep 09 03:00:29.822219 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:29.822175 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-56dbc0-759764479f-6z97x" Sep 09 03:00:29.822219 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:29.822188 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-56dbc0-759764479f-6z97x" event={"ID":"5d5fa8f3-486e-477f-b44f-f91fbb4729bd","Type":"ContainerDied","Data":"0aa93eb5f97112ff35c421d2a130bac43d33c2a795e9f27540106b29203a2973"} Sep 09 03:00:29.831626 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:29.831600 2577 scope.go:117] "RemoveContainer" containerID="45553ecbc1963bf970329b5acca05e240e2ac8f4496d7d7a60ba002ae4ae4f4b" Sep 09 03:00:29.832000 ip-10-0-136-89 kubenswrapper[2577]: E0909 03:00:29.831922 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45553ecbc1963bf970329b5acca05e240e2ac8f4496d7d7a60ba002ae4ae4f4b\": container with ID starting with 45553ecbc1963bf970329b5acca05e240e2ac8f4496d7d7a60ba002ae4ae4f4b not found: ID does not exist" containerID="45553ecbc1963bf970329b5acca05e240e2ac8f4496d7d7a60ba002ae4ae4f4b" Sep 09 03:00:29.832000 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:29.831981 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45553ecbc1963bf970329b5acca05e240e2ac8f4496d7d7a60ba002ae4ae4f4b"} err="failed to get container status \"45553ecbc1963bf970329b5acca05e240e2ac8f4496d7d7a60ba002ae4ae4f4b\": rpc error: code = NotFound desc = could not find container \"45553ecbc1963bf970329b5acca05e240e2ac8f4496d7d7a60ba002ae4ae4f4b\": container with ID starting with 45553ecbc1963bf970329b5acca05e240e2ac8f4496d7d7a60ba002ae4ae4f4b not found: ID does not exist" Sep 09 03:00:29.832154 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:29.832005 2577 scope.go:117] "RemoveContainer" containerID="e5a9bb08d4b5395908a1a6153b7420141b8af4d588b2e231443ecfb883762eae" Sep 09 03:00:29.841097 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:29.841078 2577 scope.go:117] "RemoveContainer" containerID="e5a9bb08d4b5395908a1a6153b7420141b8af4d588b2e231443ecfb883762eae" Sep 09 03:00:29.841394 ip-10-0-136-89 kubenswrapper[2577]: E0909 03:00:29.841375 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5a9bb08d4b5395908a1a6153b7420141b8af4d588b2e231443ecfb883762eae\": container with ID starting with e5a9bb08d4b5395908a1a6153b7420141b8af4d588b2e231443ecfb883762eae not found: ID does not exist" containerID="e5a9bb08d4b5395908a1a6153b7420141b8af4d588b2e231443ecfb883762eae" Sep 09 03:00:29.841431 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:29.841405 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5a9bb08d4b5395908a1a6153b7420141b8af4d588b2e231443ecfb883762eae"} err="failed to get container status \"e5a9bb08d4b5395908a1a6153b7420141b8af4d588b2e231443ecfb883762eae\": rpc error: code = NotFound desc = could not find container \"e5a9bb08d4b5395908a1a6153b7420141b8af4d588b2e231443ecfb883762eae\": container with ID starting with e5a9bb08d4b5395908a1a6153b7420141b8af4d588b2e231443ecfb883762eae not found: ID does not exist" Sep 09 03:00:29.846170 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:29.846146 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-56dbc0-59594db65d-9smp5"] Sep 09 03:00:29.852319 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:29.852295 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-56dbc0-59594db65d-9smp5"] Sep 09 03:00:29.864532 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:29.864498 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-56dbc0-759764479f-6z97x"] Sep 09 03:00:29.869084 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:29.869055 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-56dbc0-759764479f-6z97x"] Sep 09 03:00:31.166054 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:31.166017 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e293447-dee0-47d1-8868-99d0502d3dd8" path="/var/lib/kubelet/pods/3e293447-dee0-47d1-8868-99d0502d3dd8/volumes" Sep 09 03:00:31.166434 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:31.166372 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d5fa8f3-486e-477f-b44f-f91fbb4729bd" path="/var/lib/kubelet/pods/5d5fa8f3-486e-477f-b44f-f91fbb4729bd/volumes" Sep 09 03:00:31.166642 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:31.166629 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82610097-16c6-4132-9ce1-59a0abf17edd" path="/var/lib/kubelet/pods/82610097-16c6-4132-9ce1-59a0abf17edd/volumes" Sep 09 03:00:36.192806 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:36.192763 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c173ad-v5rpz"] Sep 09 03:00:36.193413 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:36.193317 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82610097-16c6-4132-9ce1-59a0abf17edd" containerName="revoke-keys" Sep 09 03:00:36.193413 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:36.193337 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="82610097-16c6-4132-9ce1-59a0abf17edd" containerName="revoke-keys" Sep 09 03:00:36.193413 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:36.193354 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d5fa8f3-486e-477f-b44f-f91fbb4729bd" containerName="payload-processing" Sep 09 03:00:36.193413 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:36.193364 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d5fa8f3-486e-477f-b44f-f91fbb4729bd" containerName="payload-processing" Sep 09 03:00:36.193413 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:36.193387 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e293447-dee0-47d1-8868-99d0502d3dd8" containerName="payload-pre-processing" Sep 09 03:00:36.193413 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:36.193396 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e293447-dee0-47d1-8868-99d0502d3dd8" containerName="payload-pre-processing" Sep 09 03:00:36.193731 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:36.193469 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e293447-dee0-47d1-8868-99d0502d3dd8" containerName="payload-pre-processing" Sep 09 03:00:36.193731 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:36.193480 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="82610097-16c6-4132-9ce1-59a0abf17edd" containerName="revoke-keys" Sep 09 03:00:36.193731 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:36.193491 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d5fa8f3-486e-477f-b44f-f91fbb4729bd" containerName="payload-processing" Sep 09 03:00:36.198272 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:36.198251 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c173ad-v5rpz" Sep 09 03:00:36.201776 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:36.201631 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 03:00:36.201776 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:36.201659 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbr52\"" Sep 09 03:00:36.204217 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:36.204190 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c173ad-v5rpz"] Sep 09 03:00:36.332691 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:36.332652 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f39b6ef-da67-48e9-9ecd-6606db8a3e12-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c173ad-v5rpz\" (UID: \"8f39b6ef-da67-48e9-9ecd-6606db8a3e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c173ad-v5rpz" Sep 09 03:00:36.433315 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:36.433275 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f39b6ef-da67-48e9-9ecd-6606db8a3e12-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c173ad-v5rpz\" (UID: \"8f39b6ef-da67-48e9-9ecd-6606db8a3e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c173ad-v5rpz" Sep 09 03:00:36.433953 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:36.433929 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f39b6ef-da67-48e9-9ecd-6606db8a3e12-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c173ad-v5rpz\" (UID: \"8f39b6ef-da67-48e9-9ecd-6606db8a3e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c173ad-v5rpz" Sep 09 03:00:36.512297 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:36.512207 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c173ad-v5rpz" Sep 09 03:00:36.789894 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:36.789851 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c173ad-v5rpz"] Sep 09 03:00:36.867184 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:36.867145 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c173ad-v5rpz" event={"ID":"8f39b6ef-da67-48e9-9ecd-6606db8a3e12","Type":"ContainerStarted","Data":"6b4def9a15ddd38c6df2c9b476a7700c71723548e1e9863cf94c4e5ea71e1fdb"} Sep 09 03:00:37.874236 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:37.874200 2577 generic.go:356] "Generic (PLEG): container finished" podID="8f39b6ef-da67-48e9-9ecd-6606db8a3e12" containerID="a2a8baa65c8aee22f7da1feeb73e22b7ebaac1ea253d23950e0ec4446c42526f" exitCode=0 Sep 09 03:00:37.874628 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:37.874286 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c173ad-v5rpz" event={"ID":"8f39b6ef-da67-48e9-9ecd-6606db8a3e12","Type":"ContainerDied","Data":"a2a8baa65c8aee22f7da1feeb73e22b7ebaac1ea253d23950e0ec4446c42526f"} Sep 09 03:00:39.013441 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:39.013406 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c173ad-v5rpz" Sep 09 03:00:39.156844 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:39.156753 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f39b6ef-da67-48e9-9ecd-6606db8a3e12-maas-api-service-ca\") pod \"8f39b6ef-da67-48e9-9ecd-6606db8a3e12\" (UID: \"8f39b6ef-da67-48e9-9ecd-6606db8a3e12\") " Sep 09 03:00:39.157174 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:39.157147 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f39b6ef-da67-48e9-9ecd-6606db8a3e12-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8f39b6ef-da67-48e9-9ecd-6606db8a3e12" (UID: "8f39b6ef-da67-48e9-9ecd-6606db8a3e12"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 03:00:39.257987 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:39.257906 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f39b6ef-da67-48e9-9ecd-6606db8a3e12-maas-api-service-ca\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 03:00:39.886587 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:39.886549 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c173ad-v5rpz" event={"ID":"8f39b6ef-da67-48e9-9ecd-6606db8a3e12","Type":"ContainerDied","Data":"6b4def9a15ddd38c6df2c9b476a7700c71723548e1e9863cf94c4e5ea71e1fdb"} Sep 09 03:00:39.886587 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:39.886570 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c173ad-v5rpz" Sep 09 03:00:39.886801 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:39.886598 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b4def9a15ddd38c6df2c9b476a7700c71723548e1e9863cf94c4e5ea71e1fdb" Sep 09 03:00:46.660712 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:46.660662 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c173ad-29815380-wmzx2"] Sep 09 03:00:46.664664 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:46.664623 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c173ad-29815380-wmzx2"] Sep 09 03:00:46.693335 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:46.693302 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c173ad-677d666774-kcl26"] Sep 09 03:00:46.693569 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:46.693544 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c173ad-677d666774-kcl26" podUID="651f46c0-a79e-4a89-b24a-42286db15ad1" containerName="payload-pre-processing" containerID="cri-o://b5f2a526a2a8f2fa4ea8227666e01c07a249fbd4d9bcade4c1fd5a2983710704" gracePeriod=30 Sep 09 03:00:46.851141 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:46.851097 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c173ad-v5rpz"] Sep 09 03:00:46.853347 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:46.853322 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c173ad-v5rpz"] Sep 09 03:00:46.928738 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:46.928655 2577 generic.go:356] "Generic (PLEG): container finished" podID="651f46c0-a79e-4a89-b24a-42286db15ad1" containerID="b5f2a526a2a8f2fa4ea8227666e01c07a249fbd4d9bcade4c1fd5a2983710704" exitCode=0 Sep 09 03:00:46.928889 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:46.928733 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c173ad-677d666774-kcl26" event={"ID":"651f46c0-a79e-4a89-b24a-42286db15ad1","Type":"ContainerDied","Data":"b5f2a526a2a8f2fa4ea8227666e01c07a249fbd4d9bcade4c1fd5a2983710704"} Sep 09 03:00:46.954028 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:46.953999 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c173ad-677d666774-kcl26" Sep 09 03:00:47.125875 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:47.125844 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/651f46c0-a79e-4a89-b24a-42286db15ad1-plugins-config-volume\") pod \"651f46c0-a79e-4a89-b24a-42286db15ad1\" (UID: \"651f46c0-a79e-4a89-b24a-42286db15ad1\") " Sep 09 03:00:47.126128 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:47.125918 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggmcb\" (UniqueName: \"kubernetes.io/projected/651f46c0-a79e-4a89-b24a-42286db15ad1-kube-api-access-ggmcb\") pod \"651f46c0-a79e-4a89-b24a-42286db15ad1\" (UID: \"651f46c0-a79e-4a89-b24a-42286db15ad1\") " Sep 09 03:00:47.126282 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:47.126257 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/651f46c0-a79e-4a89-b24a-42286db15ad1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "651f46c0-a79e-4a89-b24a-42286db15ad1" (UID: "651f46c0-a79e-4a89-b24a-42286db15ad1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 03:00:47.128107 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:47.128082 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/651f46c0-a79e-4a89-b24a-42286db15ad1-kube-api-access-ggmcb" (OuterVolumeSpecName: "kube-api-access-ggmcb") pod "651f46c0-a79e-4a89-b24a-42286db15ad1" (UID: "651f46c0-a79e-4a89-b24a-42286db15ad1"). InnerVolumeSpecName "kube-api-access-ggmcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 03:00:47.167210 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:47.167167 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b10b3c5-c216-41b3-b962-e9bf3e61468a" path="/var/lib/kubelet/pods/6b10b3c5-c216-41b3-b962-e9bf3e61468a/volumes" Sep 09 03:00:47.167584 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:47.167565 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f39b6ef-da67-48e9-9ecd-6606db8a3e12" path="/var/lib/kubelet/pods/8f39b6ef-da67-48e9-9ecd-6606db8a3e12/volumes" Sep 09 03:00:47.226640 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:47.226561 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/651f46c0-a79e-4a89-b24a-42286db15ad1-plugins-config-volume\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 03:00:47.226640 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:47.226587 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggmcb\" (UniqueName: \"kubernetes.io/projected/651f46c0-a79e-4a89-b24a-42286db15ad1-kube-api-access-ggmcb\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 03:00:47.936122 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:47.936094 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c173ad-677d666774-kcl26" Sep 09 03:00:47.936615 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:47.936089 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c173ad-677d666774-kcl26" event={"ID":"651f46c0-a79e-4a89-b24a-42286db15ad1","Type":"ContainerDied","Data":"a25b3f9c082f5536ec6357f8fca44be48a919c3102b05c8cb5c0fff5d0023361"} Sep 09 03:00:47.936615 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:47.936221 2577 scope.go:117] "RemoveContainer" containerID="b5f2a526a2a8f2fa4ea8227666e01c07a249fbd4d9bcade4c1fd5a2983710704" Sep 09 03:00:47.953351 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:47.953321 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c173ad-677d666774-kcl26"] Sep 09 03:00:47.957890 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:47.957865 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c173ad-677d666774-kcl26"] Sep 09 03:00:49.165034 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:00:49.164996 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="651f46c0-a79e-4a89-b24a-42286db15ad1" path="/var/lib/kubelet/pods/651f46c0-a79e-4a89-b24a-42286db15ad1/volumes" Sep 09 03:03:07.522583 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:03:07.522506 2577 scope.go:117] "RemoveContainer" containerID="1f953bc3e2a00d0cecfb9bc86c92bc3d61f3b6f477a9eb4c5c5232f30ec76234" Sep 09 03:03:28.573726 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:03:28.573689 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zjkq5"] Sep 09 03:03:28.574277 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:03:28.573941 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zjkq5" podUID="9c9a73be-b0a4-4c21-9fb0-e0f94826170b" containerName="manager" containerID="cri-o://9484f87cb08a0b3ad944796d65598020eb4963edc698b97d617ba7a038b95c3f" gracePeriod=10 Sep 09 03:03:29.843337 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:03:29.843304 2577 generic.go:356] "Generic (PLEG): container finished" podID="9c9a73be-b0a4-4c21-9fb0-e0f94826170b" containerID="9484f87cb08a0b3ad944796d65598020eb4963edc698b97d617ba7a038b95c3f" exitCode=0 Sep 09 03:03:29.843724 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:03:29.843378 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zjkq5" event={"ID":"9c9a73be-b0a4-4c21-9fb0-e0f94826170b","Type":"ContainerDied","Data":"9484f87cb08a0b3ad944796d65598020eb4963edc698b97d617ba7a038b95c3f"} Sep 09 03:03:30.022103 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:03:30.022081 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zjkq5" Sep 09 03:03:30.185394 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:03:30.185295 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7fhc\" (UniqueName: \"kubernetes.io/projected/9c9a73be-b0a4-4c21-9fb0-e0f94826170b-kube-api-access-j7fhc\") pod \"9c9a73be-b0a4-4c21-9fb0-e0f94826170b\" (UID: \"9c9a73be-b0a4-4c21-9fb0-e0f94826170b\") " Sep 09 03:03:30.185576 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:03:30.185402 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c9a73be-b0a4-4c21-9fb0-e0f94826170b-extensions-socket-volume\") pod \"9c9a73be-b0a4-4c21-9fb0-e0f94826170b\" (UID: \"9c9a73be-b0a4-4c21-9fb0-e0f94826170b\") " Sep 09 03:03:30.185766 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:03:30.185742 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9a73be-b0a4-4c21-9fb0-e0f94826170b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9c9a73be-b0a4-4c21-9fb0-e0f94826170b" (UID: "9c9a73be-b0a4-4c21-9fb0-e0f94826170b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 03:03:30.187590 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:03:30.187562 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c9a73be-b0a4-4c21-9fb0-e0f94826170b-kube-api-access-j7fhc" (OuterVolumeSpecName: "kube-api-access-j7fhc") pod "9c9a73be-b0a4-4c21-9fb0-e0f94826170b" (UID: "9c9a73be-b0a4-4c21-9fb0-e0f94826170b"). InnerVolumeSpecName "kube-api-access-j7fhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 03:03:30.286339 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:03:30.286301 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7fhc\" (UniqueName: \"kubernetes.io/projected/9c9a73be-b0a4-4c21-9fb0-e0f94826170b-kube-api-access-j7fhc\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 03:03:30.286339 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:03:30.286332 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c9a73be-b0a4-4c21-9fb0-e0f94826170b-extensions-socket-volume\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 03:03:30.850232 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:03:30.850195 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zjkq5" Sep 09 03:03:30.850232 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:03:30.850202 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zjkq5" event={"ID":"9c9a73be-b0a4-4c21-9fb0-e0f94826170b","Type":"ContainerDied","Data":"53963a2ada0462dabb9c06ec9a1740e439898daae4e9c9264dc0b793f9e8362e"} Sep 09 03:03:30.850747 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:03:30.850271 2577 scope.go:117] "RemoveContainer" containerID="9484f87cb08a0b3ad944796d65598020eb4963edc698b97d617ba7a038b95c3f" Sep 09 03:03:30.873546 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:03:30.873509 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zjkq5"] Sep 09 03:03:30.876953 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:03:30.876912 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zjkq5"] Sep 09 03:03:31.165197 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:03:31.165097 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c9a73be-b0a4-4c21-9fb0-e0f94826170b" path="/var/lib/kubelet/pods/9c9a73be-b0a4-4c21-9fb0-e0f94826170b/volumes" Sep 09 03:05:07.557393 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:07.557356 2577 scope.go:117] "RemoveContainer" containerID="a232447ff4eba4661f68c96b563254402fb4d388b22de365106ad5cd09e96727" Sep 09 03:05:10.549239 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:10.549200 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls"] Sep 09 03:05:10.549728 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:10.549706 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c9a73be-b0a4-4c21-9fb0-e0f94826170b" containerName="manager" Sep 09 03:05:10.549728 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:10.549726 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c9a73be-b0a4-4c21-9fb0-e0f94826170b" containerName="manager" Sep 09 03:05:10.549914 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:10.549746 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="651f46c0-a79e-4a89-b24a-42286db15ad1" containerName="payload-pre-processing" Sep 09 03:05:10.549914 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:10.549755 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="651f46c0-a79e-4a89-b24a-42286db15ad1" containerName="payload-pre-processing" Sep 09 03:05:10.549914 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:10.549767 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f39b6ef-da67-48e9-9ecd-6606db8a3e12" containerName="revoke-keys" Sep 09 03:05:10.549914 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:10.549776 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f39b6ef-da67-48e9-9ecd-6606db8a3e12" containerName="revoke-keys" Sep 09 03:05:10.549914 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:10.549863 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c9a73be-b0a4-4c21-9fb0-e0f94826170b" containerName="manager" Sep 09 03:05:10.549914 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:10.549876 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f39b6ef-da67-48e9-9ecd-6606db8a3e12" containerName="revoke-keys" Sep 09 03:05:10.549914 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:10.549886 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="651f46c0-a79e-4a89-b24a-42286db15ad1" containerName="payload-pre-processing" Sep 09 03:05:10.554517 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:10.554497 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls" Sep 09 03:05:10.557007 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:10.556986 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-40e3fdc1-kserve-self-signed-certs\"" Sep 09 03:05:10.563584 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:10.563552 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls"] Sep 09 03:05:10.583533 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:10.583496 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baf639d1-4a8d-465b-93a0-9ba44d606b76-home\") pod \"e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls\" (UID: \"baf639d1-4a8d-465b-93a0-9ba44d606b76\") " pod="llm/e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls" Sep 09 03:05:10.583533 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:10.583537 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf639d1-4a8d-465b-93a0-9ba44d606b76-model-cache\") pod \"e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls\" (UID: \"baf639d1-4a8d-465b-93a0-9ba44d606b76\") " pod="llm/e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls" Sep 09 03:05:10.583733 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:10.583613 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf639d1-4a8d-465b-93a0-9ba44d606b76-tmp-dir\") pod \"e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls\" (UID: \"baf639d1-4a8d-465b-93a0-9ba44d606b76\") " pod="llm/e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls" Sep 09 03:05:10.583733 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:10.583640 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv72v\" (UniqueName: \"kubernetes.io/projected/baf639d1-4a8d-465b-93a0-9ba44d606b76-kube-api-access-bv72v\") pod \"e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls\" (UID: \"baf639d1-4a8d-465b-93a0-9ba44d606b76\") " pod="llm/e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls" Sep 09 03:05:10.583805 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:10.583721 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf639d1-4a8d-465b-93a0-9ba44d606b76-tls-certs\") pod \"e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls\" (UID: \"baf639d1-4a8d-465b-93a0-9ba44d606b76\") " pod="llm/e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls" Sep 09 03:05:10.583805 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:10.583765 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf639d1-4a8d-465b-93a0-9ba44d606b76-dshm\") pod \"e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls\" (UID: \"baf639d1-4a8d-465b-93a0-9ba44d606b76\") " pod="llm/e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls" Sep 09 03:05:10.684764 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:10.684727 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf639d1-4a8d-465b-93a0-9ba44d606b76-tmp-dir\") pod \"e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls\" (UID: \"baf639d1-4a8d-465b-93a0-9ba44d606b76\") " pod="llm/e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls" Sep 09 03:05:10.684764 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:10.684770 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv72v\" (UniqueName: \"kubernetes.io/projected/baf639d1-4a8d-465b-93a0-9ba44d606b76-kube-api-access-bv72v\") pod \"e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls\" (UID: \"baf639d1-4a8d-465b-93a0-9ba44d606b76\") " pod="llm/e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls" Sep 09 03:05:10.685028 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:10.684905 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf639d1-4a8d-465b-93a0-9ba44d606b76-tls-certs\") pod \"e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls\" (UID: \"baf639d1-4a8d-465b-93a0-9ba44d606b76\") " pod="llm/e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls" Sep 09 03:05:10.685028 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:10.684987 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf639d1-4a8d-465b-93a0-9ba44d606b76-dshm\") pod \"e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls\" (UID: \"baf639d1-4a8d-465b-93a0-9ba44d606b76\") " pod="llm/e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls" Sep 09 03:05:10.685028 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:10.685021 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baf639d1-4a8d-465b-93a0-9ba44d606b76-home\") pod \"e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls\" (UID: \"baf639d1-4a8d-465b-93a0-9ba44d606b76\") " pod="llm/e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls" Sep 09 03:05:10.685182 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:10.685056 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf639d1-4a8d-465b-93a0-9ba44d606b76-model-cache\") pod \"e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls\" (UID: \"baf639d1-4a8d-465b-93a0-9ba44d606b76\") " pod="llm/e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls" Sep 09 03:05:10.685242 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:10.685225 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf639d1-4a8d-465b-93a0-9ba44d606b76-tmp-dir\") pod \"e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls\" (UID: \"baf639d1-4a8d-465b-93a0-9ba44d606b76\") " pod="llm/e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls" Sep 09 03:05:10.685424 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:10.685395 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf639d1-4a8d-465b-93a0-9ba44d606b76-model-cache\") pod \"e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls\" (UID: \"baf639d1-4a8d-465b-93a0-9ba44d606b76\") " pod="llm/e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls" Sep 09 03:05:10.685567 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:10.685443 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baf639d1-4a8d-465b-93a0-9ba44d606b76-home\") pod \"e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls\" (UID: \"baf639d1-4a8d-465b-93a0-9ba44d606b76\") " pod="llm/e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls" Sep 09 03:05:10.687400 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:10.687375 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf639d1-4a8d-465b-93a0-9ba44d606b76-dshm\") pod \"e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls\" (UID: \"baf639d1-4a8d-465b-93a0-9ba44d606b76\") " pod="llm/e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls" Sep 09 03:05:10.687501 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:10.687489 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf639d1-4a8d-465b-93a0-9ba44d606b76-tls-certs\") pod \"e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls\" (UID: \"baf639d1-4a8d-465b-93a0-9ba44d606b76\") " pod="llm/e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls" Sep 09 03:05:10.692606 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:10.692583 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv72v\" (UniqueName: \"kubernetes.io/projected/baf639d1-4a8d-465b-93a0-9ba44d606b76-kube-api-access-bv72v\") pod \"e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls\" (UID: \"baf639d1-4a8d-465b-93a0-9ba44d606b76\") " pod="llm/e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls" Sep 09 03:05:10.865729 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:10.865643 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls" Sep 09 03:05:11.140880 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:11.140847 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls"] Sep 09 03:05:11.142436 ip-10-0-136-89 kubenswrapper[2577]: W0909 03:05:11.142402 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbaf639d1_4a8d_465b_93a0_9ba44d606b76.slice/crio-a7fdb4a0a65189f1324ef9e120d21eb51fe8af3afc8336032dee5ccc6a05df16 WatchSource:0}: Error finding container a7fdb4a0a65189f1324ef9e120d21eb51fe8af3afc8336032dee5ccc6a05df16: Status 404 returned error can't find the container with id a7fdb4a0a65189f1324ef9e120d21eb51fe8af3afc8336032dee5ccc6a05df16 Sep 09 03:05:11.392622 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:11.392536 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls" event={"ID":"baf639d1-4a8d-465b-93a0-9ba44d606b76","Type":"ContainerStarted","Data":"07ce18ceaf8bc488beaaaef62b8f78588a6ebf370d8f669caebe780bc29b9f0f"} Sep 09 03:05:11.392622 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:11.392583 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls" event={"ID":"baf639d1-4a8d-465b-93a0-9ba44d606b76","Type":"ContainerStarted","Data":"a7fdb4a0a65189f1324ef9e120d21eb51fe8af3afc8336032dee5ccc6a05df16"} Sep 09 03:05:11.412365 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:11.412311 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls" podStartSLOduration=1.41229398 podStartE2EDuration="1.41229398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:05:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:05:11.411375744 +0000 UTC m=+2104.775710206" watchObservedRunningTime="2026-09-09 03:05:11.41229398 +0000 UTC m=+2104.776628443" Sep 09 03:05:20.866447 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:20.866400 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls" Sep 09 03:05:20.866852 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:20.866462 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls" Sep 09 03:05:20.879032 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:20.879005 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls" Sep 09 03:05:21.458207 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:21.458178 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls" Sep 09 03:05:43.310657 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:43.310617 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls"] Sep 09 03:05:58.331169 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:58.331105 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls" podUID="baf639d1-4a8d-465b-93a0-9ba44d606b76" containerName="main" containerID="cri-o://07ce18ceaf8bc488beaaaef62b8f78588a6ebf370d8f669caebe780bc29b9f0f" gracePeriod=45 Sep 09 03:05:58.574682 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:58.574657 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls" Sep 09 03:05:58.654244 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:58.654154 2577 generic.go:356] "Generic (PLEG): container finished" podID="baf639d1-4a8d-465b-93a0-9ba44d606b76" containerID="07ce18ceaf8bc488beaaaef62b8f78588a6ebf370d8f669caebe780bc29b9f0f" exitCode=0 Sep 09 03:05:58.654244 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:58.654204 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls" Sep 09 03:05:58.654428 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:58.654244 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls" event={"ID":"baf639d1-4a8d-465b-93a0-9ba44d606b76","Type":"ContainerDied","Data":"07ce18ceaf8bc488beaaaef62b8f78588a6ebf370d8f669caebe780bc29b9f0f"} Sep 09 03:05:58.654428 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:58.654293 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls" event={"ID":"baf639d1-4a8d-465b-93a0-9ba44d606b76","Type":"ContainerDied","Data":"a7fdb4a0a65189f1324ef9e120d21eb51fe8af3afc8336032dee5ccc6a05df16"} Sep 09 03:05:58.654428 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:58.654313 2577 scope.go:117] "RemoveContainer" containerID="07ce18ceaf8bc488beaaaef62b8f78588a6ebf370d8f669caebe780bc29b9f0f" Sep 09 03:05:58.664739 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:58.664703 2577 scope.go:117] "RemoveContainer" containerID="07ce18ceaf8bc488beaaaef62b8f78588a6ebf370d8f669caebe780bc29b9f0f" Sep 09 03:05:58.665352 ip-10-0-136-89 kubenswrapper[2577]: E0909 03:05:58.665320 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07ce18ceaf8bc488beaaaef62b8f78588a6ebf370d8f669caebe780bc29b9f0f\": container with ID starting with 07ce18ceaf8bc488beaaaef62b8f78588a6ebf370d8f669caebe780bc29b9f0f not found: ID does not exist" containerID="07ce18ceaf8bc488beaaaef62b8f78588a6ebf370d8f669caebe780bc29b9f0f" Sep 09 03:05:58.665474 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:58.665366 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07ce18ceaf8bc488beaaaef62b8f78588a6ebf370d8f669caebe780bc29b9f0f"} err="failed to get container status \"07ce18ceaf8bc488beaaaef62b8f78588a6ebf370d8f669caebe780bc29b9f0f\": rpc error: code = NotFound desc = could not find container \"07ce18ceaf8bc488beaaaef62b8f78588a6ebf370d8f669caebe780bc29b9f0f\": container with ID starting with 07ce18ceaf8bc488beaaaef62b8f78588a6ebf370d8f669caebe780bc29b9f0f not found: ID does not exist" Sep 09 03:05:58.684532 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:58.684505 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf639d1-4a8d-465b-93a0-9ba44d606b76-model-cache\") pod \"baf639d1-4a8d-465b-93a0-9ba44d606b76\" (UID: \"baf639d1-4a8d-465b-93a0-9ba44d606b76\") " Sep 09 03:05:58.684696 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:58.684552 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf639d1-4a8d-465b-93a0-9ba44d606b76-tmp-dir\") pod \"baf639d1-4a8d-465b-93a0-9ba44d606b76\" (UID: \"baf639d1-4a8d-465b-93a0-9ba44d606b76\") " Sep 09 03:05:58.684696 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:58.684612 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf639d1-4a8d-465b-93a0-9ba44d606b76-dshm\") pod \"baf639d1-4a8d-465b-93a0-9ba44d606b76\" (UID: \"baf639d1-4a8d-465b-93a0-9ba44d606b76\") " Sep 09 03:05:58.684696 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:58.684649 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baf639d1-4a8d-465b-93a0-9ba44d606b76-home\") pod \"baf639d1-4a8d-465b-93a0-9ba44d606b76\" (UID: \"baf639d1-4a8d-465b-93a0-9ba44d606b76\") " Sep 09 03:05:58.684696 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:58.684671 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf639d1-4a8d-465b-93a0-9ba44d606b76-tls-certs\") pod \"baf639d1-4a8d-465b-93a0-9ba44d606b76\" (UID: \"baf639d1-4a8d-465b-93a0-9ba44d606b76\") " Sep 09 03:05:58.684900 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:58.684721 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bv72v\" (UniqueName: \"kubernetes.io/projected/baf639d1-4a8d-465b-93a0-9ba44d606b76-kube-api-access-bv72v\") pod \"baf639d1-4a8d-465b-93a0-9ba44d606b76\" (UID: \"baf639d1-4a8d-465b-93a0-9ba44d606b76\") " Sep 09 03:05:58.684900 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:58.684746 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf639d1-4a8d-465b-93a0-9ba44d606b76-model-cache" (OuterVolumeSpecName: "model-cache") pod "baf639d1-4a8d-465b-93a0-9ba44d606b76" (UID: "baf639d1-4a8d-465b-93a0-9ba44d606b76"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 03:05:58.684900 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:58.684882 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf639d1-4a8d-465b-93a0-9ba44d606b76-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "baf639d1-4a8d-465b-93a0-9ba44d606b76" (UID: "baf639d1-4a8d-465b-93a0-9ba44d606b76"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 03:05:58.685040 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:58.684897 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf639d1-4a8d-465b-93a0-9ba44d606b76-home" (OuterVolumeSpecName: "home") pod "baf639d1-4a8d-465b-93a0-9ba44d606b76" (UID: "baf639d1-4a8d-465b-93a0-9ba44d606b76"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 03:05:58.685040 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:58.684994 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baf639d1-4a8d-465b-93a0-9ba44d606b76-home\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 03:05:58.685040 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:58.685007 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baf639d1-4a8d-465b-93a0-9ba44d606b76-model-cache\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 03:05:58.685040 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:58.685021 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baf639d1-4a8d-465b-93a0-9ba44d606b76-tmp-dir\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 03:05:58.687051 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:58.687025 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baf639d1-4a8d-465b-93a0-9ba44d606b76-dshm" (OuterVolumeSpecName: "dshm") pod "baf639d1-4a8d-465b-93a0-9ba44d606b76" (UID: "baf639d1-4a8d-465b-93a0-9ba44d606b76"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 03:05:58.687182 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:58.687086 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baf639d1-4a8d-465b-93a0-9ba44d606b76-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "baf639d1-4a8d-465b-93a0-9ba44d606b76" (UID: "baf639d1-4a8d-465b-93a0-9ba44d606b76"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 03:05:58.687182 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:58.687090 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baf639d1-4a8d-465b-93a0-9ba44d606b76-kube-api-access-bv72v" (OuterVolumeSpecName: "kube-api-access-bv72v") pod "baf639d1-4a8d-465b-93a0-9ba44d606b76" (UID: "baf639d1-4a8d-465b-93a0-9ba44d606b76"). InnerVolumeSpecName "kube-api-access-bv72v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 03:05:58.785406 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:58.785374 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baf639d1-4a8d-465b-93a0-9ba44d606b76-dshm\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 03:05:58.785406 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:58.785401 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf639d1-4a8d-465b-93a0-9ba44d606b76-tls-certs\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 03:05:58.785406 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:58.785412 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv72v\" (UniqueName: \"kubernetes.io/projected/baf639d1-4a8d-465b-93a0-9ba44d606b76-kube-api-access-bv72v\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 03:05:58.976971 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:58.976937 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls"] Sep 09 03:05:58.980328 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:58.980299 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-40e3fdc1-kserve-9dcfd9c5-7w7ls"] Sep 09 03:05:59.165302 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:05:59.165260 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="baf639d1-4a8d-465b-93a0-9ba44d606b76" path="/var/lib/kubelet/pods/baf639d1-4a8d-465b-93a0-9ba44d606b76/volumes" Sep 09 03:06:02.337346 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:02.337306 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-55c9b5658-qvb6h"] Sep 09 03:06:02.337815 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:02.337779 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baf639d1-4a8d-465b-93a0-9ba44d606b76" containerName="main" Sep 09 03:06:02.337815 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:02.337796 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="baf639d1-4a8d-465b-93a0-9ba44d606b76" containerName="main" Sep 09 03:06:02.337946 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:02.337883 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="baf639d1-4a8d-465b-93a0-9ba44d606b76" containerName="main" Sep 09 03:06:02.342407 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:02.342384 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55c9b5658-qvb6h" Sep 09 03:06:02.346315 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:02.346272 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 09 03:06:02.346315 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:02.346288 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 09 03:06:02.346315 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:02.346284 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 09 03:06:02.346538 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:02.346323 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 09 03:06:02.346538 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:02.346364 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hq897\"" Sep 09 03:06:02.351538 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:02.351515 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-55c9b5658-qvb6h"] Sep 09 03:06:02.516628 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:02.516599 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc-metrics-certs\") pod \"maas-api-55c9b5658-qvb6h\" (UID: \"8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc\") " pod="odh-ai-gateway-infra/maas-api-55c9b5658-qvb6h" Sep 09 03:06:02.516828 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:02.516644 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc-openshift-service-ca\") pod \"maas-api-55c9b5658-qvb6h\" (UID: \"8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc\") " pod="odh-ai-gateway-infra/maas-api-55c9b5658-qvb6h" Sep 09 03:06:02.516828 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:02.516663 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sv4j\" (UniqueName: \"kubernetes.io/projected/8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc-kube-api-access-5sv4j\") pod \"maas-api-55c9b5658-qvb6h\" (UID: \"8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc\") " pod="odh-ai-gateway-infra/maas-api-55c9b5658-qvb6h" Sep 09 03:06:02.516828 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:02.516763 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc-maas-api-tls\") pod \"maas-api-55c9b5658-qvb6h\" (UID: \"8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc\") " pod="odh-ai-gateway-infra/maas-api-55c9b5658-qvb6h" Sep 09 03:06:02.618091 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:02.618007 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc-openshift-service-ca\") pod \"maas-api-55c9b5658-qvb6h\" (UID: \"8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc\") " pod="odh-ai-gateway-infra/maas-api-55c9b5658-qvb6h" Sep 09 03:06:02.618091 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:02.618044 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sv4j\" (UniqueName: \"kubernetes.io/projected/8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc-kube-api-access-5sv4j\") pod \"maas-api-55c9b5658-qvb6h\" (UID: \"8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc\") " pod="odh-ai-gateway-infra/maas-api-55c9b5658-qvb6h" Sep 09 03:06:02.618091 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:02.618092 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc-maas-api-tls\") pod \"maas-api-55c9b5658-qvb6h\" (UID: \"8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc\") " pod="odh-ai-gateway-infra/maas-api-55c9b5658-qvb6h" Sep 09 03:06:02.618371 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:02.618139 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc-metrics-certs\") pod \"maas-api-55c9b5658-qvb6h\" (UID: \"8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc\") " pod="odh-ai-gateway-infra/maas-api-55c9b5658-qvb6h" Sep 09 03:06:02.618730 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:02.618707 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc-openshift-service-ca\") pod \"maas-api-55c9b5658-qvb6h\" (UID: \"8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc\") " pod="odh-ai-gateway-infra/maas-api-55c9b5658-qvb6h" Sep 09 03:06:02.620731 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:02.620705 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc-metrics-certs\") pod \"maas-api-55c9b5658-qvb6h\" (UID: \"8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc\") " pod="odh-ai-gateway-infra/maas-api-55c9b5658-qvb6h" Sep 09 03:06:02.620854 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:02.620830 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc-maas-api-tls\") pod \"maas-api-55c9b5658-qvb6h\" (UID: \"8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc\") " pod="odh-ai-gateway-infra/maas-api-55c9b5658-qvb6h" Sep 09 03:06:02.625413 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:02.625392 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sv4j\" (UniqueName: \"kubernetes.io/projected/8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc-kube-api-access-5sv4j\") pod \"maas-api-55c9b5658-qvb6h\" (UID: \"8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc\") " pod="odh-ai-gateway-infra/maas-api-55c9b5658-qvb6h" Sep 09 03:06:02.654312 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:02.654283 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55c9b5658-qvb6h" Sep 09 03:06:02.928679 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:02.928648 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-55c9b5658-qvb6h"] Sep 09 03:06:02.930258 ip-10-0-136-89 kubenswrapper[2577]: W0909 03:06:02.930229 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aff4bdc_b7e0_4a9f_a876_c82fb7e95dcc.slice/crio-b6ed9b6fd0d5810d00c1f06ee10e84ddde18e607a87cbd59c2048c5f34c2d306 WatchSource:0}: Error finding container b6ed9b6fd0d5810d00c1f06ee10e84ddde18e607a87cbd59c2048c5f34c2d306: Status 404 returned error can't find the container with id b6ed9b6fd0d5810d00c1f06ee10e84ddde18e607a87cbd59c2048c5f34c2d306 Sep 09 03:06:02.931670 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:02.931649 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 03:06:03.692385 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:03.692334 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55c9b5658-qvb6h" event={"ID":"8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc","Type":"ContainerStarted","Data":"b6ed9b6fd0d5810d00c1f06ee10e84ddde18e607a87cbd59c2048c5f34c2d306"} Sep 09 03:06:05.707418 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:05.707380 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55c9b5658-qvb6h" event={"ID":"8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc","Type":"ContainerStarted","Data":"34f015970532c0b3823559c99d5f7805496b7b87985ae2902e97c1b5ef33839d"} Sep 09 03:06:05.707892 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:05.707513 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-55c9b5658-qvb6h" Sep 09 03:06:05.724829 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:05.724766 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-55c9b5658-qvb6h" podStartSLOduration=1.864656536 podStartE2EDuration="3.724749692s" totalImagesPullingTime="1.860093156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:06:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:06:05.724547196 +0000 UTC m=+2159.088881661" watchObservedRunningTime="2026-09-09 03:06:05.724749692 +0000 UTC m=+2159.089084156" Sep 09 03:06:07.577999 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:07.577951 2577 scope.go:117] "RemoveContainer" containerID="35a68b61b3e01d069ec5376a107842ca9ef56f3fd996bb839f0d3c599ccd4837" Sep 09 03:06:07.587359 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:07.587340 2577 scope.go:117] "RemoveContainer" containerID="c62fdca01e58dc9f1dd4ccfd84bdee6c82b4359473620be1029e87729198e8a5" Sep 09 03:06:09.719830 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:09.719799 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-55c9b5658-qvb6h"] Sep 09 03:06:09.720252 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:09.720183 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-55c9b5658-qvb6h" podUID="8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc" containerName="maas-api" containerID="cri-o://34f015970532c0b3823559c99d5f7805496b7b87985ae2902e97c1b5ef33839d" gracePeriod=30 Sep 09 03:06:09.725779 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:09.725754 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-55c9b5658-qvb6h" Sep 09 03:06:09.752069 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:09.752038 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6f6cdc5d86-g7w6t"] Sep 09 03:06:09.755340 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:09.755320 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6f6cdc5d86-g7w6t" Sep 09 03:06:09.765278 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:09.765234 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6f6cdc5d86-g7w6t"] Sep 09 03:06:09.772755 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:09.772725 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a7d2a68-676c-4003-9374-c0865b1d0aa5-openshift-service-ca\") pod \"maas-api-6f6cdc5d86-g7w6t\" (UID: \"7a7d2a68-676c-4003-9374-c0865b1d0aa5\") " pod="odh-ai-gateway-infra/maas-api-6f6cdc5d86-g7w6t" Sep 09 03:06:09.772899 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:09.772782 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a7d2a68-676c-4003-9374-c0865b1d0aa5-maas-api-tls\") pod \"maas-api-6f6cdc5d86-g7w6t\" (UID: \"7a7d2a68-676c-4003-9374-c0865b1d0aa5\") " pod="odh-ai-gateway-infra/maas-api-6f6cdc5d86-g7w6t" Sep 09 03:06:09.772899 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:09.772838 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsbq2\" (UniqueName: \"kubernetes.io/projected/7a7d2a68-676c-4003-9374-c0865b1d0aa5-kube-api-access-zsbq2\") pod \"maas-api-6f6cdc5d86-g7w6t\" (UID: \"7a7d2a68-676c-4003-9374-c0865b1d0aa5\") " pod="odh-ai-gateway-infra/maas-api-6f6cdc5d86-g7w6t" Sep 09 03:06:09.773018 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:09.772911 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a7d2a68-676c-4003-9374-c0865b1d0aa5-metrics-certs\") pod \"maas-api-6f6cdc5d86-g7w6t\" (UID: \"7a7d2a68-676c-4003-9374-c0865b1d0aa5\") " pod="odh-ai-gateway-infra/maas-api-6f6cdc5d86-g7w6t" Sep 09 03:06:09.873578 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:09.873542 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a7d2a68-676c-4003-9374-c0865b1d0aa5-openshift-service-ca\") pod \"maas-api-6f6cdc5d86-g7w6t\" (UID: \"7a7d2a68-676c-4003-9374-c0865b1d0aa5\") " pod="odh-ai-gateway-infra/maas-api-6f6cdc5d86-g7w6t" Sep 09 03:06:09.873728 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:09.873607 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a7d2a68-676c-4003-9374-c0865b1d0aa5-maas-api-tls\") pod \"maas-api-6f6cdc5d86-g7w6t\" (UID: \"7a7d2a68-676c-4003-9374-c0865b1d0aa5\") " pod="odh-ai-gateway-infra/maas-api-6f6cdc5d86-g7w6t" Sep 09 03:06:09.873728 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:09.873634 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsbq2\" (UniqueName: \"kubernetes.io/projected/7a7d2a68-676c-4003-9374-c0865b1d0aa5-kube-api-access-zsbq2\") pod \"maas-api-6f6cdc5d86-g7w6t\" (UID: \"7a7d2a68-676c-4003-9374-c0865b1d0aa5\") " pod="odh-ai-gateway-infra/maas-api-6f6cdc5d86-g7w6t" Sep 09 03:06:09.873728 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:09.873670 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a7d2a68-676c-4003-9374-c0865b1d0aa5-metrics-certs\") pod \"maas-api-6f6cdc5d86-g7w6t\" (UID: \"7a7d2a68-676c-4003-9374-c0865b1d0aa5\") " pod="odh-ai-gateway-infra/maas-api-6f6cdc5d86-g7w6t" Sep 09 03:06:09.874420 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:09.874332 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a7d2a68-676c-4003-9374-c0865b1d0aa5-openshift-service-ca\") pod \"maas-api-6f6cdc5d86-g7w6t\" (UID: \"7a7d2a68-676c-4003-9374-c0865b1d0aa5\") " pod="odh-ai-gateway-infra/maas-api-6f6cdc5d86-g7w6t" Sep 09 03:06:09.876350 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:09.876328 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a7d2a68-676c-4003-9374-c0865b1d0aa5-maas-api-tls\") pod \"maas-api-6f6cdc5d86-g7w6t\" (UID: \"7a7d2a68-676c-4003-9374-c0865b1d0aa5\") " pod="odh-ai-gateway-infra/maas-api-6f6cdc5d86-g7w6t" Sep 09 03:06:09.876449 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:09.876370 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a7d2a68-676c-4003-9374-c0865b1d0aa5-metrics-certs\") pod \"maas-api-6f6cdc5d86-g7w6t\" (UID: \"7a7d2a68-676c-4003-9374-c0865b1d0aa5\") " pod="odh-ai-gateway-infra/maas-api-6f6cdc5d86-g7w6t" Sep 09 03:06:09.882186 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:09.882160 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsbq2\" (UniqueName: \"kubernetes.io/projected/7a7d2a68-676c-4003-9374-c0865b1d0aa5-kube-api-access-zsbq2\") pod \"maas-api-6f6cdc5d86-g7w6t\" (UID: \"7a7d2a68-676c-4003-9374-c0865b1d0aa5\") " pod="odh-ai-gateway-infra/maas-api-6f6cdc5d86-g7w6t" Sep 09 03:06:09.970029 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:09.969950 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55c9b5658-qvb6h" Sep 09 03:06:09.974388 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:09.974369 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc-metrics-certs\") pod \"8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc\" (UID: \"8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc\") " Sep 09 03:06:09.974507 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:09.974416 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sv4j\" (UniqueName: \"kubernetes.io/projected/8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc-kube-api-access-5sv4j\") pod \"8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc\" (UID: \"8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc\") " Sep 09 03:06:09.974507 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:09.974471 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc-maas-api-tls\") pod \"8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc\" (UID: \"8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc\") " Sep 09 03:06:09.974507 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:09.974497 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc-openshift-service-ca\") pod \"8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc\" (UID: \"8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc\") " Sep 09 03:06:09.974867 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:09.974842 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc" (UID: "8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 03:06:09.976754 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:09.976725 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc-kube-api-access-5sv4j" (OuterVolumeSpecName: "kube-api-access-5sv4j") pod "8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc" (UID: "8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc"). InnerVolumeSpecName "kube-api-access-5sv4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 03:06:09.977086 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:09.977065 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc" (UID: "8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 03:06:09.977086 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:09.977077 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc" (UID: "8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 03:06:10.075878 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:10.075841 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc-maas-api-tls\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 03:06:10.075878 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:10.075879 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc-openshift-service-ca\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 03:06:10.076133 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:10.075893 2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc-metrics-certs\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 03:06:10.076133 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:10.075906 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sv4j\" (UniqueName: \"kubernetes.io/projected/8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc-kube-api-access-5sv4j\") on node \"ip-10-0-136-89.ec2.internal\" DevicePath \"\"" Sep 09 03:06:10.103353 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:10.103320 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6f6cdc5d86-g7w6t" Sep 09 03:06:10.374991 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:10.374945 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6f6cdc5d86-g7w6t"] Sep 09 03:06:10.376536 ip-10-0-136-89 kubenswrapper[2577]: W0909 03:06:10.376513 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a7d2a68_676c_4003_9374_c0865b1d0aa5.slice/crio-75556b24bdb023a322d2ae2fb11be376cca1fa0aed5ebbd4cfc36574af682274 WatchSource:0}: Error finding container 75556b24bdb023a322d2ae2fb11be376cca1fa0aed5ebbd4cfc36574af682274: Status 404 returned error can't find the container with id 75556b24bdb023a322d2ae2fb11be376cca1fa0aed5ebbd4cfc36574af682274 Sep 09 03:06:10.736460 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:10.736365 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6f6cdc5d86-g7w6t" event={"ID":"7a7d2a68-676c-4003-9374-c0865b1d0aa5","Type":"ContainerStarted","Data":"75556b24bdb023a322d2ae2fb11be376cca1fa0aed5ebbd4cfc36574af682274"} Sep 09 03:06:10.737693 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:10.737664 2577 generic.go:356] "Generic (PLEG): container finished" podID="8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc" containerID="34f015970532c0b3823559c99d5f7805496b7b87985ae2902e97c1b5ef33839d" exitCode=0 Sep 09 03:06:10.737838 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:10.737714 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-55c9b5658-qvb6h" Sep 09 03:06:10.737838 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:10.737735 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55c9b5658-qvb6h" event={"ID":"8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc","Type":"ContainerDied","Data":"34f015970532c0b3823559c99d5f7805496b7b87985ae2902e97c1b5ef33839d"} Sep 09 03:06:10.737838 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:10.737781 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-55c9b5658-qvb6h" event={"ID":"8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc","Type":"ContainerDied","Data":"b6ed9b6fd0d5810d00c1f06ee10e84ddde18e607a87cbd59c2048c5f34c2d306"} Sep 09 03:06:10.737838 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:10.737809 2577 scope.go:117] "RemoveContainer" containerID="34f015970532c0b3823559c99d5f7805496b7b87985ae2902e97c1b5ef33839d" Sep 09 03:06:10.750408 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:10.750384 2577 scope.go:117] "RemoveContainer" containerID="34f015970532c0b3823559c99d5f7805496b7b87985ae2902e97c1b5ef33839d" Sep 09 03:06:10.750737 ip-10-0-136-89 kubenswrapper[2577]: E0909 03:06:10.750718 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34f015970532c0b3823559c99d5f7805496b7b87985ae2902e97c1b5ef33839d\": container with ID starting with 34f015970532c0b3823559c99d5f7805496b7b87985ae2902e97c1b5ef33839d not found: ID does not exist" containerID="34f015970532c0b3823559c99d5f7805496b7b87985ae2902e97c1b5ef33839d" Sep 09 03:06:10.750816 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:10.750754 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34f015970532c0b3823559c99d5f7805496b7b87985ae2902e97c1b5ef33839d"} err="failed to get container status \"34f015970532c0b3823559c99d5f7805496b7b87985ae2902e97c1b5ef33839d\": rpc error: code = NotFound desc = could not find container \"34f015970532c0b3823559c99d5f7805496b7b87985ae2902e97c1b5ef33839d\": container with ID starting with 34f015970532c0b3823559c99d5f7805496b7b87985ae2902e97c1b5ef33839d not found: ID does not exist" Sep 09 03:06:10.760584 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:10.760559 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-55c9b5658-qvb6h"] Sep 09 03:06:10.763949 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:10.763927 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-55c9b5658-qvb6h"] Sep 09 03:06:11.165093 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:11.165062 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc" path="/var/lib/kubelet/pods/8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc/volumes" Sep 09 03:06:11.744506 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:11.744467 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6f6cdc5d86-g7w6t" event={"ID":"7a7d2a68-676c-4003-9374-c0865b1d0aa5","Type":"ContainerStarted","Data":"0dabb9573bc7de976de7e9b096a25ddef69e83e479c3722aecf9db9be4325d7f"} Sep 09 03:06:11.744945 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:11.744537 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6f6cdc5d86-g7w6t" Sep 09 03:06:11.764408 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:11.764358 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6f6cdc5d86-g7w6t" podStartSLOduration=2.363106302 podStartE2EDuration="2.764343124s" totalImagesPullingTime="401.236822ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:06:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:06:11.761683709 +0000 UTC m=+2165.126018171" watchObservedRunningTime="2026-09-09 03:06:11.764343124 +0000 UTC m=+2165.128677586" Sep 09 03:06:17.756315 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:17.756282 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6f6cdc5d86-g7w6t" Sep 09 03:06:37.633030 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:37.632999 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-ntm9t_0112eec7-e85f-4902-8e6e-18381f89d197/manager/0.log" Sep 09 03:06:37.755844 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:37.755807 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-tbgkg_77e84108-8c09-487b-add4-4e5e27557030/manager/0.log" Sep 09 03:06:38.384345 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:38.384314 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6dfc788bdd-bt554_c4dfaff6-f558-4fd8-9e29-ac498d770593/copy-manifests/0.log" Sep 09 03:06:38.396806 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:38.396765 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6dfc788bdd-bt554_c4dfaff6-f558-4fd8-9e29-ac498d770593/manager/0.log" Sep 09 03:06:38.971675 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:38.971646 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6f6cdc5d86-g7w6t_7a7d2a68-676c-4003-9374-c0865b1d0aa5/maas-api/0.log" Sep 09 03:06:39.098199 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:39.098172 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29815365-qthmt_dd398c6a-778c-456f-b230-b41f1c9a6ac1/cleanup/0.log" Sep 09 03:06:39.210611 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:39.210585 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29815380-sd2qv_f4fa990c-e418-4112-80c4-765746776796/cleanup/0.log" Sep 09 03:06:40.401651 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:40.401622 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-vtrxx_0a4c77b7-3d67-41d7-bae0-1dd2ee694b40/manager/0.log" Sep 09 03:06:40.918761 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:40.918730 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-nw6h7_c2ece8e4-edcc-4693-bd0a-4b36498c01f0/manager/0.log" Sep 09 03:06:41.421893 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:41.421860 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-s2xz8_867e65e9-87eb-4a8d-b705-f4ede8a25d39/discovery/0.log" Sep 09 03:06:41.537564 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:41.537525 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7dc95696d9-md656_c6f2e295-79cb-4143-aa80-8ee4ddb99691/kube-auth-proxy/0.log" Sep 09 03:06:42.021078 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:42.021043 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-qd7xc_04c796d2-3ea0-4a3c-9635-4a5d5ef41cbe/payload-processing/0.log" Sep 09 03:06:42.619634 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:42.619602 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-xptr8_cd44bea0-3d05-4a14-a882-acb11ccef0e8/main/0.log" Sep 09 03:06:42.871696 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:42.871617 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-k6924_1173daaa-3545-412d-965e-ecff86901bba/main/0.log" Sep 09 03:06:43.123550 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:43.123469 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-t25pr_3eaf20f7-574a-4fbd-9559-6bcf0f575df2/main/0.log" Sep 09 03:06:51.940691 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:51.940656 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6lzgq_b974ece6-0b9d-4339-aa6f-203ce53dcd85/global-pull-secret-syncer/0.log" Sep 09 03:06:52.127717 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:52.127686 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qw4bt_dead4f18-8c41-41a5-9f37-897ddaee8563/konnectivity-agent/0.log" Sep 09 03:06:52.172289 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:52.172263 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-89.ec2.internal_38b43ea1a0636b4e4e4c403eeb25802f/haproxy/0.log" Sep 09 03:06:56.797634 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:56.797593 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-vtrxx_0a4c77b7-3d67-41d7-bae0-1dd2ee694b40/manager/0.log" Sep 09 03:06:56.966080 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:56.966047 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-nw6h7_c2ece8e4-edcc-4693-bd0a-4b36498c01f0/manager/0.log" Sep 09 03:06:58.963056 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:58.963026 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h7tvf_1b3ae68f-235f-41fe-ba91-9af0909ccdcf/node-exporter/0.log" Sep 09 03:06:58.992315 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:58.992288 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h7tvf_1b3ae68f-235f-41fe-ba91-9af0909ccdcf/kube-rbac-proxy/0.log" Sep 09 03:06:59.040139 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:06:59.040111 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-h7tvf_1b3ae68f-235f-41fe-ba91-9af0909ccdcf/init-textfile/0.log" Sep 09 03:07:01.935173 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:01.935135 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qh97t/perf-node-gather-daemonset-9v7dd"] Sep 09 03:07:01.935564 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:01.935471 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc" containerName="maas-api" Sep 09 03:07:01.935564 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:01.935482 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc" containerName="maas-api" Sep 09 03:07:01.935564 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:01.935537 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aff4bdc-b7e0-4a9f-a876-c82fb7e95dcc" containerName="maas-api" Sep 09 03:07:01.938476 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:01.938454 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-9v7dd" Sep 09 03:07:01.941304 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:01.941280 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qh97t\"/\"openshift-service-ca.crt\"" Sep 09 03:07:01.941501 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:01.941480 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qh97t\"/\"default-dockercfg-kzxwx\"" Sep 09 03:07:01.942532 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:01.942511 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qh97t\"/\"kube-root-ca.crt\"" Sep 09 03:07:01.949814 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:01.949791 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qh97t/perf-node-gather-daemonset-9v7dd"] Sep 09 03:07:02.125755 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:02.125718 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/860e71c3-34cd-4521-b077-c8469c0977dd-sys\") pod \"perf-node-gather-daemonset-9v7dd\" (UID: \"860e71c3-34cd-4521-b077-c8469c0977dd\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-9v7dd" Sep 09 03:07:02.125985 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:02.125770 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/860e71c3-34cd-4521-b077-c8469c0977dd-lib-modules\") pod \"perf-node-gather-daemonset-9v7dd\" (UID: \"860e71c3-34cd-4521-b077-c8469c0977dd\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-9v7dd" Sep 09 03:07:02.125985 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:02.125786 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/860e71c3-34cd-4521-b077-c8469c0977dd-podres\") pod \"perf-node-gather-daemonset-9v7dd\" (UID: \"860e71c3-34cd-4521-b077-c8469c0977dd\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-9v7dd" Sep 09 03:07:02.125985 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:02.125810 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/860e71c3-34cd-4521-b077-c8469c0977dd-proc\") pod \"perf-node-gather-daemonset-9v7dd\" (UID: \"860e71c3-34cd-4521-b077-c8469c0977dd\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-9v7dd" Sep 09 03:07:02.125985 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:02.125826 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcft4\" (UniqueName: \"kubernetes.io/projected/860e71c3-34cd-4521-b077-c8469c0977dd-kube-api-access-vcft4\") pod \"perf-node-gather-daemonset-9v7dd\" (UID: \"860e71c3-34cd-4521-b077-c8469c0977dd\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-9v7dd" Sep 09 03:07:02.226671 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:02.226590 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/860e71c3-34cd-4521-b077-c8469c0977dd-sys\") pod \"perf-node-gather-daemonset-9v7dd\" (UID: \"860e71c3-34cd-4521-b077-c8469c0977dd\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-9v7dd" Sep 09 03:07:02.226671 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:02.226636 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/860e71c3-34cd-4521-b077-c8469c0977dd-lib-modules\") pod \"perf-node-gather-daemonset-9v7dd\" (UID: \"860e71c3-34cd-4521-b077-c8469c0977dd\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-9v7dd" Sep 09 03:07:02.226671 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:02.226653 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/860e71c3-34cd-4521-b077-c8469c0977dd-podres\") pod \"perf-node-gather-daemonset-9v7dd\" (UID: \"860e71c3-34cd-4521-b077-c8469c0977dd\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-9v7dd" Sep 09 03:07:02.226891 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:02.226675 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/860e71c3-34cd-4521-b077-c8469c0977dd-proc\") pod \"perf-node-gather-daemonset-9v7dd\" (UID: \"860e71c3-34cd-4521-b077-c8469c0977dd\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-9v7dd" Sep 09 03:07:02.226891 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:02.226726 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/860e71c3-34cd-4521-b077-c8469c0977dd-proc\") pod \"perf-node-gather-daemonset-9v7dd\" (UID: \"860e71c3-34cd-4521-b077-c8469c0977dd\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-9v7dd" Sep 09 03:07:02.226891 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:02.226729 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/860e71c3-34cd-4521-b077-c8469c0977dd-sys\") pod \"perf-node-gather-daemonset-9v7dd\" (UID: \"860e71c3-34cd-4521-b077-c8469c0977dd\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-9v7dd" Sep 09 03:07:02.226891 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:02.226753 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcft4\" (UniqueName: \"kubernetes.io/projected/860e71c3-34cd-4521-b077-c8469c0977dd-kube-api-access-vcft4\") pod \"perf-node-gather-daemonset-9v7dd\" (UID: \"860e71c3-34cd-4521-b077-c8469c0977dd\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-9v7dd" Sep 09 03:07:02.226891 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:02.226779 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/860e71c3-34cd-4521-b077-c8469c0977dd-lib-modules\") pod \"perf-node-gather-daemonset-9v7dd\" (UID: \"860e71c3-34cd-4521-b077-c8469c0977dd\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-9v7dd" Sep 09 03:07:02.226891 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:02.226783 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/860e71c3-34cd-4521-b077-c8469c0977dd-podres\") pod \"perf-node-gather-daemonset-9v7dd\" (UID: \"860e71c3-34cd-4521-b077-c8469c0977dd\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-9v7dd" Sep 09 03:07:02.242163 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:02.242131 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcft4\" (UniqueName: \"kubernetes.io/projected/860e71c3-34cd-4521-b077-c8469c0977dd-kube-api-access-vcft4\") pod \"perf-node-gather-daemonset-9v7dd\" (UID: \"860e71c3-34cd-4521-b077-c8469c0977dd\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-9v7dd" Sep 09 03:07:02.249047 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:02.249020 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-9v7dd" Sep 09 03:07:02.523525 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:02.523498 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qh97t/perf-node-gather-daemonset-9v7dd"] Sep 09 03:07:02.524942 ip-10-0-136-89 kubenswrapper[2577]: W0909 03:07:02.524906 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod860e71c3_34cd_4521_b077_c8469c0977dd.slice/crio-d82d50b019396f1cec236f81a721e7b765ea56bdf8bf0507beb60745566d7adb WatchSource:0}: Error finding container d82d50b019396f1cec236f81a721e7b765ea56bdf8bf0507beb60745566d7adb: Status 404 returned error can't find the container with id d82d50b019396f1cec236f81a721e7b765ea56bdf8bf0507beb60745566d7adb Sep 09 03:07:03.022007 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:03.021949 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-9v7dd" event={"ID":"860e71c3-34cd-4521-b077-c8469c0977dd","Type":"ContainerStarted","Data":"790c1142ce2940063ebde221fea8e70b4b9d341b4cd338fcccf54b270a8cb60c"} Sep 09 03:07:03.022460 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:03.022023 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-9v7dd" event={"ID":"860e71c3-34cd-4521-b077-c8469c0977dd","Type":"ContainerStarted","Data":"d82d50b019396f1cec236f81a721e7b765ea56bdf8bf0507beb60745566d7adb"} Sep 09 03:07:03.022460 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:03.022175 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-9v7dd" Sep 09 03:07:03.045399 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:03.045351 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-9v7dd" podStartSLOduration=2.045333557 podStartE2EDuration="2.045333557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:07:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:07:03.042920628 +0000 UTC m=+2216.407255093" watchObservedRunningTime="2026-09-09 03:07:03.045333557 +0000 UTC m=+2216.409668020" Sep 09 03:07:04.626721 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:04.626688 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-frnzb_98d0408e-2648-467e-a2fa-a553b001c70c/dns/0.log" Sep 09 03:07:04.652049 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:04.652019 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-frnzb_98d0408e-2648-467e-a2fa-a553b001c70c/kube-rbac-proxy/0.log" Sep 09 03:07:04.812514 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:04.812481 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pd8dm_19afdc10-5a52-4e9d-a84f-04df6136a1c0/dns-node-resolver/0.log" Sep 09 03:07:05.525746 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:05.525720 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tvhb5_54c26b32-0189-4a6a-a569-c2f859d87c8b/node-ca/0.log" Sep 09 03:07:06.793230 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:06.793190 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-s2xz8_867e65e9-87eb-4a8d-b705-f4ede8a25d39/discovery/0.log" Sep 09 03:07:06.816917 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:06.816890 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7dc95696d9-md656_c6f2e295-79cb-4143-aa80-8ee4ddb99691/kube-auth-proxy/0.log" Sep 09 03:07:06.976142 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:06.976109 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-qd7xc_04c796d2-3ea0-4a3c-9635-4a5d5ef41cbe/payload-processing/0.log" Sep 09 03:07:07.616126 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:07.616095 2577 scope.go:117] "RemoveContainer" containerID="9b847d836d711780156e5c59bba154b5b1fb46c61023c67122d3cb51e9a0a097" Sep 09 03:07:07.625132 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:07.625109 2577 scope.go:117] "RemoveContainer" containerID="a2a8baa65c8aee22f7da1feeb73e22b7ebaac1ea253d23950e0ec4446c42526f" Sep 09 03:07:07.843342 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:07.843300 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pkdtz_1ceefd67-0082-4ecb-9d62-48f62ac00b58/serve-healthcheck-canary/0.log" Sep 09 03:07:08.468166 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:08.468121 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w65x5_d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7/kube-rbac-proxy/0.log" Sep 09 03:07:08.489981 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:08.489940 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w65x5_d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7/exporter/0.log" Sep 09 03:07:08.508229 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:08.508209 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w65x5_d4c6d9d2-8b3b-4f28-beff-d4c2936e4df7/extractor/0.log" Sep 09 03:07:09.038162 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:09.038131 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-9v7dd" Sep 09 03:07:10.676422 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:10.676388 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-ntm9t_0112eec7-e85f-4902-8e6e-18381f89d197/manager/0.log" Sep 09 03:07:10.851599 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:10.851565 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-tbgkg_77e84108-8c09-487b-add4-4e5e27557030/manager/0.log" Sep 09 03:07:11.173348 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:11.173268 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6dfc788bdd-bt554_c4dfaff6-f558-4fd8-9e29-ac498d770593/manager/0.log" Sep 09 03:07:11.189422 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:11.189398 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6dfc788bdd-bt554_c4dfaff6-f558-4fd8-9e29-ac498d770593/copy-manifests/0.log" Sep 09 03:07:12.624012 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:12.623981 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-s2xz8_867e65e9-87eb-4a8d-b705-f4ede8a25d39/discovery/0.log" Sep 09 03:07:12.636967 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:12.636940 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7dc95696d9-md656_c6f2e295-79cb-4143-aa80-8ee4ddb99691/kube-auth-proxy/0.log" Sep 09 03:07:12.766405 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:12.766374 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-qd7xc_04c796d2-3ea0-4a3c-9635-4a5d5ef41cbe/payload-processing/0.log" Sep 09 03:07:14.446641 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:14.446603 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-s2xz8_867e65e9-87eb-4a8d-b705-f4ede8a25d39/discovery/0.log" Sep 09 03:07:14.463591 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:14.463562 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7dc95696d9-md656_c6f2e295-79cb-4143-aa80-8ee4ddb99691/kube-auth-proxy/0.log" Sep 09 03:07:14.591157 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:14.591128 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-qd7xc_04c796d2-3ea0-4a3c-9635-4a5d5ef41cbe/payload-processing/0.log" Sep 09 03:07:17.912216 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:17.912181 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-2lgf6_d06cd08c-40fb-449c-9b6a-78736d4c362e/kube-storage-version-migrator-operator/1.log" Sep 09 03:07:17.913285 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:17.913266 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-2lgf6_d06cd08c-40fb-449c-9b6a-78736d4c362e/kube-storage-version-migrator-operator/0.log" Sep 09 03:07:19.144064 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:19.144035 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zjstd_95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1/kube-multus-additional-cni-plugins/0.log" Sep 09 03:07:19.167573 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:19.167516 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zjstd_95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1/egress-router-binary-copy/0.log" Sep 09 03:07:19.187845 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:19.187820 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zjstd_95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1/cni-plugins/0.log" Sep 09 03:07:19.204776 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:19.204756 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zjstd_95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1/bond-cni-plugin/0.log" Sep 09 03:07:19.222785 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:19.222757 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zjstd_95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1/routeoverride-cni/0.log" Sep 09 03:07:19.239636 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:19.239612 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zjstd_95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1/whereabouts-cni-bincopy/0.log" Sep 09 03:07:19.257765 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:19.257733 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zjstd_95b9fbf9-fb9c-44c5-bdde-9306b4be5eb1/whereabouts-cni/0.log" Sep 09 03:07:19.455703 ip-10-0-136-89 kubenswrapper[2577]: I0909 03:07:19.455591 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dj8dq_b7a1ff67-afbc-4e76-aa24-d1a8cebb3888/kube-multus/0.log"