Sep 04 22:06:37.118868 ip-10-0-137-219 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 04 22:06:37.118879 ip-10-0-137-219 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 04 22:06:37.118886 ip-10-0-137-219 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 04 22:06:37.119118 ip-10-0-137-219 systemd[1]: Failed to start Kubernetes Kubelet. Sep 04 22:06:46.303149 ip-10-0-137-219 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found. Sep 04 22:06:46.303161 ip-10-0-137-219 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 9304516009304cb4aebed2969cc9d11c -- Sep 04 22:08:16.109824 ip-10-0-137-219 systemd[1]: Starting Kubernetes Kubelet... Sep 04 22:08:16.553674 ip-10-0-137-219 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 04 22:08:16.553674 ip-10-0-137-219 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 04 22:08:16.553674 ip-10-0-137-219 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 04 22:08:16.553674 ip-10-0-137-219 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 04 22:08:16.553674 ip-10-0-137-219 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 04 22:08:16.555651 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.555564 2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 04 22:08:16.560410 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560381 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 04 22:08:16.560410 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560401 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 04 22:08:16.560410 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560405 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 04 22:08:16.560410 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560410 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 04 22:08:16.560410 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560415 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 04 22:08:16.560410 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560418 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 04 22:08:16.560410 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560421 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 04 22:08:16.560701 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560424 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 04 22:08:16.560701 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560428 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 04 22:08:16.560701 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560431 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 04 22:08:16.560701 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560434 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 04 22:08:16.560701 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560437 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 04 22:08:16.560701 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560440 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 04 22:08:16.560701 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560442 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 04 22:08:16.560701 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560445 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 04 22:08:16.560701 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560448 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 04 22:08:16.560701 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560451 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 04 22:08:16.560701 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560454 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 04 22:08:16.560701 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560457 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 04 22:08:16.560701 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560459 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 04 22:08:16.560701 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560462 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 04 22:08:16.560701 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560465 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 04 22:08:16.560701 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560468 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 04 22:08:16.560701 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560472 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 04 22:08:16.560701 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560475 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 04 22:08:16.560701 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560485 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 04 22:08:16.561163 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560488 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 04 22:08:16.561163 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560490 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 04 22:08:16.561163 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560493 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 04 22:08:16.561163 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560495 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 04 22:08:16.561163 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560498 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 04 22:08:16.561163 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560500 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 04 22:08:16.561163 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560503 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 04 22:08:16.561163 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560505 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 04 22:08:16.561163 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560508 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 04 22:08:16.561163 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560510 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 04 22:08:16.561163 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560513 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 04 22:08:16.561163 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560515 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 04 22:08:16.561163 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560517 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 04 22:08:16.561163 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560520 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 04 22:08:16.561163 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560523 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 04 22:08:16.561163 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560526 2574 feature_gate.go:328] unrecognized feature gate: Example Sep 04 22:08:16.561163 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560529 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 04 22:08:16.561163 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560532 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 04 22:08:16.561163 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560534 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 04 22:08:16.561163 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560537 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 04 22:08:16.561672 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560539 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 04 22:08:16.561672 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560542 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 04 22:08:16.561672 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560544 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 04 22:08:16.561672 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560561 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 04 22:08:16.561672 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560564 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 04 22:08:16.561672 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560567 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 04 22:08:16.561672 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560569 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 04 22:08:16.561672 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560572 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 04 22:08:16.561672 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560574 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 04 22:08:16.561672 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560577 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 04 22:08:16.561672 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560580 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 04 22:08:16.561672 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560583 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 04 22:08:16.561672 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560587 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 04 22:08:16.561672 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560590 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 04 22:08:16.561672 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560593 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 04 22:08:16.561672 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560596 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 04 22:08:16.561672 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560599 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 04 22:08:16.561672 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560602 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 04 22:08:16.561672 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560605 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 04 22:08:16.562120 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560608 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 04 22:08:16.562120 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560610 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 04 22:08:16.562120 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560613 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 04 22:08:16.562120 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560616 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 04 22:08:16.562120 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560619 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 04 22:08:16.562120 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560621 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 04 22:08:16.562120 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560623 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 04 22:08:16.562120 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560627 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 04 22:08:16.562120 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560630 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 04 22:08:16.562120 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560633 2574 feature_gate.go:328] unrecognized feature gate: Example2 Sep 04 22:08:16.562120 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560636 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 04 22:08:16.562120 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560639 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 04 22:08:16.562120 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560641 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 04 22:08:16.562120 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560644 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 04 22:08:16.562120 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560647 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 04 22:08:16.562120 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560649 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 04 22:08:16.562120 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560652 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 04 22:08:16.562120 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560654 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 04 22:08:16.562120 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560658 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 04 22:08:16.562120 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560661 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 04 22:08:16.562690 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560663 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 04 22:08:16.562690 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560666 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 04 22:08:16.562690 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560668 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 04 22:08:16.562690 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560671 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 04 22:08:16.562690 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560673 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 04 22:08:16.562690 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560676 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 04 22:08:16.562690 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560679 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 04 22:08:16.562690 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560682 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 04 22:08:16.562690 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560684 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 04 22:08:16.562690 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560687 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 04 22:08:16.562690 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560689 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 04 22:08:16.562690 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560692 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 04 22:08:16.562690 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560694 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 04 22:08:16.562690 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.560697 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 04 22:08:16.562690 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561420 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 04 22:08:16.562690 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561428 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 04 22:08:16.562690 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561431 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 04 22:08:16.562690 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561436 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 04 22:08:16.562690 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561439 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 04 22:08:16.562690 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561441 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 04 22:08:16.563187 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561444 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 04 22:08:16.563187 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561447 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 04 22:08:16.563187 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561450 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 04 22:08:16.563187 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561453 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 04 22:08:16.563187 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561455 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 04 22:08:16.563187 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561458 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 04 22:08:16.563187 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561460 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 04 22:08:16.563187 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561463 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 04 22:08:16.563187 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561465 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 04 22:08:16.563187 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561468 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 04 22:08:16.563187 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561470 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 04 22:08:16.563187 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561474 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 04 22:08:16.563187 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561477 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 04 22:08:16.563187 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561479 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 04 22:08:16.563187 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561482 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 04 22:08:16.563187 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561485 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 04 22:08:16.563187 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561487 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 04 22:08:16.563187 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561490 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 04 22:08:16.563187 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561493 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 04 22:08:16.563187 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561495 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 04 22:08:16.563699 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561498 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 04 22:08:16.563699 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561500 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 04 22:08:16.563699 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561503 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 04 22:08:16.563699 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561506 2574 feature_gate.go:328] unrecognized feature gate: Example2 Sep 04 22:08:16.563699 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561509 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 04 22:08:16.563699 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561512 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 04 22:08:16.563699 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561514 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 04 22:08:16.563699 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561517 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 04 22:08:16.563699 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561519 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 04 22:08:16.563699 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561523 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 04 22:08:16.563699 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561526 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 04 22:08:16.563699 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561528 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 04 22:08:16.563699 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561531 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 04 22:08:16.563699 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561533 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 04 22:08:16.563699 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561537 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 04 22:08:16.563699 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561540 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 04 22:08:16.563699 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561543 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 04 22:08:16.563699 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561558 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 04 22:08:16.563699 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561561 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 04 22:08:16.563699 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561564 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 04 22:08:16.564191 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561566 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 04 22:08:16.564191 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561569 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 04 22:08:16.564191 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561571 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 04 22:08:16.564191 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561574 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 04 22:08:16.564191 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561578 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 04 22:08:16.564191 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561581 2574 feature_gate.go:328] unrecognized feature gate: Example Sep 04 22:08:16.564191 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561583 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 04 22:08:16.564191 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561586 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 04 22:08:16.564191 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561588 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 04 22:08:16.564191 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561591 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 04 22:08:16.564191 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561593 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 04 22:08:16.564191 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561596 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 04 22:08:16.564191 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561598 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 04 22:08:16.564191 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561601 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 04 22:08:16.564191 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561603 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 04 22:08:16.564191 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561606 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 04 22:08:16.564191 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561609 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 04 22:08:16.564191 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561611 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 04 22:08:16.564191 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561614 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 04 22:08:16.564191 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561617 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 04 22:08:16.564717 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561620 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 04 22:08:16.564717 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561623 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 04 22:08:16.564717 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561625 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 04 22:08:16.564717 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561628 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 04 22:08:16.564717 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561631 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 04 22:08:16.564717 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561633 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 04 22:08:16.564717 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561636 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 04 22:08:16.564717 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561638 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 04 22:08:16.564717 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561641 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 04 22:08:16.564717 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561643 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 04 22:08:16.564717 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561645 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 04 22:08:16.564717 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561648 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 04 22:08:16.564717 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561651 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 04 22:08:16.564717 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561653 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 04 22:08:16.564717 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561656 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 04 22:08:16.564717 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561658 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 04 22:08:16.564717 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561661 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 04 22:08:16.564717 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561664 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 04 22:08:16.564717 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561667 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 04 22:08:16.565189 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561672 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 04 22:08:16.565189 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561676 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 04 22:08:16.565189 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561679 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 04 22:08:16.565189 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561682 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 04 22:08:16.565189 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561685 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 04 22:08:16.565189 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561688 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 04 22:08:16.565189 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561691 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 04 22:08:16.565189 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561694 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 04 22:08:16.565189 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561697 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 04 22:08:16.565189 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561699 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 04 22:08:16.565189 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561702 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 04 22:08:16.565189 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561706 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 04 22:08:16.565189 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561709 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 04 22:08:16.565189 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.561712 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 04 22:08:16.565189 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562108 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 04 22:08:16.565189 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562114 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 04 22:08:16.565189 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562117 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 04 22:08:16.565189 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562119 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 04 22:08:16.565189 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562122 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 04 22:08:16.565674 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562125 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 04 22:08:16.565674 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562128 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 04 22:08:16.565674 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562131 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 04 22:08:16.565674 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562133 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 04 22:08:16.565674 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562136 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 04 22:08:16.565674 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562139 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 04 22:08:16.565674 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562142 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 04 22:08:16.565674 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562144 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 04 22:08:16.565674 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562147 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 04 22:08:16.565674 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562150 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 04 22:08:16.565674 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562153 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 04 22:08:16.565674 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562156 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 04 22:08:16.565674 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562159 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 04 22:08:16.565674 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562163 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 04 22:08:16.565674 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562166 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 04 22:08:16.565674 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562170 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 04 22:08:16.565674 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562173 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 04 22:08:16.565674 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562176 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 04 22:08:16.565674 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562179 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 04 22:08:16.566145 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562182 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 04 22:08:16.566145 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562185 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 04 22:08:16.566145 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562188 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 04 22:08:16.566145 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562191 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 04 22:08:16.566145 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562193 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 04 22:08:16.566145 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562195 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 04 22:08:16.566145 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562198 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 04 22:08:16.566145 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562200 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 04 22:08:16.566145 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562203 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 04 22:08:16.566145 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562205 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 04 22:08:16.566145 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562208 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 04 22:08:16.566145 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562210 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 04 22:08:16.566145 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562214 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 04 22:08:16.566145 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562217 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 04 22:08:16.566145 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562220 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 04 22:08:16.566145 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562223 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 04 22:08:16.566145 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562226 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 04 22:08:16.566145 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562229 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 04 22:08:16.566145 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562232 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 04 22:08:16.566622 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562234 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 04 22:08:16.566622 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562237 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 04 22:08:16.566622 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562242 2574 feature_gate.go:328] unrecognized feature gate: Example Sep 04 22:08:16.566622 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562245 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 04 22:08:16.566622 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562248 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 04 22:08:16.566622 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562250 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 04 22:08:16.566622 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562253 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 04 22:08:16.566622 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562256 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 04 22:08:16.566622 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562259 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 04 22:08:16.566622 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562262 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 04 22:08:16.566622 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562264 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 04 22:08:16.566622 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562267 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 04 22:08:16.566622 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562270 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 04 22:08:16.566622 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562272 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 04 22:08:16.566622 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562275 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 04 22:08:16.566622 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562277 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 04 22:08:16.566622 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562280 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 04 22:08:16.566622 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562283 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 04 22:08:16.566622 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562285 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 04 22:08:16.566622 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562287 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 04 22:08:16.567111 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562290 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 04 22:08:16.567111 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562292 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 04 22:08:16.567111 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562295 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 04 22:08:16.567111 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562297 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 04 22:08:16.567111 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562300 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 04 22:08:16.567111 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562302 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 04 22:08:16.567111 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562305 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 04 22:08:16.567111 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562307 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 04 22:08:16.567111 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562310 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 04 22:08:16.567111 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562312 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 04 22:08:16.567111 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562315 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 04 22:08:16.567111 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562317 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 04 22:08:16.567111 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562319 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 04 22:08:16.567111 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562321 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 04 22:08:16.567111 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562325 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 04 22:08:16.567111 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562328 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 04 22:08:16.567111 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562330 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 04 22:08:16.567111 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562333 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 04 22:08:16.567111 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562335 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 04 22:08:16.567111 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562338 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 04 22:08:16.567695 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562342 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 04 22:08:16.567695 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562345 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 04 22:08:16.567695 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562347 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 04 22:08:16.567695 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562351 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 04 22:08:16.567695 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562354 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 04 22:08:16.567695 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562359 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 04 22:08:16.567695 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562361 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 04 22:08:16.567695 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562364 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 04 22:08:16.567695 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562366 2574 feature_gate.go:328] unrecognized feature gate: Example2 Sep 04 22:08:16.567695 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562369 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 04 22:08:16.567695 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562371 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 04 22:08:16.567695 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562374 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 04 22:08:16.567695 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562376 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 04 22:08:16.567695 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562379 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 04 22:08:16.567695 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562382 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 04 22:08:16.567695 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.562384 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 04 22:08:16.567695 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562447 2574 flags.go:64] FLAG: --address="0.0.0.0" Sep 04 22:08:16.567695 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562483 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 04 22:08:16.567695 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562490 2574 flags.go:64] FLAG: --anonymous-auth="true" Sep 04 22:08:16.567695 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562494 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 04 22:08:16.567695 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562498 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 04 22:08:16.568217 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562501 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 04 22:08:16.568217 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562505 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 04 22:08:16.568217 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562509 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 04 22:08:16.568217 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562512 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 04 22:08:16.568217 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562515 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 04 22:08:16.568217 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562519 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 04 22:08:16.568217 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562523 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 04 22:08:16.568217 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562526 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 04 22:08:16.568217 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562529 2574 flags.go:64] FLAG: --cgroup-root="" Sep 04 22:08:16.568217 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562531 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 04 22:08:16.568217 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562534 2574 flags.go:64] FLAG: --client-ca-file="" Sep 04 22:08:16.568217 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562561 2574 flags.go:64] FLAG: --cloud-provider="external" Sep 04 22:08:16.568217 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562567 2574 flags.go:64] FLAG: --cluster-dns="[]" Sep 04 22:08:16.568217 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562572 2574 flags.go:64] FLAG: --cluster-domain="" Sep 04 22:08:16.568217 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562577 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 04 22:08:16.568217 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562581 2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 04 22:08:16.568217 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562584 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 04 22:08:16.568217 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562587 2574 flags.go:64] FLAG: --container-log-max-files="5" Sep 04 22:08:16.568217 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562597 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 04 22:08:16.568217 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562600 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 04 22:08:16.568217 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562603 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 04 22:08:16.568217 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562606 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 04 22:08:16.568217 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562609 2574 flags.go:64] FLAG: --contention-profiling="false" Sep 04 22:08:16.568217 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562612 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 04 22:08:16.568810 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562615 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 04 22:08:16.568810 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562618 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 04 22:08:16.568810 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562621 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 04 22:08:16.568810 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562625 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 04 22:08:16.568810 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562628 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 04 22:08:16.568810 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562631 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 04 22:08:16.568810 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562633 2574 flags.go:64] FLAG: --enable-load-reader="false" Sep 04 22:08:16.568810 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562636 2574 flags.go:64] FLAG: --enable-server="true" Sep 04 22:08:16.568810 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562639 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 04 22:08:16.568810 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562647 2574 flags.go:64] FLAG: --event-burst="100" Sep 04 22:08:16.568810 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562650 2574 flags.go:64] FLAG: --event-qps="50" Sep 04 22:08:16.568810 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562652 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 04 22:08:16.568810 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562655 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 04 22:08:16.568810 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562660 2574 flags.go:64] FLAG: --eviction-hard="" Sep 04 22:08:16.568810 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562664 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 04 22:08:16.568810 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562666 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 04 22:08:16.568810 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562669 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 04 22:08:16.568810 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562672 2574 flags.go:64] FLAG: --eviction-soft="" Sep 04 22:08:16.568810 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562675 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 04 22:08:16.568810 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562679 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 04 22:08:16.568810 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562687 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 04 22:08:16.568810 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562691 2574 flags.go:64] FLAG: --experimental-mounter-path="" Sep 04 22:08:16.568810 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562694 2574 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 04 22:08:16.568810 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562697 2574 flags.go:64] FLAG: --fail-swap-on="true" Sep 04 22:08:16.568810 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562700 2574 flags.go:64] FLAG: --feature-gates="" Sep 04 22:08:16.569477 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562704 2574 flags.go:64] FLAG: --file-check-frequency="20s" Sep 04 22:08:16.569477 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562707 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 04 22:08:16.569477 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562710 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 04 22:08:16.569477 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562713 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 04 22:08:16.569477 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562716 2574 flags.go:64] FLAG: --healthz-port="10248" Sep 04 22:08:16.569477 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562719 2574 flags.go:64] FLAG: --help="false" Sep 04 22:08:16.569477 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562722 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-137-219.ec2.internal" Sep 04 22:08:16.569477 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562725 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 04 22:08:16.569477 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562728 2574 flags.go:64] FLAG: --http-check-frequency="20s" Sep 04 22:08:16.569477 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562731 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 04 22:08:16.569477 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562734 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 04 22:08:16.569477 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562737 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 04 22:08:16.569477 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562740 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 04 22:08:16.569477 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562742 2574 flags.go:64] FLAG: --image-service-endpoint="" Sep 04 22:08:16.569477 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562745 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 04 22:08:16.569477 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562748 2574 flags.go:64] FLAG: --kube-api-burst="100" Sep 04 22:08:16.569477 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562751 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 04 22:08:16.569477 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562754 2574 flags.go:64] FLAG: --kube-api-qps="50" Sep 04 22:08:16.569477 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562757 2574 flags.go:64] FLAG: --kube-reserved="" Sep 04 22:08:16.569477 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562759 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 04 22:08:16.569477 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562763 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 04 22:08:16.569477 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562765 2574 flags.go:64] FLAG: --kubelet-cgroups="" Sep 04 22:08:16.569477 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562768 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 04 22:08:16.569477 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562771 2574 flags.go:64] FLAG: --lock-file="" Sep 04 22:08:16.570075 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562774 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 04 22:08:16.570075 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562776 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 04 22:08:16.570075 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562780 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 04 22:08:16.570075 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562786 2574 flags.go:64] FLAG: --log-json-split-stream="false" Sep 04 22:08:16.570075 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562794 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 04 22:08:16.570075 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562797 2574 flags.go:64] FLAG: --log-text-split-stream="false" Sep 04 22:08:16.570075 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562800 2574 flags.go:64] FLAG: --logging-format="text" Sep 04 22:08:16.570075 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562802 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 04 22:08:16.570075 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562806 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 04 22:08:16.570075 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562808 2574 flags.go:64] FLAG: --manifest-url="" Sep 04 22:08:16.570075 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562811 2574 flags.go:64] FLAG: --manifest-url-header="" Sep 04 22:08:16.570075 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562815 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 04 22:08:16.570075 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562818 2574 flags.go:64] FLAG: --max-open-files="1000000" Sep 04 22:08:16.570075 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562822 2574 flags.go:64] FLAG: --max-pods="110" Sep 04 22:08:16.570075 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562825 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 04 22:08:16.570075 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562827 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 04 22:08:16.570075 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562830 2574 flags.go:64] FLAG: --memory-manager-policy="None" Sep 04 22:08:16.570075 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562833 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 04 22:08:16.570075 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562836 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 04 22:08:16.570075 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562839 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 04 22:08:16.570075 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562842 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 04 22:08:16.570075 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562849 2574 flags.go:64] FLAG: --node-status-max-images="50" Sep 04 22:08:16.570075 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562851 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 04 22:08:16.570075 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562854 2574 flags.go:64] FLAG: --oom-score-adj="-999" Sep 04 22:08:16.570075 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562857 2574 flags.go:64] FLAG: --pod-cidr="" Sep 04 22:08:16.570708 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562860 2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 04 22:08:16.570708 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562865 2574 flags.go:64] FLAG: --pod-manifest-path="" Sep 04 22:08:16.570708 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562868 2574 flags.go:64] FLAG: --pod-max-pids="-1" Sep 04 22:08:16.570708 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562871 2574 flags.go:64] FLAG: --pods-per-core="0" Sep 04 22:08:16.570708 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562874 2574 flags.go:64] FLAG: --port="10250" Sep 04 22:08:16.570708 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562877 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 04 22:08:16.570708 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562879 2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03f38de74a089506a" Sep 04 22:08:16.570708 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562882 2574 flags.go:64] FLAG: --qos-reserved="" Sep 04 22:08:16.570708 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562885 2574 flags.go:64] FLAG: --read-only-port="10255" Sep 04 22:08:16.570708 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562888 2574 flags.go:64] FLAG: --register-node="true" Sep 04 22:08:16.570708 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562893 2574 flags.go:64] FLAG: --register-with-taints="" Sep 04 22:08:16.570708 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562896 2574 flags.go:64] FLAG: --registry-burst="10" Sep 04 22:08:16.570708 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562900 2574 flags.go:64] FLAG: --registry-qps="5" Sep 04 22:08:16.570708 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562902 2574 flags.go:64] FLAG: --reserved-cpus="" Sep 04 22:08:16.570708 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562905 2574 flags.go:64] FLAG: --reserved-memory="" Sep 04 22:08:16.570708 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562909 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 04 22:08:16.570708 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562912 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 04 22:08:16.570708 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562915 2574 flags.go:64] FLAG: --rotate-certificates="false" Sep 04 22:08:16.570708 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562918 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 04 22:08:16.570708 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562921 2574 flags.go:64] FLAG: --runonce="false" Sep 04 22:08:16.570708 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562923 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 04 22:08:16.570708 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562926 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 04 22:08:16.570708 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562929 2574 flags.go:64] FLAG: --seccomp-default="false" Sep 04 22:08:16.570708 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562932 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 04 22:08:16.570708 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562935 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 04 22:08:16.571336 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562938 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 04 22:08:16.571336 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562941 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 04 22:08:16.571336 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562944 2574 flags.go:64] FLAG: --storage-driver-password="root" Sep 04 22:08:16.571336 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562947 2574 flags.go:64] FLAG: --storage-driver-secure="false" Sep 04 22:08:16.571336 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562950 2574 flags.go:64] FLAG: --storage-driver-table="stats" Sep 04 22:08:16.571336 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562953 2574 flags.go:64] FLAG: --storage-driver-user="root" Sep 04 22:08:16.571336 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562956 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 04 22:08:16.571336 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562959 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 04 22:08:16.571336 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562962 2574 flags.go:64] FLAG: --system-cgroups="" Sep 04 22:08:16.571336 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562965 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 04 22:08:16.571336 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562970 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 04 22:08:16.571336 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562973 2574 flags.go:64] FLAG: --tls-cert-file="" Sep 04 22:08:16.571336 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562976 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 04 22:08:16.571336 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562982 2574 flags.go:64] FLAG: --tls-min-version="" Sep 04 22:08:16.571336 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562985 2574 flags.go:64] FLAG: --tls-private-key-file="" Sep 04 22:08:16.571336 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562987 2574 flags.go:64] FLAG: --topology-manager-policy="none" Sep 04 22:08:16.571336 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562990 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 04 22:08:16.571336 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562994 2574 flags.go:64] FLAG: --topology-manager-scope="container" Sep 04 22:08:16.571336 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.562997 2574 flags.go:64] FLAG: --v="2" Sep 04 22:08:16.571336 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.563001 2574 flags.go:64] FLAG: --version="false" Sep 04 22:08:16.571336 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.563005 2574 flags.go:64] FLAG: --vmodule="" Sep 04 22:08:16.571336 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.563008 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 04 22:08:16.571336 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.563012 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 04 22:08:16.571336 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563136 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 04 22:08:16.571336 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563140 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 04 22:08:16.571954 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563143 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 04 22:08:16.571954 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563147 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 04 22:08:16.571954 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563151 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 04 22:08:16.571954 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563155 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 04 22:08:16.571954 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563157 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 04 22:08:16.571954 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563160 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 04 22:08:16.571954 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563163 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 04 22:08:16.571954 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563165 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 04 22:08:16.571954 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563168 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 04 22:08:16.571954 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563171 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 04 22:08:16.571954 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563173 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 04 22:08:16.571954 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563175 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 04 22:08:16.571954 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563178 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 04 22:08:16.571954 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563181 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 04 22:08:16.571954 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563183 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 04 22:08:16.571954 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563186 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 04 22:08:16.571954 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563189 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 04 22:08:16.571954 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563192 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 04 22:08:16.571954 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563195 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 04 22:08:16.571954 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563197 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 04 22:08:16.572433 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563200 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 04 22:08:16.572433 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563202 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 04 22:08:16.572433 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563205 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 04 22:08:16.572433 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563207 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 04 22:08:16.572433 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563211 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 04 22:08:16.572433 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563214 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 04 22:08:16.572433 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563216 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 04 22:08:16.572433 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563219 2574 feature_gate.go:328] unrecognized feature gate: Example2 Sep 04 22:08:16.572433 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563227 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 04 22:08:16.572433 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563230 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 04 22:08:16.572433 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563233 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 04 22:08:16.572433 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563235 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 04 22:08:16.572433 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563238 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 04 22:08:16.572433 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563240 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 04 22:08:16.572433 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563243 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 04 22:08:16.572433 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563247 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 04 22:08:16.572433 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563250 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 04 22:08:16.572433 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563253 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 04 22:08:16.572433 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563255 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 04 22:08:16.572926 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563258 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 04 22:08:16.572926 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563261 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 04 22:08:16.572926 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563263 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 04 22:08:16.572926 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563266 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 04 22:08:16.572926 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563268 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 04 22:08:16.572926 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563270 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 04 22:08:16.572926 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563273 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 04 22:08:16.572926 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563276 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 04 22:08:16.572926 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563281 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 04 22:08:16.572926 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563283 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 04 22:08:16.572926 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563286 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 04 22:08:16.572926 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563289 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 04 22:08:16.572926 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563291 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 04 22:08:16.572926 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563294 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 04 22:08:16.572926 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563296 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 04 22:08:16.572926 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563298 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 04 22:08:16.572926 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563301 2574 feature_gate.go:328] unrecognized feature gate: Example Sep 04 22:08:16.572926 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563305 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 04 22:08:16.572926 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563307 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 04 22:08:16.572926 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563310 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 04 22:08:16.573425 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563312 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 04 22:08:16.573425 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563315 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 04 22:08:16.573425 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563323 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 04 22:08:16.573425 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563326 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 04 22:08:16.573425 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563328 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 04 22:08:16.573425 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563331 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 04 22:08:16.573425 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563333 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 04 22:08:16.573425 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563336 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 04 22:08:16.573425 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563338 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 04 22:08:16.573425 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563341 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 04 22:08:16.573425 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563343 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 04 22:08:16.573425 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563346 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 04 22:08:16.573425 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563348 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 04 22:08:16.573425 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563351 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 04 22:08:16.573425 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563353 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 04 22:08:16.573425 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563356 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 04 22:08:16.573425 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563358 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 04 22:08:16.573425 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563361 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 04 22:08:16.573425 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563363 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 04 22:08:16.573425 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563366 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 04 22:08:16.574015 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563369 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 04 22:08:16.574015 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563372 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 04 22:08:16.574015 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563374 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 04 22:08:16.574015 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563377 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 04 22:08:16.574015 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563381 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 04 22:08:16.574015 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563384 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 04 22:08:16.574015 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563397 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 04 22:08:16.574015 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563400 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 04 22:08:16.574015 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563403 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 04 22:08:16.574015 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563407 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 04 22:08:16.574015 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563410 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 04 22:08:16.574015 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563412 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 04 22:08:16.574015 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563416 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 04 22:08:16.574015 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563418 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 04 22:08:16.574015 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563421 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 04 22:08:16.574015 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563424 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 04 22:08:16.574015 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563427 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 04 22:08:16.574015 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.563429 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 04 22:08:16.574515 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.563437 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 04 22:08:16.574515 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.570872 2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 04 22:08:16.574515 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.570888 2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 04 22:08:16.574515 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.571821 2574 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 04 22:08:16.574515 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.571832 2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 04 22:08:16.574515 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.571887 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 04 22:08:16.574515 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.571893 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 04 22:08:16.574515 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.571896 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 04 22:08:16.574515 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.571899 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 04 22:08:16.574515 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.571902 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 04 22:08:16.574515 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.571905 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 04 22:08:16.574515 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.571908 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 04 22:08:16.574515 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.571911 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 04 22:08:16.574515 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.571914 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 04 22:08:16.574892 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.571917 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 04 22:08:16.574892 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.571920 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 04 22:08:16.574892 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.571923 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 04 22:08:16.574892 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.571926 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 04 22:08:16.574892 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.571928 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 04 22:08:16.574892 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.571931 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 04 22:08:16.574892 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.571934 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 04 22:08:16.574892 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.571937 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 04 22:08:16.574892 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.571940 2574 feature_gate.go:328] unrecognized feature gate: Example2 Sep 04 22:08:16.574892 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.571943 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 04 22:08:16.574892 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.571945 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 04 22:08:16.574892 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.571948 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 04 22:08:16.574892 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.571951 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 04 22:08:16.574892 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.571953 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 04 22:08:16.574892 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.571956 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 04 22:08:16.574892 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.571959 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 04 22:08:16.574892 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.571961 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 04 22:08:16.574892 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.571964 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 04 22:08:16.574892 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.571967 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 04 22:08:16.574892 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.571970 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 04 22:08:16.575385 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.571974 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 04 22:08:16.575385 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.571977 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 04 22:08:16.575385 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.571980 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 04 22:08:16.575385 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.571982 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 04 22:08:16.575385 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.571985 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 04 22:08:16.575385 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.571988 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 04 22:08:16.575385 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.571991 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 04 22:08:16.575385 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.571994 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 04 22:08:16.575385 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.571996 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 04 22:08:16.575385 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.571999 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 04 22:08:16.575385 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572002 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 04 22:08:16.575385 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572004 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 04 22:08:16.575385 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572007 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 04 22:08:16.575385 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572009 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 04 22:08:16.575385 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572012 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 04 22:08:16.575385 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572014 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 04 22:08:16.575385 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572017 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 04 22:08:16.575385 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572020 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 04 22:08:16.575385 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572024 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 04 22:08:16.575848 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572028 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 04 22:08:16.575848 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572031 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 04 22:08:16.575848 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572034 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 04 22:08:16.575848 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572037 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 04 22:08:16.575848 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572040 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 04 22:08:16.575848 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572042 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 04 22:08:16.575848 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572045 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 04 22:08:16.575848 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572047 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 04 22:08:16.575848 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572050 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 04 22:08:16.575848 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572053 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 04 22:08:16.575848 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572055 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 04 22:08:16.575848 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572058 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 04 22:08:16.575848 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572060 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 04 22:08:16.575848 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572064 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 04 22:08:16.575848 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572067 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 04 22:08:16.575848 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572070 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 04 22:08:16.575848 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572073 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 04 22:08:16.575848 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572075 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 04 22:08:16.575848 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572078 2574 feature_gate.go:328] unrecognized feature gate: Example Sep 04 22:08:16.575848 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572081 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 04 22:08:16.576330 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572083 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 04 22:08:16.576330 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572086 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 04 22:08:16.576330 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572088 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 04 22:08:16.576330 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572091 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 04 22:08:16.576330 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572093 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 04 22:08:16.576330 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572096 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 04 22:08:16.576330 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572098 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 04 22:08:16.576330 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572101 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 04 22:08:16.576330 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572105 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 04 22:08:16.576330 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572108 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 04 22:08:16.576330 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572111 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 04 22:08:16.576330 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572114 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 04 22:08:16.576330 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572116 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 04 22:08:16.576330 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572119 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 04 22:08:16.576330 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572122 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 04 22:08:16.576330 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572125 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 04 22:08:16.576330 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572128 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 04 22:08:16.576330 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572130 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 04 22:08:16.576330 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572133 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 04 22:08:16.576882 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572136 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 04 22:08:16.576882 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572138 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 04 22:08:16.576882 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572141 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 04 22:08:16.576882 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572143 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 04 22:08:16.576882 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572146 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 04 22:08:16.576882 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572149 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 04 22:08:16.576882 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572152 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 04 22:08:16.576882 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572156 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 04 22:08:16.576882 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572161 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 04 22:08:16.576882 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572164 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 04 22:08:16.576882 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572166 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 04 22:08:16.576882 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572169 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 04 22:08:16.576882 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.572174 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 04 22:08:16.576882 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572285 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 04 22:08:16.577238 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572289 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 04 22:08:16.577238 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572292 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 04 22:08:16.577238 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572295 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 04 22:08:16.577238 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572297 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 04 22:08:16.577238 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572300 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 04 22:08:16.577238 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572303 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 04 22:08:16.577238 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572305 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 04 22:08:16.577238 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572308 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 04 22:08:16.577238 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572311 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 04 22:08:16.577238 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572313 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 04 22:08:16.577238 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572316 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 04 22:08:16.577238 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572318 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 04 22:08:16.577238 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572321 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 04 22:08:16.577238 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572323 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 04 22:08:16.577238 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572325 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 04 22:08:16.577238 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572328 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 04 22:08:16.577238 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572330 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 04 22:08:16.577238 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572333 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 04 22:08:16.577238 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572335 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 04 22:08:16.577238 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572338 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 04 22:08:16.577740 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572341 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 04 22:08:16.577740 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572343 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 04 22:08:16.577740 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572346 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 04 22:08:16.577740 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572349 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 04 22:08:16.577740 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572351 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 04 22:08:16.577740 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572354 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 04 22:08:16.577740 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572357 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 04 22:08:16.577740 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572360 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 04 22:08:16.577740 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572362 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 04 22:08:16.577740 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572365 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 04 22:08:16.577740 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572368 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 04 22:08:16.577740 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572370 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 04 22:08:16.577740 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572374 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 04 22:08:16.577740 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572378 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 04 22:08:16.577740 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572381 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 04 22:08:16.577740 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572384 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 04 22:08:16.577740 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572386 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 04 22:08:16.577740 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572389 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 04 22:08:16.577740 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572392 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 04 22:08:16.577740 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572394 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 04 22:08:16.578217 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572397 2574 feature_gate.go:328] unrecognized feature gate: Example Sep 04 22:08:16.578217 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572400 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 04 22:08:16.578217 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572403 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 04 22:08:16.578217 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572406 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 04 22:08:16.578217 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572408 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 04 22:08:16.578217 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572412 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 04 22:08:16.578217 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572415 2574 feature_gate.go:328] unrecognized feature gate: Example2 Sep 04 22:08:16.578217 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572418 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 04 22:08:16.578217 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572421 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 04 22:08:16.578217 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572424 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 04 22:08:16.578217 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572426 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 04 22:08:16.578217 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572429 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 04 22:08:16.578217 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572432 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 04 22:08:16.578217 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572434 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 04 22:08:16.578217 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572437 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 04 22:08:16.578217 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572440 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 04 22:08:16.578217 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572443 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 04 22:08:16.578217 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572446 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 04 22:08:16.578217 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572448 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 04 22:08:16.578689 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572451 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 04 22:08:16.578689 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572454 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 04 22:08:16.578689 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572457 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 04 22:08:16.578689 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572460 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 04 22:08:16.578689 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572462 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 04 22:08:16.578689 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572465 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 04 22:08:16.578689 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572467 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 04 22:08:16.578689 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572470 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 04 22:08:16.578689 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572472 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 04 22:08:16.578689 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572475 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 04 22:08:16.578689 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572477 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 04 22:08:16.578689 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572480 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 04 22:08:16.578689 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572482 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 04 22:08:16.578689 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572485 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 04 22:08:16.578689 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572487 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 04 22:08:16.578689 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572489 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 04 22:08:16.578689 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572492 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 04 22:08:16.578689 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572495 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 04 22:08:16.578689 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572497 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 04 22:08:16.578689 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572499 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 04 22:08:16.579183 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572502 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 04 22:08:16.579183 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572505 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 04 22:08:16.579183 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572507 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 04 22:08:16.579183 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572510 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 04 22:08:16.579183 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572512 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 04 22:08:16.579183 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572515 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 04 22:08:16.579183 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572518 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 04 22:08:16.579183 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572522 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 04 22:08:16.579183 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572525 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 04 22:08:16.579183 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572527 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 04 22:08:16.579183 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572530 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 04 22:08:16.579183 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572533 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 04 22:08:16.579183 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572535 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 04 22:08:16.579183 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572538 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 04 22:08:16.579183 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572560 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 04 22:08:16.579183 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572564 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 04 22:08:16.579183 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572567 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 04 22:08:16.579183 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572570 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 04 22:08:16.579183 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:16.572572 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 04 22:08:16.579654 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.572577 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 04 22:08:16.579654 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.572677 2574 server.go:962] "Client rotation is on, will bootstrap in background" Sep 04 22:08:16.579654 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.575111 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 04 22:08:16.579654 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.576149 2574 server.go:1019] "Starting client certificate rotation" Sep 04 22:08:16.579654 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.576244 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 04 22:08:16.579654 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.576287 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 04 22:08:16.579654 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.578982 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 04 22:08:16.581506 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.581488 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 04 22:08:16.595425 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.595405 2574 log.go:25] "Validated CRI v1 runtime API" Sep 04 22:08:16.599463 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.599449 2574 log.go:25] "Validated CRI v1 image API" Sep 04 22:08:16.600998 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.600977 2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 04 22:08:16.604346 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.604325 2574 fs.go:135] Filesystem UUIDs: map[5d96b9a2-8160-4f3f-927d-8cbfd991a820:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b4e1722f-98bd-441c-a636-d7894c32b9fb:/dev/nvme0n1p4] Sep 04 22:08:16.604412 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.604344 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 04 22:08:16.608619 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.608495 2574 manager.go:216] Machine: {Timestamp:2026-09-04 22:08:16.608016348 +0000 UTC m=+0.372491972 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3106230 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f4ee897e1182d1679e08145a4fa1c SystemUUID:ec2f4ee8-97e1-182d-1679-e08145a4fa1c BootID:93045160-0930-4cb4-aebe-d2969cc9d11c Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:92:f1:ea:46:f3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:92:f1:ea:46:f3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ee:64:dd:06:be:30 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 04 22:08:16.608619 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.608613 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 04 22:08:16.608761 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.608688 2574 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 04 22:08:16.610448 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.610420 2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 04 22:08:16.610614 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.610450 2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-219.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 04 22:08:16.611388 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.611376 2574 topology_manager.go:138] "Creating topology manager with none policy" Sep 04 22:08:16.611423 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.611391 2574 container_manager_linux.go:309] "Creating device plugin manager" Sep 04 22:08:16.611423 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.611399 2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 04 22:08:16.612415 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.612404 2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 04 22:08:16.614652 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.614642 2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 04 22:08:16.619568 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.619543 2574 state_mem.go:36] "Initialized new in-memory state store" Sep 04 22:08:16.619672 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.619663 2574 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 04 22:08:16.619713 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.619695 2574 kubelet.go:486] "Attempting to sync node with API server" Sep 04 22:08:16.619713 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.619705 2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 04 22:08:16.619780 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.619715 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 04 22:08:16.619780 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.619724 2574 kubelet.go:398] "Adding apiserver pod source" Sep 04 22:08:16.619780 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.619733 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 04 22:08:16.620085 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.620074 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 04 22:08:16.620130 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.620092 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 04 22:08:16.622490 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.622466 2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 04 22:08:16.625394 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.625378 2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 04 22:08:16.625456 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.625408 2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 04 22:08:16.625576 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.625566 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 04 22:08:16.625620 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.625581 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 04 22:08:16.625620 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.625587 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 04 22:08:16.625620 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.625592 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 04 22:08:16.625620 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.625600 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 04 22:08:16.625620 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.625606 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 04 22:08:16.626316 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.626307 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 04 22:08:16.626345 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.626317 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 04 22:08:16.626345 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.626324 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 04 22:08:16.626345 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.626330 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 04 22:08:16.626345 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.626338 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 04 22:08:16.626345 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.626346 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 04 22:08:16.627034 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.627025 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 04 22:08:16.627034 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.627034 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 04 22:08:16.631042 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.630915 2574 server.go:1268] "Started kubelet" Sep 04 22:08:16.631125 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.631002 2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 04 22:08:16.631125 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.630995 2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 04 22:08:16.631125 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.631085 2574 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 04 22:08:16.631732 ip-10-0-137-219 systemd[1]: Started Kubernetes Kubelet. Sep 04 22:08:16.635606 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.635344 2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 04 22:08:16.637171 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.636853 2574 server.go:310] "Adding debug handlers to kubelet server" Sep 04 22:08:16.637171 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:16.637088 2574 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 04 22:08:16.639306 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.639288 2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-219.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 04 22:08:16.639428 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:16.639358 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 04 22:08:16.640289 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.640273 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 04 22:08:16.642279 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.642261 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 04 22:08:16.643027 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.643011 2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 04 22:08:16.643123 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:16.643098 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-219.ec2.internal\" not found" Sep 04 22:08:16.643770 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:16.643748 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-219.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 04 22:08:16.643848 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.643794 2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 04 22:08:16.643848 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.643796 2574 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 04 22:08:16.643848 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.643821 2574 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 04 22:08:16.643985 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.643807 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 04 22:08:16.643985 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.643869 2574 factory.go:55] Registering systemd factory Sep 04 22:08:16.643985 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.643880 2574 factory.go:223] Registration of the systemd container factory successfully Sep 04 22:08:16.643985 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.643906 2574 reconstruct.go:97] "Volume reconstruction finished" Sep 04 22:08:16.643985 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.643915 2574 reconciler.go:29] "Reconciler: start to sync state" Sep 04 22:08:16.644211 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.644173 2574 factory.go:153] Registering CRI-O factory Sep 04 22:08:16.644211 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.644188 2574 factory.go:223] Registration of the crio container factory successfully Sep 04 22:08:16.644211 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.644209 2574 factory.go:103] Registering Raw factory Sep 04 22:08:16.644344 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.644224 2574 manager.go:1196] Started watching for new ooms in manager Sep 04 22:08:16.644954 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:16.643807 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-219.ec2.internal.18d23d916143b531 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-219.ec2.internal,UID:ip-10-0-137-219.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-219.ec2.internal,},FirstTimestamp:2026-09-04 22:08:16.630887729 +0000 UTC m=+0.395363333,LastTimestamp:2026-09-04 22:08:16.630887729 +0000 UTC m=+0.395363333,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-219.ec2.internal,}" Sep 04 22:08:16.645114 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.645101 2574 manager.go:349] Starting recovery of all containers Sep 04 22:08:16.652997 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:16.652974 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 04 22:08:16.653106 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:16.653081 2574 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-219.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 04 22:08:16.655864 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.655849 2574 manager.go:354] Recovery completed Sep 04 22:08:16.660076 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.660063 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 04 22:08:16.661699 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.661671 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-219.ec2.internal" event="NodeHasSufficientMemory" Sep 04 22:08:16.661758 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.661720 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-219.ec2.internal" event="NodeHasNoDiskPressure" Sep 04 22:08:16.661758 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.661736 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-219.ec2.internal" event="NodeHasSufficientPID" Sep 04 22:08:16.662337 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.662321 2574 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 04 22:08:16.662337 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.662336 2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 04 22:08:16.662437 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.662357 2574 state_mem.go:36] "Initialized new in-memory state store" Sep 04 22:08:16.662437 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.662323 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 04 22:08:16.663078 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.663064 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-219.ec2.internal" event="NodeHasSufficientMemory" Sep 04 22:08:16.663160 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.663088 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-219.ec2.internal" event="NodeHasNoDiskPressure" Sep 04 22:08:16.663160 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.663098 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-219.ec2.internal" event="NodeHasSufficientPID" Sep 04 22:08:16.664595 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:16.664519 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-219.ec2.internal.18d23d916319dae5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-219.ec2.internal,UID:ip-10-0-137-219.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-219.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-219.ec2.internal,},FirstTimestamp:2026-09-04 22:08:16.661699301 +0000 UTC m=+0.426174909,LastTimestamp:2026-09-04 22:08:16.661699301 +0000 UTC m=+0.426174909,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-219.ec2.internal,}" Sep 04 22:08:16.664676 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.664614 2574 policy_none.go:49] "None policy: Start" Sep 04 22:08:16.664676 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.664629 2574 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 04 22:08:16.664676 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.664644 2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 04 22:08:16.666647 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.666635 2574 policy_none.go:47] "Start" Sep 04 22:08:16.673512 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:16.673433 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-219.ec2.internal.18d23d91631a490c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-219.ec2.internal,UID:ip-10-0-137-219.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-219.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-219.ec2.internal,},FirstTimestamp:2026-09-04 22:08:16.6617275 +0000 UTC m=+0.426203105,LastTimestamp:2026-09-04 22:08:16.6617275 +0000 UTC m=+0.426203105,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-219.ec2.internal,}" Sep 04 22:08:16.683612 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:16.683510 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-219.ec2.internal.18d23d91631a86b0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-219.ec2.internal,UID:ip-10-0-137-219.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-137-219.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-137-219.ec2.internal,},FirstTimestamp:2026-09-04 22:08:16.66174328 +0000 UTC m=+0.426218885,LastTimestamp:2026-09-04 22:08:16.66174328 +0000 UTC m=+0.426218885,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-219.ec2.internal,}" Sep 04 22:08:16.693581 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.693560 2574 manager.go:337] "Starting Device Plugin manager" Sep 04 22:08:16.720975 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:16.693606 2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 04 22:08:16.720975 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.693620 2574 server.go:87] "Starting device plugin registration server" Sep 04 22:08:16.720975 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.693953 2574 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 04 22:08:16.720975 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.693967 2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 04 22:08:16.720975 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.694047 2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 04 22:08:16.720975 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.694133 2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 04 22:08:16.720975 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.694142 2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 04 22:08:16.720975 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:16.694689 2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 04 22:08:16.720975 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:16.694732 2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-219.ec2.internal\" not found" Sep 04 22:08:16.720975 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:16.695275 2574 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-219.ec2.internal.18d23d916319dae5\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-219.ec2.internal.18d23d916319dae5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-219.ec2.internal,UID:ip-10-0-137-219.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-219.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-219.ec2.internal,},FirstTimestamp:2026-09-04 22:08:16.661699301 +0000 UTC m=+0.426174909,LastTimestamp:2026-09-04 22:08:16.663076826 +0000 UTC m=+0.427552428,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-219.ec2.internal,}" Sep 04 22:08:16.720975 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:16.705796 2574 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-219.ec2.internal.18d23d91631a490c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-219.ec2.internal.18d23d91631a490c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-219.ec2.internal,UID:ip-10-0-137-219.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-219.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-219.ec2.internal,},FirstTimestamp:2026-09-04 22:08:16.6617275 +0000 UTC m=+0.426203105,LastTimestamp:2026-09-04 22:08:16.663092995 +0000 UTC m=+0.427568598,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-219.ec2.internal,}" Sep 04 22:08:16.720975 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.707344 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 04 22:08:16.720975 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.708538 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 04 22:08:16.720975 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.708594 2574 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 04 22:08:16.720975 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.708613 2574 kubelet.go:2448] "Starting kubelet main sync loop" Sep 04 22:08:16.720975 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:16.708652 2574 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 04 22:08:16.720975 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:16.716748 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" Sep 04 22:08:16.720975 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:16.717408 2574 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-219.ec2.internal.18d23d91631a86b0\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-219.ec2.internal.18d23d91631a86b0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-219.ec2.internal,UID:ip-10-0-137-219.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-137-219.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-137-219.ec2.internal,},FirstTimestamp:2026-09-04 22:08:16.66174328 +0000 UTC m=+0.426218885,LastTimestamp:2026-09-04 22:08:16.663102435 +0000 UTC m=+0.427578038,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-219.ec2.internal,}" Sep 04 22:08:16.721783 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:16.721718 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-219.ec2.internal.18d23d9165229998 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-219.ec2.internal,UID:ip-10-0-137-219.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-137-219.ec2.internal,},FirstTimestamp:2026-09-04 22:08:16.69582684 +0000 UTC m=+0.460302435,LastTimestamp:2026-09-04 22:08:16.69582684 +0000 UTC m=+0.460302435,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-219.ec2.internal,}" Sep 04 22:08:16.794855 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.794826 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 04 22:08:16.795769 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.795747 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-219.ec2.internal" event="NodeHasSufficientMemory" Sep 04 22:08:16.795869 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.795777 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-219.ec2.internal" event="NodeHasNoDiskPressure" Sep 04 22:08:16.795869 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.795789 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-219.ec2.internal" event="NodeHasSufficientPID" Sep 04 22:08:16.795869 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.795814 2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-219.ec2.internal" Sep 04 22:08:16.800232 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.800215 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7qjps" Sep 04 22:08:16.800293 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:16.800155 2574 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-219.ec2.internal.18d23d916319dae5\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-219.ec2.internal.18d23d916319dae5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-219.ec2.internal,UID:ip-10-0-137-219.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-219.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-219.ec2.internal,},FirstTimestamp:2026-09-04 22:08:16.661699301 +0000 UTC m=+0.426174909,LastTimestamp:2026-09-04 22:08:16.795763177 +0000 UTC m=+0.560238781,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-219.ec2.internal,}" Sep 04 22:08:16.808994 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.808922 2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-219.ec2.internal"] Sep 04 22:08:16.808994 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.808990 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 04 22:08:16.810614 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.810599 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-219.ec2.internal" event="NodeHasSufficientMemory" Sep 04 22:08:16.810681 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.810626 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-219.ec2.internal" event="NodeHasNoDiskPressure" Sep 04 22:08:16.810681 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.810637 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-219.ec2.internal" event="NodeHasSufficientPID" Sep 04 22:08:16.812775 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:16.812746 2574 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-137-219.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-137-219.ec2.internal" Sep 04 22:08:16.812840 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:16.812779 2574 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-137-219.ec2.internal.18d23d91631a490c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-219.ec2.internal.18d23d91631a490c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-219.ec2.internal,UID:ip-10-0-137-219.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-219.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-219.ec2.internal,},FirstTimestamp:2026-09-04 22:08:16.6617275 +0000 UTC m=+0.426203105,LastTimestamp:2026-09-04 22:08:16.795781797 +0000 UTC m=+0.560257400,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-219.ec2.internal,}" Sep 04 22:08:16.813390 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.813375 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7qjps" Sep 04 22:08:16.814324 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.814310 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 04 22:08:16.814443 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.814432 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal" Sep 04 22:08:16.814480 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.814459 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 04 22:08:16.815151 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.815138 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-219.ec2.internal" event="NodeHasSufficientMemory" Sep 04 22:08:16.815214 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.815160 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-219.ec2.internal" event="NodeHasSufficientMemory" Sep 04 22:08:16.815214 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.815184 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-219.ec2.internal" event="NodeHasNoDiskPressure" Sep 04 22:08:16.815214 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.815193 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-219.ec2.internal" event="NodeHasSufficientPID" Sep 04 22:08:16.815307 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.815163 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-219.ec2.internal" event="NodeHasNoDiskPressure" Sep 04 22:08:16.815307 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.815247 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-219.ec2.internal" event="NodeHasSufficientPID" Sep 04 22:08:16.817339 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.817325 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-219.ec2.internal" Sep 04 22:08:16.817405 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.817347 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 04 22:08:16.818026 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.818014 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-219.ec2.internal" event="NodeHasSufficientMemory" Sep 04 22:08:16.818073 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.818040 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-219.ec2.internal" event="NodeHasNoDiskPressure" Sep 04 22:08:16.818073 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.818049 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-219.ec2.internal" event="NodeHasSufficientPID" Sep 04 22:08:16.844771 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.844751 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1c571934cf12fb6136a9351d29e54859-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal\" (UID: \"1c571934cf12fb6136a9351d29e54859\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal" Sep 04 22:08:16.844871 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.844774 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c571934cf12fb6136a9351d29e54859-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal\" (UID: \"1c571934cf12fb6136a9351d29e54859\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal" Sep 04 22:08:16.844871 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.844791 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5c4c794f844e4275466dd646fd4a6840-config\") pod \"kube-apiserver-proxy-ip-10-0-137-219.ec2.internal\" (UID: \"5c4c794f844e4275466dd646fd4a6840\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-219.ec2.internal" Sep 04 22:08:16.849164 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:16.849141 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-219.ec2.internal\" not found" node="ip-10-0-137-219.ec2.internal" Sep 04 22:08:16.853571 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:16.853540 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-219.ec2.internal\" not found" node="ip-10-0-137-219.ec2.internal" Sep 04 22:08:16.871466 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:16.871446 2574 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-137-219.ec2.internal\" not found" node="ip-10-0-137-219.ec2.internal" Sep 04 22:08:16.944898 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.944859 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1c571934cf12fb6136a9351d29e54859-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal\" (UID: \"1c571934cf12fb6136a9351d29e54859\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal" Sep 04 22:08:16.945067 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.944906 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c571934cf12fb6136a9351d29e54859-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal\" (UID: \"1c571934cf12fb6136a9351d29e54859\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal" Sep 04 22:08:16.945067 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.944921 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1c571934cf12fb6136a9351d29e54859-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal\" (UID: \"1c571934cf12fb6136a9351d29e54859\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal" Sep 04 22:08:16.945067 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.944934 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5c4c794f844e4275466dd646fd4a6840-config\") pod \"kube-apiserver-proxy-ip-10-0-137-219.ec2.internal\" (UID: \"5c4c794f844e4275466dd646fd4a6840\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-219.ec2.internal" Sep 04 22:08:16.945067 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.944965 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c571934cf12fb6136a9351d29e54859-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal\" (UID: \"1c571934cf12fb6136a9351d29e54859\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal" Sep 04 22:08:16.945067 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:16.944984 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5c4c794f844e4275466dd646fd4a6840-config\") pod \"kube-apiserver-proxy-ip-10-0-137-219.ec2.internal\" (UID: \"5c4c794f844e4275466dd646fd4a6840\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-219.ec2.internal" Sep 04 22:08:17.013179 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:17.013156 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 04 22:08:17.014414 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:17.014395 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-219.ec2.internal" event="NodeHasSufficientMemory" Sep 04 22:08:17.014503 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:17.014425 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-219.ec2.internal" event="NodeHasNoDiskPressure" Sep 04 22:08:17.014503 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:17.014448 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-219.ec2.internal" event="NodeHasSufficientPID" Sep 04 22:08:17.014503 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:17.014477 2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-219.ec2.internal" Sep 04 22:08:17.030860 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:17.030847 2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-219.ec2.internal" Sep 04 22:08:17.030927 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:17.030866 2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-219.ec2.internal\": node \"ip-10-0-137-219.ec2.internal\" not found" Sep 04 22:08:17.080513 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:17.080454 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-219.ec2.internal\" not found" Sep 04 22:08:17.152976 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:17.152957 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal" Sep 04 22:08:17.158394 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:17.158177 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-219.ec2.internal" Sep 04 22:08:17.181300 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:17.181277 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-219.ec2.internal\" not found" Sep 04 22:08:17.281826 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:17.281797 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-219.ec2.internal\" not found" Sep 04 22:08:17.382313 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:17.382249 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-219.ec2.internal\" not found" Sep 04 22:08:17.482926 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:17.482889 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-219.ec2.internal\" not found" Sep 04 22:08:17.577482 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:17.577451 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 04 22:08:17.583112 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:17.583090 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-219.ec2.internal\" not found" Sep 04 22:08:17.670794 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:17.670711 2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 04 22:08:17.743571 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:17.743530 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal" Sep 04 22:08:17.761443 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:17.761417 2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 04 22:08:17.790824 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:17.790806 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 04 22:08:17.792848 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:17.792832 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-219.ec2.internal" Sep 04 22:08:17.808875 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:17.808853 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 04 22:08:17.816440 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:17.816411 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-03 22:03:17 +0000 UTC" deadline="2028-06-12 03:23:12.605195572 +0000 UTC" Sep 04 22:08:17.816440 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:17.816437 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15509h14m54.788762025s" Sep 04 22:08:17.943140 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:17.942973 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c571934cf12fb6136a9351d29e54859.slice/crio-ddfa3ed9327c67bb20035874932af9a54fd8a0e8a7e12878d4e233a1df417a75 WatchSource:0}: Error finding container ddfa3ed9327c67bb20035874932af9a54fd8a0e8a7e12878d4e233a1df417a75: Status 404 returned error can't find the container with id ddfa3ed9327c67bb20035874932af9a54fd8a0e8a7e12878d4e233a1df417a75 Sep 04 22:08:17.945131 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:17.945107 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c4c794f844e4275466dd646fd4a6840.slice/crio-5b4e0cd467c42d677e7416aabdea8e3a75420867c8dbc868465cedd4e6c74e74 WatchSource:0}: Error finding container 5b4e0cd467c42d677e7416aabdea8e3a75420867c8dbc868465cedd4e6c74e74: Status 404 returned error can't find the container with id 5b4e0cd467c42d677e7416aabdea8e3a75420867c8dbc868465cedd4e6c74e74 Sep 04 22:08:17.952601 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:17.952575 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 04 22:08:18.025130 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.025107 2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 04 22:08:18.266106 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.266014 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 04 22:08:18.621483 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.621253 2574 apiserver.go:52] "Watching apiserver" Sep 04 22:08:18.631264 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.631235 2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 04 22:08:18.631851 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.631824 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-v7kt4","kube-system/konnectivity-agent-9gcxl","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xks49","openshift-cluster-node-tuning-operator/tuned-jdkml","openshift-dns/node-resolver-p9mdt","openshift-multus/network-metrics-daemon-qvwc7","openshift-ovn-kubernetes/ovnkube-node-gv845","kube-system/global-pull-secret-syncer-ndzbl","kube-system/kube-apiserver-proxy-ip-10-0-137-219.ec2.internal","openshift-image-registry/node-ca-d5j8h","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal","openshift-multus/multus-additional-cni-plugins-b49j5","openshift-multus/multus-sbhpf","openshift-network-diagnostics/network-check-target-ggpqf"] Sep 04 22:08:18.635053 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.635031 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b49j5" Sep 04 22:08:18.637309 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.637288 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9gcxl" Sep 04 22:08:18.639702 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.639414 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggpqf" Sep 04 22:08:18.639702 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:18.639490 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggpqf" podUID="6200aa24-5731-4d1a-865b-6c7c1cd218d1" Sep 04 22:08:18.642218 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.641780 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.642471 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.642452 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 04 22:08:18.642715 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.642695 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 04 22:08:18.643059 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.643017 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 04 22:08:18.643144 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.643103 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 04 22:08:18.643307 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.643289 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 04 22:08:18.643451 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.643428 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 04 22:08:18.643551 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.643515 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 04 22:08:18.643795 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.643777 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-k8d8k\"" Sep 04 22:08:18.645048 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.644803 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-p9mdt" Sep 04 22:08:18.645048 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.644884 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-n8zhp\"" Sep 04 22:08:18.645048 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.644929 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 04 22:08:18.645691 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.645637 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-scc45\"" Sep 04 22:08:18.646332 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.646312 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 04 22:08:18.648358 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.648328 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvwc7" Sep 04 22:08:18.648454 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:18.648394 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvwc7" podUID="c0972118-6d5c-4d3a-a979-497ab05ca339" Sep 04 22:08:18.649486 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.649448 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 04 22:08:18.650554 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.650532 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v7kt4" Sep 04 22:08:18.655208 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.652498 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 04 22:08:18.655208 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.652947 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lnwtj\"" Sep 04 22:08:18.655208 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.653619 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/666e8bf3-3b4b-4dc6-8b12-5546378326ea-host-run-netns\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.655208 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.653691 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krwrw\" (UniqueName: \"kubernetes.io/projected/95ade170-0175-41e0-b78b-ede111ef37af-kube-api-access-krwrw\") pod \"multus-additional-cni-plugins-b49j5\" (UID: \"95ade170-0175-41e0-b78b-ede111ef37af\") " pod="openshift-multus/multus-additional-cni-plugins-b49j5" Sep 04 22:08:18.655208 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.653726 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1373f05f-60fa-4414-a0da-93106632c578-agent-certs\") pod \"konnectivity-agent-9gcxl\" (UID: \"1373f05f-60fa-4414-a0da-93106632c578\") " pod="kube-system/konnectivity-agent-9gcxl" Sep 04 22:08:18.655208 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.653754 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/666e8bf3-3b4b-4dc6-8b12-5546378326ea-cni-binary-copy\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.655208 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.653825 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wczkj\" (UniqueName: \"kubernetes.io/projected/c8319608-4bcc-405e-b607-e3fea2c4e9a8-kube-api-access-wczkj\") pod \"node-resolver-p9mdt\" (UID: \"c8319608-4bcc-405e-b607-e3fea2c4e9a8\") " pod="openshift-dns/node-resolver-p9mdt" Sep 04 22:08:18.655208 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.653883 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8319608-4bcc-405e-b607-e3fea2c4e9a8-tmp-dir\") pod \"node-resolver-p9mdt\" (UID: \"c8319608-4bcc-405e-b607-e3fea2c4e9a8\") " pod="openshift-dns/node-resolver-p9mdt" Sep 04 22:08:18.655208 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.653973 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/95ade170-0175-41e0-b78b-ede111ef37af-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b49j5\" (UID: \"95ade170-0175-41e0-b78b-ede111ef37af\") " pod="openshift-multus/multus-additional-cni-plugins-b49j5" Sep 04 22:08:18.655208 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.654004 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1373f05f-60fa-4414-a0da-93106632c578-konnectivity-ca\") pod \"konnectivity-agent-9gcxl\" (UID: \"1373f05f-60fa-4414-a0da-93106632c578\") " pod="kube-system/konnectivity-agent-9gcxl" Sep 04 22:08:18.655208 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.654036 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/95ade170-0175-41e0-b78b-ede111ef37af-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b49j5\" (UID: \"95ade170-0175-41e0-b78b-ede111ef37af\") " pod="openshift-multus/multus-additional-cni-plugins-b49j5" Sep 04 22:08:18.655208 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.654068 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzgbw\" (UniqueName: \"kubernetes.io/projected/6200aa24-5731-4d1a-865b-6c7c1cd218d1-kube-api-access-mzgbw\") pod \"network-check-target-ggpqf\" (UID: \"6200aa24-5731-4d1a-865b-6c7c1cd218d1\") " pod="openshift-network-diagnostics/network-check-target-ggpqf" Sep 04 22:08:18.655208 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.654098 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/666e8bf3-3b4b-4dc6-8b12-5546378326ea-host-var-lib-cni-multus\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.655208 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.654161 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c8319608-4bcc-405e-b607-e3fea2c4e9a8-hosts-file\") pod \"node-resolver-p9mdt\" (UID: \"c8319608-4bcc-405e-b607-e3fea2c4e9a8\") " pod="openshift-dns/node-resolver-p9mdt" Sep 04 22:08:18.655208 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.654202 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/666e8bf3-3b4b-4dc6-8b12-5546378326ea-host-var-lib-cni-bin\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.655208 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.654237 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/666e8bf3-3b4b-4dc6-8b12-5546378326ea-host-run-multus-certs\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.655208 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.654268 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/95ade170-0175-41e0-b78b-ede111ef37af-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b49j5\" (UID: \"95ade170-0175-41e0-b78b-ede111ef37af\") " pod="openshift-multus/multus-additional-cni-plugins-b49j5" Sep 04 22:08:18.656122 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.654303 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/666e8bf3-3b4b-4dc6-8b12-5546378326ea-multus-cni-dir\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.656122 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.654334 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/666e8bf3-3b4b-4dc6-8b12-5546378326ea-multus-socket-dir-parent\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.656122 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.654368 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/95ade170-0175-41e0-b78b-ede111ef37af-cnibin\") pod \"multus-additional-cni-plugins-b49j5\" (UID: \"95ade170-0175-41e0-b78b-ede111ef37af\") " pod="openshift-multus/multus-additional-cni-plugins-b49j5" Sep 04 22:08:18.656122 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.654398 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/666e8bf3-3b4b-4dc6-8b12-5546378326ea-cnibin\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.656122 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.654437 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/666e8bf3-3b4b-4dc6-8b12-5546378326ea-host-run-k8s-cni-cncf-io\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.656122 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.654472 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/666e8bf3-3b4b-4dc6-8b12-5546378326ea-host-var-lib-kubelet\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.656122 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.654506 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/666e8bf3-3b4b-4dc6-8b12-5546378326ea-hostroot\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.656122 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.654538 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/666e8bf3-3b4b-4dc6-8b12-5546378326ea-multus-daemon-config\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.656122 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.654607 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/666e8bf3-3b4b-4dc6-8b12-5546378326ea-etc-kubernetes\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.656122 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.654642 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/666e8bf3-3b4b-4dc6-8b12-5546378326ea-system-cni-dir\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.656122 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.654677 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95ade170-0175-41e0-b78b-ede111ef37af-system-cni-dir\") pod \"multus-additional-cni-plugins-b49j5\" (UID: \"95ade170-0175-41e0-b78b-ede111ef37af\") " pod="openshift-multus/multus-additional-cni-plugins-b49j5" Sep 04 22:08:18.656122 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.654710 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/95ade170-0175-41e0-b78b-ede111ef37af-cni-binary-copy\") pod \"multus-additional-cni-plugins-b49j5\" (UID: \"95ade170-0175-41e0-b78b-ede111ef37af\") " pod="openshift-multus/multus-additional-cni-plugins-b49j5" Sep 04 22:08:18.656122 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.654742 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/666e8bf3-3b4b-4dc6-8b12-5546378326ea-multus-conf-dir\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.656122 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.654778 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dkzz\" (UniqueName: \"kubernetes.io/projected/666e8bf3-3b4b-4dc6-8b12-5546378326ea-kube-api-access-4dkzz\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.656932 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.656474 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.658534 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.658039 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d5j8h" Sep 04 22:08:18.659028 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.659005 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 04 22:08:18.659116 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.659075 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 04 22:08:18.659232 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.659214 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8gqfs\"" Sep 04 22:08:18.659348 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.659295 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 04 22:08:18.661000 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.660611 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndzbl" Sep 04 22:08:18.661000 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:18.660686 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndzbl" podUID="62c2d50e-f517-47e6-8042-31b2455981fb" Sep 04 22:08:18.661000 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.660852 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 04 22:08:18.662227 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.662207 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 04 22:08:18.663323 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.663301 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-m4z7k\"" Sep 04 22:08:18.663582 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.663564 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 04 22:08:18.663916 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.663841 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 04 22:08:18.664031 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.664006 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 04 22:08:18.664095 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.664058 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8lpx4\"" Sep 04 22:08:18.664291 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.664274 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 04 22:08:18.664470 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.664447 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 04 22:08:18.664531 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.664503 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 04 22:08:18.664688 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.664671 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jdkml" Sep 04 22:08:18.666042 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.666019 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xks49" Sep 04 22:08:18.671801 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.671782 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 04 22:08:18.672237 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.672107 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 04 22:08:18.672237 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.672124 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 04 22:08:18.672237 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.672169 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 04 22:08:18.672237 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.672176 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 04 22:08:18.672519 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.672303 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-klqb5\"" Sep 04 22:08:18.672519 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.672494 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 04 22:08:18.673439 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.673422 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pr64g\"" Sep 04 22:08:18.677821 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.677797 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 04 22:08:18.714155 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.714100 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-219.ec2.internal" event={"ID":"5c4c794f844e4275466dd646fd4a6840","Type":"ContainerStarted","Data":"5b4e0cd467c42d677e7416aabdea8e3a75420867c8dbc868465cedd4e6c74e74"} Sep 04 22:08:18.717350 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.717327 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9gpn5" Sep 04 22:08:18.717803 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.717756 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal" event={"ID":"1c571934cf12fb6136a9351d29e54859","Type":"ContainerStarted","Data":"ddfa3ed9327c67bb20035874932af9a54fd8a0e8a7e12878d4e233a1df417a75"} Sep 04 22:08:18.735317 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.735285 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9gpn5" Sep 04 22:08:18.744915 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.744846 2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 04 22:08:18.755825 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.755709 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-host-run-ovn-kubernetes\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.755825 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.755748 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62c2d50e-f517-47e6-8042-31b2455981fb-original-pull-secret\") pod \"global-pull-secret-syncer-ndzbl\" (UID: \"62c2d50e-f517-47e6-8042-31b2455981fb\") " pod="kube-system/global-pull-secret-syncer-ndzbl" Sep 04 22:08:18.755825 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.755776 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0972118-6d5c-4d3a-a979-497ab05ca339-metrics-certs\") pod \"network-metrics-daemon-qvwc7\" (UID: \"c0972118-6d5c-4d3a-a979-497ab05ca339\") " pod="openshift-multus/network-metrics-daemon-qvwc7" Sep 04 22:08:18.755825 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.755802 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-host-kubelet\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.756087 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.755836 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-systemd-units\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.756087 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.755863 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-host-run-netns\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.756087 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.755885 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e424da81-8acd-4c1a-80c0-d1087d78ae44-etc-tuned\") pod \"tuned-jdkml\" (UID: \"e424da81-8acd-4c1a-80c0-d1087d78ae44\") " pod="openshift-cluster-node-tuning-operator/tuned-jdkml" Sep 04 22:08:18.756087 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.755915 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dkzz\" (UniqueName: \"kubernetes.io/projected/666e8bf3-3b4b-4dc6-8b12-5546378326ea-kube-api-access-4dkzz\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.756087 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.755939 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e424da81-8acd-4c1a-80c0-d1087d78ae44-run\") pod \"tuned-jdkml\" (UID: \"e424da81-8acd-4c1a-80c0-d1087d78ae44\") " pod="openshift-cluster-node-tuning-operator/tuned-jdkml" Sep 04 22:08:18.756087 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.755963 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/666e8bf3-3b4b-4dc6-8b12-5546378326ea-host-run-netns\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.756087 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.755988 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krwrw\" (UniqueName: \"kubernetes.io/projected/95ade170-0175-41e0-b78b-ede111ef37af-kube-api-access-krwrw\") pod \"multus-additional-cni-plugins-b49j5\" (UID: \"95ade170-0175-41e0-b78b-ede111ef37af\") " pod="openshift-multus/multus-additional-cni-plugins-b49j5" Sep 04 22:08:18.756087 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.756026 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1373f05f-60fa-4414-a0da-93106632c578-agent-certs\") pod \"konnectivity-agent-9gcxl\" (UID: \"1373f05f-60fa-4414-a0da-93106632c578\") " pod="kube-system/konnectivity-agent-9gcxl" Sep 04 22:08:18.756087 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.756049 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/666e8bf3-3b4b-4dc6-8b12-5546378326ea-cni-binary-copy\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.756087 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.756077 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wczkj\" (UniqueName: \"kubernetes.io/projected/c8319608-4bcc-405e-b607-e3fea2c4e9a8-kube-api-access-wczkj\") pod \"node-resolver-p9mdt\" (UID: \"c8319608-4bcc-405e-b607-e3fea2c4e9a8\") " pod="openshift-dns/node-resolver-p9mdt" Sep 04 22:08:18.756672 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.756102 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-etc-openvswitch\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.756672 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.756131 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e424da81-8acd-4c1a-80c0-d1087d78ae44-etc-sysconfig\") pod \"tuned-jdkml\" (UID: \"e424da81-8acd-4c1a-80c0-d1087d78ae44\") " pod="openshift-cluster-node-tuning-operator/tuned-jdkml" Sep 04 22:08:18.756672 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.756154 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e424da81-8acd-4c1a-80c0-d1087d78ae44-etc-systemd\") pod \"tuned-jdkml\" (UID: \"e424da81-8acd-4c1a-80c0-d1087d78ae44\") " pod="openshift-cluster-node-tuning-operator/tuned-jdkml" Sep 04 22:08:18.756672 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.756188 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8319608-4bcc-405e-b607-e3fea2c4e9a8-tmp-dir\") pod \"node-resolver-p9mdt\" (UID: \"c8319608-4bcc-405e-b607-e3fea2c4e9a8\") " pod="openshift-dns/node-resolver-p9mdt" Sep 04 22:08:18.756672 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.756213 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/76bd7339-f02c-4c10-af0a-af3010cfbc30-iptables-alerter-script\") pod \"iptables-alerter-v7kt4\" (UID: \"76bd7339-f02c-4c10-af0a-af3010cfbc30\") " pod="openshift-network-operator/iptables-alerter-v7kt4" Sep 04 22:08:18.756672 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.756278 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e424da81-8acd-4c1a-80c0-d1087d78ae44-etc-kubernetes\") pod \"tuned-jdkml\" (UID: \"e424da81-8acd-4c1a-80c0-d1087d78ae44\") " pod="openshift-cluster-node-tuning-operator/tuned-jdkml" Sep 04 22:08:18.756672 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.756303 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e424da81-8acd-4c1a-80c0-d1087d78ae44-var-lib-kubelet\") pod \"tuned-jdkml\" (UID: \"e424da81-8acd-4c1a-80c0-d1087d78ae44\") " pod="openshift-cluster-node-tuning-operator/tuned-jdkml" Sep 04 22:08:18.756672 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.756332 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xks49\" (UID: \"6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xks49" Sep 04 22:08:18.756672 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.756355 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef-etc-selinux\") pod \"aws-ebs-csi-driver-node-xks49\" (UID: \"6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xks49" Sep 04 22:08:18.756672 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.756377 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-node-log\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.756672 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.756400 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e424da81-8acd-4c1a-80c0-d1087d78ae44-lib-modules\") pod \"tuned-jdkml\" (UID: \"e424da81-8acd-4c1a-80c0-d1087d78ae44\") " pod="openshift-cluster-node-tuning-operator/tuned-jdkml" Sep 04 22:08:18.756672 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.756423 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef-sys-fs\") pod \"aws-ebs-csi-driver-node-xks49\" (UID: \"6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xks49" Sep 04 22:08:18.756672 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.756451 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/95ade170-0175-41e0-b78b-ede111ef37af-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b49j5\" (UID: \"95ade170-0175-41e0-b78b-ede111ef37af\") " pod="openshift-multus/multus-additional-cni-plugins-b49j5" Sep 04 22:08:18.756672 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.756476 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c8319608-4bcc-405e-b607-e3fea2c4e9a8-hosts-file\") pod \"node-resolver-p9mdt\" (UID: \"c8319608-4bcc-405e-b607-e3fea2c4e9a8\") " pod="openshift-dns/node-resolver-p9mdt" Sep 04 22:08:18.756672 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.756504 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-run-openvswitch\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.756672 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.756523 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/666e8bf3-3b4b-4dc6-8b12-5546378326ea-host-run-netns\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.756672 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.756528 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e424da81-8acd-4c1a-80c0-d1087d78ae44-etc-modprobe-d\") pod \"tuned-jdkml\" (UID: \"e424da81-8acd-4c1a-80c0-d1087d78ae44\") " pod="openshift-cluster-node-tuning-operator/tuned-jdkml" Sep 04 22:08:18.757714 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.756557 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e424da81-8acd-4c1a-80c0-d1087d78ae44-tmp\") pod \"tuned-jdkml\" (UID: \"e424da81-8acd-4c1a-80c0-d1087d78ae44\") " pod="openshift-cluster-node-tuning-operator/tuned-jdkml" Sep 04 22:08:18.757714 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.756595 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/666e8bf3-3b4b-4dc6-8b12-5546378326ea-host-run-multus-certs\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.757714 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.756699 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/95ade170-0175-41e0-b78b-ede111ef37af-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b49j5\" (UID: \"95ade170-0175-41e0-b78b-ede111ef37af\") " pod="openshift-multus/multus-additional-cni-plugins-b49j5" Sep 04 22:08:18.757714 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.756745 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/54f49670-6d36-4464-b9c4-87718e384fb2-serviceca\") pod \"node-ca-d5j8h\" (UID: \"54f49670-6d36-4464-b9c4-87718e384fb2\") " pod="openshift-image-registry/node-ca-d5j8h" Sep 04 22:08:18.757714 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.756773 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c8319608-4bcc-405e-b607-e3fea2c4e9a8-hosts-file\") pod \"node-resolver-p9mdt\" (UID: \"c8319608-4bcc-405e-b607-e3fea2c4e9a8\") " pod="openshift-dns/node-resolver-p9mdt" Sep 04 22:08:18.757714 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.756787 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8zsr\" (UniqueName: \"kubernetes.io/projected/c0972118-6d5c-4d3a-a979-497ab05ca339-kube-api-access-m8zsr\") pod \"network-metrics-daemon-qvwc7\" (UID: \"c0972118-6d5c-4d3a-a979-497ab05ca339\") " pod="openshift-multus/network-metrics-daemon-qvwc7" Sep 04 22:08:18.757714 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.756841 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/666e8bf3-3b4b-4dc6-8b12-5546378326ea-host-run-multus-certs\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.757714 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.756880 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/95ade170-0175-41e0-b78b-ede111ef37af-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b49j5\" (UID: \"95ade170-0175-41e0-b78b-ede111ef37af\") " pod="openshift-multus/multus-additional-cni-plugins-b49j5" Sep 04 22:08:18.757714 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.756894 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/37958fa0-bfd6-402f-9c5c-faa146b7d886-ovnkube-script-lib\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.757714 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.756934 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/95ade170-0175-41e0-b78b-ede111ef37af-cnibin\") pod \"multus-additional-cni-plugins-b49j5\" (UID: \"95ade170-0175-41e0-b78b-ede111ef37af\") " pod="openshift-multus/multus-additional-cni-plugins-b49j5" Sep 04 22:08:18.757714 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.756963 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/666e8bf3-3b4b-4dc6-8b12-5546378326ea-cnibin\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.757714 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.756991 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/666e8bf3-3b4b-4dc6-8b12-5546378326ea-host-run-k8s-cni-cncf-io\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.757714 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.757022 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/666e8bf3-3b4b-4dc6-8b12-5546378326ea-hostroot\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.757714 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.757046 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/666e8bf3-3b4b-4dc6-8b12-5546378326ea-host-run-k8s-cni-cncf-io\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.757714 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.757050 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/666e8bf3-3b4b-4dc6-8b12-5546378326ea-multus-daemon-config\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.757714 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.757085 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/62c2d50e-f517-47e6-8042-31b2455981fb-kubelet-config\") pod \"global-pull-secret-syncer-ndzbl\" (UID: \"62c2d50e-f517-47e6-8042-31b2455981fb\") " pod="kube-system/global-pull-secret-syncer-ndzbl" Sep 04 22:08:18.757714 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.757111 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvfjv\" (UniqueName: \"kubernetes.io/projected/76bd7339-f02c-4c10-af0a-af3010cfbc30-kube-api-access-lvfjv\") pod \"iptables-alerter-v7kt4\" (UID: \"76bd7339-f02c-4c10-af0a-af3010cfbc30\") " pod="openshift-network-operator/iptables-alerter-v7kt4" Sep 04 22:08:18.758465 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.757136 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-var-lib-openvswitch\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.758465 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.757155 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8319608-4bcc-405e-b607-e3fea2c4e9a8-tmp-dir\") pod \"node-resolver-p9mdt\" (UID: \"c8319608-4bcc-405e-b607-e3fea2c4e9a8\") " pod="openshift-dns/node-resolver-p9mdt" Sep 04 22:08:18.758465 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.757164 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/666e8bf3-3b4b-4dc6-8b12-5546378326ea-system-cni-dir\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.758465 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.757191 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-run-systemd\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.758465 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.757215 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-run-ovn\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.758465 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.757240 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-host-cni-netd\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.758465 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.757246 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/95ade170-0175-41e0-b78b-ede111ef37af-cnibin\") pod \"multus-additional-cni-plugins-b49j5\" (UID: \"95ade170-0175-41e0-b78b-ede111ef37af\") " pod="openshift-multus/multus-additional-cni-plugins-b49j5" Sep 04 22:08:18.758465 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.757265 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.758465 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.757289 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/666e8bf3-3b4b-4dc6-8b12-5546378326ea-hostroot\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.758465 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.757292 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/37958fa0-bfd6-402f-9c5c-faa146b7d886-env-overrides\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.758465 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.757319 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/666e8bf3-3b4b-4dc6-8b12-5546378326ea-cni-binary-copy\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.758465 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.757379 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/666e8bf3-3b4b-4dc6-8b12-5546378326ea-cnibin\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.758465 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.757403 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/666e8bf3-3b4b-4dc6-8b12-5546378326ea-system-cni-dir\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.758465 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.757436 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e424da81-8acd-4c1a-80c0-d1087d78ae44-etc-sysctl-d\") pod \"tuned-jdkml\" (UID: \"e424da81-8acd-4c1a-80c0-d1087d78ae44\") " pod="openshift-cluster-node-tuning-operator/tuned-jdkml" Sep 04 22:08:18.758465 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.757463 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-host-cni-bin\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.758465 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.757492 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95ade170-0175-41e0-b78b-ede111ef37af-system-cni-dir\") pod \"multus-additional-cni-plugins-b49j5\" (UID: \"95ade170-0175-41e0-b78b-ede111ef37af\") " pod="openshift-multus/multus-additional-cni-plugins-b49j5" Sep 04 22:08:18.758465 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.757523 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/95ade170-0175-41e0-b78b-ede111ef37af-cni-binary-copy\") pod \"multus-additional-cni-plugins-b49j5\" (UID: \"95ade170-0175-41e0-b78b-ede111ef37af\") " pod="openshift-multus/multus-additional-cni-plugins-b49j5" Sep 04 22:08:18.759295 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.757534 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/666e8bf3-3b4b-4dc6-8b12-5546378326ea-multus-daemon-config\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.759295 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.757554 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/666e8bf3-3b4b-4dc6-8b12-5546378326ea-multus-conf-dir\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.759295 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.757566 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/95ade170-0175-41e0-b78b-ede111ef37af-system-cni-dir\") pod \"multus-additional-cni-plugins-b49j5\" (UID: \"95ade170-0175-41e0-b78b-ede111ef37af\") " pod="openshift-multus/multus-additional-cni-plugins-b49j5" Sep 04 22:08:18.759295 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.757580 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54f49670-6d36-4464-b9c4-87718e384fb2-host\") pod \"node-ca-d5j8h\" (UID: \"54f49670-6d36-4464-b9c4-87718e384fb2\") " pod="openshift-image-registry/node-ca-d5j8h" Sep 04 22:08:18.759295 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.757625 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/666e8bf3-3b4b-4dc6-8b12-5546378326ea-multus-conf-dir\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.759295 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.757659 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/37958fa0-bfd6-402f-9c5c-faa146b7d886-ovnkube-config\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.759295 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.757687 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/37958fa0-bfd6-402f-9c5c-faa146b7d886-ovn-node-metrics-cert\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.759295 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.757716 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef-socket-dir\") pod \"aws-ebs-csi-driver-node-xks49\" (UID: \"6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xks49" Sep 04 22:08:18.759295 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.757741 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4gf8\" (UniqueName: \"kubernetes.io/projected/e424da81-8acd-4c1a-80c0-d1087d78ae44-kube-api-access-b4gf8\") pod \"tuned-jdkml\" (UID: \"e424da81-8acd-4c1a-80c0-d1087d78ae44\") " pod="openshift-cluster-node-tuning-operator/tuned-jdkml" Sep 04 22:08:18.759295 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.757768 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef-registration-dir\") pod \"aws-ebs-csi-driver-node-xks49\" (UID: \"6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xks49" Sep 04 22:08:18.759295 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.757810 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxwmr\" (UniqueName: \"kubernetes.io/projected/6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef-kube-api-access-kxwmr\") pod \"aws-ebs-csi-driver-node-xks49\" (UID: \"6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xks49" Sep 04 22:08:18.759295 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.757842 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfpf2\" (UniqueName: \"kubernetes.io/projected/54f49670-6d36-4464-b9c4-87718e384fb2-kube-api-access-gfpf2\") pod \"node-ca-d5j8h\" (UID: \"54f49670-6d36-4464-b9c4-87718e384fb2\") " pod="openshift-image-registry/node-ca-d5j8h" Sep 04 22:08:18.759295 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.757872 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/95ade170-0175-41e0-b78b-ede111ef37af-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b49j5\" (UID: \"95ade170-0175-41e0-b78b-ede111ef37af\") " pod="openshift-multus/multus-additional-cni-plugins-b49j5" Sep 04 22:08:18.759295 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.757899 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1373f05f-60fa-4414-a0da-93106632c578-konnectivity-ca\") pod \"konnectivity-agent-9gcxl\" (UID: \"1373f05f-60fa-4414-a0da-93106632c578\") " pod="kube-system/konnectivity-agent-9gcxl" Sep 04 22:08:18.759295 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.757925 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-host-slash\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.759295 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.757948 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e424da81-8acd-4c1a-80c0-d1087d78ae44-host\") pod \"tuned-jdkml\" (UID: \"e424da81-8acd-4c1a-80c0-d1087d78ae44\") " pod="openshift-cluster-node-tuning-operator/tuned-jdkml" Sep 04 22:08:18.760177 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.757958 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/95ade170-0175-41e0-b78b-ede111ef37af-cni-binary-copy\") pod \"multus-additional-cni-plugins-b49j5\" (UID: \"95ade170-0175-41e0-b78b-ede111ef37af\") " pod="openshift-multus/multus-additional-cni-plugins-b49j5" Sep 04 22:08:18.760177 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.757974 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef-device-dir\") pod \"aws-ebs-csi-driver-node-xks49\" (UID: \"6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xks49" Sep 04 22:08:18.760177 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.757949 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/95ade170-0175-41e0-b78b-ede111ef37af-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b49j5\" (UID: \"95ade170-0175-41e0-b78b-ede111ef37af\") " pod="openshift-multus/multus-additional-cni-plugins-b49j5" Sep 04 22:08:18.760177 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.758087 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzgbw\" (UniqueName: \"kubernetes.io/projected/6200aa24-5731-4d1a-865b-6c7c1cd218d1-kube-api-access-mzgbw\") pod \"network-check-target-ggpqf\" (UID: \"6200aa24-5731-4d1a-865b-6c7c1cd218d1\") " pod="openshift-network-diagnostics/network-check-target-ggpqf" Sep 04 22:08:18.760177 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.758117 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/666e8bf3-3b4b-4dc6-8b12-5546378326ea-host-var-lib-cni-multus\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.760177 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.758141 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-log-socket\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.760177 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.758166 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/666e8bf3-3b4b-4dc6-8b12-5546378326ea-host-var-lib-cni-bin\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.760177 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.758212 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/666e8bf3-3b4b-4dc6-8b12-5546378326ea-multus-cni-dir\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.760177 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.758238 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/666e8bf3-3b4b-4dc6-8b12-5546378326ea-multus-socket-dir-parent\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.760177 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.758257 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/95ade170-0175-41e0-b78b-ede111ef37af-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b49j5\" (UID: \"95ade170-0175-41e0-b78b-ede111ef37af\") " pod="openshift-multus/multus-additional-cni-plugins-b49j5" Sep 04 22:08:18.760177 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.758262 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e424da81-8acd-4c1a-80c0-d1087d78ae44-etc-sysctl-conf\") pod \"tuned-jdkml\" (UID: \"e424da81-8acd-4c1a-80c0-d1087d78ae44\") " pod="openshift-cluster-node-tuning-operator/tuned-jdkml" Sep 04 22:08:18.760177 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.758293 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/666e8bf3-3b4b-4dc6-8b12-5546378326ea-host-var-lib-kubelet\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.760177 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.758320 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/666e8bf3-3b4b-4dc6-8b12-5546378326ea-etc-kubernetes\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.760177 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.758346 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/62c2d50e-f517-47e6-8042-31b2455981fb-dbus\") pod \"global-pull-secret-syncer-ndzbl\" (UID: \"62c2d50e-f517-47e6-8042-31b2455981fb\") " pod="kube-system/global-pull-secret-syncer-ndzbl" Sep 04 22:08:18.760177 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.758401 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/76bd7339-f02c-4c10-af0a-af3010cfbc30-host-slash\") pod \"iptables-alerter-v7kt4\" (UID: \"76bd7339-f02c-4c10-af0a-af3010cfbc30\") " pod="openshift-network-operator/iptables-alerter-v7kt4" Sep 04 22:08:18.760177 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.758457 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/666e8bf3-3b4b-4dc6-8b12-5546378326ea-host-var-lib-cni-bin\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.760177 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.758468 2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 04 22:08:18.761134 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.758585 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/666e8bf3-3b4b-4dc6-8b12-5546378326ea-multus-cni-dir\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.761134 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.758610 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/666e8bf3-3b4b-4dc6-8b12-5546378326ea-multus-socket-dir-parent\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.761134 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.758653 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/666e8bf3-3b4b-4dc6-8b12-5546378326ea-host-var-lib-kubelet\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.761134 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.758684 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/666e8bf3-3b4b-4dc6-8b12-5546378326ea-etc-kubernetes\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.761134 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.758705 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hm6x\" (UniqueName: \"kubernetes.io/projected/37958fa0-bfd6-402f-9c5c-faa146b7d886-kube-api-access-9hm6x\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.761134 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.758721 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e424da81-8acd-4c1a-80c0-d1087d78ae44-sys\") pod \"tuned-jdkml\" (UID: \"e424da81-8acd-4c1a-80c0-d1087d78ae44\") " pod="openshift-cluster-node-tuning-operator/tuned-jdkml" Sep 04 22:08:18.761134 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.758460 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/666e8bf3-3b4b-4dc6-8b12-5546378326ea-host-var-lib-cni-multus\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.761134 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.759037 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1373f05f-60fa-4414-a0da-93106632c578-konnectivity-ca\") pod \"konnectivity-agent-9gcxl\" (UID: \"1373f05f-60fa-4414-a0da-93106632c578\") " pod="kube-system/konnectivity-agent-9gcxl" Sep 04 22:08:18.768733 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.768713 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1373f05f-60fa-4414-a0da-93106632c578-agent-certs\") pod \"konnectivity-agent-9gcxl\" (UID: \"1373f05f-60fa-4414-a0da-93106632c578\") " pod="kube-system/konnectivity-agent-9gcxl" Sep 04 22:08:18.769711 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.769686 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krwrw\" (UniqueName: \"kubernetes.io/projected/95ade170-0175-41e0-b78b-ede111ef37af-kube-api-access-krwrw\") pod \"multus-additional-cni-plugins-b49j5\" (UID: \"95ade170-0175-41e0-b78b-ede111ef37af\") " pod="openshift-multus/multus-additional-cni-plugins-b49j5" Sep 04 22:08:18.771427 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.771404 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wczkj\" (UniqueName: \"kubernetes.io/projected/c8319608-4bcc-405e-b607-e3fea2c4e9a8-kube-api-access-wczkj\") pod \"node-resolver-p9mdt\" (UID: \"c8319608-4bcc-405e-b607-e3fea2c4e9a8\") " pod="openshift-dns/node-resolver-p9mdt" Sep 04 22:08:18.782909 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:18.782887 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 04 22:08:18.783013 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:18.782916 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 04 22:08:18.783013 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:18.782931 2574 projected.go:196] Error preparing data for projected volume kube-api-access-mzgbw for pod openshift-network-diagnostics/network-check-target-ggpqf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 04 22:08:18.783013 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:18.783012 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6200aa24-5731-4d1a-865b-6c7c1cd218d1-kube-api-access-mzgbw podName:6200aa24-5731-4d1a-865b-6c7c1cd218d1 nodeName:}" failed. No retries permitted until 2026-09-04 22:08:19.282989348 +0000 UTC m=+3.047464964 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mzgbw" (UniqueName: "kubernetes.io/projected/6200aa24-5731-4d1a-865b-6c7c1cd218d1-kube-api-access-mzgbw") pod "network-check-target-ggpqf" (UID: "6200aa24-5731-4d1a-865b-6c7c1cd218d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 04 22:08:18.785241 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.785217 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dkzz\" (UniqueName: \"kubernetes.io/projected/666e8bf3-3b4b-4dc6-8b12-5546378326ea-kube-api-access-4dkzz\") pod \"multus-sbhpf\" (UID: \"666e8bf3-3b4b-4dc6-8b12-5546378326ea\") " pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.859272 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.859234 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/62c2d50e-f517-47e6-8042-31b2455981fb-kubelet-config\") pod \"global-pull-secret-syncer-ndzbl\" (UID: \"62c2d50e-f517-47e6-8042-31b2455981fb\") " pod="kube-system/global-pull-secret-syncer-ndzbl" Sep 04 22:08:18.859403 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.859280 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvfjv\" (UniqueName: \"kubernetes.io/projected/76bd7339-f02c-4c10-af0a-af3010cfbc30-kube-api-access-lvfjv\") pod \"iptables-alerter-v7kt4\" (UID: \"76bd7339-f02c-4c10-af0a-af3010cfbc30\") " pod="openshift-network-operator/iptables-alerter-v7kt4" Sep 04 22:08:18.859403 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.859306 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-var-lib-openvswitch\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.859403 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.859332 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-run-systemd\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.859403 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.859357 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-run-ovn\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.859403 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.859374 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/62c2d50e-f517-47e6-8042-31b2455981fb-kubelet-config\") pod \"global-pull-secret-syncer-ndzbl\" (UID: \"62c2d50e-f517-47e6-8042-31b2455981fb\") " pod="kube-system/global-pull-secret-syncer-ndzbl" Sep 04 22:08:18.859403 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.859397 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-run-systemd\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.859768 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.859383 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-host-cni-netd\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.859768 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.859426 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-run-ovn\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.859768 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.859438 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-host-cni-netd\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.859768 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.859411 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-var-lib-openvswitch\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.859768 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.859456 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.859768 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.859486 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/37958fa0-bfd6-402f-9c5c-faa146b7d886-env-overrides\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.859768 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.859496 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.859768 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.859512 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e424da81-8acd-4c1a-80c0-d1087d78ae44-etc-sysctl-d\") pod \"tuned-jdkml\" (UID: \"e424da81-8acd-4c1a-80c0-d1087d78ae44\") " pod="openshift-cluster-node-tuning-operator/tuned-jdkml" Sep 04 22:08:18.859768 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.859539 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-host-cni-bin\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.859768 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.859583 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54f49670-6d36-4464-b9c4-87718e384fb2-host\") pod \"node-ca-d5j8h\" (UID: \"54f49670-6d36-4464-b9c4-87718e384fb2\") " pod="openshift-image-registry/node-ca-d5j8h" Sep 04 22:08:18.859768 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.859609 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/37958fa0-bfd6-402f-9c5c-faa146b7d886-ovnkube-config\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.859768 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.859633 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/37958fa0-bfd6-402f-9c5c-faa146b7d886-ovn-node-metrics-cert\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.859768 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.859653 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e424da81-8acd-4c1a-80c0-d1087d78ae44-etc-sysctl-d\") pod \"tuned-jdkml\" (UID: \"e424da81-8acd-4c1a-80c0-d1087d78ae44\") " pod="openshift-cluster-node-tuning-operator/tuned-jdkml" Sep 04 22:08:18.859768 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.859662 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef-socket-dir\") pod \"aws-ebs-csi-driver-node-xks49\" (UID: \"6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xks49" Sep 04 22:08:18.859768 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.859688 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4gf8\" (UniqueName: \"kubernetes.io/projected/e424da81-8acd-4c1a-80c0-d1087d78ae44-kube-api-access-b4gf8\") pod \"tuned-jdkml\" (UID: \"e424da81-8acd-4c1a-80c0-d1087d78ae44\") " pod="openshift-cluster-node-tuning-operator/tuned-jdkml" Sep 04 22:08:18.859768 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.859699 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54f49670-6d36-4464-b9c4-87718e384fb2-host\") pod \"node-ca-d5j8h\" (UID: \"54f49670-6d36-4464-b9c4-87718e384fb2\") " pod="openshift-image-registry/node-ca-d5j8h" Sep 04 22:08:18.859768 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.859715 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef-registration-dir\") pod \"aws-ebs-csi-driver-node-xks49\" (UID: \"6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xks49" Sep 04 22:08:18.860660 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.859741 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxwmr\" (UniqueName: \"kubernetes.io/projected/6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef-kube-api-access-kxwmr\") pod \"aws-ebs-csi-driver-node-xks49\" (UID: \"6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xks49" Sep 04 22:08:18.860660 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.859742 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-host-cni-bin\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.860660 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.859853 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef-socket-dir\") pod \"aws-ebs-csi-driver-node-xks49\" (UID: \"6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xks49" Sep 04 22:08:18.860660 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.859885 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfpf2\" (UniqueName: \"kubernetes.io/projected/54f49670-6d36-4464-b9c4-87718e384fb2-kube-api-access-gfpf2\") pod \"node-ca-d5j8h\" (UID: \"54f49670-6d36-4464-b9c4-87718e384fb2\") " pod="openshift-image-registry/node-ca-d5j8h" Sep 04 22:08:18.860660 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.859913 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-host-slash\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.860660 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.859936 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e424da81-8acd-4c1a-80c0-d1087d78ae44-host\") pod \"tuned-jdkml\" (UID: \"e424da81-8acd-4c1a-80c0-d1087d78ae44\") " pod="openshift-cluster-node-tuning-operator/tuned-jdkml" Sep 04 22:08:18.860660 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.859939 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/37958fa0-bfd6-402f-9c5c-faa146b7d886-env-overrides\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.860660 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.859960 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef-device-dir\") pod \"aws-ebs-csi-driver-node-xks49\" (UID: \"6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xks49" Sep 04 22:08:18.860660 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860001 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-log-socket\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.860660 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860028 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e424da81-8acd-4c1a-80c0-d1087d78ae44-etc-sysctl-conf\") pod \"tuned-jdkml\" (UID: \"e424da81-8acd-4c1a-80c0-d1087d78ae44\") " pod="openshift-cluster-node-tuning-operator/tuned-jdkml" Sep 04 22:08:18.860660 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860055 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/62c2d50e-f517-47e6-8042-31b2455981fb-dbus\") pod \"global-pull-secret-syncer-ndzbl\" (UID: \"62c2d50e-f517-47e6-8042-31b2455981fb\") " pod="kube-system/global-pull-secret-syncer-ndzbl" Sep 04 22:08:18.860660 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860061 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef-registration-dir\") pod \"aws-ebs-csi-driver-node-xks49\" (UID: \"6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xks49" Sep 04 22:08:18.860660 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860081 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/76bd7339-f02c-4c10-af0a-af3010cfbc30-host-slash\") pod \"iptables-alerter-v7kt4\" (UID: \"76bd7339-f02c-4c10-af0a-af3010cfbc30\") " pod="openshift-network-operator/iptables-alerter-v7kt4" Sep 04 22:08:18.860660 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860106 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hm6x\" (UniqueName: \"kubernetes.io/projected/37958fa0-bfd6-402f-9c5c-faa146b7d886-kube-api-access-9hm6x\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.860660 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860129 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e424da81-8acd-4c1a-80c0-d1087d78ae44-sys\") pod \"tuned-jdkml\" (UID: \"e424da81-8acd-4c1a-80c0-d1087d78ae44\") " pod="openshift-cluster-node-tuning-operator/tuned-jdkml" Sep 04 22:08:18.860660 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860155 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-host-run-ovn-kubernetes\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.860660 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860170 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-host-slash\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.861571 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860180 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62c2d50e-f517-47e6-8042-31b2455981fb-original-pull-secret\") pod \"global-pull-secret-syncer-ndzbl\" (UID: \"62c2d50e-f517-47e6-8042-31b2455981fb\") " pod="kube-system/global-pull-secret-syncer-ndzbl" Sep 04 22:08:18.861571 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860206 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0972118-6d5c-4d3a-a979-497ab05ca339-metrics-certs\") pod \"network-metrics-daemon-qvwc7\" (UID: \"c0972118-6d5c-4d3a-a979-497ab05ca339\") " pod="openshift-multus/network-metrics-daemon-qvwc7" Sep 04 22:08:18.861571 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860222 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-host-kubelet\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.861571 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860224 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e424da81-8acd-4c1a-80c0-d1087d78ae44-host\") pod \"tuned-jdkml\" (UID: \"e424da81-8acd-4c1a-80c0-d1087d78ae44\") " pod="openshift-cluster-node-tuning-operator/tuned-jdkml" Sep 04 22:08:18.861571 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860130 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef-device-dir\") pod \"aws-ebs-csi-driver-node-xks49\" (UID: \"6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xks49" Sep 04 22:08:18.861571 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860247 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-systemd-units\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.861571 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860272 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/37958fa0-bfd6-402f-9c5c-faa146b7d886-ovnkube-config\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.861571 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860285 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-log-socket\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.861571 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860291 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-host-run-netns\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.861571 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860323 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e424da81-8acd-4c1a-80c0-d1087d78ae44-etc-tuned\") pod \"tuned-jdkml\" (UID: \"e424da81-8acd-4c1a-80c0-d1087d78ae44\") " pod="openshift-cluster-node-tuning-operator/tuned-jdkml" Sep 04 22:08:18.861571 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:18.860330 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 04 22:08:18.861571 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860345 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e424da81-8acd-4c1a-80c0-d1087d78ae44-run\") pod \"tuned-jdkml\" (UID: \"e424da81-8acd-4c1a-80c0-d1087d78ae44\") " pod="openshift-cluster-node-tuning-operator/tuned-jdkml" Sep 04 22:08:18.861571 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860343 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/62c2d50e-f517-47e6-8042-31b2455981fb-dbus\") pod \"global-pull-secret-syncer-ndzbl\" (UID: \"62c2d50e-f517-47e6-8042-31b2455981fb\") " pod="kube-system/global-pull-secret-syncer-ndzbl" Sep 04 22:08:18.861571 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:18.860361 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 04 22:08:18.861571 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860374 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-etc-openvswitch\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.861571 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:18.860396 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62c2d50e-f517-47e6-8042-31b2455981fb-original-pull-secret podName:62c2d50e-f517-47e6-8042-31b2455981fb nodeName:}" failed. No retries permitted until 2026-09-04 22:08:19.360377945 +0000 UTC m=+3.124853561 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62c2d50e-f517-47e6-8042-31b2455981fb-original-pull-secret") pod "global-pull-secret-syncer-ndzbl" (UID: "62c2d50e-f517-47e6-8042-31b2455981fb") : object "kube-system"/"original-pull-secret" not registered Sep 04 22:08:18.861571 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860419 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-etc-openvswitch\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.861571 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860431 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e424da81-8acd-4c1a-80c0-d1087d78ae44-etc-sysconfig\") pod \"tuned-jdkml\" (UID: \"e424da81-8acd-4c1a-80c0-d1087d78ae44\") " pod="openshift-cluster-node-tuning-operator/tuned-jdkml" Sep 04 22:08:18.862398 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860433 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-host-run-netns\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.862398 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:18.860447 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0972118-6d5c-4d3a-a979-497ab05ca339-metrics-certs podName:c0972118-6d5c-4d3a-a979-497ab05ca339 nodeName:}" failed. No retries permitted until 2026-09-04 22:08:19.360430645 +0000 UTC m=+3.124906256 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0972118-6d5c-4d3a-a979-497ab05ca339-metrics-certs") pod "network-metrics-daemon-qvwc7" (UID: "c0972118-6d5c-4d3a-a979-497ab05ca339") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 04 22:08:18.862398 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860444 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/76bd7339-f02c-4c10-af0a-af3010cfbc30-host-slash\") pod \"iptables-alerter-v7kt4\" (UID: \"76bd7339-f02c-4c10-af0a-af3010cfbc30\") " pod="openshift-network-operator/iptables-alerter-v7kt4" Sep 04 22:08:18.862398 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860348 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e424da81-8acd-4c1a-80c0-d1087d78ae44-sys\") pod \"tuned-jdkml\" (UID: \"e424da81-8acd-4c1a-80c0-d1087d78ae44\") " pod="openshift-cluster-node-tuning-operator/tuned-jdkml" Sep 04 22:08:18.862398 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860469 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e424da81-8acd-4c1a-80c0-d1087d78ae44-etc-sysctl-conf\") pod \"tuned-jdkml\" (UID: \"e424da81-8acd-4c1a-80c0-d1087d78ae44\") " pod="openshift-cluster-node-tuning-operator/tuned-jdkml" Sep 04 22:08:18.862398 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860394 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-host-run-ovn-kubernetes\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.862398 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860483 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-host-kubelet\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.862398 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860475 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-systemd-units\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.862398 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860500 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e424da81-8acd-4c1a-80c0-d1087d78ae44-etc-systemd\") pod \"tuned-jdkml\" (UID: \"e424da81-8acd-4c1a-80c0-d1087d78ae44\") " pod="openshift-cluster-node-tuning-operator/tuned-jdkml" Sep 04 22:08:18.862398 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860522 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e424da81-8acd-4c1a-80c0-d1087d78ae44-run\") pod \"tuned-jdkml\" (UID: \"e424da81-8acd-4c1a-80c0-d1087d78ae44\") " pod="openshift-cluster-node-tuning-operator/tuned-jdkml" Sep 04 22:08:18.862398 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860534 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/76bd7339-f02c-4c10-af0a-af3010cfbc30-iptables-alerter-script\") pod \"iptables-alerter-v7kt4\" (UID: \"76bd7339-f02c-4c10-af0a-af3010cfbc30\") " pod="openshift-network-operator/iptables-alerter-v7kt4" Sep 04 22:08:18.862398 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860547 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e424da81-8acd-4c1a-80c0-d1087d78ae44-etc-systemd\") pod \"tuned-jdkml\" (UID: \"e424da81-8acd-4c1a-80c0-d1087d78ae44\") " pod="openshift-cluster-node-tuning-operator/tuned-jdkml" Sep 04 22:08:18.862398 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860563 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e424da81-8acd-4c1a-80c0-d1087d78ae44-etc-kubernetes\") pod \"tuned-jdkml\" (UID: \"e424da81-8acd-4c1a-80c0-d1087d78ae44\") " pod="openshift-cluster-node-tuning-operator/tuned-jdkml" Sep 04 22:08:18.862398 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860569 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e424da81-8acd-4c1a-80c0-d1087d78ae44-etc-sysconfig\") pod \"tuned-jdkml\" (UID: \"e424da81-8acd-4c1a-80c0-d1087d78ae44\") " pod="openshift-cluster-node-tuning-operator/tuned-jdkml" Sep 04 22:08:18.862398 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860588 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e424da81-8acd-4c1a-80c0-d1087d78ae44-var-lib-kubelet\") pod \"tuned-jdkml\" (UID: \"e424da81-8acd-4c1a-80c0-d1087d78ae44\") " pod="openshift-cluster-node-tuning-operator/tuned-jdkml" Sep 04 22:08:18.862398 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860615 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xks49\" (UID: \"6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xks49" Sep 04 22:08:18.862398 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860622 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e424da81-8acd-4c1a-80c0-d1087d78ae44-etc-kubernetes\") pod \"tuned-jdkml\" (UID: \"e424da81-8acd-4c1a-80c0-d1087d78ae44\") " pod="openshift-cluster-node-tuning-operator/tuned-jdkml" Sep 04 22:08:18.863234 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860657 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef-etc-selinux\") pod \"aws-ebs-csi-driver-node-xks49\" (UID: \"6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xks49" Sep 04 22:08:18.863234 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860681 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-node-log\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.863234 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860694 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xks49\" (UID: \"6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xks49" Sep 04 22:08:18.863234 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860696 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e424da81-8acd-4c1a-80c0-d1087d78ae44-var-lib-kubelet\") pod \"tuned-jdkml\" (UID: \"e424da81-8acd-4c1a-80c0-d1087d78ae44\") " pod="openshift-cluster-node-tuning-operator/tuned-jdkml" Sep 04 22:08:18.863234 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860705 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e424da81-8acd-4c1a-80c0-d1087d78ae44-lib-modules\") pod \"tuned-jdkml\" (UID: \"e424da81-8acd-4c1a-80c0-d1087d78ae44\") " pod="openshift-cluster-node-tuning-operator/tuned-jdkml" Sep 04 22:08:18.863234 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860743 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-node-log\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.863234 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860742 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef-sys-fs\") pod \"aws-ebs-csi-driver-node-xks49\" (UID: \"6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xks49" Sep 04 22:08:18.863234 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860787 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef-sys-fs\") pod \"aws-ebs-csi-driver-node-xks49\" (UID: \"6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xks49" Sep 04 22:08:18.863234 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860787 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-run-openvswitch\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.863234 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860806 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e424da81-8acd-4c1a-80c0-d1087d78ae44-lib-modules\") pod \"tuned-jdkml\" (UID: \"e424da81-8acd-4c1a-80c0-d1087d78ae44\") " pod="openshift-cluster-node-tuning-operator/tuned-jdkml" Sep 04 22:08:18.863234 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860819 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37958fa0-bfd6-402f-9c5c-faa146b7d886-run-openvswitch\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.863234 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860819 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e424da81-8acd-4c1a-80c0-d1087d78ae44-etc-modprobe-d\") pod \"tuned-jdkml\" (UID: \"e424da81-8acd-4c1a-80c0-d1087d78ae44\") " pod="openshift-cluster-node-tuning-operator/tuned-jdkml" Sep 04 22:08:18.863234 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860849 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e424da81-8acd-4c1a-80c0-d1087d78ae44-tmp\") pod \"tuned-jdkml\" (UID: \"e424da81-8acd-4c1a-80c0-d1087d78ae44\") " pod="openshift-cluster-node-tuning-operator/tuned-jdkml" Sep 04 22:08:18.863234 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860880 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/54f49670-6d36-4464-b9c4-87718e384fb2-serviceca\") pod \"node-ca-d5j8h\" (UID: \"54f49670-6d36-4464-b9c4-87718e384fb2\") " pod="openshift-image-registry/node-ca-d5j8h" Sep 04 22:08:18.863234 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860901 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef-etc-selinux\") pod \"aws-ebs-csi-driver-node-xks49\" (UID: \"6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xks49" Sep 04 22:08:18.863234 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860907 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e424da81-8acd-4c1a-80c0-d1087d78ae44-etc-modprobe-d\") pod \"tuned-jdkml\" (UID: \"e424da81-8acd-4c1a-80c0-d1087d78ae44\") " pod="openshift-cluster-node-tuning-operator/tuned-jdkml" Sep 04 22:08:18.863234 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.860905 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8zsr\" (UniqueName: \"kubernetes.io/projected/c0972118-6d5c-4d3a-a979-497ab05ca339-kube-api-access-m8zsr\") pod \"network-metrics-daemon-qvwc7\" (UID: \"c0972118-6d5c-4d3a-a979-497ab05ca339\") " pod="openshift-multus/network-metrics-daemon-qvwc7" Sep 04 22:08:18.863996 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.861070 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/37958fa0-bfd6-402f-9c5c-faa146b7d886-ovnkube-script-lib\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.863996 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.861143 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/76bd7339-f02c-4c10-af0a-af3010cfbc30-iptables-alerter-script\") pod \"iptables-alerter-v7kt4\" (UID: \"76bd7339-f02c-4c10-af0a-af3010cfbc30\") " pod="openshift-network-operator/iptables-alerter-v7kt4" Sep 04 22:08:18.863996 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.861433 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/54f49670-6d36-4464-b9c4-87718e384fb2-serviceca\") pod \"node-ca-d5j8h\" (UID: \"54f49670-6d36-4464-b9c4-87718e384fb2\") " pod="openshift-image-registry/node-ca-d5j8h" Sep 04 22:08:18.863996 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.861601 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/37958fa0-bfd6-402f-9c5c-faa146b7d886-ovnkube-script-lib\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.863996 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.862625 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/37958fa0-bfd6-402f-9c5c-faa146b7d886-ovn-node-metrics-cert\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.864496 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.864466 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e424da81-8acd-4c1a-80c0-d1087d78ae44-tmp\") pod \"tuned-jdkml\" (UID: \"e424da81-8acd-4c1a-80c0-d1087d78ae44\") " pod="openshift-cluster-node-tuning-operator/tuned-jdkml" Sep 04 22:08:18.864952 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.864910 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e424da81-8acd-4c1a-80c0-d1087d78ae44-etc-tuned\") pod \"tuned-jdkml\" (UID: \"e424da81-8acd-4c1a-80c0-d1087d78ae44\") " pod="openshift-cluster-node-tuning-operator/tuned-jdkml" Sep 04 22:08:18.880862 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.880797 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvfjv\" (UniqueName: \"kubernetes.io/projected/76bd7339-f02c-4c10-af0a-af3010cfbc30-kube-api-access-lvfjv\") pod \"iptables-alerter-v7kt4\" (UID: \"76bd7339-f02c-4c10-af0a-af3010cfbc30\") " pod="openshift-network-operator/iptables-alerter-v7kt4" Sep 04 22:08:18.882652 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.882590 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfpf2\" (UniqueName: \"kubernetes.io/projected/54f49670-6d36-4464-b9c4-87718e384fb2-kube-api-access-gfpf2\") pod \"node-ca-d5j8h\" (UID: \"54f49670-6d36-4464-b9c4-87718e384fb2\") " pod="openshift-image-registry/node-ca-d5j8h" Sep 04 22:08:18.882944 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.882925 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hm6x\" (UniqueName: \"kubernetes.io/projected/37958fa0-bfd6-402f-9c5c-faa146b7d886-kube-api-access-9hm6x\") pod \"ovnkube-node-gv845\" (UID: \"37958fa0-bfd6-402f-9c5c-faa146b7d886\") " pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:18.883456 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.883430 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4gf8\" (UniqueName: \"kubernetes.io/projected/e424da81-8acd-4c1a-80c0-d1087d78ae44-kube-api-access-b4gf8\") pod \"tuned-jdkml\" (UID: \"e424da81-8acd-4c1a-80c0-d1087d78ae44\") " pod="openshift-cluster-node-tuning-operator/tuned-jdkml" Sep 04 22:08:18.884573 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.884554 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxwmr\" (UniqueName: \"kubernetes.io/projected/6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef-kube-api-access-kxwmr\") pod \"aws-ebs-csi-driver-node-xks49\" (UID: \"6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xks49" Sep 04 22:08:18.885149 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.885125 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8zsr\" (UniqueName: \"kubernetes.io/projected/c0972118-6d5c-4d3a-a979-497ab05ca339-kube-api-access-m8zsr\") pod \"network-metrics-daemon-qvwc7\" (UID: \"c0972118-6d5c-4d3a-a979-497ab05ca339\") " pod="openshift-multus/network-metrics-daemon-qvwc7" Sep 04 22:08:18.950286 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.950256 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b49j5" Sep 04 22:08:18.961592 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.961570 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9gcxl" Sep 04 22:08:18.973447 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.973429 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sbhpf" Sep 04 22:08:18.981252 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.981234 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-p9mdt" Sep 04 22:08:18.991054 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.991034 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v7kt4" Sep 04 22:08:18.999755 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:18.999734 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:19.009075 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:19.009052 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d5j8h" Sep 04 22:08:19.018007 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:19.017987 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jdkml" Sep 04 22:08:19.025241 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:19.025221 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xks49" Sep 04 22:08:19.365426 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:19.365394 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62c2d50e-f517-47e6-8042-31b2455981fb-original-pull-secret\") pod \"global-pull-secret-syncer-ndzbl\" (UID: \"62c2d50e-f517-47e6-8042-31b2455981fb\") " pod="kube-system/global-pull-secret-syncer-ndzbl" Sep 04 22:08:19.365639 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:19.365431 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0972118-6d5c-4d3a-a979-497ab05ca339-metrics-certs\") pod \"network-metrics-daemon-qvwc7\" (UID: \"c0972118-6d5c-4d3a-a979-497ab05ca339\") " pod="openshift-multus/network-metrics-daemon-qvwc7" Sep 04 22:08:19.365639 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:19.365496 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzgbw\" (UniqueName: \"kubernetes.io/projected/6200aa24-5731-4d1a-865b-6c7c1cd218d1-kube-api-access-mzgbw\") pod \"network-check-target-ggpqf\" (UID: \"6200aa24-5731-4d1a-865b-6c7c1cd218d1\") " pod="openshift-network-diagnostics/network-check-target-ggpqf" Sep 04 22:08:19.365639 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:19.365577 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 04 22:08:19.365639 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:19.365611 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 04 22:08:19.365639 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:19.365630 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 04 22:08:19.365639 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:19.365643 2574 projected.go:196] Error preparing data for projected volume kube-api-access-mzgbw for pod openshift-network-diagnostics/network-check-target-ggpqf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 04 22:08:19.365906 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:19.365650 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62c2d50e-f517-47e6-8042-31b2455981fb-original-pull-secret podName:62c2d50e-f517-47e6-8042-31b2455981fb nodeName:}" failed. No retries permitted until 2026-09-04 22:08:20.365629219 +0000 UTC m=+4.130104831 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62c2d50e-f517-47e6-8042-31b2455981fb-original-pull-secret") pod "global-pull-secret-syncer-ndzbl" (UID: "62c2d50e-f517-47e6-8042-31b2455981fb") : object "kube-system"/"original-pull-secret" not registered Sep 04 22:08:19.365906 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:19.365666 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 04 22:08:19.365906 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:19.365682 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6200aa24-5731-4d1a-865b-6c7c1cd218d1-kube-api-access-mzgbw podName:6200aa24-5731-4d1a-865b-6c7c1cd218d1 nodeName:}" failed. No retries permitted until 2026-09-04 22:08:20.365671919 +0000 UTC m=+4.130147526 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mzgbw" (UniqueName: "kubernetes.io/projected/6200aa24-5731-4d1a-865b-6c7c1cd218d1-kube-api-access-mzgbw") pod "network-check-target-ggpqf" (UID: "6200aa24-5731-4d1a-865b-6c7c1cd218d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 04 22:08:19.365906 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:19.365736 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0972118-6d5c-4d3a-a979-497ab05ca339-metrics-certs podName:c0972118-6d5c-4d3a-a979-497ab05ca339 nodeName:}" failed. No retries permitted until 2026-09-04 22:08:20.365715734 +0000 UTC m=+4.130191344 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0972118-6d5c-4d3a-a979-497ab05ca339-metrics-certs") pod "network-metrics-daemon-qvwc7" (UID: "c0972118-6d5c-4d3a-a979-497ab05ca339") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 04 22:08:19.738812 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:19.738718 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-03 22:03:19 +0000 UTC" deadline="2028-02-05 20:20:55.69980253 +0000 UTC" Sep 04 22:08:19.738812 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:19.738765 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12454h12m35.961040489s" Sep 04 22:08:19.992544 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:19.992519 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1373f05f_60fa_4414_a0da_93106632c578.slice/crio-28210357f8c6f20bde595ccd17f726fd98c9a28b3c78dee1d907e7cffa7800f4 WatchSource:0}: Error finding container 28210357f8c6f20bde595ccd17f726fd98c9a28b3c78dee1d907e7cffa7800f4: Status 404 returned error can't find the container with id 28210357f8c6f20bde595ccd17f726fd98c9a28b3c78dee1d907e7cffa7800f4 Sep 04 22:08:19.994389 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:19.994358 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8319608_4bcc_405e_b607_e3fea2c4e9a8.slice/crio-3f243f86c224b2c8497d986c40d89868e7e30430a87bc0467f153873fc0b932e WatchSource:0}: Error finding container 3f243f86c224b2c8497d986c40d89868e7e30430a87bc0467f153873fc0b932e: Status 404 returned error can't find the container with id 3f243f86c224b2c8497d986c40d89868e7e30430a87bc0467f153873fc0b932e Sep 04 22:08:19.998868 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:19.998835 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode424da81_8acd_4c1a_80c0_d1087d78ae44.slice/crio-daa8854af802ab595db4546fbfaf95549b8d5fed7d525fab09f46d13376fe700 WatchSource:0}: Error finding container daa8854af802ab595db4546fbfaf95549b8d5fed7d525fab09f46d13376fe700: Status 404 returned error can't find the container with id daa8854af802ab595db4546fbfaf95549b8d5fed7d525fab09f46d13376fe700 Sep 04 22:08:20.000198 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:20.000150 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37958fa0_bfd6_402f_9c5c_faa146b7d886.slice/crio-60a8f2a0d0eb363ab1064ea0af0f101a9cb5d1b7fdd17fe5cd2cda7b030da9b9 WatchSource:0}: Error finding container 60a8f2a0d0eb363ab1064ea0af0f101a9cb5d1b7fdd17fe5cd2cda7b030da9b9: Status 404 returned error can't find the container with id 60a8f2a0d0eb363ab1064ea0af0f101a9cb5d1b7fdd17fe5cd2cda7b030da9b9 Sep 04 22:08:20.001149 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:20.000932 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod666e8bf3_3b4b_4dc6_8b12_5546378326ea.slice/crio-cb5419eeefaea21c21eab375fddd5482540924652e439d1c4d453f7c0367f81c WatchSource:0}: Error finding container cb5419eeefaea21c21eab375fddd5482540924652e439d1c4d453f7c0367f81c: Status 404 returned error can't find the container with id cb5419eeefaea21c21eab375fddd5482540924652e439d1c4d453f7c0367f81c Sep 04 22:08:20.005214 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:20.002747 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a87f1c3_6ff0_4e6f_9a69_85f3ac7b93ef.slice/crio-335ed2392d88f3dd2d7be9482207727538b721d8402fc8cad621b75db9c601d9 WatchSource:0}: Error finding container 335ed2392d88f3dd2d7be9482207727538b721d8402fc8cad621b75db9c601d9: Status 404 returned error can't find the container with id 335ed2392d88f3dd2d7be9482207727538b721d8402fc8cad621b75db9c601d9 Sep 04 22:08:20.373146 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:20.373112 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzgbw\" (UniqueName: \"kubernetes.io/projected/6200aa24-5731-4d1a-865b-6c7c1cd218d1-kube-api-access-mzgbw\") pod \"network-check-target-ggpqf\" (UID: \"6200aa24-5731-4d1a-865b-6c7c1cd218d1\") " pod="openshift-network-diagnostics/network-check-target-ggpqf" Sep 04 22:08:20.373146 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:20.373148 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62c2d50e-f517-47e6-8042-31b2455981fb-original-pull-secret\") pod \"global-pull-secret-syncer-ndzbl\" (UID: \"62c2d50e-f517-47e6-8042-31b2455981fb\") " pod="kube-system/global-pull-secret-syncer-ndzbl" Sep 04 22:08:20.373336 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:20.373169 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0972118-6d5c-4d3a-a979-497ab05ca339-metrics-certs\") pod \"network-metrics-daemon-qvwc7\" (UID: \"c0972118-6d5c-4d3a-a979-497ab05ca339\") " pod="openshift-multus/network-metrics-daemon-qvwc7" Sep 04 22:08:20.373336 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:20.373279 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 04 22:08:20.373336 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:20.373284 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 04 22:08:20.373336 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:20.373301 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 04 22:08:20.373336 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:20.373319 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 04 22:08:20.373336 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:20.373331 2574 projected.go:196] Error preparing data for projected volume kube-api-access-mzgbw for pod openshift-network-diagnostics/network-check-target-ggpqf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 04 22:08:20.373530 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:20.373334 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0972118-6d5c-4d3a-a979-497ab05ca339-metrics-certs podName:c0972118-6d5c-4d3a-a979-497ab05ca339 nodeName:}" failed. No retries permitted until 2026-09-04 22:08:22.373319891 +0000 UTC m=+6.137795484 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0972118-6d5c-4d3a-a979-497ab05ca339-metrics-certs") pod "network-metrics-daemon-qvwc7" (UID: "c0972118-6d5c-4d3a-a979-497ab05ca339") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 04 22:08:20.373530 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:20.373378 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62c2d50e-f517-47e6-8042-31b2455981fb-original-pull-secret podName:62c2d50e-f517-47e6-8042-31b2455981fb nodeName:}" failed. No retries permitted until 2026-09-04 22:08:22.373365294 +0000 UTC m=+6.137840887 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62c2d50e-f517-47e6-8042-31b2455981fb-original-pull-secret") pod "global-pull-secret-syncer-ndzbl" (UID: "62c2d50e-f517-47e6-8042-31b2455981fb") : object "kube-system"/"original-pull-secret" not registered Sep 04 22:08:20.373530 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:20.373392 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6200aa24-5731-4d1a-865b-6c7c1cd218d1-kube-api-access-mzgbw podName:6200aa24-5731-4d1a-865b-6c7c1cd218d1 nodeName:}" failed. No retries permitted until 2026-09-04 22:08:22.373385273 +0000 UTC m=+6.137860865 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mzgbw" (UniqueName: "kubernetes.io/projected/6200aa24-5731-4d1a-865b-6c7c1cd218d1-kube-api-access-mzgbw") pod "network-check-target-ggpqf" (UID: "6200aa24-5731-4d1a-865b-6c7c1cd218d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 04 22:08:20.710843 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:20.710754 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggpqf" Sep 04 22:08:20.711021 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:20.710899 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggpqf" podUID="6200aa24-5731-4d1a-865b-6c7c1cd218d1" Sep 04 22:08:20.711340 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:20.711316 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvwc7" Sep 04 22:08:20.711467 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:20.711444 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvwc7" podUID="c0972118-6d5c-4d3a-a979-497ab05ca339" Sep 04 22:08:20.713285 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:20.713149 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndzbl" Sep 04 22:08:20.713285 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:20.713245 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndzbl" podUID="62c2d50e-f517-47e6-8042-31b2455981fb" Sep 04 22:08:20.734284 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:20.734245 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b49j5" event={"ID":"95ade170-0175-41e0-b78b-ede111ef37af","Type":"ContainerStarted","Data":"54be975c903acbe62dfe0eb99ef15cabe89b26d4cb833a2a8dbcf82a62282774"} Sep 04 22:08:20.739596 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:20.739542 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-03 22:03:19 +0000 UTC" deadline="2028-04-01 07:49:53.18982803 +0000 UTC" Sep 04 22:08:20.739596 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:20.739588 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13785h41m32.45024523s" Sep 04 22:08:20.756619 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:20.756506 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gv845" event={"ID":"37958fa0-bfd6-402f-9c5c-faa146b7d886","Type":"ContainerStarted","Data":"60a8f2a0d0eb363ab1064ea0af0f101a9cb5d1b7fdd17fe5cd2cda7b030da9b9"} Sep 04 22:08:20.765497 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:20.764770 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-219.ec2.internal" event={"ID":"5c4c794f844e4275466dd646fd4a6840","Type":"ContainerStarted","Data":"28a3379e70594e0894632b65a9a21fc9ccba3e3139d6e21768cfd1b9e9b71ad4"} Sep 04 22:08:20.769856 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:20.769828 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d5j8h" event={"ID":"54f49670-6d36-4464-b9c4-87718e384fb2","Type":"ContainerStarted","Data":"5f5ebb65a5067da0b9165470861ea4e9bec26cd90c0b1b89a953552540f06e1d"} Sep 04 22:08:20.780094 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:20.780066 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xks49" event={"ID":"6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef","Type":"ContainerStarted","Data":"335ed2392d88f3dd2d7be9482207727538b721d8402fc8cad621b75db9c601d9"} Sep 04 22:08:20.788417 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:20.788383 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sbhpf" event={"ID":"666e8bf3-3b4b-4dc6-8b12-5546378326ea","Type":"ContainerStarted","Data":"cb5419eeefaea21c21eab375fddd5482540924652e439d1c4d453f7c0367f81c"} Sep 04 22:08:20.810687 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:20.810517 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jdkml" event={"ID":"e424da81-8acd-4c1a-80c0-d1087d78ae44","Type":"ContainerStarted","Data":"daa8854af802ab595db4546fbfaf95549b8d5fed7d525fab09f46d13376fe700"} Sep 04 22:08:20.826062 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:20.825988 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v7kt4" event={"ID":"76bd7339-f02c-4c10-af0a-af3010cfbc30","Type":"ContainerStarted","Data":"ab61fe3879776c03c15b268e4cc20038caac04f5eeac37c8ff216488dd2f4a09"} Sep 04 22:08:20.845725 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:20.845688 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-p9mdt" event={"ID":"c8319608-4bcc-405e-b607-e3fea2c4e9a8","Type":"ContainerStarted","Data":"3f243f86c224b2c8497d986c40d89868e7e30430a87bc0467f153873fc0b932e"} Sep 04 22:08:20.862172 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:20.862135 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9gcxl" event={"ID":"1373f05f-60fa-4414-a0da-93106632c578","Type":"ContainerStarted","Data":"28210357f8c6f20bde595ccd17f726fd98c9a28b3c78dee1d907e7cffa7800f4"} Sep 04 22:08:20.873094 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:20.873057 2574 generic.go:356] "Generic (PLEG): container finished" podID="1c571934cf12fb6136a9351d29e54859" containerID="5bc1ac052e5f79760292f100b6b8e66135f2be43b144502dee902df769d8b3d6" exitCode=0 Sep 04 22:08:20.873243 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:20.873105 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal" event={"ID":"1c571934cf12fb6136a9351d29e54859","Type":"ContainerDied","Data":"5bc1ac052e5f79760292f100b6b8e66135f2be43b144502dee902df769d8b3d6"} Sep 04 22:08:20.899283 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:20.899205 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-219.ec2.internal" podStartSLOduration=2.899183156 podStartE2EDuration="2.899183156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 22:08:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:08:20.803094781 +0000 UTC m=+4.567570400" watchObservedRunningTime="2026-09-04 22:08:20.899183156 +0000 UTC m=+4.663658774" Sep 04 22:08:21.893933 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:21.893696 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal" event={"ID":"1c571934cf12fb6136a9351d29e54859","Type":"ContainerStarted","Data":"af07a0af69cdf5e285a9a18eb4b06bc6e032932fda4c03f87e679b6d3b93a300"} Sep 04 22:08:22.395558 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:22.395523 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzgbw\" (UniqueName: \"kubernetes.io/projected/6200aa24-5731-4d1a-865b-6c7c1cd218d1-kube-api-access-mzgbw\") pod \"network-check-target-ggpqf\" (UID: \"6200aa24-5731-4d1a-865b-6c7c1cd218d1\") " pod="openshift-network-diagnostics/network-check-target-ggpqf" Sep 04 22:08:22.395761 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:22.395588 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62c2d50e-f517-47e6-8042-31b2455981fb-original-pull-secret\") pod \"global-pull-secret-syncer-ndzbl\" (UID: \"62c2d50e-f517-47e6-8042-31b2455981fb\") " pod="kube-system/global-pull-secret-syncer-ndzbl" Sep 04 22:08:22.395761 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:22.395621 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0972118-6d5c-4d3a-a979-497ab05ca339-metrics-certs\") pod \"network-metrics-daemon-qvwc7\" (UID: \"c0972118-6d5c-4d3a-a979-497ab05ca339\") " pod="openshift-multus/network-metrics-daemon-qvwc7" Sep 04 22:08:22.395761 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:22.395719 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 04 22:08:22.395953 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:22.395775 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 04 22:08:22.395953 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:22.395719 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 04 22:08:22.395953 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:22.395813 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0972118-6d5c-4d3a-a979-497ab05ca339-metrics-certs podName:c0972118-6d5c-4d3a-a979-497ab05ca339 nodeName:}" failed. No retries permitted until 2026-09-04 22:08:26.395765453 +0000 UTC m=+10.160241071 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0972118-6d5c-4d3a-a979-497ab05ca339-metrics-certs") pod "network-metrics-daemon-qvwc7" (UID: "c0972118-6d5c-4d3a-a979-497ab05ca339") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 04 22:08:22.395953 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:22.395830 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 04 22:08:22.395953 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:22.395833 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62c2d50e-f517-47e6-8042-31b2455981fb-original-pull-secret podName:62c2d50e-f517-47e6-8042-31b2455981fb nodeName:}" failed. No retries permitted until 2026-09-04 22:08:26.395822464 +0000 UTC m=+10.160298060 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62c2d50e-f517-47e6-8042-31b2455981fb-original-pull-secret") pod "global-pull-secret-syncer-ndzbl" (UID: "62c2d50e-f517-47e6-8042-31b2455981fb") : object "kube-system"/"original-pull-secret" not registered Sep 04 22:08:22.395953 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:22.395845 2574 projected.go:196] Error preparing data for projected volume kube-api-access-mzgbw for pod openshift-network-diagnostics/network-check-target-ggpqf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 04 22:08:22.395953 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:22.395880 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6200aa24-5731-4d1a-865b-6c7c1cd218d1-kube-api-access-mzgbw podName:6200aa24-5731-4d1a-865b-6c7c1cd218d1 nodeName:}" failed. No retries permitted until 2026-09-04 22:08:26.395868696 +0000 UTC m=+10.160344305 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mzgbw" (UniqueName: "kubernetes.io/projected/6200aa24-5731-4d1a-865b-6c7c1cd218d1-kube-api-access-mzgbw") pod "network-check-target-ggpqf" (UID: "6200aa24-5731-4d1a-865b-6c7c1cd218d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 04 22:08:22.710095 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:22.709087 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndzbl" Sep 04 22:08:22.710095 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:22.709235 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndzbl" podUID="62c2d50e-f517-47e6-8042-31b2455981fb" Sep 04 22:08:22.710095 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:22.709691 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggpqf" Sep 04 22:08:22.710095 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:22.709787 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggpqf" podUID="6200aa24-5731-4d1a-865b-6c7c1cd218d1" Sep 04 22:08:22.710095 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:22.709890 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvwc7" Sep 04 22:08:22.710095 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:22.709998 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvwc7" podUID="c0972118-6d5c-4d3a-a979-497ab05ca339" Sep 04 22:08:24.709374 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:24.709325 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndzbl" Sep 04 22:08:24.709916 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:24.709327 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggpqf" Sep 04 22:08:24.709916 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:24.709477 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndzbl" podUID="62c2d50e-f517-47e6-8042-31b2455981fb" Sep 04 22:08:24.709916 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:24.709577 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggpqf" podUID="6200aa24-5731-4d1a-865b-6c7c1cd218d1" Sep 04 22:08:24.709916 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:24.709334 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvwc7" Sep 04 22:08:24.709916 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:24.709696 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvwc7" podUID="c0972118-6d5c-4d3a-a979-497ab05ca339" Sep 04 22:08:26.432428 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:26.432050 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzgbw\" (UniqueName: \"kubernetes.io/projected/6200aa24-5731-4d1a-865b-6c7c1cd218d1-kube-api-access-mzgbw\") pod \"network-check-target-ggpqf\" (UID: \"6200aa24-5731-4d1a-865b-6c7c1cd218d1\") " pod="openshift-network-diagnostics/network-check-target-ggpqf" Sep 04 22:08:26.432428 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:26.432111 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62c2d50e-f517-47e6-8042-31b2455981fb-original-pull-secret\") pod \"global-pull-secret-syncer-ndzbl\" (UID: \"62c2d50e-f517-47e6-8042-31b2455981fb\") " pod="kube-system/global-pull-secret-syncer-ndzbl" Sep 04 22:08:26.432428 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:26.432143 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0972118-6d5c-4d3a-a979-497ab05ca339-metrics-certs\") pod \"network-metrics-daemon-qvwc7\" (UID: \"c0972118-6d5c-4d3a-a979-497ab05ca339\") " pod="openshift-multus/network-metrics-daemon-qvwc7" Sep 04 22:08:26.432428 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:26.432292 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 04 22:08:26.432428 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:26.432310 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 04 22:08:26.432428 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:26.432329 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 04 22:08:26.432428 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:26.432342 2574 projected.go:196] Error preparing data for projected volume kube-api-access-mzgbw for pod openshift-network-diagnostics/network-check-target-ggpqf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 04 22:08:26.432428 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:26.432337 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 04 22:08:26.432428 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:26.432364 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0972118-6d5c-4d3a-a979-497ab05ca339-metrics-certs podName:c0972118-6d5c-4d3a-a979-497ab05ca339 nodeName:}" failed. No retries permitted until 2026-09-04 22:08:34.432344619 +0000 UTC m=+18.196820213 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0972118-6d5c-4d3a-a979-497ab05ca339-metrics-certs") pod "network-metrics-daemon-qvwc7" (UID: "c0972118-6d5c-4d3a-a979-497ab05ca339") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 04 22:08:26.432428 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:26.432381 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6200aa24-5731-4d1a-865b-6c7c1cd218d1-kube-api-access-mzgbw podName:6200aa24-5731-4d1a-865b-6c7c1cd218d1 nodeName:}" failed. No retries permitted until 2026-09-04 22:08:34.432372489 +0000 UTC m=+18.196848082 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mzgbw" (UniqueName: "kubernetes.io/projected/6200aa24-5731-4d1a-865b-6c7c1cd218d1-kube-api-access-mzgbw") pod "network-check-target-ggpqf" (UID: "6200aa24-5731-4d1a-865b-6c7c1cd218d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 04 22:08:26.432428 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:26.432429 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62c2d50e-f517-47e6-8042-31b2455981fb-original-pull-secret podName:62c2d50e-f517-47e6-8042-31b2455981fb nodeName:}" failed. No retries permitted until 2026-09-04 22:08:34.432411132 +0000 UTC m=+18.196886737 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62c2d50e-f517-47e6-8042-31b2455981fb-original-pull-secret") pod "global-pull-secret-syncer-ndzbl" (UID: "62c2d50e-f517-47e6-8042-31b2455981fb") : object "kube-system"/"original-pull-secret" not registered Sep 04 22:08:26.710468 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:26.709958 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndzbl" Sep 04 22:08:26.710468 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:26.710052 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggpqf" Sep 04 22:08:26.710468 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:26.710073 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndzbl" podUID="62c2d50e-f517-47e6-8042-31b2455981fb" Sep 04 22:08:26.710468 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:26.710128 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggpqf" podUID="6200aa24-5731-4d1a-865b-6c7c1cd218d1" Sep 04 22:08:26.710468 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:26.710184 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvwc7" Sep 04 22:08:26.710468 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:26.710318 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvwc7" podUID="c0972118-6d5c-4d3a-a979-497ab05ca339" Sep 04 22:08:28.710567 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:28.710469 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggpqf" Sep 04 22:08:28.711035 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:28.710609 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggpqf" podUID="6200aa24-5731-4d1a-865b-6c7c1cd218d1" Sep 04 22:08:28.711035 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:28.711002 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvwc7" Sep 04 22:08:28.711149 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:28.711115 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvwc7" podUID="c0972118-6d5c-4d3a-a979-497ab05ca339" Sep 04 22:08:28.711201 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:28.711167 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndzbl" Sep 04 22:08:28.711287 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:28.711237 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndzbl" podUID="62c2d50e-f517-47e6-8042-31b2455981fb" Sep 04 22:08:30.709693 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:30.709665 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndzbl" Sep 04 22:08:30.709693 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:30.709680 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggpqf" Sep 04 22:08:30.710106 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:30.709787 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndzbl" podUID="62c2d50e-f517-47e6-8042-31b2455981fb" Sep 04 22:08:30.710106 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:30.709853 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvwc7" Sep 04 22:08:30.710106 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:30.709848 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggpqf" podUID="6200aa24-5731-4d1a-865b-6c7c1cd218d1" Sep 04 22:08:30.710106 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:30.709919 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvwc7" podUID="c0972118-6d5c-4d3a-a979-497ab05ca339" Sep 04 22:08:32.709774 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:32.709741 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggpqf" Sep 04 22:08:32.709774 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:32.709772 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndzbl" Sep 04 22:08:32.710259 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:32.709746 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvwc7" Sep 04 22:08:32.710259 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:32.709852 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggpqf" podUID="6200aa24-5731-4d1a-865b-6c7c1cd218d1" Sep 04 22:08:32.710259 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:32.709966 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvwc7" podUID="c0972118-6d5c-4d3a-a979-497ab05ca339" Sep 04 22:08:32.710259 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:32.710031 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndzbl" podUID="62c2d50e-f517-47e6-8042-31b2455981fb" Sep 04 22:08:34.489678 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:34.489648 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzgbw\" (UniqueName: \"kubernetes.io/projected/6200aa24-5731-4d1a-865b-6c7c1cd218d1-kube-api-access-mzgbw\") pod \"network-check-target-ggpqf\" (UID: \"6200aa24-5731-4d1a-865b-6c7c1cd218d1\") " pod="openshift-network-diagnostics/network-check-target-ggpqf" Sep 04 22:08:34.490164 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:34.489694 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62c2d50e-f517-47e6-8042-31b2455981fb-original-pull-secret\") pod \"global-pull-secret-syncer-ndzbl\" (UID: \"62c2d50e-f517-47e6-8042-31b2455981fb\") " pod="kube-system/global-pull-secret-syncer-ndzbl" Sep 04 22:08:34.490164 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:34.489717 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0972118-6d5c-4d3a-a979-497ab05ca339-metrics-certs\") pod \"network-metrics-daemon-qvwc7\" (UID: \"c0972118-6d5c-4d3a-a979-497ab05ca339\") " pod="openshift-multus/network-metrics-daemon-qvwc7" Sep 04 22:08:34.490164 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:34.489794 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 04 22:08:34.490164 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:34.489822 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 04 22:08:34.490164 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:34.489834 2574 projected.go:196] Error preparing data for projected volume kube-api-access-mzgbw for pod openshift-network-diagnostics/network-check-target-ggpqf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 04 22:08:34.490164 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:34.489842 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 04 22:08:34.490164 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:34.489842 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 04 22:08:34.490164 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:34.489893 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6200aa24-5731-4d1a-865b-6c7c1cd218d1-kube-api-access-mzgbw podName:6200aa24-5731-4d1a-865b-6c7c1cd218d1 nodeName:}" failed. No retries permitted until 2026-09-04 22:08:50.489875658 +0000 UTC m=+34.254351250 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mzgbw" (UniqueName: "kubernetes.io/projected/6200aa24-5731-4d1a-865b-6c7c1cd218d1-kube-api-access-mzgbw") pod "network-check-target-ggpqf" (UID: "6200aa24-5731-4d1a-865b-6c7c1cd218d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 04 22:08:34.490164 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:34.489912 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0972118-6d5c-4d3a-a979-497ab05ca339-metrics-certs podName:c0972118-6d5c-4d3a-a979-497ab05ca339 nodeName:}" failed. No retries permitted until 2026-09-04 22:08:50.489903527 +0000 UTC m=+34.254379117 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0972118-6d5c-4d3a-a979-497ab05ca339-metrics-certs") pod "network-metrics-daemon-qvwc7" (UID: "c0972118-6d5c-4d3a-a979-497ab05ca339") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 04 22:08:34.490164 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:34.489927 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62c2d50e-f517-47e6-8042-31b2455981fb-original-pull-secret podName:62c2d50e-f517-47e6-8042-31b2455981fb nodeName:}" failed. No retries permitted until 2026-09-04 22:08:50.48991923 +0000 UTC m=+34.254394824 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62c2d50e-f517-47e6-8042-31b2455981fb-original-pull-secret") pod "global-pull-secret-syncer-ndzbl" (UID: "62c2d50e-f517-47e6-8042-31b2455981fb") : object "kube-system"/"original-pull-secret" not registered Sep 04 22:08:34.709924 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:34.709892 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndzbl" Sep 04 22:08:34.710108 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:34.709894 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggpqf" Sep 04 22:08:34.710108 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:34.710012 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvwc7" Sep 04 22:08:34.710108 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:34.710013 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndzbl" podUID="62c2d50e-f517-47e6-8042-31b2455981fb" Sep 04 22:08:34.710108 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:34.710098 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggpqf" podUID="6200aa24-5731-4d1a-865b-6c7c1cd218d1" Sep 04 22:08:34.710293 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:34.710174 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvwc7" podUID="c0972118-6d5c-4d3a-a979-497ab05ca339" Sep 04 22:08:36.709929 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:36.709900 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggpqf" Sep 04 22:08:36.710382 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:36.709978 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvwc7" Sep 04 22:08:36.710382 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:36.709994 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggpqf" podUID="6200aa24-5731-4d1a-865b-6c7c1cd218d1" Sep 04 22:08:36.710382 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:36.710104 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvwc7" podUID="c0972118-6d5c-4d3a-a979-497ab05ca339" Sep 04 22:08:36.710382 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:36.710138 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndzbl" Sep 04 22:08:36.710382 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:36.710210 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndzbl" podUID="62c2d50e-f517-47e6-8042-31b2455981fb" Sep 04 22:08:38.709899 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:38.709727 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndzbl" Sep 04 22:08:38.710770 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:38.709727 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggpqf" Sep 04 22:08:38.710770 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:38.709975 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndzbl" podUID="62c2d50e-f517-47e6-8042-31b2455981fb" Sep 04 22:08:38.710770 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:38.710069 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggpqf" podUID="6200aa24-5731-4d1a-865b-6c7c1cd218d1" Sep 04 22:08:38.710770 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:38.709747 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvwc7" Sep 04 22:08:38.710770 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:38.710180 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvwc7" podUID="c0972118-6d5c-4d3a-a979-497ab05ca339" Sep 04 22:08:38.946131 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:38.946054 2574 generic.go:356] "Generic (PLEG): container finished" podID="95ade170-0175-41e0-b78b-ede111ef37af" containerID="c9260abb5ab8ee5c80f48ed2857f79f891d3898f046b60b71a6f5aa81e2ff7ae" exitCode=0 Sep 04 22:08:38.946319 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:38.946149 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b49j5" event={"ID":"95ade170-0175-41e0-b78b-ede111ef37af","Type":"ContainerDied","Data":"c9260abb5ab8ee5c80f48ed2857f79f891d3898f046b60b71a6f5aa81e2ff7ae"} Sep 04 22:08:38.949408 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:38.949374 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gv845" event={"ID":"37958fa0-bfd6-402f-9c5c-faa146b7d886","Type":"ContainerStarted","Data":"92b120116ccbdbbf390c92e65aafb9077149deb7cba4789eee2e98f35e25eab2"} Sep 04 22:08:38.949484 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:38.949470 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gv845" event={"ID":"37958fa0-bfd6-402f-9c5c-faa146b7d886","Type":"ContainerStarted","Data":"f958ac636c0c2d51fd34a0d11214681291b36cb73c681f32720b214b6b71b8b6"} Sep 04 22:08:38.949591 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:38.949501 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gv845" event={"ID":"37958fa0-bfd6-402f-9c5c-faa146b7d886","Type":"ContainerStarted","Data":"8fdc1401296b13229ca0df119da210035e0b33040236d9ed7ed7bbf18d381803"} Sep 04 22:08:38.949591 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:38.949530 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gv845" event={"ID":"37958fa0-bfd6-402f-9c5c-faa146b7d886","Type":"ContainerStarted","Data":"2ae601687a9173370f07cf353abe54cd0141db220dca8cea38ab9bcdd8f4bc2a"} Sep 04 22:08:38.949591 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:38.949582 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gv845" event={"ID":"37958fa0-bfd6-402f-9c5c-faa146b7d886","Type":"ContainerStarted","Data":"86b61997a09a529197bd32d8100d520b0f54390166ec50c6465ef5c5e86bb7e4"} Sep 04 22:08:38.949797 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:38.949614 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gv845" event={"ID":"37958fa0-bfd6-402f-9c5c-faa146b7d886","Type":"ContainerStarted","Data":"07b14b7fdc6d9834624ed722b183d9b6b3087b37328d68a27ef432adcb53a131"} Sep 04 22:08:38.951140 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:38.951104 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d5j8h" event={"ID":"54f49670-6d36-4464-b9c4-87718e384fb2","Type":"ContainerStarted","Data":"0f8fcfba45bb233c1d7826c33099cef8193f5f26b456978493d5bf64c2110f21"} Sep 04 22:08:38.956084 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:38.956058 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xks49" event={"ID":"6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef","Type":"ContainerStarted","Data":"e04ae33595203a672f03264971d71c6cd0d51c3bc265464f98b137de5928347a"} Sep 04 22:08:38.957806 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:38.957781 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sbhpf" event={"ID":"666e8bf3-3b4b-4dc6-8b12-5546378326ea","Type":"ContainerStarted","Data":"628e8e1c5fe32eed38810247942f0a3540b123e780e0ed1adc090fd855821a4d"} Sep 04 22:08:38.959353 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:38.959331 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jdkml" event={"ID":"e424da81-8acd-4c1a-80c0-d1087d78ae44","Type":"ContainerStarted","Data":"9684de45f3b1f9549bd08239bcc044c54cc417468403dfaca6fde5c022abb9d0"} Sep 04 22:08:38.960934 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:38.960909 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-p9mdt" event={"ID":"c8319608-4bcc-405e-b607-e3fea2c4e9a8","Type":"ContainerStarted","Data":"2990e7ee07282845219c0f2eddb4cdbdc475416cb54ace6f9cafbcc1b1434b5c"} Sep 04 22:08:38.962339 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:38.962320 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9gcxl" event={"ID":"1373f05f-60fa-4414-a0da-93106632c578","Type":"ContainerStarted","Data":"2c8f3d833c909984d8490e4f5782dbbc8d2e65d70fcc5a18a7bf7858a3ab1878"} Sep 04 22:08:38.991333 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:38.991290 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-219.ec2.internal" podStartSLOduration=20.991278067 podStartE2EDuration="20.991278067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 22:08:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:08:21.917156768 +0000 UTC m=+5.681632388" watchObservedRunningTime="2026-09-04 22:08:38.991278067 +0000 UTC m=+22.755753679" Sep 04 22:08:39.048729 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:39.048679 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-sbhpf" podStartSLOduration=3.924670877 podStartE2EDuration="22.048663605s" totalImagesPullingTime="18.123992728s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 22:08:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:08:39.048008927 +0000 UTC m=+22.812484540" watchObservedRunningTime="2026-09-04 22:08:39.048663605 +0000 UTC m=+22.813139218" Sep 04 22:08:39.049029 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:39.048999 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jdkml" podStartSLOduration=2.948734603 podStartE2EDuration="21.048986514s" totalImagesPullingTime="18.100251911s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 22:08:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:08:39.023330878 +0000 UTC m=+22.787806491" watchObservedRunningTime="2026-09-04 22:08:39.048986514 +0000 UTC m=+22.813462126" Sep 04 22:08:39.078337 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:39.078284 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-p9mdt" podStartSLOduration=4.008767766 podStartE2EDuration="22.078266738s" totalImagesPullingTime="18.069498972s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 22:08:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:08:39.077901089 +0000 UTC m=+22.842376705" watchObservedRunningTime="2026-09-04 22:08:39.078266738 +0000 UTC m=+22.842742351" Sep 04 22:08:39.148094 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:39.148047 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9gcxl" podStartSLOduration=11.4106819 podStartE2EDuration="22.148032338s" totalImagesPullingTime="10.737350438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:08:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:08:39.104448366 +0000 UTC m=+22.868923979" watchObservedRunningTime="2026-09-04 22:08:39.148032338 +0000 UTC m=+22.912507951" Sep 04 22:08:39.582319 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:39.582278 2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 04 22:08:39.705835 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:39.705632 2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-04T22:08:39.582300972Z","UUID":"f402b639-7dbc-4b46-a707-2f60f422d04d","Handler":null,"Name":"","Endpoint":""} Sep 04 22:08:39.707113 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:39.707088 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 04 22:08:39.707243 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:39.707124 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 04 22:08:39.967511 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:39.967424 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xks49" event={"ID":"6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef","Type":"ContainerStarted","Data":"7f7775231b990a95a21a398f1c849b3526d910d9d86d109ff0464dba9916c320"} Sep 04 22:08:39.969252 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:39.969218 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v7kt4" event={"ID":"76bd7339-f02c-4c10-af0a-af3010cfbc30","Type":"ContainerStarted","Data":"7ded730d5e9b31deeac1c4cbe0100b1c4a4a27c0a189e87ad2ad9fc536016818"} Sep 04 22:08:39.998288 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:39.998235 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-d5j8h" podStartSLOduration=3.937890575 podStartE2EDuration="21.998217415s" totalImagesPullingTime="18.06032684s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 22:08:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:08:39.147865135 +0000 UTC m=+22.912340748" watchObservedRunningTime="2026-09-04 22:08:39.998217415 +0000 UTC m=+23.762693027" Sep 04 22:08:39.998541 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:39.998513 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-v7kt4" podStartSLOduration=4.9288024230000005 podStartE2EDuration="22.998506012s" totalImagesPullingTime="18.069703589s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 22:08:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:08:39.998122047 +0000 UTC m=+23.762597661" watchObservedRunningTime="2026-09-04 22:08:39.998506012 +0000 UTC m=+23.762981624" Sep 04 22:08:40.709534 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:40.709508 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggpqf" Sep 04 22:08:40.709695 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:40.709513 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvwc7" Sep 04 22:08:40.709695 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:40.709619 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggpqf" podUID="6200aa24-5731-4d1a-865b-6c7c1cd218d1" Sep 04 22:08:40.709695 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:40.709681 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvwc7" podUID="c0972118-6d5c-4d3a-a979-497ab05ca339" Sep 04 22:08:40.709861 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:40.709514 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndzbl" Sep 04 22:08:40.709861 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:40.709768 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndzbl" podUID="62c2d50e-f517-47e6-8042-31b2455981fb" Sep 04 22:08:40.974302 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:40.974217 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xks49" event={"ID":"6a87f1c3-6ff0-4e6f-9a69-85f3ac7b93ef","Type":"ContainerStarted","Data":"ad9be3bc7bc1ba41578d0757033b7f60074cabdf38f4d8b01ef91c2ce0ed219b"} Sep 04 22:08:40.978337 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:40.978298 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gv845" event={"ID":"37958fa0-bfd6-402f-9c5c-faa146b7d886","Type":"ContainerStarted","Data":"b39c6b79ca55e6359a043074c730c0f28a720cac90c542e098ef04aeace3ee77"} Sep 04 22:08:41.000196 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:41.000156 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xks49" podStartSLOduration=2.726231041 podStartE2EDuration="23.000144779s" totalImagesPullingTime="20.273913738s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 22:08:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:08:40.999881537 +0000 UTC m=+24.764357149" watchObservedRunningTime="2026-09-04 22:08:41.000144779 +0000 UTC m=+24.764620392" Sep 04 22:08:41.808424 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:41.808393 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9gcxl" Sep 04 22:08:41.809006 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:41.808983 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9gcxl" Sep 04 22:08:41.823972 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:41.823944 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9gcxl" Sep 04 22:08:41.824447 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:41.824429 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9gcxl" Sep 04 22:08:42.709410 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:42.709369 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndzbl" Sep 04 22:08:42.709885 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:42.709375 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggpqf" Sep 04 22:08:42.709885 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:42.709378 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvwc7" Sep 04 22:08:42.709885 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:42.709576 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndzbl" podUID="62c2d50e-f517-47e6-8042-31b2455981fb" Sep 04 22:08:42.709885 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:42.709598 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggpqf" podUID="6200aa24-5731-4d1a-865b-6c7c1cd218d1" Sep 04 22:08:42.709885 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:42.709676 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvwc7" podUID="c0972118-6d5c-4d3a-a979-497ab05ca339" Sep 04 22:08:43.988963 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:43.988718 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gv845" event={"ID":"37958fa0-bfd6-402f-9c5c-faa146b7d886","Type":"ContainerStarted","Data":"56ca050f57722bc2544f76576da89212fbbe4b67e22ef1e2e95c49c6ca455120"} Sep 04 22:08:43.989498 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:43.989009 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:43.989498 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:43.989058 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:44.003772 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:44.003749 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:44.045444 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:44.045403 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gv845" podStartSLOduration=7.9236303679999995 podStartE2EDuration="26.045389309s" totalImagesPullingTime="18.121758941s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 22:08:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:08:44.042163166 +0000 UTC m=+27.806638792" watchObservedRunningTime="2026-09-04 22:08:44.045389309 +0000 UTC m=+27.809864922" Sep 04 22:08:44.709094 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:44.709070 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndzbl" Sep 04 22:08:44.709094 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:44.709085 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggpqf" Sep 04 22:08:44.709307 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:44.709085 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvwc7" Sep 04 22:08:44.709307 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:44.709162 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndzbl" podUID="62c2d50e-f517-47e6-8042-31b2455981fb" Sep 04 22:08:44.709307 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:44.709255 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggpqf" podUID="6200aa24-5731-4d1a-865b-6c7c1cd218d1" Sep 04 22:08:44.709413 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:44.709324 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvwc7" podUID="c0972118-6d5c-4d3a-a979-497ab05ca339" Sep 04 22:08:44.994120 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:44.994034 2574 generic.go:356] "Generic (PLEG): container finished" podID="95ade170-0175-41e0-b78b-ede111ef37af" containerID="c340f5e3e2a174ee20641603afd7a202441906b7081c74ef94a7a7af9ca07600" exitCode=0 Sep 04 22:08:44.994120 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:44.994099 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b49j5" event={"ID":"95ade170-0175-41e0-b78b-ede111ef37af","Type":"ContainerDied","Data":"c340f5e3e2a174ee20641603afd7a202441906b7081c74ef94a7a7af9ca07600"} Sep 04 22:08:44.994906 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:44.994744 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:45.015276 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:45.015249 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:08:46.026125 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:46.026099 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qvwc7"] Sep 04 22:08:46.026512 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:46.026226 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvwc7" Sep 04 22:08:46.026512 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:46.026360 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvwc7" podUID="c0972118-6d5c-4d3a-a979-497ab05ca339" Sep 04 22:08:46.028922 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:46.028892 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ggpqf"] Sep 04 22:08:46.029038 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:46.029022 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggpqf" Sep 04 22:08:46.029163 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:46.029133 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggpqf" podUID="6200aa24-5731-4d1a-865b-6c7c1cd218d1" Sep 04 22:08:46.029792 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:46.029767 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ndzbl"] Sep 04 22:08:46.029898 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:46.029888 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndzbl" Sep 04 22:08:46.030021 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:46.030000 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndzbl" podUID="62c2d50e-f517-47e6-8042-31b2455981fb" Sep 04 22:08:46.999340 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:46.999302 2574 generic.go:356] "Generic (PLEG): container finished" podID="95ade170-0175-41e0-b78b-ede111ef37af" containerID="4cd8436b95287dc0943aa07502eae27bb100711ac28b623e543cc2ef40ab9ffa" exitCode=0 Sep 04 22:08:46.999464 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:46.999393 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b49j5" event={"ID":"95ade170-0175-41e0-b78b-ede111ef37af","Type":"ContainerDied","Data":"4cd8436b95287dc0943aa07502eae27bb100711ac28b623e543cc2ef40ab9ffa"} Sep 04 22:08:47.709238 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:47.709207 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvwc7" Sep 04 22:08:47.709735 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:47.709246 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndzbl" Sep 04 22:08:47.709735 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:47.709209 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggpqf" Sep 04 22:08:47.709735 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:47.709328 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvwc7" podUID="c0972118-6d5c-4d3a-a979-497ab05ca339" Sep 04 22:08:47.709735 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:47.709433 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggpqf" podUID="6200aa24-5731-4d1a-865b-6c7c1cd218d1" Sep 04 22:08:47.709735 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:47.709516 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndzbl" podUID="62c2d50e-f517-47e6-8042-31b2455981fb" Sep 04 22:08:48.003852 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:48.003814 2574 generic.go:356] "Generic (PLEG): container finished" podID="95ade170-0175-41e0-b78b-ede111ef37af" containerID="f14682d6c572369c77301d7a54c1ce6eaee641034be61ba7db6e68163c464144" exitCode=0 Sep 04 22:08:48.004027 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:48.003873 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b49j5" event={"ID":"95ade170-0175-41e0-b78b-ede111ef37af","Type":"ContainerDied","Data":"f14682d6c572369c77301d7a54c1ce6eaee641034be61ba7db6e68163c464144"} Sep 04 22:08:49.709016 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:49.708930 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndzbl" Sep 04 22:08:49.709016 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:49.708967 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggpqf" Sep 04 22:08:49.709016 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:49.708973 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvwc7" Sep 04 22:08:49.709483 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:49.709058 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndzbl" podUID="62c2d50e-f517-47e6-8042-31b2455981fb" Sep 04 22:08:49.709483 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:49.709155 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggpqf" podUID="6200aa24-5731-4d1a-865b-6c7c1cd218d1" Sep 04 22:08:49.709483 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:49.709250 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qvwc7" podUID="c0972118-6d5c-4d3a-a979-497ab05ca339" Sep 04 22:08:50.510792 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:50.510754 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62c2d50e-f517-47e6-8042-31b2455981fb-original-pull-secret\") pod \"global-pull-secret-syncer-ndzbl\" (UID: \"62c2d50e-f517-47e6-8042-31b2455981fb\") " pod="kube-system/global-pull-secret-syncer-ndzbl" Sep 04 22:08:50.510963 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:50.510802 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0972118-6d5c-4d3a-a979-497ab05ca339-metrics-certs\") pod \"network-metrics-daemon-qvwc7\" (UID: \"c0972118-6d5c-4d3a-a979-497ab05ca339\") " pod="openshift-multus/network-metrics-daemon-qvwc7" Sep 04 22:08:50.510963 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:50.510859 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzgbw\" (UniqueName: \"kubernetes.io/projected/6200aa24-5731-4d1a-865b-6c7c1cd218d1-kube-api-access-mzgbw\") pod \"network-check-target-ggpqf\" (UID: \"6200aa24-5731-4d1a-865b-6c7c1cd218d1\") " pod="openshift-network-diagnostics/network-check-target-ggpqf" Sep 04 22:08:50.510963 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:50.510920 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 04 22:08:50.510963 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:50.510961 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 04 22:08:50.511143 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:50.510979 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 04 22:08:50.511143 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:50.510992 2574 projected.go:196] Error preparing data for projected volume kube-api-access-mzgbw for pod openshift-network-diagnostics/network-check-target-ggpqf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 04 22:08:50.511143 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:50.511000 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62c2d50e-f517-47e6-8042-31b2455981fb-original-pull-secret podName:62c2d50e-f517-47e6-8042-31b2455981fb nodeName:}" failed. No retries permitted until 2026-09-04 22:09:22.510975058 +0000 UTC m=+66.275450661 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62c2d50e-f517-47e6-8042-31b2455981fb-original-pull-secret") pod "global-pull-secret-syncer-ndzbl" (UID: "62c2d50e-f517-47e6-8042-31b2455981fb") : object "kube-system"/"original-pull-secret" not registered Sep 04 22:08:50.511143 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:50.511016 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 04 22:08:50.511143 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:50.511040 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6200aa24-5731-4d1a-865b-6c7c1cd218d1-kube-api-access-mzgbw podName:6200aa24-5731-4d1a-865b-6c7c1cd218d1 nodeName:}" failed. No retries permitted until 2026-09-04 22:09:22.511024057 +0000 UTC m=+66.275499654 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mzgbw" (UniqueName: "kubernetes.io/projected/6200aa24-5731-4d1a-865b-6c7c1cd218d1-kube-api-access-mzgbw") pod "network-check-target-ggpqf" (UID: "6200aa24-5731-4d1a-865b-6c7c1cd218d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 04 22:08:50.511143 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:50.511059 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0972118-6d5c-4d3a-a979-497ab05ca339-metrics-certs podName:c0972118-6d5c-4d3a-a979-497ab05ca339 nodeName:}" failed. No retries permitted until 2026-09-04 22:09:22.511046522 +0000 UTC m=+66.275522130 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0972118-6d5c-4d3a-a979-497ab05ca339-metrics-certs") pod "network-metrics-daemon-qvwc7" (UID: "c0972118-6d5c-4d3a-a979-497ab05ca339") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 04 22:08:51.049137 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.049106 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-219.ec2.internal" event="NodeReady" Sep 04 22:08:51.049649 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.049282 2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 04 22:08:51.109264 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.109231 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7bdccbfc7-xzs6l"] Sep 04 22:08:51.139097 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.139061 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7bdccbfc7-xzs6l"] Sep 04 22:08:51.139097 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.139095 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-c2jbn"] Sep 04 22:08:51.139311 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.139173 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bdccbfc7-xzs6l" Sep 04 22:08:51.143016 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.142992 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 04 22:08:51.143123 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.143028 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2bk2j\"" Sep 04 22:08:51.150021 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.150000 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 04 22:08:51.150260 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.150243 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 04 22:08:51.150345 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.150317 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 04 22:08:51.152946 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.152928 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9jzzc"] Sep 04 22:08:51.153076 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.153059 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-c2jbn" Sep 04 22:08:51.163535 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.163514 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 04 22:08:51.163698 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.163680 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 04 22:08:51.164532 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.164510 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-psbpz\"" Sep 04 22:08:51.170256 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.170236 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 04 22:08:51.173116 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.173098 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-c2jbn"] Sep 04 22:08:51.173203 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.173128 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9jzzc"] Sep 04 22:08:51.173263 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.173213 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9jzzc" Sep 04 22:08:51.178309 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.178289 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 04 22:08:51.178462 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.178337 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wcrtk\"" Sep 04 22:08:51.178462 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.178363 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 04 22:08:51.216247 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.216224 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-ca-trust-extracted\") pod \"image-registry-7bdccbfc7-xzs6l\" (UID: \"50ea3996-6bbf-41f2-854b-d99d1af4cf6c\") " pod="openshift-image-registry/image-registry-7bdccbfc7-xzs6l" Sep 04 22:08:51.216360 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.216259 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-installation-pull-secrets\") pod \"image-registry-7bdccbfc7-xzs6l\" (UID: \"50ea3996-6bbf-41f2-854b-d99d1af4cf6c\") " pod="openshift-image-registry/image-registry-7bdccbfc7-xzs6l" Sep 04 22:08:51.216360 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.216297 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-image-registry-private-configuration\") pod \"image-registry-7bdccbfc7-xzs6l\" (UID: \"50ea3996-6bbf-41f2-854b-d99d1af4cf6c\") " pod="openshift-image-registry/image-registry-7bdccbfc7-xzs6l" Sep 04 22:08:51.216479 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.216384 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-registry-certificates\") pod \"image-registry-7bdccbfc7-xzs6l\" (UID: \"50ea3996-6bbf-41f2-854b-d99d1af4cf6c\") " pod="openshift-image-registry/image-registry-7bdccbfc7-xzs6l" Sep 04 22:08:51.216479 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.216420 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n25sx\" (UniqueName: \"kubernetes.io/projected/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-kube-api-access-n25sx\") pod \"image-registry-7bdccbfc7-xzs6l\" (UID: \"50ea3996-6bbf-41f2-854b-d99d1af4cf6c\") " pod="openshift-image-registry/image-registry-7bdccbfc7-xzs6l" Sep 04 22:08:51.216479 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.216468 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-trusted-ca\") pod \"image-registry-7bdccbfc7-xzs6l\" (UID: \"50ea3996-6bbf-41f2-854b-d99d1af4cf6c\") " pod="openshift-image-registry/image-registry-7bdccbfc7-xzs6l" Sep 04 22:08:51.216620 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.216578 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-registry-tls\") pod \"image-registry-7bdccbfc7-xzs6l\" (UID: \"50ea3996-6bbf-41f2-854b-d99d1af4cf6c\") " pod="openshift-image-registry/image-registry-7bdccbfc7-xzs6l" Sep 04 22:08:51.216620 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.216614 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-bound-sa-token\") pod \"image-registry-7bdccbfc7-xzs6l\" (UID: \"50ea3996-6bbf-41f2-854b-d99d1af4cf6c\") " pod="openshift-image-registry/image-registry-7bdccbfc7-xzs6l" Sep 04 22:08:51.317263 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.317231 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-ca-trust-extracted\") pod \"image-registry-7bdccbfc7-xzs6l\" (UID: \"50ea3996-6bbf-41f2-854b-d99d1af4cf6c\") " pod="openshift-image-registry/image-registry-7bdccbfc7-xzs6l" Sep 04 22:08:51.317263 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.317264 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-installation-pull-secrets\") pod \"image-registry-7bdccbfc7-xzs6l\" (UID: \"50ea3996-6bbf-41f2-854b-d99d1af4cf6c\") " pod="openshift-image-registry/image-registry-7bdccbfc7-xzs6l" Sep 04 22:08:51.317490 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.317294 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-image-registry-private-configuration\") pod \"image-registry-7bdccbfc7-xzs6l\" (UID: \"50ea3996-6bbf-41f2-854b-d99d1af4cf6c\") " pod="openshift-image-registry/image-registry-7bdccbfc7-xzs6l" Sep 04 22:08:51.317490 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.317425 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/059b83a0-e9ee-4f9f-a3ae-250d85b3387d-metrics-tls\") pod \"dns-default-9jzzc\" (UID: \"059b83a0-e9ee-4f9f-a3ae-250d85b3387d\") " pod="openshift-dns/dns-default-9jzzc" Sep 04 22:08:51.317632 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.317490 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-registry-certificates\") pod \"image-registry-7bdccbfc7-xzs6l\" (UID: \"50ea3996-6bbf-41f2-854b-d99d1af4cf6c\") " pod="openshift-image-registry/image-registry-7bdccbfc7-xzs6l" Sep 04 22:08:51.317632 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.317537 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n25sx\" (UniqueName: \"kubernetes.io/projected/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-kube-api-access-n25sx\") pod \"image-registry-7bdccbfc7-xzs6l\" (UID: \"50ea3996-6bbf-41f2-854b-d99d1af4cf6c\") " pod="openshift-image-registry/image-registry-7bdccbfc7-xzs6l" Sep 04 22:08:51.317632 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.317596 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/059b83a0-e9ee-4f9f-a3ae-250d85b3387d-config-volume\") pod \"dns-default-9jzzc\" (UID: \"059b83a0-e9ee-4f9f-a3ae-250d85b3387d\") " pod="openshift-dns/dns-default-9jzzc" Sep 04 22:08:51.317632 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.317625 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d30375e-bff3-4f2c-9322-6e97a3c0e109-cert\") pod \"ingress-canary-c2jbn\" (UID: \"6d30375e-bff3-4f2c-9322-6e97a3c0e109\") " pod="openshift-ingress-canary/ingress-canary-c2jbn" Sep 04 22:08:51.317819 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.317658 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-trusted-ca\") pod \"image-registry-7bdccbfc7-xzs6l\" (UID: \"50ea3996-6bbf-41f2-854b-d99d1af4cf6c\") " pod="openshift-image-registry/image-registry-7bdccbfc7-xzs6l" Sep 04 22:08:51.317819 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.317715 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs8t6\" (UniqueName: \"kubernetes.io/projected/6d30375e-bff3-4f2c-9322-6e97a3c0e109-kube-api-access-vs8t6\") pod \"ingress-canary-c2jbn\" (UID: \"6d30375e-bff3-4f2c-9322-6e97a3c0e109\") " pod="openshift-ingress-canary/ingress-canary-c2jbn" Sep 04 22:08:51.317819 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.317741 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pvwr\" (UniqueName: \"kubernetes.io/projected/059b83a0-e9ee-4f9f-a3ae-250d85b3387d-kube-api-access-2pvwr\") pod \"dns-default-9jzzc\" (UID: \"059b83a0-e9ee-4f9f-a3ae-250d85b3387d\") " pod="openshift-dns/dns-default-9jzzc" Sep 04 22:08:51.317975 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.317767 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-registry-tls\") pod \"image-registry-7bdccbfc7-xzs6l\" (UID: \"50ea3996-6bbf-41f2-854b-d99d1af4cf6c\") " pod="openshift-image-registry/image-registry-7bdccbfc7-xzs6l" Sep 04 22:08:51.318087 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.317987 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/059b83a0-e9ee-4f9f-a3ae-250d85b3387d-tmp-dir\") pod \"dns-default-9jzzc\" (UID: \"059b83a0-e9ee-4f9f-a3ae-250d85b3387d\") " pod="openshift-dns/dns-default-9jzzc" Sep 04 22:08:51.318087 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.318010 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-bound-sa-token\") pod \"image-registry-7bdccbfc7-xzs6l\" (UID: \"50ea3996-6bbf-41f2-854b-d99d1af4cf6c\") " pod="openshift-image-registry/image-registry-7bdccbfc7-xzs6l" Sep 04 22:08:51.318087 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.318080 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-ca-trust-extracted\") pod \"image-registry-7bdccbfc7-xzs6l\" (UID: \"50ea3996-6bbf-41f2-854b-d99d1af4cf6c\") " pod="openshift-image-registry/image-registry-7bdccbfc7-xzs6l" Sep 04 22:08:51.318254 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:51.317953 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 04 22:08:51.318254 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:51.318113 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bdccbfc7-xzs6l: secret "image-registry-tls" not found Sep 04 22:08:51.318254 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:51.318187 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-registry-tls podName:50ea3996-6bbf-41f2-854b-d99d1af4cf6c nodeName:}" failed. No retries permitted until 2026-09-04 22:08:51.818169795 +0000 UTC m=+35.582645391 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-registry-tls") pod "image-registry-7bdccbfc7-xzs6l" (UID: "50ea3996-6bbf-41f2-854b-d99d1af4cf6c") : secret "image-registry-tls" not found Sep 04 22:08:51.318254 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.318221 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-registry-certificates\") pod \"image-registry-7bdccbfc7-xzs6l\" (UID: \"50ea3996-6bbf-41f2-854b-d99d1af4cf6c\") " pod="openshift-image-registry/image-registry-7bdccbfc7-xzs6l" Sep 04 22:08:51.318820 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.318798 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-trusted-ca\") pod \"image-registry-7bdccbfc7-xzs6l\" (UID: \"50ea3996-6bbf-41f2-854b-d99d1af4cf6c\") " pod="openshift-image-registry/image-registry-7bdccbfc7-xzs6l" Sep 04 22:08:51.321815 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.321797 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-installation-pull-secrets\") pod \"image-registry-7bdccbfc7-xzs6l\" (UID: \"50ea3996-6bbf-41f2-854b-d99d1af4cf6c\") " pod="openshift-image-registry/image-registry-7bdccbfc7-xzs6l" Sep 04 22:08:51.321912 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.321802 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-image-registry-private-configuration\") pod \"image-registry-7bdccbfc7-xzs6l\" (UID: \"50ea3996-6bbf-41f2-854b-d99d1af4cf6c\") " pod="openshift-image-registry/image-registry-7bdccbfc7-xzs6l" Sep 04 22:08:51.342000 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.341981 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-bound-sa-token\") pod \"image-registry-7bdccbfc7-xzs6l\" (UID: \"50ea3996-6bbf-41f2-854b-d99d1af4cf6c\") " pod="openshift-image-registry/image-registry-7bdccbfc7-xzs6l" Sep 04 22:08:51.342439 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.342418 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n25sx\" (UniqueName: \"kubernetes.io/projected/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-kube-api-access-n25sx\") pod \"image-registry-7bdccbfc7-xzs6l\" (UID: \"50ea3996-6bbf-41f2-854b-d99d1af4cf6c\") " pod="openshift-image-registry/image-registry-7bdccbfc7-xzs6l" Sep 04 22:08:51.419423 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.419345 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs8t6\" (UniqueName: \"kubernetes.io/projected/6d30375e-bff3-4f2c-9322-6e97a3c0e109-kube-api-access-vs8t6\") pod \"ingress-canary-c2jbn\" (UID: \"6d30375e-bff3-4f2c-9322-6e97a3c0e109\") " pod="openshift-ingress-canary/ingress-canary-c2jbn" Sep 04 22:08:51.419423 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.419388 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pvwr\" (UniqueName: \"kubernetes.io/projected/059b83a0-e9ee-4f9f-a3ae-250d85b3387d-kube-api-access-2pvwr\") pod \"dns-default-9jzzc\" (UID: \"059b83a0-e9ee-4f9f-a3ae-250d85b3387d\") " pod="openshift-dns/dns-default-9jzzc" Sep 04 22:08:51.419423 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.419427 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/059b83a0-e9ee-4f9f-a3ae-250d85b3387d-tmp-dir\") pod \"dns-default-9jzzc\" (UID: \"059b83a0-e9ee-4f9f-a3ae-250d85b3387d\") " pod="openshift-dns/dns-default-9jzzc" Sep 04 22:08:51.419709 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.419479 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/059b83a0-e9ee-4f9f-a3ae-250d85b3387d-metrics-tls\") pod \"dns-default-9jzzc\" (UID: \"059b83a0-e9ee-4f9f-a3ae-250d85b3387d\") " pod="openshift-dns/dns-default-9jzzc" Sep 04 22:08:51.419709 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:51.419590 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 04 22:08:51.419709 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.419637 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/059b83a0-e9ee-4f9f-a3ae-250d85b3387d-config-volume\") pod \"dns-default-9jzzc\" (UID: \"059b83a0-e9ee-4f9f-a3ae-250d85b3387d\") " pod="openshift-dns/dns-default-9jzzc" Sep 04 22:08:51.419709 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:51.419655 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/059b83a0-e9ee-4f9f-a3ae-250d85b3387d-metrics-tls podName:059b83a0-e9ee-4f9f-a3ae-250d85b3387d nodeName:}" failed. No retries permitted until 2026-09-04 22:08:51.919636847 +0000 UTC m=+35.684112455 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/059b83a0-e9ee-4f9f-a3ae-250d85b3387d-metrics-tls") pod "dns-default-9jzzc" (UID: "059b83a0-e9ee-4f9f-a3ae-250d85b3387d") : secret "dns-default-metrics-tls" not found Sep 04 22:08:51.419918 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.419781 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/059b83a0-e9ee-4f9f-a3ae-250d85b3387d-tmp-dir\") pod \"dns-default-9jzzc\" (UID: \"059b83a0-e9ee-4f9f-a3ae-250d85b3387d\") " pod="openshift-dns/dns-default-9jzzc" Sep 04 22:08:51.419918 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.419800 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d30375e-bff3-4f2c-9322-6e97a3c0e109-cert\") pod \"ingress-canary-c2jbn\" (UID: \"6d30375e-bff3-4f2c-9322-6e97a3c0e109\") " pod="openshift-ingress-canary/ingress-canary-c2jbn" Sep 04 22:08:51.420210 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.420181 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/059b83a0-e9ee-4f9f-a3ae-250d85b3387d-config-volume\") pod \"dns-default-9jzzc\" (UID: \"059b83a0-e9ee-4f9f-a3ae-250d85b3387d\") " pod="openshift-dns/dns-default-9jzzc" Sep 04 22:08:51.422500 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.422480 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6d30375e-bff3-4f2c-9322-6e97a3c0e109-cert\") pod \"ingress-canary-c2jbn\" (UID: \"6d30375e-bff3-4f2c-9322-6e97a3c0e109\") " pod="openshift-ingress-canary/ingress-canary-c2jbn" Sep 04 22:08:51.435179 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.435160 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs8t6\" (UniqueName: \"kubernetes.io/projected/6d30375e-bff3-4f2c-9322-6e97a3c0e109-kube-api-access-vs8t6\") pod \"ingress-canary-c2jbn\" (UID: \"6d30375e-bff3-4f2c-9322-6e97a3c0e109\") " pod="openshift-ingress-canary/ingress-canary-c2jbn" Sep 04 22:08:51.436832 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.436804 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pvwr\" (UniqueName: \"kubernetes.io/projected/059b83a0-e9ee-4f9f-a3ae-250d85b3387d-kube-api-access-2pvwr\") pod \"dns-default-9jzzc\" (UID: \"059b83a0-e9ee-4f9f-a3ae-250d85b3387d\") " pod="openshift-dns/dns-default-9jzzc" Sep 04 22:08:51.461923 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.461901 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-c2jbn" Sep 04 22:08:51.557060 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.557026 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gcxl_1373f05f-60fa-4414-a0da-93106632c578/konnectivity-agent/0.log" limit=1024 Sep 04 22:08:51.708909 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.708793 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggpqf" Sep 04 22:08:51.708909 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.708902 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndzbl" Sep 04 22:08:51.709130 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.708927 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvwc7" Sep 04 22:08:51.713539 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.713508 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 04 22:08:51.713683 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.713644 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5pc9f\"" Sep 04 22:08:51.713978 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.713939 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 04 22:08:51.714474 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.714458 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 04 22:08:51.714924 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.714903 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 04 22:08:51.718153 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.718136 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-c64pf\"" Sep 04 22:08:51.749409 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.749382 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gcxl_1373f05f-60fa-4414-a0da-93106632c578/konnectivity-agent/0.log" limit=1024 Sep 04 22:08:51.823845 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.823808 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-registry-tls\") pod \"image-registry-7bdccbfc7-xzs6l\" (UID: \"50ea3996-6bbf-41f2-854b-d99d1af4cf6c\") " pod="openshift-image-registry/image-registry-7bdccbfc7-xzs6l" Sep 04 22:08:51.824021 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:51.823952 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 04 22:08:51.824021 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:51.823972 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bdccbfc7-xzs6l: secret "image-registry-tls" not found Sep 04 22:08:51.824105 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:51.824041 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-registry-tls podName:50ea3996-6bbf-41f2-854b-d99d1af4cf6c nodeName:}" failed. No retries permitted until 2026-09-04 22:08:52.824022119 +0000 UTC m=+36.588497724 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-registry-tls") pod "image-registry-7bdccbfc7-xzs6l" (UID: "50ea3996-6bbf-41f2-854b-d99d1af4cf6c") : secret "image-registry-tls" not found Sep 04 22:08:51.925781 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:51.925738 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/059b83a0-e9ee-4f9f-a3ae-250d85b3387d-metrics-tls\") pod \"dns-default-9jzzc\" (UID: \"059b83a0-e9ee-4f9f-a3ae-250d85b3387d\") " pod="openshift-dns/dns-default-9jzzc" Sep 04 22:08:51.928291 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:51.928202 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 04 22:08:51.928444 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:51.928344 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/059b83a0-e9ee-4f9f-a3ae-250d85b3387d-metrics-tls podName:059b83a0-e9ee-4f9f-a3ae-250d85b3387d nodeName:}" failed. No retries permitted until 2026-09-04 22:08:52.928269299 +0000 UTC m=+36.692744904 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/059b83a0-e9ee-4f9f-a3ae-250d85b3387d-metrics-tls") pod "dns-default-9jzzc" (UID: "059b83a0-e9ee-4f9f-a3ae-250d85b3387d") : secret "dns-default-metrics-tls" not found Sep 04 22:08:52.581119 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:52.581018 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-c2jbn"] Sep 04 22:08:52.586393 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:08:52.586368 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d30375e_bff3_4f2c_9322_6e97a3c0e109.slice/crio-5bd416d12ada88b07bb324355fd0c59ac98e3843c078264a37cc126ba3087d9b WatchSource:0}: Error finding container 5bd416d12ada88b07bb324355fd0c59ac98e3843c078264a37cc126ba3087d9b: Status 404 returned error can't find the container with id 5bd416d12ada88b07bb324355fd0c59ac98e3843c078264a37cc126ba3087d9b Sep 04 22:08:52.832865 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:52.832796 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-registry-tls\") pod \"image-registry-7bdccbfc7-xzs6l\" (UID: \"50ea3996-6bbf-41f2-854b-d99d1af4cf6c\") " pod="openshift-image-registry/image-registry-7bdccbfc7-xzs6l" Sep 04 22:08:52.832966 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:52.832945 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 04 22:08:52.833008 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:52.832968 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bdccbfc7-xzs6l: secret "image-registry-tls" not found Sep 04 22:08:52.833043 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:52.833038 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-registry-tls podName:50ea3996-6bbf-41f2-854b-d99d1af4cf6c nodeName:}" failed. No retries permitted until 2026-09-04 22:08:54.833023544 +0000 UTC m=+38.597499135 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-registry-tls") pod "image-registry-7bdccbfc7-xzs6l" (UID: "50ea3996-6bbf-41f2-854b-d99d1af4cf6c") : secret "image-registry-tls" not found Sep 04 22:08:52.933944 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:52.933911 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/059b83a0-e9ee-4f9f-a3ae-250d85b3387d-metrics-tls\") pod \"dns-default-9jzzc\" (UID: \"059b83a0-e9ee-4f9f-a3ae-250d85b3387d\") " pod="openshift-dns/dns-default-9jzzc" Sep 04 22:08:52.934068 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:52.934033 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 04 22:08:52.934116 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:52.934082 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/059b83a0-e9ee-4f9f-a3ae-250d85b3387d-metrics-tls podName:059b83a0-e9ee-4f9f-a3ae-250d85b3387d nodeName:}" failed. No retries permitted until 2026-09-04 22:08:54.934070315 +0000 UTC m=+38.698545905 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/059b83a0-e9ee-4f9f-a3ae-250d85b3387d-metrics-tls") pod "dns-default-9jzzc" (UID: "059b83a0-e9ee-4f9f-a3ae-250d85b3387d") : secret "dns-default-metrics-tls" not found Sep 04 22:08:53.017772 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:53.017740 2574 generic.go:356] "Generic (PLEG): container finished" podID="95ade170-0175-41e0-b78b-ede111ef37af" containerID="b4cf378be86582e66229070b46069e9b59d63b31814552c0e64136dc474893d9" exitCode=0 Sep 04 22:08:53.017917 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:53.017819 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b49j5" event={"ID":"95ade170-0175-41e0-b78b-ede111ef37af","Type":"ContainerDied","Data":"b4cf378be86582e66229070b46069e9b59d63b31814552c0e64136dc474893d9"} Sep 04 22:08:53.019089 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:53.019069 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-c2jbn" event={"ID":"6d30375e-bff3-4f2c-9322-6e97a3c0e109","Type":"ContainerStarted","Data":"5bd416d12ada88b07bb324355fd0c59ac98e3843c078264a37cc126ba3087d9b"} Sep 04 22:08:53.269204 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:53.269172 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gcxl_1373f05f-60fa-4414-a0da-93106632c578/konnectivity-agent/0.log" limit=1024 Sep 04 22:08:53.387183 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:53.387112 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gcxl_1373f05f-60fa-4414-a0da-93106632c578/konnectivity-agent/0.log" limit=1024 Sep 04 22:08:54.025683 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:54.025642 2574 generic.go:356] "Generic (PLEG): container finished" podID="95ade170-0175-41e0-b78b-ede111ef37af" containerID="6dabe3d17b47a93d7f504ac7e48d6952d2bc40bb83843cc1cda56ee1d2d7e91d" exitCode=0 Sep 04 22:08:54.026078 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:54.025699 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b49j5" event={"ID":"95ade170-0175-41e0-b78b-ede111ef37af","Type":"ContainerDied","Data":"6dabe3d17b47a93d7f504ac7e48d6952d2bc40bb83843cc1cda56ee1d2d7e91d"} Sep 04 22:08:54.850343 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:54.850306 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-registry-tls\") pod \"image-registry-7bdccbfc7-xzs6l\" (UID: \"50ea3996-6bbf-41f2-854b-d99d1af4cf6c\") " pod="openshift-image-registry/image-registry-7bdccbfc7-xzs6l" Sep 04 22:08:54.850527 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:54.850503 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 04 22:08:54.850588 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:54.850532 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bdccbfc7-xzs6l: secret "image-registry-tls" not found Sep 04 22:08:54.850648 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:54.850615 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-registry-tls podName:50ea3996-6bbf-41f2-854b-d99d1af4cf6c nodeName:}" failed. No retries permitted until 2026-09-04 22:08:58.85059559 +0000 UTC m=+42.615071181 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-registry-tls") pod "image-registry-7bdccbfc7-xzs6l" (UID: "50ea3996-6bbf-41f2-854b-d99d1af4cf6c") : secret "image-registry-tls" not found Sep 04 22:08:54.950964 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:54.950927 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/059b83a0-e9ee-4f9f-a3ae-250d85b3387d-metrics-tls\") pod \"dns-default-9jzzc\" (UID: \"059b83a0-e9ee-4f9f-a3ae-250d85b3387d\") " pod="openshift-dns/dns-default-9jzzc" Sep 04 22:08:54.951116 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:54.951097 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 04 22:08:54.951192 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:54.951175 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/059b83a0-e9ee-4f9f-a3ae-250d85b3387d-metrics-tls podName:059b83a0-e9ee-4f9f-a3ae-250d85b3387d nodeName:}" failed. No retries permitted until 2026-09-04 22:08:58.951154435 +0000 UTC m=+42.715630026 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/059b83a0-e9ee-4f9f-a3ae-250d85b3387d-metrics-tls") pod "dns-default-9jzzc" (UID: "059b83a0-e9ee-4f9f-a3ae-250d85b3387d") : secret "dns-default-metrics-tls" not found Sep 04 22:08:55.032241 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:55.032203 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b49j5" event={"ID":"95ade170-0175-41e0-b78b-ede111ef37af","Type":"ContainerStarted","Data":"938c1447ddebe0f115a655ae1ac98db778e8fd60f461c4110d4d560f08491823"} Sep 04 22:08:55.061012 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:55.060953 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-b49j5" podStartSLOduration=8.596344517 podStartE2EDuration="38.060910251s" totalImagesPullingTime="29.464565734s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 22:08:17 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:08:55.059522202 +0000 UTC m=+38.823997815" watchObservedRunningTime="2026-09-04 22:08:55.060910251 +0000 UTC m=+38.825385865" Sep 04 22:08:55.208460 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:55.208342 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gcxl_1373f05f-60fa-4414-a0da-93106632c578/konnectivity-agent/0.log" limit=1024 Sep 04 22:08:55.328811 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:55.328770 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gcxl_1373f05f-60fa-4414-a0da-93106632c578/konnectivity-agent/0.log" limit=1024 Sep 04 22:08:56.035529 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:56.035492 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-c2jbn" event={"ID":"6d30375e-bff3-4f2c-9322-6e97a3c0e109","Type":"ContainerStarted","Data":"631399c774180a41795a857afefb028ae3bc99c2a853194615d37f95a5430ad5"} Sep 04 22:08:56.056715 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:56.056668 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-c2jbn" podStartSLOduration=1.929080668 podStartE2EDuration="5.056653071s" totalImagesPullingTime="3.127572403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:08:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:08:56.056439621 +0000 UTC m=+39.820915235" watchObservedRunningTime="2026-09-04 22:08:56.056653071 +0000 UTC m=+39.821128684" Sep 04 22:08:56.540987 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:56.540955 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gcxl_1373f05f-60fa-4414-a0da-93106632c578/konnectivity-agent/0.log" limit=1024 Sep 04 22:08:56.622236 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:56.622204 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gcxl_1373f05f-60fa-4414-a0da-93106632c578/konnectivity-agent/0.log" limit=1024 Sep 04 22:08:58.878616 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:58.878570 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-registry-tls\") pod \"image-registry-7bdccbfc7-xzs6l\" (UID: \"50ea3996-6bbf-41f2-854b-d99d1af4cf6c\") " pod="openshift-image-registry/image-registry-7bdccbfc7-xzs6l" Sep 04 22:08:58.879009 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:58.878716 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 04 22:08:58.879009 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:58.878740 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bdccbfc7-xzs6l: secret "image-registry-tls" not found Sep 04 22:08:58.879009 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:58.878796 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-registry-tls podName:50ea3996-6bbf-41f2-854b-d99d1af4cf6c nodeName:}" failed. No retries permitted until 2026-09-04 22:09:06.878779512 +0000 UTC m=+50.643255103 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-registry-tls") pod "image-registry-7bdccbfc7-xzs6l" (UID: "50ea3996-6bbf-41f2-854b-d99d1af4cf6c") : secret "image-registry-tls" not found Sep 04 22:08:58.979814 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:08:58.979779 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/059b83a0-e9ee-4f9f-a3ae-250d85b3387d-metrics-tls\") pod \"dns-default-9jzzc\" (UID: \"059b83a0-e9ee-4f9f-a3ae-250d85b3387d\") " pod="openshift-dns/dns-default-9jzzc" Sep 04 22:08:58.979977 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:58.979896 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 04 22:08:58.979977 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:08:58.979944 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/059b83a0-e9ee-4f9f-a3ae-250d85b3387d-metrics-tls podName:059b83a0-e9ee-4f9f-a3ae-250d85b3387d nodeName:}" failed. No retries permitted until 2026-09-04 22:09:06.979931664 +0000 UTC m=+50.744407255 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/059b83a0-e9ee-4f9f-a3ae-250d85b3387d-metrics-tls") pod "dns-default-9jzzc" (UID: "059b83a0-e9ee-4f9f-a3ae-250d85b3387d") : secret "dns-default-metrics-tls" not found Sep 04 22:09:06.932132 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:09:06.932055 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-registry-tls\") pod \"image-registry-7bdccbfc7-xzs6l\" (UID: \"50ea3996-6bbf-41f2-854b-d99d1af4cf6c\") " pod="openshift-image-registry/image-registry-7bdccbfc7-xzs6l" Sep 04 22:09:06.932520 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:09:06.932206 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 04 22:09:06.932520 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:09:06.932220 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bdccbfc7-xzs6l: secret "image-registry-tls" not found Sep 04 22:09:06.932520 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:09:06.932286 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-registry-tls podName:50ea3996-6bbf-41f2-854b-d99d1af4cf6c nodeName:}" failed. No retries permitted until 2026-09-04 22:09:22.93226834 +0000 UTC m=+66.696743939 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-registry-tls") pod "image-registry-7bdccbfc7-xzs6l" (UID: "50ea3996-6bbf-41f2-854b-d99d1af4cf6c") : secret "image-registry-tls" not found Sep 04 22:09:07.033352 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:09:07.033322 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/059b83a0-e9ee-4f9f-a3ae-250d85b3387d-metrics-tls\") pod \"dns-default-9jzzc\" (UID: \"059b83a0-e9ee-4f9f-a3ae-250d85b3387d\") " pod="openshift-dns/dns-default-9jzzc" Sep 04 22:09:07.033492 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:09:07.033472 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 04 22:09:07.033569 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:09:07.033543 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/059b83a0-e9ee-4f9f-a3ae-250d85b3387d-metrics-tls podName:059b83a0-e9ee-4f9f-a3ae-250d85b3387d nodeName:}" failed. No retries permitted until 2026-09-04 22:09:23.033527474 +0000 UTC m=+66.798003065 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/059b83a0-e9ee-4f9f-a3ae-250d85b3387d-metrics-tls") pod "dns-default-9jzzc" (UID: "059b83a0-e9ee-4f9f-a3ae-250d85b3387d") : secret "dns-default-metrics-tls" not found Sep 04 22:09:17.009183 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:09:17.009155 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gv845" Sep 04 22:09:22.533315 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:09:22.533278 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzgbw\" (UniqueName: \"kubernetes.io/projected/6200aa24-5731-4d1a-865b-6c7c1cd218d1-kube-api-access-mzgbw\") pod \"network-check-target-ggpqf\" (UID: \"6200aa24-5731-4d1a-865b-6c7c1cd218d1\") " pod="openshift-network-diagnostics/network-check-target-ggpqf" Sep 04 22:09:22.533315 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:09:22.533325 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62c2d50e-f517-47e6-8042-31b2455981fb-original-pull-secret\") pod \"global-pull-secret-syncer-ndzbl\" (UID: \"62c2d50e-f517-47e6-8042-31b2455981fb\") " pod="kube-system/global-pull-secret-syncer-ndzbl" Sep 04 22:09:22.533838 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:09:22.533342 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0972118-6d5c-4d3a-a979-497ab05ca339-metrics-certs\") pod \"network-metrics-daemon-qvwc7\" (UID: \"c0972118-6d5c-4d3a-a979-497ab05ca339\") " pod="openshift-multus/network-metrics-daemon-qvwc7" Sep 04 22:09:22.543277 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:09:22.543252 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 04 22:09:22.543370 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:09:22.543302 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 04 22:09:22.543370 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:09:22.543265 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 04 22:09:22.543500 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:09:22.543486 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 04 22:09:22.543543 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:09:22.543534 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0972118-6d5c-4d3a-a979-497ab05ca339-metrics-certs podName:c0972118-6d5c-4d3a-a979-497ab05ca339 nodeName:}" failed. No retries permitted until 2026-09-04 22:10:26.54352054 +0000 UTC m=+130.307996131 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0972118-6d5c-4d3a-a979-497ab05ca339-metrics-certs") pod "network-metrics-daemon-qvwc7" (UID: "c0972118-6d5c-4d3a-a979-497ab05ca339") : secret "metrics-daemon-secret" not found Sep 04 22:09:22.546935 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:09:22.546911 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62c2d50e-f517-47e6-8042-31b2455981fb-original-pull-secret\") pod \"global-pull-secret-syncer-ndzbl\" (UID: \"62c2d50e-f517-47e6-8042-31b2455981fb\") " pod="kube-system/global-pull-secret-syncer-ndzbl" Sep 04 22:09:22.547643 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:09:22.547629 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 04 22:09:22.555869 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:09:22.555852 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzgbw\" (UniqueName: \"kubernetes.io/projected/6200aa24-5731-4d1a-865b-6c7c1cd218d1-kube-api-access-mzgbw\") pod \"network-check-target-ggpqf\" (UID: \"6200aa24-5731-4d1a-865b-6c7c1cd218d1\") " pod="openshift-network-diagnostics/network-check-target-ggpqf" Sep 04 22:09:22.623386 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:09:22.623365 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5pc9f\"" Sep 04 22:09:22.627598 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:09:22.627580 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndzbl" Sep 04 22:09:22.631151 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:09:22.631129 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggpqf" Sep 04 22:09:22.906970 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:09:22.906886 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ggpqf"] Sep 04 22:09:22.910032 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:09:22.910004 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ndzbl"] Sep 04 22:09:22.911466 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:09:22.911445 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6200aa24_5731_4d1a_865b_6c7c1cd218d1.slice/crio-53ce7e29b89244c0d745359bbac84a56dd012d32f0b09500add6038673257f3e WatchSource:0}: Error finding container 53ce7e29b89244c0d745359bbac84a56dd012d32f0b09500add6038673257f3e: Status 404 returned error can't find the container with id 53ce7e29b89244c0d745359bbac84a56dd012d32f0b09500add6038673257f3e Sep 04 22:09:22.913520 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:09:22.913492 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62c2d50e_f517_47e6_8042_31b2455981fb.slice/crio-79c97bc3f13e4f74ce547e6ff466c23f151653309e6b06398a13b4c7eef70ef2 WatchSource:0}: Error finding container 79c97bc3f13e4f74ce547e6ff466c23f151653309e6b06398a13b4c7eef70ef2: Status 404 returned error can't find the container with id 79c97bc3f13e4f74ce547e6ff466c23f151653309e6b06398a13b4c7eef70ef2 Sep 04 22:09:22.935928 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:09:22.935900 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-registry-tls\") pod \"image-registry-7bdccbfc7-xzs6l\" (UID: \"50ea3996-6bbf-41f2-854b-d99d1af4cf6c\") " pod="openshift-image-registry/image-registry-7bdccbfc7-xzs6l" Sep 04 22:09:22.936019 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:09:22.935991 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 04 22:09:22.936019 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:09:22.936001 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bdccbfc7-xzs6l: secret "image-registry-tls" not found Sep 04 22:09:22.936098 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:09:22.936042 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-registry-tls podName:50ea3996-6bbf-41f2-854b-d99d1af4cf6c nodeName:}" failed. No retries permitted until 2026-09-04 22:09:54.936029511 +0000 UTC m=+98.700505102 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-registry-tls") pod "image-registry-7bdccbfc7-xzs6l" (UID: "50ea3996-6bbf-41f2-854b-d99d1af4cf6c") : secret "image-registry-tls" not found Sep 04 22:09:23.036397 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:09:23.036363 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/059b83a0-e9ee-4f9f-a3ae-250d85b3387d-metrics-tls\") pod \"dns-default-9jzzc\" (UID: \"059b83a0-e9ee-4f9f-a3ae-250d85b3387d\") " pod="openshift-dns/dns-default-9jzzc" Sep 04 22:09:23.036902 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:09:23.036869 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 04 22:09:23.037028 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:09:23.036973 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/059b83a0-e9ee-4f9f-a3ae-250d85b3387d-metrics-tls podName:059b83a0-e9ee-4f9f-a3ae-250d85b3387d nodeName:}" failed. No retries permitted until 2026-09-04 22:09:55.036949399 +0000 UTC m=+98.801425003 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/059b83a0-e9ee-4f9f-a3ae-250d85b3387d-metrics-tls") pod "dns-default-9jzzc" (UID: "059b83a0-e9ee-4f9f-a3ae-250d85b3387d") : secret "dns-default-metrics-tls" not found Sep 04 22:09:23.101539 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:09:23.101506 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ggpqf" event={"ID":"6200aa24-5731-4d1a-865b-6c7c1cd218d1","Type":"ContainerStarted","Data":"53ce7e29b89244c0d745359bbac84a56dd012d32f0b09500add6038673257f3e"} Sep 04 22:09:23.102533 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:09:23.102511 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ndzbl" event={"ID":"62c2d50e-f517-47e6-8042-31b2455981fb","Type":"ContainerStarted","Data":"79c97bc3f13e4f74ce547e6ff466c23f151653309e6b06398a13b4c7eef70ef2"} Sep 04 22:09:28.119089 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:09:28.119000 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ggpqf" event={"ID":"6200aa24-5731-4d1a-865b-6c7c1cd218d1","Type":"ContainerStarted","Data":"9ab5d00c57f45e3ca144027397c172ede5026f0857e9b16e315b7088c6837b16"} Sep 04 22:09:28.119523 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:09:28.119121 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ggpqf" Sep 04 22:09:28.120445 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:09:28.120419 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ndzbl" event={"ID":"62c2d50e-f517-47e6-8042-31b2455981fb","Type":"ContainerStarted","Data":"79473082f5a78d19f53534d058cb0b875de6f22dd44e9c724cbad62dfa627333"} Sep 04 22:09:28.135328 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:09:28.135276 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ggpqf" podStartSLOduration=66.280559862 podStartE2EDuration="1m11.135264104s" totalImagesPullingTime="4.854704242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:08:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:09:28.135080529 +0000 UTC m=+71.899556141" watchObservedRunningTime="2026-09-04 22:09:28.135264104 +0000 UTC m=+71.899739716" Sep 04 22:09:28.151316 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:09:28.151278 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-ndzbl" podStartSLOduration=65.286151971 podStartE2EDuration="1m10.151265551s" totalImagesPullingTime="4.86511358s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 22:08:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:09:28.15106907 +0000 UTC m=+71.915544683" watchObservedRunningTime="2026-09-04 22:09:28.151265551 +0000 UTC m=+71.915741164" Sep 04 22:09:54.949144 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:09:54.949016 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-registry-tls\") pod \"image-registry-7bdccbfc7-xzs6l\" (UID: \"50ea3996-6bbf-41f2-854b-d99d1af4cf6c\") " pod="openshift-image-registry/image-registry-7bdccbfc7-xzs6l" Sep 04 22:09:54.949144 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:09:54.949135 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 04 22:09:54.949144 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:09:54.949147 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bdccbfc7-xzs6l: secret "image-registry-tls" not found Sep 04 22:09:54.949695 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:09:54.949207 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-registry-tls podName:50ea3996-6bbf-41f2-854b-d99d1af4cf6c nodeName:}" failed. No retries permitted until 2026-09-04 22:10:58.949191107 +0000 UTC m=+162.713666701 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-registry-tls") pod "image-registry-7bdccbfc7-xzs6l" (UID: "50ea3996-6bbf-41f2-854b-d99d1af4cf6c") : secret "image-registry-tls" not found Sep 04 22:09:55.050202 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:09:55.050170 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/059b83a0-e9ee-4f9f-a3ae-250d85b3387d-metrics-tls\") pod \"dns-default-9jzzc\" (UID: \"059b83a0-e9ee-4f9f-a3ae-250d85b3387d\") " pod="openshift-dns/dns-default-9jzzc" Sep 04 22:09:55.050359 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:09:55.050291 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 04 22:09:55.050359 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:09:55.050350 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/059b83a0-e9ee-4f9f-a3ae-250d85b3387d-metrics-tls podName:059b83a0-e9ee-4f9f-a3ae-250d85b3387d nodeName:}" failed. No retries permitted until 2026-09-04 22:10:59.050336613 +0000 UTC m=+162.814812204 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/059b83a0-e9ee-4f9f-a3ae-250d85b3387d-metrics-tls") pod "dns-default-9jzzc" (UID: "059b83a0-e9ee-4f9f-a3ae-250d85b3387d") : secret "dns-default-metrics-tls" not found Sep 04 22:09:59.125504 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:09:59.125469 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ggpqf" Sep 04 22:10:00.379657 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:00.379618 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-599484b46b-xdvdt"] Sep 04 22:10:00.402494 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:00.402467 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-599484b46b-xdvdt"] Sep 04 22:10:00.402673 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:00.402603 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-599484b46b-xdvdt" Sep 04 22:10:00.405158 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:00.405137 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 04 22:10:00.405415 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:00.405397 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 04 22:10:00.405609 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:00.405583 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 04 22:10:00.405727 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:00.405678 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 04 22:10:00.405795 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:00.405749 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 04 22:10:00.405851 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:00.405797 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 04 22:10:00.406118 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:00.406098 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-z77tv\"" Sep 04 22:10:00.483396 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:00.483364 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-service-ca-bundle\") pod \"router-default-599484b46b-xdvdt\" (UID: \"de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94\") " pod="openshift-ingress/router-default-599484b46b-xdvdt" Sep 04 22:10:00.483396 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:00.483396 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-stats-auth\") pod \"router-default-599484b46b-xdvdt\" (UID: \"de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94\") " pod="openshift-ingress/router-default-599484b46b-xdvdt" Sep 04 22:10:00.483625 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:00.483420 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-default-certificate\") pod \"router-default-599484b46b-xdvdt\" (UID: \"de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94\") " pod="openshift-ingress/router-default-599484b46b-xdvdt" Sep 04 22:10:00.483625 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:00.483454 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrm8f\" (UniqueName: \"kubernetes.io/projected/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-kube-api-access-vrm8f\") pod \"router-default-599484b46b-xdvdt\" (UID: \"de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94\") " pod="openshift-ingress/router-default-599484b46b-xdvdt" Sep 04 22:10:00.483625 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:00.483516 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-metrics-certs\") pod \"router-default-599484b46b-xdvdt\" (UID: \"de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94\") " pod="openshift-ingress/router-default-599484b46b-xdvdt" Sep 04 22:10:00.584783 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:00.584762 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-metrics-certs\") pod \"router-default-599484b46b-xdvdt\" (UID: \"de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94\") " pod="openshift-ingress/router-default-599484b46b-xdvdt" Sep 04 22:10:00.584886 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:00.584810 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-service-ca-bundle\") pod \"router-default-599484b46b-xdvdt\" (UID: \"de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94\") " pod="openshift-ingress/router-default-599484b46b-xdvdt" Sep 04 22:10:00.584886 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:00.584828 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-stats-auth\") pod \"router-default-599484b46b-xdvdt\" (UID: \"de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94\") " pod="openshift-ingress/router-default-599484b46b-xdvdt" Sep 04 22:10:00.584992 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:00.584893 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-default-certificate\") pod \"router-default-599484b46b-xdvdt\" (UID: \"de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94\") " pod="openshift-ingress/router-default-599484b46b-xdvdt" Sep 04 22:10:00.584992 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:00.584936 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrm8f\" (UniqueName: \"kubernetes.io/projected/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-kube-api-access-vrm8f\") pod \"router-default-599484b46b-xdvdt\" (UID: \"de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94\") " pod="openshift-ingress/router-default-599484b46b-xdvdt" Sep 04 22:10:00.584992 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:00.584971 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-service-ca-bundle podName:de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94 nodeName:}" failed. No retries permitted until 2026-09-04 22:10:01.084948075 +0000 UTC m=+104.849423682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-service-ca-bundle") pod "router-default-599484b46b-xdvdt" (UID: "de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94") : configmap references non-existent config key: service-ca.crt Sep 04 22:10:00.585138 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:00.585075 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 04 22:10:00.585185 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:00.585145 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-metrics-certs podName:de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94 nodeName:}" failed. No retries permitted until 2026-09-04 22:10:01.085127464 +0000 UTC m=+104.849603055 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-metrics-certs") pod "router-default-599484b46b-xdvdt" (UID: "de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94") : secret "router-metrics-certs-default" not found Sep 04 22:10:00.587229 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:00.587204 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-default-certificate\") pod \"router-default-599484b46b-xdvdt\" (UID: \"de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94\") " pod="openshift-ingress/router-default-599484b46b-xdvdt" Sep 04 22:10:00.587382 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:00.587364 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-stats-auth\") pod \"router-default-599484b46b-xdvdt\" (UID: \"de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94\") " pod="openshift-ingress/router-default-599484b46b-xdvdt" Sep 04 22:10:00.596518 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:00.596494 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrm8f\" (UniqueName: \"kubernetes.io/projected/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-kube-api-access-vrm8f\") pod \"router-default-599484b46b-xdvdt\" (UID: \"de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94\") " pod="openshift-ingress/router-default-599484b46b-xdvdt" Sep 04 22:10:01.089649 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:01.089614 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-metrics-certs\") pod \"router-default-599484b46b-xdvdt\" (UID: \"de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94\") " pod="openshift-ingress/router-default-599484b46b-xdvdt" Sep 04 22:10:01.089825 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:01.089669 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-service-ca-bundle\") pod \"router-default-599484b46b-xdvdt\" (UID: \"de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94\") " pod="openshift-ingress/router-default-599484b46b-xdvdt" Sep 04 22:10:01.089825 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:01.089762 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 04 22:10:01.089825 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:01.089798 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-service-ca-bundle podName:de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94 nodeName:}" failed. No retries permitted until 2026-09-04 22:10:02.089782654 +0000 UTC m=+105.854258265 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-service-ca-bundle") pod "router-default-599484b46b-xdvdt" (UID: "de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94") : configmap references non-existent config key: service-ca.crt Sep 04 22:10:01.089932 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:01.089827 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-metrics-certs podName:de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94 nodeName:}" failed. No retries permitted until 2026-09-04 22:10:02.089813113 +0000 UTC m=+105.854288703 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-metrics-certs") pod "router-default-599484b46b-xdvdt" (UID: "de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94") : secret "router-metrics-certs-default" not found Sep 04 22:10:02.097843 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:02.097788 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-metrics-certs\") pod \"router-default-599484b46b-xdvdt\" (UID: \"de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94\") " pod="openshift-ingress/router-default-599484b46b-xdvdt" Sep 04 22:10:02.097843 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:02.097861 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-service-ca-bundle\") pod \"router-default-599484b46b-xdvdt\" (UID: \"de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94\") " pod="openshift-ingress/router-default-599484b46b-xdvdt" Sep 04 22:10:02.098284 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:02.097948 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 04 22:10:02.098284 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:02.097978 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-service-ca-bundle podName:de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94 nodeName:}" failed. No retries permitted until 2026-09-04 22:10:04.097965736 +0000 UTC m=+107.862441327 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-service-ca-bundle") pod "router-default-599484b46b-xdvdt" (UID: "de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94") : configmap references non-existent config key: service-ca.crt Sep 04 22:10:02.098284 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:02.097999 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-metrics-certs podName:de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94 nodeName:}" failed. No retries permitted until 2026-09-04 22:10:04.097987056 +0000 UTC m=+107.862462647 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-metrics-certs") pod "router-default-599484b46b-xdvdt" (UID: "de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94") : secret "router-metrics-certs-default" not found Sep 04 22:10:04.109782 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:04.109727 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-metrics-certs\") pod \"router-default-599484b46b-xdvdt\" (UID: \"de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94\") " pod="openshift-ingress/router-default-599484b46b-xdvdt" Sep 04 22:10:04.109782 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:04.109798 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-service-ca-bundle\") pod \"router-default-599484b46b-xdvdt\" (UID: \"de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94\") " pod="openshift-ingress/router-default-599484b46b-xdvdt" Sep 04 22:10:04.110197 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:04.109872 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 04 22:10:04.110197 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:04.109919 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-service-ca-bundle podName:de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94 nodeName:}" failed. No retries permitted until 2026-09-04 22:10:08.109905062 +0000 UTC m=+111.874380653 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-service-ca-bundle") pod "router-default-599484b46b-xdvdt" (UID: "de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94") : configmap references non-existent config key: service-ca.crt Sep 04 22:10:04.110197 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:04.109934 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-metrics-certs podName:de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94 nodeName:}" failed. No retries permitted until 2026-09-04 22:10:08.109927881 +0000 UTC m=+111.874403473 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-metrics-certs") pod "router-default-599484b46b-xdvdt" (UID: "de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94") : secret "router-metrics-certs-default" not found Sep 04 22:10:04.312773 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:04.312739 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-84kv4"] Sep 04 22:10:04.314781 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:04.314767 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-84kv4" Sep 04 22:10:04.317261 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:04.317238 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 04 22:10:04.317368 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:04.317328 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 04 22:10:04.317423 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:04.317392 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fhpsq\"" Sep 04 22:10:04.317465 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:04.317435 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 04 22:10:04.318208 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:04.318175 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 04 22:10:04.327480 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:04.327457 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-84kv4"] Sep 04 22:10:04.411741 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:04.411661 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdjgg\" (UniqueName: \"kubernetes.io/projected/c932b83b-a9b3-46dc-963d-af429480dc45-kube-api-access-xdjgg\") pod \"insights-runtime-extractor-84kv4\" (UID: \"c932b83b-a9b3-46dc-963d-af429480dc45\") " pod="openshift-insights/insights-runtime-extractor-84kv4" Sep 04 22:10:04.411741 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:04.411734 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c932b83b-a9b3-46dc-963d-af429480dc45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-84kv4\" (UID: \"c932b83b-a9b3-46dc-963d-af429480dc45\") " pod="openshift-insights/insights-runtime-extractor-84kv4" Sep 04 22:10:04.411912 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:04.411792 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c932b83b-a9b3-46dc-963d-af429480dc45-crio-socket\") pod \"insights-runtime-extractor-84kv4\" (UID: \"c932b83b-a9b3-46dc-963d-af429480dc45\") " pod="openshift-insights/insights-runtime-extractor-84kv4" Sep 04 22:10:04.411912 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:04.411845 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c932b83b-a9b3-46dc-963d-af429480dc45-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-84kv4\" (UID: \"c932b83b-a9b3-46dc-963d-af429480dc45\") " pod="openshift-insights/insights-runtime-extractor-84kv4" Sep 04 22:10:04.412003 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:04.411918 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c932b83b-a9b3-46dc-963d-af429480dc45-data-volume\") pod \"insights-runtime-extractor-84kv4\" (UID: \"c932b83b-a9b3-46dc-963d-af429480dc45\") " pod="openshift-insights/insights-runtime-extractor-84kv4" Sep 04 22:10:04.512963 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:04.512929 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c932b83b-a9b3-46dc-963d-af429480dc45-crio-socket\") pod \"insights-runtime-extractor-84kv4\" (UID: \"c932b83b-a9b3-46dc-963d-af429480dc45\") " pod="openshift-insights/insights-runtime-extractor-84kv4" Sep 04 22:10:04.512963 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:04.512969 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c932b83b-a9b3-46dc-963d-af429480dc45-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-84kv4\" (UID: \"c932b83b-a9b3-46dc-963d-af429480dc45\") " pod="openshift-insights/insights-runtime-extractor-84kv4" Sep 04 22:10:04.513111 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:04.513002 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c932b83b-a9b3-46dc-963d-af429480dc45-data-volume\") pod \"insights-runtime-extractor-84kv4\" (UID: \"c932b83b-a9b3-46dc-963d-af429480dc45\") " pod="openshift-insights/insights-runtime-extractor-84kv4" Sep 04 22:10:04.513111 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:04.513029 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdjgg\" (UniqueName: \"kubernetes.io/projected/c932b83b-a9b3-46dc-963d-af429480dc45-kube-api-access-xdjgg\") pod \"insights-runtime-extractor-84kv4\" (UID: \"c932b83b-a9b3-46dc-963d-af429480dc45\") " pod="openshift-insights/insights-runtime-extractor-84kv4" Sep 04 22:10:04.513111 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:04.513059 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c932b83b-a9b3-46dc-963d-af429480dc45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-84kv4\" (UID: \"c932b83b-a9b3-46dc-963d-af429480dc45\") " pod="openshift-insights/insights-runtime-extractor-84kv4" Sep 04 22:10:04.513111 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:04.513059 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c932b83b-a9b3-46dc-963d-af429480dc45-crio-socket\") pod \"insights-runtime-extractor-84kv4\" (UID: \"c932b83b-a9b3-46dc-963d-af429480dc45\") " pod="openshift-insights/insights-runtime-extractor-84kv4" Sep 04 22:10:04.513111 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:04.513077 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 04 22:10:04.513340 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:04.513146 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c932b83b-a9b3-46dc-963d-af429480dc45-kube-rbac-proxy-cm podName:c932b83b-a9b3-46dc-963d-af429480dc45 nodeName:}" failed. No retries permitted until 2026-09-04 22:10:05.013128177 +0000 UTC m=+108.777603783 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c932b83b-a9b3-46dc-963d-af429480dc45-kube-rbac-proxy-cm") pod "insights-runtime-extractor-84kv4" (UID: "c932b83b-a9b3-46dc-963d-af429480dc45") : configmap "kube-rbac-proxy" not found Sep 04 22:10:04.513340 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:04.513147 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 04 22:10:04.513340 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:04.513175 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c932b83b-a9b3-46dc-963d-af429480dc45-insights-runtime-extractor-tls podName:c932b83b-a9b3-46dc-963d-af429480dc45 nodeName:}" failed. No retries permitted until 2026-09-04 22:10:05.013169101 +0000 UTC m=+108.777644691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c932b83b-a9b3-46dc-963d-af429480dc45-insights-runtime-extractor-tls") pod "insights-runtime-extractor-84kv4" (UID: "c932b83b-a9b3-46dc-963d-af429480dc45") : secret "insights-runtime-extractor-tls" not found Sep 04 22:10:04.513490 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:04.513339 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c932b83b-a9b3-46dc-963d-af429480dc45-data-volume\") pod \"insights-runtime-extractor-84kv4\" (UID: \"c932b83b-a9b3-46dc-963d-af429480dc45\") " pod="openshift-insights/insights-runtime-extractor-84kv4" Sep 04 22:10:04.521722 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:04.521703 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdjgg\" (UniqueName: \"kubernetes.io/projected/c932b83b-a9b3-46dc-963d-af429480dc45-kube-api-access-xdjgg\") pod \"insights-runtime-extractor-84kv4\" (UID: \"c932b83b-a9b3-46dc-963d-af429480dc45\") " pod="openshift-insights/insights-runtime-extractor-84kv4" Sep 04 22:10:05.015754 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:05.015723 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c932b83b-a9b3-46dc-963d-af429480dc45-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-84kv4\" (UID: \"c932b83b-a9b3-46dc-963d-af429480dc45\") " pod="openshift-insights/insights-runtime-extractor-84kv4" Sep 04 22:10:05.015934 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:05.015803 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c932b83b-a9b3-46dc-963d-af429480dc45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-84kv4\" (UID: \"c932b83b-a9b3-46dc-963d-af429480dc45\") " pod="openshift-insights/insights-runtime-extractor-84kv4" Sep 04 22:10:05.015934 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:05.015849 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 04 22:10:05.015934 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:05.015912 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 04 22:10:05.015934 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:05.015934 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c932b83b-a9b3-46dc-963d-af429480dc45-kube-rbac-proxy-cm podName:c932b83b-a9b3-46dc-963d-af429480dc45 nodeName:}" failed. No retries permitted until 2026-09-04 22:10:06.015914245 +0000 UTC m=+109.780389856 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c932b83b-a9b3-46dc-963d-af429480dc45-kube-rbac-proxy-cm") pod "insights-runtime-extractor-84kv4" (UID: "c932b83b-a9b3-46dc-963d-af429480dc45") : configmap "kube-rbac-proxy" not found Sep 04 22:10:05.016150 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:05.015959 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c932b83b-a9b3-46dc-963d-af429480dc45-insights-runtime-extractor-tls podName:c932b83b-a9b3-46dc-963d-af429480dc45 nodeName:}" failed. No retries permitted until 2026-09-04 22:10:06.015948305 +0000 UTC m=+109.780423896 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c932b83b-a9b3-46dc-963d-af429480dc45-insights-runtime-extractor-tls") pod "insights-runtime-extractor-84kv4" (UID: "c932b83b-a9b3-46dc-963d-af429480dc45") : secret "insights-runtime-extractor-tls" not found Sep 04 22:10:06.023721 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:06.023668 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c932b83b-a9b3-46dc-963d-af429480dc45-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-84kv4\" (UID: \"c932b83b-a9b3-46dc-963d-af429480dc45\") " pod="openshift-insights/insights-runtime-extractor-84kv4" Sep 04 22:10:06.024149 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:06.023766 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c932b83b-a9b3-46dc-963d-af429480dc45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-84kv4\" (UID: \"c932b83b-a9b3-46dc-963d-af429480dc45\") " pod="openshift-insights/insights-runtime-extractor-84kv4" Sep 04 22:10:06.024149 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:06.023787 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 04 22:10:06.024149 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:06.023849 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 04 22:10:06.024149 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:06.023867 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c932b83b-a9b3-46dc-963d-af429480dc45-kube-rbac-proxy-cm podName:c932b83b-a9b3-46dc-963d-af429480dc45 nodeName:}" failed. No retries permitted until 2026-09-04 22:10:08.023850767 +0000 UTC m=+111.788326357 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c932b83b-a9b3-46dc-963d-af429480dc45-kube-rbac-proxy-cm") pod "insights-runtime-extractor-84kv4" (UID: "c932b83b-a9b3-46dc-963d-af429480dc45") : configmap "kube-rbac-proxy" not found Sep 04 22:10:06.024149 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:06.023896 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c932b83b-a9b3-46dc-963d-af429480dc45-insights-runtime-extractor-tls podName:c932b83b-a9b3-46dc-963d-af429480dc45 nodeName:}" failed. No retries permitted until 2026-09-04 22:10:08.023885467 +0000 UTC m=+111.788361058 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c932b83b-a9b3-46dc-963d-af429480dc45-insights-runtime-extractor-tls") pod "insights-runtime-extractor-84kv4" (UID: "c932b83b-a9b3-46dc-963d-af429480dc45") : secret "insights-runtime-extractor-tls" not found Sep 04 22:10:06.473989 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:06.473913 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-nzhrn"] Sep 04 22:10:06.474131 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:06.474058 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-p9mdt_c8319608-4bcc-405e-b607-e3fea2c4e9a8/dns-node-resolver/0.log" Sep 04 22:10:06.475777 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:06.475763 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-nzhrn" Sep 04 22:10:06.478162 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:06.478142 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-8hldz\"" Sep 04 22:10:06.486799 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:06.486772 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-nzhrn"] Sep 04 22:10:06.527664 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:06.527632 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6wms\" (UniqueName: \"kubernetes.io/projected/6639be45-021d-4491-9390-87bfa56a46b3-kube-api-access-b6wms\") pod \"network-check-source-685456655f-nzhrn\" (UID: \"6639be45-021d-4491-9390-87bfa56a46b3\") " pod="openshift-network-diagnostics/network-check-source-685456655f-nzhrn" Sep 04 22:10:06.628430 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:06.628401 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6wms\" (UniqueName: \"kubernetes.io/projected/6639be45-021d-4491-9390-87bfa56a46b3-kube-api-access-b6wms\") pod \"network-check-source-685456655f-nzhrn\" (UID: \"6639be45-021d-4491-9390-87bfa56a46b3\") " pod="openshift-network-diagnostics/network-check-source-685456655f-nzhrn" Sep 04 22:10:06.638829 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:06.638790 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6wms\" (UniqueName: \"kubernetes.io/projected/6639be45-021d-4491-9390-87bfa56a46b3-kube-api-access-b6wms\") pod \"network-check-source-685456655f-nzhrn\" (UID: \"6639be45-021d-4491-9390-87bfa56a46b3\") " pod="openshift-network-diagnostics/network-check-source-685456655f-nzhrn" Sep 04 22:10:06.784020 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:06.783990 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-nzhrn" Sep 04 22:10:07.049568 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:07.049352 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-nzhrn"] Sep 04 22:10:07.051909 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:10:07.051886 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6639be45_021d_4491_9390_87bfa56a46b3.slice/crio-b70368a4951b648b23c7e3781d89226ece9b0fc0b13c7d3c69e0899ca8ef318e WatchSource:0}: Error finding container b70368a4951b648b23c7e3781d89226ece9b0fc0b13c7d3c69e0899ca8ef318e: Status 404 returned error can't find the container with id b70368a4951b648b23c7e3781d89226ece9b0fc0b13c7d3c69e0899ca8ef318e Sep 04 22:10:07.070919 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:07.070900 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d5j8h_54f49670-6d36-4464-b9c4-87718e384fb2/node-ca/0.log" Sep 04 22:10:07.215942 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:07.215911 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-nzhrn" event={"ID":"6639be45-021d-4491-9390-87bfa56a46b3","Type":"ContainerStarted","Data":"11df0abb3aa1b31dc3b5d90f11351e7fbd8227fa7d8b359eeceb3eb4558af12b"} Sep 04 22:10:07.216107 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:07.215956 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-nzhrn" event={"ID":"6639be45-021d-4491-9390-87bfa56a46b3","Type":"ContainerStarted","Data":"b70368a4951b648b23c7e3781d89226ece9b0fc0b13c7d3c69e0899ca8ef318e"} Sep 04 22:10:07.233569 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:07.233509 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-nzhrn" podStartSLOduration=1.233493816 podStartE2EDuration="1.233493816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:10:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:10:07.233370681 +0000 UTC m=+110.997846285" watchObservedRunningTime="2026-09-04 22:10:07.233493816 +0000 UTC m=+110.997969430" Sep 04 22:10:08.038373 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:08.038337 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c932b83b-a9b3-46dc-963d-af429480dc45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-84kv4\" (UID: \"c932b83b-a9b3-46dc-963d-af429480dc45\") " pod="openshift-insights/insights-runtime-extractor-84kv4" Sep 04 22:10:08.038542 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:08.038395 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c932b83b-a9b3-46dc-963d-af429480dc45-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-84kv4\" (UID: \"c932b83b-a9b3-46dc-963d-af429480dc45\") " pod="openshift-insights/insights-runtime-extractor-84kv4" Sep 04 22:10:08.038542 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:08.038475 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 04 22:10:08.038542 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:08.038497 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 04 22:10:08.038542 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:08.038530 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c932b83b-a9b3-46dc-963d-af429480dc45-kube-rbac-proxy-cm podName:c932b83b-a9b3-46dc-963d-af429480dc45 nodeName:}" failed. No retries permitted until 2026-09-04 22:10:12.038516041 +0000 UTC m=+115.802991631 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c932b83b-a9b3-46dc-963d-af429480dc45-kube-rbac-proxy-cm") pod "insights-runtime-extractor-84kv4" (UID: "c932b83b-a9b3-46dc-963d-af429480dc45") : configmap "kube-rbac-proxy" not found Sep 04 22:10:08.038717 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:08.038578 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c932b83b-a9b3-46dc-963d-af429480dc45-insights-runtime-extractor-tls podName:c932b83b-a9b3-46dc-963d-af429480dc45 nodeName:}" failed. No retries permitted until 2026-09-04 22:10:12.038543061 +0000 UTC m=+115.803018653 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c932b83b-a9b3-46dc-963d-af429480dc45-insights-runtime-extractor-tls") pod "insights-runtime-extractor-84kv4" (UID: "c932b83b-a9b3-46dc-963d-af429480dc45") : secret "insights-runtime-extractor-tls" not found Sep 04 22:10:08.139401 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:08.139372 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-metrics-certs\") pod \"router-default-599484b46b-xdvdt\" (UID: \"de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94\") " pod="openshift-ingress/router-default-599484b46b-xdvdt" Sep 04 22:10:08.139765 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:08.139443 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-service-ca-bundle\") pod \"router-default-599484b46b-xdvdt\" (UID: \"de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94\") " pod="openshift-ingress/router-default-599484b46b-xdvdt" Sep 04 22:10:08.139765 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:08.139533 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 04 22:10:08.139765 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:08.139562 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-service-ca-bundle podName:de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94 nodeName:}" failed. No retries permitted until 2026-09-04 22:10:16.139529329 +0000 UTC m=+119.904004941 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-service-ca-bundle") pod "router-default-599484b46b-xdvdt" (UID: "de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94") : configmap references non-existent config key: service-ca.crt Sep 04 22:10:08.139765 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:08.139622 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-metrics-certs podName:de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94 nodeName:}" failed. No retries permitted until 2026-09-04 22:10:16.139609492 +0000 UTC m=+119.904085083 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-metrics-certs") pod "router-default-599484b46b-xdvdt" (UID: "de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94") : secret "router-metrics-certs-default" not found Sep 04 22:10:08.270573 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:08.270490 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-c2jbn_6d30375e-bff3-4f2c-9322-6e97a3c0e109/serve-healthcheck-canary/0.log" Sep 04 22:10:12.068455 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:12.068413 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c932b83b-a9b3-46dc-963d-af429480dc45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-84kv4\" (UID: \"c932b83b-a9b3-46dc-963d-af429480dc45\") " pod="openshift-insights/insights-runtime-extractor-84kv4" Sep 04 22:10:12.068853 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:12.068470 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c932b83b-a9b3-46dc-963d-af429480dc45-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-84kv4\" (UID: \"c932b83b-a9b3-46dc-963d-af429480dc45\") " pod="openshift-insights/insights-runtime-extractor-84kv4" Sep 04 22:10:12.068853 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:12.068533 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 04 22:10:12.068853 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:12.068571 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 04 22:10:12.068853 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:12.068613 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c932b83b-a9b3-46dc-963d-af429480dc45-kube-rbac-proxy-cm podName:c932b83b-a9b3-46dc-963d-af429480dc45 nodeName:}" failed. No retries permitted until 2026-09-04 22:10:20.068598251 +0000 UTC m=+123.833073843 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c932b83b-a9b3-46dc-963d-af429480dc45-kube-rbac-proxy-cm") pod "insights-runtime-extractor-84kv4" (UID: "c932b83b-a9b3-46dc-963d-af429480dc45") : configmap "kube-rbac-proxy" not found Sep 04 22:10:12.068853 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:12.068633 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c932b83b-a9b3-46dc-963d-af429480dc45-insights-runtime-extractor-tls podName:c932b83b-a9b3-46dc-963d-af429480dc45 nodeName:}" failed. No retries permitted until 2026-09-04 22:10:20.068619889 +0000 UTC m=+123.833095480 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c932b83b-a9b3-46dc-963d-af429480dc45-insights-runtime-extractor-tls") pod "insights-runtime-extractor-84kv4" (UID: "c932b83b-a9b3-46dc-963d-af429480dc45") : secret "insights-runtime-extractor-tls" not found Sep 04 22:10:16.198626 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:16.198582 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-service-ca-bundle\") pod \"router-default-599484b46b-xdvdt\" (UID: \"de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94\") " pod="openshift-ingress/router-default-599484b46b-xdvdt" Sep 04 22:10:16.199003 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:16.198663 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-metrics-certs\") pod \"router-default-599484b46b-xdvdt\" (UID: \"de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94\") " pod="openshift-ingress/router-default-599484b46b-xdvdt" Sep 04 22:10:16.199003 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:16.198766 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-service-ca-bundle podName:de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94 nodeName:}" failed. No retries permitted until 2026-09-04 22:10:32.19874921 +0000 UTC m=+135.963224806 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-service-ca-bundle") pod "router-default-599484b46b-xdvdt" (UID: "de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94") : configmap references non-existent config key: service-ca.crt Sep 04 22:10:16.199003 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:16.198782 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 04 22:10:16.199003 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:16.198832 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-metrics-certs podName:de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94 nodeName:}" failed. No retries permitted until 2026-09-04 22:10:32.198821627 +0000 UTC m=+135.963297218 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-metrics-certs") pod "router-default-599484b46b-xdvdt" (UID: "de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94") : secret "router-metrics-certs-default" not found Sep 04 22:10:20.127895 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:20.127855 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c932b83b-a9b3-46dc-963d-af429480dc45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-84kv4\" (UID: \"c932b83b-a9b3-46dc-963d-af429480dc45\") " pod="openshift-insights/insights-runtime-extractor-84kv4" Sep 04 22:10:20.128256 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:20.127914 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c932b83b-a9b3-46dc-963d-af429480dc45-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-84kv4\" (UID: \"c932b83b-a9b3-46dc-963d-af429480dc45\") " pod="openshift-insights/insights-runtime-extractor-84kv4" Sep 04 22:10:20.128256 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:20.127989 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 04 22:10:20.128256 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:20.128049 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c932b83b-a9b3-46dc-963d-af429480dc45-insights-runtime-extractor-tls podName:c932b83b-a9b3-46dc-963d-af429480dc45 nodeName:}" failed. No retries permitted until 2026-09-04 22:10:36.128034466 +0000 UTC m=+139.892510057 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c932b83b-a9b3-46dc-963d-af429480dc45-insights-runtime-extractor-tls") pod "insights-runtime-extractor-84kv4" (UID: "c932b83b-a9b3-46dc-963d-af429480dc45") : secret "insights-runtime-extractor-tls" not found Sep 04 22:10:20.128362 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:20.128339 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c932b83b-a9b3-46dc-963d-af429480dc45-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-84kv4\" (UID: \"c932b83b-a9b3-46dc-963d-af429480dc45\") " pod="openshift-insights/insights-runtime-extractor-84kv4" Sep 04 22:10:26.573639 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:26.573601 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0972118-6d5c-4d3a-a979-497ab05ca339-metrics-certs\") pod \"network-metrics-daemon-qvwc7\" (UID: \"c0972118-6d5c-4d3a-a979-497ab05ca339\") " pod="openshift-multus/network-metrics-daemon-qvwc7" Sep 04 22:10:26.574211 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:26.573735 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 04 22:10:26.574211 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:26.573796 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0972118-6d5c-4d3a-a979-497ab05ca339-metrics-certs podName:c0972118-6d5c-4d3a-a979-497ab05ca339 nodeName:}" failed. No retries permitted until 2026-09-04 22:12:28.573778862 +0000 UTC m=+252.338254458 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c0972118-6d5c-4d3a-a979-497ab05ca339-metrics-certs") pod "network-metrics-daemon-qvwc7" (UID: "c0972118-6d5c-4d3a-a979-497ab05ca339") : secret "metrics-daemon-secret" not found Sep 04 22:10:32.218727 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:32.218679 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-service-ca-bundle\") pod \"router-default-599484b46b-xdvdt\" (UID: \"de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94\") " pod="openshift-ingress/router-default-599484b46b-xdvdt" Sep 04 22:10:32.219237 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:32.218757 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-metrics-certs\") pod \"router-default-599484b46b-xdvdt\" (UID: \"de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94\") " pod="openshift-ingress/router-default-599484b46b-xdvdt" Sep 04 22:10:32.219461 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:32.219436 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-service-ca-bundle\") pod \"router-default-599484b46b-xdvdt\" (UID: \"de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94\") " pod="openshift-ingress/router-default-599484b46b-xdvdt" Sep 04 22:10:32.221027 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:32.221009 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94-metrics-certs\") pod \"router-default-599484b46b-xdvdt\" (UID: \"de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94\") " pod="openshift-ingress/router-default-599484b46b-xdvdt" Sep 04 22:10:32.514226 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:32.514197 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-z77tv\"" Sep 04 22:10:32.522335 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:32.522310 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-599484b46b-xdvdt" Sep 04 22:10:32.795068 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:32.794987 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-599484b46b-xdvdt"] Sep 04 22:10:32.797629 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:10:32.797597 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde0ec74b_29ce_42dc_bb2d_be0dcfdc0a94.slice/crio-aed8edcd74bd95f546ec471ff0fd9a33f7dcb1928b55bd47d9c0e821267fd8c7 WatchSource:0}: Error finding container aed8edcd74bd95f546ec471ff0fd9a33f7dcb1928b55bd47d9c0e821267fd8c7: Status 404 returned error can't find the container with id aed8edcd74bd95f546ec471ff0fd9a33f7dcb1928b55bd47d9c0e821267fd8c7 Sep 04 22:10:33.278265 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:33.278233 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-599484b46b-xdvdt" event={"ID":"de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94","Type":"ContainerStarted","Data":"dddab13a769370d3a093780570d3b59274d07d318d10cbef4e70dc088b1d229b"} Sep 04 22:10:33.278706 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:33.278285 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-599484b46b-xdvdt" event={"ID":"de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94","Type":"ContainerStarted","Data":"aed8edcd74bd95f546ec471ff0fd9a33f7dcb1928b55bd47d9c0e821267fd8c7"} Sep 04 22:10:33.305729 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:33.304118 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-599484b46b-xdvdt" podStartSLOduration=33.304097183 podStartE2EDuration="33.304097183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:10:33.303593074 +0000 UTC m=+137.068068687" watchObservedRunningTime="2026-09-04 22:10:33.304097183 +0000 UTC m=+137.068572796" Sep 04 22:10:33.522478 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:33.522444 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-599484b46b-xdvdt" Sep 04 22:10:33.525028 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:33.525003 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-599484b46b-xdvdt" Sep 04 22:10:34.281299 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:34.281269 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-599484b46b-xdvdt" Sep 04 22:10:34.282495 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:34.282475 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-599484b46b-xdvdt" Sep 04 22:10:35.081606 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:35.081573 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85bc9c578d-chx2p"] Sep 04 22:10:35.083825 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:35.083807 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85bc9c578d-chx2p" Sep 04 22:10:35.086664 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:35.086632 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 04 22:10:35.086664 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:35.086641 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Sep 04 22:10:35.086890 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:35.086778 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 04 22:10:35.087631 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:35.087613 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Sep 04 22:10:35.087726 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:35.087639 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 04 22:10:35.087726 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:35.087704 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Sep 04 22:10:35.087836 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:35.087815 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Sep 04 22:10:35.097742 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:35.097717 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85bc9c578d-chx2p"] Sep 04 22:10:35.238455 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:35.238421 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/39bf5e97-d923-4349-a47c-c3ec884a146d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-85bc9c578d-chx2p\" (UID: \"39bf5e97-d923-4349-a47c-c3ec884a146d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85bc9c578d-chx2p" Sep 04 22:10:35.238662 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:35.238522 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sgtp\" (UniqueName: \"kubernetes.io/projected/39bf5e97-d923-4349-a47c-c3ec884a146d-kube-api-access-7sgtp\") pod \"cluster-proxy-proxy-agent-85bc9c578d-chx2p\" (UID: \"39bf5e97-d923-4349-a47c-c3ec884a146d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85bc9c578d-chx2p" Sep 04 22:10:35.238662 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:35.238582 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/39bf5e97-d923-4349-a47c-c3ec884a146d-ca\") pod \"cluster-proxy-proxy-agent-85bc9c578d-chx2p\" (UID: \"39bf5e97-d923-4349-a47c-c3ec884a146d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85bc9c578d-chx2p" Sep 04 22:10:35.238662 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:35.238624 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/39bf5e97-d923-4349-a47c-c3ec884a146d-hub\") pod \"cluster-proxy-proxy-agent-85bc9c578d-chx2p\" (UID: \"39bf5e97-d923-4349-a47c-c3ec884a146d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85bc9c578d-chx2p" Sep 04 22:10:35.238662 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:35.238646 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/39bf5e97-d923-4349-a47c-c3ec884a146d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-85bc9c578d-chx2p\" (UID: \"39bf5e97-d923-4349-a47c-c3ec884a146d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85bc9c578d-chx2p" Sep 04 22:10:35.238843 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:35.238692 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/39bf5e97-d923-4349-a47c-c3ec884a146d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-85bc9c578d-chx2p\" (UID: \"39bf5e97-d923-4349-a47c-c3ec884a146d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85bc9c578d-chx2p" Sep 04 22:10:35.339839 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:35.339766 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/39bf5e97-d923-4349-a47c-c3ec884a146d-hub\") pod \"cluster-proxy-proxy-agent-85bc9c578d-chx2p\" (UID: \"39bf5e97-d923-4349-a47c-c3ec884a146d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85bc9c578d-chx2p" Sep 04 22:10:35.339839 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:35.339793 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/39bf5e97-d923-4349-a47c-c3ec884a146d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-85bc9c578d-chx2p\" (UID: \"39bf5e97-d923-4349-a47c-c3ec884a146d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85bc9c578d-chx2p" Sep 04 22:10:35.339839 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:35.339825 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/39bf5e97-d923-4349-a47c-c3ec884a146d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-85bc9c578d-chx2p\" (UID: \"39bf5e97-d923-4349-a47c-c3ec884a146d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85bc9c578d-chx2p" Sep 04 22:10:35.340336 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:35.340024 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/39bf5e97-d923-4349-a47c-c3ec884a146d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-85bc9c578d-chx2p\" (UID: \"39bf5e97-d923-4349-a47c-c3ec884a146d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85bc9c578d-chx2p" Sep 04 22:10:35.340336 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:35.340119 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sgtp\" (UniqueName: \"kubernetes.io/projected/39bf5e97-d923-4349-a47c-c3ec884a146d-kube-api-access-7sgtp\") pod \"cluster-proxy-proxy-agent-85bc9c578d-chx2p\" (UID: \"39bf5e97-d923-4349-a47c-c3ec884a146d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85bc9c578d-chx2p" Sep 04 22:10:35.340336 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:35.340152 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/39bf5e97-d923-4349-a47c-c3ec884a146d-ca\") pod \"cluster-proxy-proxy-agent-85bc9c578d-chx2p\" (UID: \"39bf5e97-d923-4349-a47c-c3ec884a146d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85bc9c578d-chx2p" Sep 04 22:10:35.340540 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:35.340516 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/39bf5e97-d923-4349-a47c-c3ec884a146d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-85bc9c578d-chx2p\" (UID: \"39bf5e97-d923-4349-a47c-c3ec884a146d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85bc9c578d-chx2p" Sep 04 22:10:35.342513 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:35.342485 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/39bf5e97-d923-4349-a47c-c3ec884a146d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-85bc9c578d-chx2p\" (UID: \"39bf5e97-d923-4349-a47c-c3ec884a146d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85bc9c578d-chx2p" Sep 04 22:10:35.342628 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:35.342538 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/39bf5e97-d923-4349-a47c-c3ec884a146d-hub\") pod \"cluster-proxy-proxy-agent-85bc9c578d-chx2p\" (UID: \"39bf5e97-d923-4349-a47c-c3ec884a146d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85bc9c578d-chx2p" Sep 04 22:10:35.342795 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:35.342771 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/39bf5e97-d923-4349-a47c-c3ec884a146d-ca\") pod \"cluster-proxy-proxy-agent-85bc9c578d-chx2p\" (UID: \"39bf5e97-d923-4349-a47c-c3ec884a146d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85bc9c578d-chx2p" Sep 04 22:10:35.342898 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:35.342882 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/39bf5e97-d923-4349-a47c-c3ec884a146d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-85bc9c578d-chx2p\" (UID: \"39bf5e97-d923-4349-a47c-c3ec884a146d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85bc9c578d-chx2p" Sep 04 22:10:35.354105 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:35.354085 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sgtp\" (UniqueName: \"kubernetes.io/projected/39bf5e97-d923-4349-a47c-c3ec884a146d-kube-api-access-7sgtp\") pod \"cluster-proxy-proxy-agent-85bc9c578d-chx2p\" (UID: \"39bf5e97-d923-4349-a47c-c3ec884a146d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85bc9c578d-chx2p" Sep 04 22:10:35.408242 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:35.408218 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85bc9c578d-chx2p" Sep 04 22:10:35.676613 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:35.676521 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85bc9c578d-chx2p"] Sep 04 22:10:35.680216 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:10:35.680189 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39bf5e97_d923_4349_a47c_c3ec884a146d.slice/crio-edc7774efbb1113c5cc4f29c37eb4c6fd4bb496e20adfb79108fcef36151b0cb WatchSource:0}: Error finding container edc7774efbb1113c5cc4f29c37eb4c6fd4bb496e20adfb79108fcef36151b0cb: Status 404 returned error can't find the container with id edc7774efbb1113c5cc4f29c37eb4c6fd4bb496e20adfb79108fcef36151b0cb Sep 04 22:10:36.146726 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:36.146689 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c932b83b-a9b3-46dc-963d-af429480dc45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-84kv4\" (UID: \"c932b83b-a9b3-46dc-963d-af429480dc45\") " pod="openshift-insights/insights-runtime-extractor-84kv4" Sep 04 22:10:36.148989 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:36.148971 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c932b83b-a9b3-46dc-963d-af429480dc45-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-84kv4\" (UID: \"c932b83b-a9b3-46dc-963d-af429480dc45\") " pod="openshift-insights/insights-runtime-extractor-84kv4" Sep 04 22:10:36.286698 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:36.286659 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85bc9c578d-chx2p" event={"ID":"39bf5e97-d923-4349-a47c-c3ec884a146d","Type":"ContainerStarted","Data":"edc7774efbb1113c5cc4f29c37eb4c6fd4bb496e20adfb79108fcef36151b0cb"} Sep 04 22:10:36.426339 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:36.426266 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fhpsq\"" Sep 04 22:10:36.434152 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:36.434129 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-84kv4" Sep 04 22:10:36.702895 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:36.702816 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-84kv4"] Sep 04 22:10:36.705752 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:10:36.705724 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc932b83b_a9b3_46dc_963d_af429480dc45.slice/crio-55ed420ad09445000acd621c55be226b9dc20ff6ab459502b70caedd6a9be637 WatchSource:0}: Error finding container 55ed420ad09445000acd621c55be226b9dc20ff6ab459502b70caedd6a9be637: Status 404 returned error can't find the container with id 55ed420ad09445000acd621c55be226b9dc20ff6ab459502b70caedd6a9be637 Sep 04 22:10:37.293515 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:37.293475 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-84kv4" event={"ID":"c932b83b-a9b3-46dc-963d-af429480dc45","Type":"ContainerStarted","Data":"8ac95273f44958f6bd39cb4004ae26521f57a94227fe14de7ee81ec62fb16a37"} Sep 04 22:10:37.293708 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:37.293567 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-84kv4" event={"ID":"c932b83b-a9b3-46dc-963d-af429480dc45","Type":"ContainerStarted","Data":"55ed420ad09445000acd621c55be226b9dc20ff6ab459502b70caedd6a9be637"} Sep 04 22:10:38.298992 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:38.298957 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-84kv4" event={"ID":"c932b83b-a9b3-46dc-963d-af429480dc45","Type":"ContainerStarted","Data":"a456afb3c21a8de38d71a481734cd0d9e6180d74b91da6aa2907e96d7b9c2c46"} Sep 04 22:10:40.307631 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:40.307527 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-84kv4" event={"ID":"c932b83b-a9b3-46dc-963d-af429480dc45","Type":"ContainerStarted","Data":"07d9a7fd3223d8942260fa1b465632d1f618b6bf7702cf686bac9281d71dd9fe"} Sep 04 22:10:40.309563 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:40.309531 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85bc9c578d-chx2p" event={"ID":"39bf5e97-d923-4349-a47c-c3ec884a146d","Type":"ContainerStarted","Data":"f1eb7c9bd182e928e6825f1ff9fdd9fae50318ada084b3ac48e1e60cf78e2bb6"} Sep 04 22:10:40.309665 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:40.309576 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85bc9c578d-chx2p" event={"ID":"39bf5e97-d923-4349-a47c-c3ec884a146d","Type":"ContainerStarted","Data":"b1cd06cb167140f71236185769b85a58ef7f9ec5f328c3b1bcc7c4426d728286"} Sep 04 22:10:40.309665 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:40.309590 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85bc9c578d-chx2p" event={"ID":"39bf5e97-d923-4349-a47c-c3ec884a146d","Type":"ContainerStarted","Data":"f29e857c14c274223c19de84a23f73b92db6228a47d2bf18e8552a76230960c3"} Sep 04 22:10:40.328736 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:40.328697 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-84kv4" podStartSLOduration=33.108542775 podStartE2EDuration="36.328685244s" totalImagesPullingTime="3.220142469s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 22:10:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:10:40.327140814 +0000 UTC m=+144.091616426" watchObservedRunningTime="2026-09-04 22:10:40.328685244 +0000 UTC m=+144.093160857" Sep 04 22:10:40.347507 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:40.347463 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85bc9c578d-chx2p" podStartSLOduration=0.980214005 podStartE2EDuration="5.347450689s" totalImagesPullingTime="4.367236684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:10:40.346598027 +0000 UTC m=+144.111073640" watchObservedRunningTime="2026-09-04 22:10:40.347450689 +0000 UTC m=+144.111926302" Sep 04 22:10:46.440727 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:46.440689 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-k7vtb"] Sep 04 22:10:46.444250 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:46.444229 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-k7vtb" Sep 04 22:10:46.447303 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:46.447281 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 04 22:10:46.447402 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:46.447327 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 04 22:10:46.447502 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:46.447489 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 04 22:10:46.447580 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:46.447564 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 04 22:10:46.447821 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:46.447805 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fxd5g\"" Sep 04 22:10:46.448099 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:46.448079 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 04 22:10:46.448205 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:46.448101 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 04 22:10:46.524305 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:46.524273 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e8026ebe-4c89-4066-9ef4-39769575e3c3-node-exporter-accelerators-collector-config\") pod \"node-exporter-k7vtb\" (UID: \"e8026ebe-4c89-4066-9ef4-39769575e3c3\") " pod="openshift-monitoring/node-exporter-k7vtb" Sep 04 22:10:46.524305 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:46.524313 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e8026ebe-4c89-4066-9ef4-39769575e3c3-node-exporter-wtmp\") pod \"node-exporter-k7vtb\" (UID: \"e8026ebe-4c89-4066-9ef4-39769575e3c3\") " pod="openshift-monitoring/node-exporter-k7vtb" Sep 04 22:10:46.524465 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:46.524382 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wgwg\" (UniqueName: \"kubernetes.io/projected/e8026ebe-4c89-4066-9ef4-39769575e3c3-kube-api-access-9wgwg\") pod \"node-exporter-k7vtb\" (UID: \"e8026ebe-4c89-4066-9ef4-39769575e3c3\") " pod="openshift-monitoring/node-exporter-k7vtb" Sep 04 22:10:46.524465 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:46.524401 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8026ebe-4c89-4066-9ef4-39769575e3c3-sys\") pod \"node-exporter-k7vtb\" (UID: \"e8026ebe-4c89-4066-9ef4-39769575e3c3\") " pod="openshift-monitoring/node-exporter-k7vtb" Sep 04 22:10:46.524465 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:46.524421 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e8026ebe-4c89-4066-9ef4-39769575e3c3-node-exporter-textfile\") pod \"node-exporter-k7vtb\" (UID: \"e8026ebe-4c89-4066-9ef4-39769575e3c3\") " pod="openshift-monitoring/node-exporter-k7vtb" Sep 04 22:10:46.524465 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:46.524440 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e8026ebe-4c89-4066-9ef4-39769575e3c3-node-exporter-tls\") pod \"node-exporter-k7vtb\" (UID: \"e8026ebe-4c89-4066-9ef4-39769575e3c3\") " pod="openshift-monitoring/node-exporter-k7vtb" Sep 04 22:10:46.524465 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:46.524459 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8026ebe-4c89-4066-9ef4-39769575e3c3-metrics-client-ca\") pod \"node-exporter-k7vtb\" (UID: \"e8026ebe-4c89-4066-9ef4-39769575e3c3\") " pod="openshift-monitoring/node-exporter-k7vtb" Sep 04 22:10:46.524659 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:46.524500 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8026ebe-4c89-4066-9ef4-39769575e3c3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k7vtb\" (UID: \"e8026ebe-4c89-4066-9ef4-39769575e3c3\") " pod="openshift-monitoring/node-exporter-k7vtb" Sep 04 22:10:46.524659 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:46.524525 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e8026ebe-4c89-4066-9ef4-39769575e3c3-root\") pod \"node-exporter-k7vtb\" (UID: \"e8026ebe-4c89-4066-9ef4-39769575e3c3\") " pod="openshift-monitoring/node-exporter-k7vtb" Sep 04 22:10:46.624854 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:46.624827 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8026ebe-4c89-4066-9ef4-39769575e3c3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k7vtb\" (UID: \"e8026ebe-4c89-4066-9ef4-39769575e3c3\") " pod="openshift-monitoring/node-exporter-k7vtb" Sep 04 22:10:46.624854 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:46.624854 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e8026ebe-4c89-4066-9ef4-39769575e3c3-root\") pod \"node-exporter-k7vtb\" (UID: \"e8026ebe-4c89-4066-9ef4-39769575e3c3\") " pod="openshift-monitoring/node-exporter-k7vtb" Sep 04 22:10:46.624997 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:46.624875 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e8026ebe-4c89-4066-9ef4-39769575e3c3-node-exporter-accelerators-collector-config\") pod \"node-exporter-k7vtb\" (UID: \"e8026ebe-4c89-4066-9ef4-39769575e3c3\") " pod="openshift-monitoring/node-exporter-k7vtb" Sep 04 22:10:46.624997 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:46.624894 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e8026ebe-4c89-4066-9ef4-39769575e3c3-node-exporter-wtmp\") pod \"node-exporter-k7vtb\" (UID: \"e8026ebe-4c89-4066-9ef4-39769575e3c3\") " pod="openshift-monitoring/node-exporter-k7vtb" Sep 04 22:10:46.624997 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:46.624936 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e8026ebe-4c89-4066-9ef4-39769575e3c3-root\") pod \"node-exporter-k7vtb\" (UID: \"e8026ebe-4c89-4066-9ef4-39769575e3c3\") " pod="openshift-monitoring/node-exporter-k7vtb" Sep 04 22:10:46.624997 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:46.624946 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wgwg\" (UniqueName: \"kubernetes.io/projected/e8026ebe-4c89-4066-9ef4-39769575e3c3-kube-api-access-9wgwg\") pod \"node-exporter-k7vtb\" (UID: \"e8026ebe-4c89-4066-9ef4-39769575e3c3\") " pod="openshift-monitoring/node-exporter-k7vtb" Sep 04 22:10:46.624997 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:46.624985 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8026ebe-4c89-4066-9ef4-39769575e3c3-sys\") pod \"node-exporter-k7vtb\" (UID: \"e8026ebe-4c89-4066-9ef4-39769575e3c3\") " pod="openshift-monitoring/node-exporter-k7vtb" Sep 04 22:10:46.625204 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:46.625000 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e8026ebe-4c89-4066-9ef4-39769575e3c3-node-exporter-textfile\") pod \"node-exporter-k7vtb\" (UID: \"e8026ebe-4c89-4066-9ef4-39769575e3c3\") " pod="openshift-monitoring/node-exporter-k7vtb" Sep 04 22:10:46.625204 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:46.625059 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8026ebe-4c89-4066-9ef4-39769575e3c3-sys\") pod \"node-exporter-k7vtb\" (UID: \"e8026ebe-4c89-4066-9ef4-39769575e3c3\") " pod="openshift-monitoring/node-exporter-k7vtb" Sep 04 22:10:46.625204 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:46.625103 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e8026ebe-4c89-4066-9ef4-39769575e3c3-node-exporter-wtmp\") pod \"node-exporter-k7vtb\" (UID: \"e8026ebe-4c89-4066-9ef4-39769575e3c3\") " pod="openshift-monitoring/node-exporter-k7vtb" Sep 04 22:10:46.625204 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:46.625099 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e8026ebe-4c89-4066-9ef4-39769575e3c3-node-exporter-tls\") pod \"node-exporter-k7vtb\" (UID: \"e8026ebe-4c89-4066-9ef4-39769575e3c3\") " pod="openshift-monitoring/node-exporter-k7vtb" Sep 04 22:10:46.625204 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:46.625155 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8026ebe-4c89-4066-9ef4-39769575e3c3-metrics-client-ca\") pod \"node-exporter-k7vtb\" (UID: \"e8026ebe-4c89-4066-9ef4-39769575e3c3\") " pod="openshift-monitoring/node-exporter-k7vtb" Sep 04 22:10:46.625379 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:46.625293 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e8026ebe-4c89-4066-9ef4-39769575e3c3-node-exporter-textfile\") pod \"node-exporter-k7vtb\" (UID: \"e8026ebe-4c89-4066-9ef4-39769575e3c3\") " pod="openshift-monitoring/node-exporter-k7vtb" Sep 04 22:10:46.625471 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:46.625452 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e8026ebe-4c89-4066-9ef4-39769575e3c3-node-exporter-accelerators-collector-config\") pod \"node-exporter-k7vtb\" (UID: \"e8026ebe-4c89-4066-9ef4-39769575e3c3\") " pod="openshift-monitoring/node-exporter-k7vtb" Sep 04 22:10:46.625713 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:46.625693 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8026ebe-4c89-4066-9ef4-39769575e3c3-metrics-client-ca\") pod \"node-exporter-k7vtb\" (UID: \"e8026ebe-4c89-4066-9ef4-39769575e3c3\") " pod="openshift-monitoring/node-exporter-k7vtb" Sep 04 22:10:46.627318 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:46.627297 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e8026ebe-4c89-4066-9ef4-39769575e3c3-node-exporter-tls\") pod \"node-exporter-k7vtb\" (UID: \"e8026ebe-4c89-4066-9ef4-39769575e3c3\") " pod="openshift-monitoring/node-exporter-k7vtb" Sep 04 22:10:46.627405 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:46.627325 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8026ebe-4c89-4066-9ef4-39769575e3c3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k7vtb\" (UID: \"e8026ebe-4c89-4066-9ef4-39769575e3c3\") " pod="openshift-monitoring/node-exporter-k7vtb" Sep 04 22:10:46.633999 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:46.633970 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wgwg\" (UniqueName: \"kubernetes.io/projected/e8026ebe-4c89-4066-9ef4-39769575e3c3-kube-api-access-9wgwg\") pod \"node-exporter-k7vtb\" (UID: \"e8026ebe-4c89-4066-9ef4-39769575e3c3\") " pod="openshift-monitoring/node-exporter-k7vtb" Sep 04 22:10:46.753351 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:46.753327 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-k7vtb" Sep 04 22:10:46.763710 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:10:46.763688 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8026ebe_4c89_4066_9ef4_39769575e3c3.slice/crio-3e8e4b5903cbd2d673873b0ba9107255cfbf11eedd4aee82b4d4a07d65e7cd94 WatchSource:0}: Error finding container 3e8e4b5903cbd2d673873b0ba9107255cfbf11eedd4aee82b4d4a07d65e7cd94: Status 404 returned error can't find the container with id 3e8e4b5903cbd2d673873b0ba9107255cfbf11eedd4aee82b4d4a07d65e7cd94 Sep 04 22:10:47.331979 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:47.331950 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k7vtb" event={"ID":"e8026ebe-4c89-4066-9ef4-39769575e3c3","Type":"ContainerStarted","Data":"3e8e4b5903cbd2d673873b0ba9107255cfbf11eedd4aee82b4d4a07d65e7cd94"} Sep 04 22:10:48.336466 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:48.336433 2574 generic.go:356] "Generic (PLEG): container finished" podID="e8026ebe-4c89-4066-9ef4-39769575e3c3" containerID="9064ab17eb8b7644480c5245621464eaac80db0ad615313855b81b45ffd9e5d4" exitCode=0 Sep 04 22:10:48.336832 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:48.336510 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k7vtb" event={"ID":"e8026ebe-4c89-4066-9ef4-39769575e3c3","Type":"ContainerDied","Data":"9064ab17eb8b7644480c5245621464eaac80db0ad615313855b81b45ffd9e5d4"} Sep 04 22:10:49.342376 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:49.342331 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k7vtb" event={"ID":"e8026ebe-4c89-4066-9ef4-39769575e3c3","Type":"ContainerStarted","Data":"8f2163d44dc00af7402241cf3f955927feb940e4a2b11b53fa5f7c5b5d67487e"} Sep 04 22:10:49.342751 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:49.342404 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k7vtb" event={"ID":"e8026ebe-4c89-4066-9ef4-39769575e3c3","Type":"ContainerStarted","Data":"d0affed38e782eac3b453b6bc4a5f8aaa425ca8d77ed81e1b98aaa95504165df"} Sep 04 22:10:49.368436 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:49.368391 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-k7vtb" podStartSLOduration=2.245338651 podStartE2EDuration="3.368376317s" totalImagesPullingTime="1.064233649s" totalInitContainerRuntime="58.804017ms" isStatefulPod=true podCreationTimestamp="2026-09-04 22:10:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:10:49.366688889 +0000 UTC m=+153.131164502" watchObservedRunningTime="2026-09-04 22:10:49.368376317 +0000 UTC m=+153.132851930" Sep 04 22:10:54.151004 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:54.150965 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-7bdccbfc7-xzs6l" podUID="50ea3996-6bbf-41f2-854b-d99d1af4cf6c" Sep 04 22:10:54.182357 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:54.182328 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-9jzzc" podUID="059b83a0-e9ee-4f9f-a3ae-250d85b3387d" Sep 04 22:10:54.358639 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:54.358612 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9jzzc" Sep 04 22:10:54.734653 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:10:54.734599 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-qvwc7" podUID="c0972118-6d5c-4d3a-a979-497ab05ca339" Sep 04 22:10:57.246934 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:57.246897 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7bdccbfc7-xzs6l"] Sep 04 22:10:57.247361 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:57.247001 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bdccbfc7-xzs6l" Sep 04 22:10:57.253488 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:57.253470 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bdccbfc7-xzs6l" Sep 04 22:10:57.293070 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:57.293051 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-bound-sa-token\") pod \"50ea3996-6bbf-41f2-854b-d99d1af4cf6c\" (UID: \"50ea3996-6bbf-41f2-854b-d99d1af4cf6c\") " Sep 04 22:10:57.293174 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:57.293084 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n25sx\" (UniqueName: \"kubernetes.io/projected/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-kube-api-access-n25sx\") pod \"50ea3996-6bbf-41f2-854b-d99d1af4cf6c\" (UID: \"50ea3996-6bbf-41f2-854b-d99d1af4cf6c\") " Sep 04 22:10:57.293174 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:57.293105 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-ca-trust-extracted\") pod \"50ea3996-6bbf-41f2-854b-d99d1af4cf6c\" (UID: \"50ea3996-6bbf-41f2-854b-d99d1af4cf6c\") " Sep 04 22:10:57.293174 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:57.293122 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-trusted-ca\") pod \"50ea3996-6bbf-41f2-854b-d99d1af4cf6c\" (UID: \"50ea3996-6bbf-41f2-854b-d99d1af4cf6c\") " Sep 04 22:10:57.293174 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:57.293151 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-registry-certificates\") pod \"50ea3996-6bbf-41f2-854b-d99d1af4cf6c\" (UID: \"50ea3996-6bbf-41f2-854b-d99d1af4cf6c\") " Sep 04 22:10:57.293342 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:57.293193 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-installation-pull-secrets\") pod \"50ea3996-6bbf-41f2-854b-d99d1af4cf6c\" (UID: \"50ea3996-6bbf-41f2-854b-d99d1af4cf6c\") " Sep 04 22:10:57.293342 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:57.293245 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-image-registry-private-configuration\") pod \"50ea3996-6bbf-41f2-854b-d99d1af4cf6c\" (UID: \"50ea3996-6bbf-41f2-854b-d99d1af4cf6c\") " Sep 04 22:10:57.293491 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:57.293444 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "50ea3996-6bbf-41f2-854b-d99d1af4cf6c" (UID: "50ea3996-6bbf-41f2-854b-d99d1af4cf6c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 04 22:10:57.293593 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:57.293565 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "50ea3996-6bbf-41f2-854b-d99d1af4cf6c" (UID: "50ea3996-6bbf-41f2-854b-d99d1af4cf6c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:10:57.293674 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:57.293646 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "50ea3996-6bbf-41f2-854b-d99d1af4cf6c" (UID: "50ea3996-6bbf-41f2-854b-d99d1af4cf6c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:10:57.295413 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:57.295389 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-kube-api-access-n25sx" (OuterVolumeSpecName: "kube-api-access-n25sx") pod "50ea3996-6bbf-41f2-854b-d99d1af4cf6c" (UID: "50ea3996-6bbf-41f2-854b-d99d1af4cf6c"). InnerVolumeSpecName "kube-api-access-n25sx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:10:57.295519 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:57.295446 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "50ea3996-6bbf-41f2-854b-d99d1af4cf6c" (UID: "50ea3996-6bbf-41f2-854b-d99d1af4cf6c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:10:57.295519 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:57.295468 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "50ea3996-6bbf-41f2-854b-d99d1af4cf6c" (UID: "50ea3996-6bbf-41f2-854b-d99d1af4cf6c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 04 22:10:57.295519 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:57.295513 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "50ea3996-6bbf-41f2-854b-d99d1af4cf6c" (UID: "50ea3996-6bbf-41f2-854b-d99d1af4cf6c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 04 22:10:57.368869 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:57.368845 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bdccbfc7-xzs6l" Sep 04 22:10:57.394632 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:57.394609 2574 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-registry-certificates\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:10:57.394632 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:57.394631 2574 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-installation-pull-secrets\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:10:57.394752 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:57.394646 2574 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-image-registry-private-configuration\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:10:57.394752 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:57.394661 2574 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-bound-sa-token\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:10:57.394752 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:57.394670 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n25sx\" (UniqueName: \"kubernetes.io/projected/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-kube-api-access-n25sx\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:10:57.394752 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:57.394679 2574 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-ca-trust-extracted\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:10:57.394752 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:57.394687 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-trusted-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:10:57.431564 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:57.431533 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7bdccbfc7-xzs6l"] Sep 04 22:10:57.441905 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:57.441883 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7bdccbfc7-xzs6l"] Sep 04 22:10:57.495067 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:57.495045 2574 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/50ea3996-6bbf-41f2-854b-d99d1af4cf6c-registry-tls\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:10:58.713323 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:58.713282 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50ea3996-6bbf-41f2-854b-d99d1af4cf6c" path="/var/lib/kubelet/pods/50ea3996-6bbf-41f2-854b-d99d1af4cf6c/volumes" Sep 04 22:10:59.106274 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:59.106225 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/059b83a0-e9ee-4f9f-a3ae-250d85b3387d-metrics-tls\") pod \"dns-default-9jzzc\" (UID: \"059b83a0-e9ee-4f9f-a3ae-250d85b3387d\") " pod="openshift-dns/dns-default-9jzzc" Sep 04 22:10:59.108560 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:59.108523 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/059b83a0-e9ee-4f9f-a3ae-250d85b3387d-metrics-tls\") pod \"dns-default-9jzzc\" (UID: \"059b83a0-e9ee-4f9f-a3ae-250d85b3387d\") " pod="openshift-dns/dns-default-9jzzc" Sep 04 22:10:59.162042 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:59.162022 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wcrtk\"" Sep 04 22:10:59.169892 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:59.169874 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9jzzc" Sep 04 22:10:59.444378 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:10:59.444308 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9jzzc"] Sep 04 22:10:59.447748 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:10:59.447721 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod059b83a0_e9ee_4f9f_a3ae_250d85b3387d.slice/crio-bc8eaed9dad68a4f7f831121cee42c8b0848527f333b98501625aa95dfd17fe0 WatchSource:0}: Error finding container bc8eaed9dad68a4f7f831121cee42c8b0848527f333b98501625aa95dfd17fe0: Status 404 returned error can't find the container with id bc8eaed9dad68a4f7f831121cee42c8b0848527f333b98501625aa95dfd17fe0 Sep 04 22:11:00.380465 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:00.380430 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9jzzc" event={"ID":"059b83a0-e9ee-4f9f-a3ae-250d85b3387d","Type":"ContainerStarted","Data":"bc8eaed9dad68a4f7f831121cee42c8b0848527f333b98501625aa95dfd17fe0"} Sep 04 22:11:01.385972 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:01.385905 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9jzzc" event={"ID":"059b83a0-e9ee-4f9f-a3ae-250d85b3387d","Type":"ContainerStarted","Data":"74e6722eae2287bb4c68db691dd339f98e69304f53732fcb9fe0f92fef1415ad"} Sep 04 22:11:01.385972 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:01.385963 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9jzzc" event={"ID":"059b83a0-e9ee-4f9f-a3ae-250d85b3387d","Type":"ContainerStarted","Data":"d775e62e583eae81aeff1faa641c618c919a8073313d233ff9ba3c68171005ce"} Sep 04 22:11:01.386305 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:01.386010 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9jzzc" Sep 04 22:11:01.411318 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:01.411275 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9jzzc" podStartSLOduration=128.732069627 podStartE2EDuration="2m10.411256483s" totalImagesPullingTime="1.679186856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:08:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:11:01.410176792 +0000 UTC m=+165.174652406" watchObservedRunningTime="2026-09-04 22:11:01.411256483 +0000 UTC m=+165.175732099" Sep 04 22:11:03.065801 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:03.065765 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d78c49dbf-4qqfn"] Sep 04 22:11:03.068758 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:03.068739 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d78c49dbf-4qqfn" Sep 04 22:11:03.071887 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:03.071864 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 04 22:11:03.071887 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:03.071880 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 04 22:11:03.072071 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:03.071906 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 04 22:11:03.072071 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:03.071862 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 04 22:11:03.072071 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:03.071952 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6dt64\"" Sep 04 22:11:03.072071 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:03.072010 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 04 22:11:03.072245 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:03.072106 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 04 22:11:03.072245 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:03.072164 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 04 22:11:03.085868 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:03.085840 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d78c49dbf-4qqfn"] Sep 04 22:11:03.135431 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:03.135409 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c26d316e-96e5-4678-a904-71942982a97b-service-ca\") pod \"console-7d78c49dbf-4qqfn\" (UID: \"c26d316e-96e5-4678-a904-71942982a97b\") " pod="openshift-console/console-7d78c49dbf-4qqfn" Sep 04 22:11:03.135538 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:03.135451 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c26d316e-96e5-4678-a904-71942982a97b-console-serving-cert\") pod \"console-7d78c49dbf-4qqfn\" (UID: \"c26d316e-96e5-4678-a904-71942982a97b\") " pod="openshift-console/console-7d78c49dbf-4qqfn" Sep 04 22:11:03.135538 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:03.135468 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb67k\" (UniqueName: \"kubernetes.io/projected/c26d316e-96e5-4678-a904-71942982a97b-kube-api-access-wb67k\") pod \"console-7d78c49dbf-4qqfn\" (UID: \"c26d316e-96e5-4678-a904-71942982a97b\") " pod="openshift-console/console-7d78c49dbf-4qqfn" Sep 04 22:11:03.135538 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:03.135493 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c26d316e-96e5-4678-a904-71942982a97b-console-oauth-config\") pod \"console-7d78c49dbf-4qqfn\" (UID: \"c26d316e-96e5-4678-a904-71942982a97b\") " pod="openshift-console/console-7d78c49dbf-4qqfn" Sep 04 22:11:03.135681 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:03.135587 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c26d316e-96e5-4678-a904-71942982a97b-console-config\") pod \"console-7d78c49dbf-4qqfn\" (UID: \"c26d316e-96e5-4678-a904-71942982a97b\") " pod="openshift-console/console-7d78c49dbf-4qqfn" Sep 04 22:11:03.135681 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:03.135616 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c26d316e-96e5-4678-a904-71942982a97b-oauth-serving-cert\") pod \"console-7d78c49dbf-4qqfn\" (UID: \"c26d316e-96e5-4678-a904-71942982a97b\") " pod="openshift-console/console-7d78c49dbf-4qqfn" Sep 04 22:11:03.236265 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:03.236237 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c26d316e-96e5-4678-a904-71942982a97b-oauth-serving-cert\") pod \"console-7d78c49dbf-4qqfn\" (UID: \"c26d316e-96e5-4678-a904-71942982a97b\") " pod="openshift-console/console-7d78c49dbf-4qqfn" Sep 04 22:11:03.236400 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:03.236276 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c26d316e-96e5-4678-a904-71942982a97b-service-ca\") pod \"console-7d78c49dbf-4qqfn\" (UID: \"c26d316e-96e5-4678-a904-71942982a97b\") " pod="openshift-console/console-7d78c49dbf-4qqfn" Sep 04 22:11:03.236400 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:03.236342 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c26d316e-96e5-4678-a904-71942982a97b-console-serving-cert\") pod \"console-7d78c49dbf-4qqfn\" (UID: \"c26d316e-96e5-4678-a904-71942982a97b\") " pod="openshift-console/console-7d78c49dbf-4qqfn" Sep 04 22:11:03.236400 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:03.236367 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb67k\" (UniqueName: \"kubernetes.io/projected/c26d316e-96e5-4678-a904-71942982a97b-kube-api-access-wb67k\") pod \"console-7d78c49dbf-4qqfn\" (UID: \"c26d316e-96e5-4678-a904-71942982a97b\") " pod="openshift-console/console-7d78c49dbf-4qqfn" Sep 04 22:11:03.236577 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:03.236406 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c26d316e-96e5-4678-a904-71942982a97b-console-oauth-config\") pod \"console-7d78c49dbf-4qqfn\" (UID: \"c26d316e-96e5-4678-a904-71942982a97b\") " pod="openshift-console/console-7d78c49dbf-4qqfn" Sep 04 22:11:03.236577 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:03.236451 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c26d316e-96e5-4678-a904-71942982a97b-console-config\") pod \"console-7d78c49dbf-4qqfn\" (UID: \"c26d316e-96e5-4678-a904-71942982a97b\") " pod="openshift-console/console-7d78c49dbf-4qqfn" Sep 04 22:11:03.236987 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:03.236962 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c26d316e-96e5-4678-a904-71942982a97b-oauth-serving-cert\") pod \"console-7d78c49dbf-4qqfn\" (UID: \"c26d316e-96e5-4678-a904-71942982a97b\") " pod="openshift-console/console-7d78c49dbf-4qqfn" Sep 04 22:11:03.237074 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:03.237038 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c26d316e-96e5-4678-a904-71942982a97b-console-config\") pod \"console-7d78c49dbf-4qqfn\" (UID: \"c26d316e-96e5-4678-a904-71942982a97b\") " pod="openshift-console/console-7d78c49dbf-4qqfn" Sep 04 22:11:03.237074 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:03.237047 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c26d316e-96e5-4678-a904-71942982a97b-service-ca\") pod \"console-7d78c49dbf-4qqfn\" (UID: \"c26d316e-96e5-4678-a904-71942982a97b\") " pod="openshift-console/console-7d78c49dbf-4qqfn" Sep 04 22:11:03.238689 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:03.238674 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c26d316e-96e5-4678-a904-71942982a97b-console-serving-cert\") pod \"console-7d78c49dbf-4qqfn\" (UID: \"c26d316e-96e5-4678-a904-71942982a97b\") " pod="openshift-console/console-7d78c49dbf-4qqfn" Sep 04 22:11:03.238790 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:03.238774 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c26d316e-96e5-4678-a904-71942982a97b-console-oauth-config\") pod \"console-7d78c49dbf-4qqfn\" (UID: \"c26d316e-96e5-4678-a904-71942982a97b\") " pod="openshift-console/console-7d78c49dbf-4qqfn" Sep 04 22:11:03.247665 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:03.247643 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb67k\" (UniqueName: \"kubernetes.io/projected/c26d316e-96e5-4678-a904-71942982a97b-kube-api-access-wb67k\") pod \"console-7d78c49dbf-4qqfn\" (UID: \"c26d316e-96e5-4678-a904-71942982a97b\") " pod="openshift-console/console-7d78c49dbf-4qqfn" Sep 04 22:11:03.377346 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:03.377280 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d78c49dbf-4qqfn" Sep 04 22:11:03.654028 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:03.653951 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d78c49dbf-4qqfn"] Sep 04 22:11:03.656723 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:11:03.656693 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc26d316e_96e5_4678_a904_71942982a97b.slice/crio-fe30cf6dd16819baacb9f7e5a4b870bed128078ce22033b418ba015244cf5df4 WatchSource:0}: Error finding container fe30cf6dd16819baacb9f7e5a4b870bed128078ce22033b418ba015244cf5df4: Status 404 returned error can't find the container with id fe30cf6dd16819baacb9f7e5a4b870bed128078ce22033b418ba015244cf5df4 Sep 04 22:11:04.404311 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:04.404274 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d78c49dbf-4qqfn" event={"ID":"c26d316e-96e5-4678-a904-71942982a97b","Type":"ContainerStarted","Data":"fe30cf6dd16819baacb9f7e5a4b870bed128078ce22033b418ba015244cf5df4"} Sep 04 22:11:07.416602 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:07.416511 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d78c49dbf-4qqfn" event={"ID":"c26d316e-96e5-4678-a904-71942982a97b","Type":"ContainerStarted","Data":"1be61c3ef3e552259e855824e874e96994941a48286d5edbab7713080b636c5e"} Sep 04 22:11:07.441346 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:07.441303 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7d78c49dbf-4qqfn" podStartSLOduration=0.985605143 podStartE2EDuration="4.44128919s" totalImagesPullingTime="3.455684047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:11:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:11:07.439700597 +0000 UTC m=+171.204176210" watchObservedRunningTime="2026-09-04 22:11:07.44128919 +0000 UTC m=+171.205764794" Sep 04 22:11:08.709604 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:08.709566 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvwc7" Sep 04 22:11:11.391531 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:11.391463 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9jzzc" Sep 04 22:11:12.817943 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:12.817913 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c5dfdb68-8rflk"] Sep 04 22:11:12.819927 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:12.819910 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c5dfdb68-8rflk" Sep 04 22:11:12.827302 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:12.827283 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 04 22:11:12.830771 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:12.830745 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c5dfdb68-8rflk"] Sep 04 22:11:12.905616 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:12.905589 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a06b2a76-9cfa-4dc9-9446-3c07b7250fea-service-ca\") pod \"console-7c5dfdb68-8rflk\" (UID: \"a06b2a76-9cfa-4dc9-9446-3c07b7250fea\") " pod="openshift-console/console-7c5dfdb68-8rflk" Sep 04 22:11:12.905733 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:12.905647 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a06b2a76-9cfa-4dc9-9446-3c07b7250fea-trusted-ca-bundle\") pod \"console-7c5dfdb68-8rflk\" (UID: \"a06b2a76-9cfa-4dc9-9446-3c07b7250fea\") " pod="openshift-console/console-7c5dfdb68-8rflk" Sep 04 22:11:12.905733 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:12.905676 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a06b2a76-9cfa-4dc9-9446-3c07b7250fea-oauth-serving-cert\") pod \"console-7c5dfdb68-8rflk\" (UID: \"a06b2a76-9cfa-4dc9-9446-3c07b7250fea\") " pod="openshift-console/console-7c5dfdb68-8rflk" Sep 04 22:11:12.905733 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:12.905707 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skr2f\" (UniqueName: \"kubernetes.io/projected/a06b2a76-9cfa-4dc9-9446-3c07b7250fea-kube-api-access-skr2f\") pod \"console-7c5dfdb68-8rflk\" (UID: \"a06b2a76-9cfa-4dc9-9446-3c07b7250fea\") " pod="openshift-console/console-7c5dfdb68-8rflk" Sep 04 22:11:12.905879 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:12.905750 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a06b2a76-9cfa-4dc9-9446-3c07b7250fea-console-serving-cert\") pod \"console-7c5dfdb68-8rflk\" (UID: \"a06b2a76-9cfa-4dc9-9446-3c07b7250fea\") " pod="openshift-console/console-7c5dfdb68-8rflk" Sep 04 22:11:12.905879 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:12.905794 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a06b2a76-9cfa-4dc9-9446-3c07b7250fea-console-oauth-config\") pod \"console-7c5dfdb68-8rflk\" (UID: \"a06b2a76-9cfa-4dc9-9446-3c07b7250fea\") " pod="openshift-console/console-7c5dfdb68-8rflk" Sep 04 22:11:12.905879 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:12.905845 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a06b2a76-9cfa-4dc9-9446-3c07b7250fea-console-config\") pod \"console-7c5dfdb68-8rflk\" (UID: \"a06b2a76-9cfa-4dc9-9446-3c07b7250fea\") " pod="openshift-console/console-7c5dfdb68-8rflk" Sep 04 22:11:13.006588 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:13.006561 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a06b2a76-9cfa-4dc9-9446-3c07b7250fea-console-serving-cert\") pod \"console-7c5dfdb68-8rflk\" (UID: \"a06b2a76-9cfa-4dc9-9446-3c07b7250fea\") " pod="openshift-console/console-7c5dfdb68-8rflk" Sep 04 22:11:13.006704 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:13.006601 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a06b2a76-9cfa-4dc9-9446-3c07b7250fea-console-oauth-config\") pod \"console-7c5dfdb68-8rflk\" (UID: \"a06b2a76-9cfa-4dc9-9446-3c07b7250fea\") " pod="openshift-console/console-7c5dfdb68-8rflk" Sep 04 22:11:13.006704 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:13.006620 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a06b2a76-9cfa-4dc9-9446-3c07b7250fea-console-config\") pod \"console-7c5dfdb68-8rflk\" (UID: \"a06b2a76-9cfa-4dc9-9446-3c07b7250fea\") " pod="openshift-console/console-7c5dfdb68-8rflk" Sep 04 22:11:13.006704 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:13.006637 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a06b2a76-9cfa-4dc9-9446-3c07b7250fea-service-ca\") pod \"console-7c5dfdb68-8rflk\" (UID: \"a06b2a76-9cfa-4dc9-9446-3c07b7250fea\") " pod="openshift-console/console-7c5dfdb68-8rflk" Sep 04 22:11:13.006704 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:13.006686 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a06b2a76-9cfa-4dc9-9446-3c07b7250fea-trusted-ca-bundle\") pod \"console-7c5dfdb68-8rflk\" (UID: \"a06b2a76-9cfa-4dc9-9446-3c07b7250fea\") " pod="openshift-console/console-7c5dfdb68-8rflk" Sep 04 22:11:13.006895 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:13.006716 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a06b2a76-9cfa-4dc9-9446-3c07b7250fea-oauth-serving-cert\") pod \"console-7c5dfdb68-8rflk\" (UID: \"a06b2a76-9cfa-4dc9-9446-3c07b7250fea\") " pod="openshift-console/console-7c5dfdb68-8rflk" Sep 04 22:11:13.006895 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:13.006745 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skr2f\" (UniqueName: \"kubernetes.io/projected/a06b2a76-9cfa-4dc9-9446-3c07b7250fea-kube-api-access-skr2f\") pod \"console-7c5dfdb68-8rflk\" (UID: \"a06b2a76-9cfa-4dc9-9446-3c07b7250fea\") " pod="openshift-console/console-7c5dfdb68-8rflk" Sep 04 22:11:13.007419 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:13.007399 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a06b2a76-9cfa-4dc9-9446-3c07b7250fea-console-config\") pod \"console-7c5dfdb68-8rflk\" (UID: \"a06b2a76-9cfa-4dc9-9446-3c07b7250fea\") " pod="openshift-console/console-7c5dfdb68-8rflk" Sep 04 22:11:13.007538 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:13.007516 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a06b2a76-9cfa-4dc9-9446-3c07b7250fea-service-ca\") pod \"console-7c5dfdb68-8rflk\" (UID: \"a06b2a76-9cfa-4dc9-9446-3c07b7250fea\") " pod="openshift-console/console-7c5dfdb68-8rflk" Sep 04 22:11:13.007622 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:13.007524 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a06b2a76-9cfa-4dc9-9446-3c07b7250fea-oauth-serving-cert\") pod \"console-7c5dfdb68-8rflk\" (UID: \"a06b2a76-9cfa-4dc9-9446-3c07b7250fea\") " pod="openshift-console/console-7c5dfdb68-8rflk" Sep 04 22:11:13.007774 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:13.007751 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a06b2a76-9cfa-4dc9-9446-3c07b7250fea-trusted-ca-bundle\") pod \"console-7c5dfdb68-8rflk\" (UID: \"a06b2a76-9cfa-4dc9-9446-3c07b7250fea\") " pod="openshift-console/console-7c5dfdb68-8rflk" Sep 04 22:11:13.009111 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:13.009091 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a06b2a76-9cfa-4dc9-9446-3c07b7250fea-console-oauth-config\") pod \"console-7c5dfdb68-8rflk\" (UID: \"a06b2a76-9cfa-4dc9-9446-3c07b7250fea\") " pod="openshift-console/console-7c5dfdb68-8rflk" Sep 04 22:11:13.009276 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:13.009258 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a06b2a76-9cfa-4dc9-9446-3c07b7250fea-console-serving-cert\") pod \"console-7c5dfdb68-8rflk\" (UID: \"a06b2a76-9cfa-4dc9-9446-3c07b7250fea\") " pod="openshift-console/console-7c5dfdb68-8rflk" Sep 04 22:11:13.016120 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:13.016092 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skr2f\" (UniqueName: \"kubernetes.io/projected/a06b2a76-9cfa-4dc9-9446-3c07b7250fea-kube-api-access-skr2f\") pod \"console-7c5dfdb68-8rflk\" (UID: \"a06b2a76-9cfa-4dc9-9446-3c07b7250fea\") " pod="openshift-console/console-7c5dfdb68-8rflk" Sep 04 22:11:13.129642 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:13.129532 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c5dfdb68-8rflk" Sep 04 22:11:13.378195 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:13.378166 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d78c49dbf-4qqfn" Sep 04 22:11:13.378354 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:13.378234 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7d78c49dbf-4qqfn" Sep 04 22:11:13.383057 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:13.383002 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7d78c49dbf-4qqfn" Sep 04 22:11:13.405452 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:13.401491 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c5dfdb68-8rflk"] Sep 04 22:11:13.406936 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:11:13.406912 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda06b2a76_9cfa_4dc9_9446_3c07b7250fea.slice/crio-5ba628296c150671b506beedd5958aba7b455b8be5a652d6cdaad75d94c7c455 WatchSource:0}: Error finding container 5ba628296c150671b506beedd5958aba7b455b8be5a652d6cdaad75d94c7c455: Status 404 returned error can't find the container with id 5ba628296c150671b506beedd5958aba7b455b8be5a652d6cdaad75d94c7c455 Sep 04 22:11:13.438217 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:13.438183 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c5dfdb68-8rflk" event={"ID":"a06b2a76-9cfa-4dc9-9446-3c07b7250fea","Type":"ContainerStarted","Data":"5ba628296c150671b506beedd5958aba7b455b8be5a652d6cdaad75d94c7c455"} Sep 04 22:11:13.443214 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:13.443188 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7d78c49dbf-4qqfn" Sep 04 22:11:14.442627 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:14.442592 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c5dfdb68-8rflk" event={"ID":"a06b2a76-9cfa-4dc9-9446-3c07b7250fea","Type":"ContainerStarted","Data":"d3ee6983a97904627e51b84eba99177b439da6dab342d2e1a1d8a83c3c9abf50"} Sep 04 22:11:14.460943 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:14.460891 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7c5dfdb68-8rflk" podStartSLOduration=2.460872043 podStartE2EDuration="2.460872043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:11:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:11:14.460274263 +0000 UTC m=+178.224749877" watchObservedRunningTime="2026-09-04 22:11:14.460872043 +0000 UTC m=+178.225347658" Sep 04 22:11:19.679803 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:19.679776 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k7vtb_e8026ebe-4c89-4066-9ef4-39769575e3c3/init-textfile/0.log" Sep 04 22:11:19.880754 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:19.880723 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k7vtb_e8026ebe-4c89-4066-9ef4-39769575e3c3/node-exporter/0.log" Sep 04 22:11:20.081864 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:20.081840 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k7vtb_e8026ebe-4c89-4066-9ef4-39769575e3c3/kube-rbac-proxy/0.log" Sep 04 22:11:23.129709 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:23.129675 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c5dfdb68-8rflk" Sep 04 22:11:23.130096 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:23.129731 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c5dfdb68-8rflk" Sep 04 22:11:23.134527 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:23.134506 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c5dfdb68-8rflk" Sep 04 22:11:23.474240 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:23.474163 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c5dfdb68-8rflk" Sep 04 22:11:23.544590 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:23.544540 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d78c49dbf-4qqfn"] Sep 04 22:11:25.880420 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:25.880387 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7c5dfdb68-8rflk_a06b2a76-9cfa-4dc9-9446-3c07b7250fea/console/0.log" Sep 04 22:11:26.080606 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:26.080583 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7d78c49dbf-4qqfn_c26d316e-96e5-4678-a904-71942982a97b/console/0.log" Sep 04 22:11:26.879796 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:26.879767 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9jzzc_059b83a0-e9ee-4f9f-a3ae-250d85b3387d/dns/0.log" Sep 04 22:11:27.079964 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:27.079942 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9jzzc_059b83a0-e9ee-4f9f-a3ae-250d85b3387d/kube-rbac-proxy/0.log" Sep 04 22:11:28.081584 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:28.081532 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-p9mdt_c8319608-4bcc-405e-b607-e3fea2c4e9a8/dns-node-resolver/0.log" Sep 04 22:11:28.280923 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:28.280893 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-599484b46b-xdvdt_de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94/router/0.log" Sep 04 22:11:28.880644 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:28.880615 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-c2jbn_6d30375e-bff3-4f2c-9322-6e97a3c0e109/serve-healthcheck-canary/0.log" Sep 04 22:11:48.563562 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:48.563503 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7d78c49dbf-4qqfn" podUID="c26d316e-96e5-4678-a904-71942982a97b" containerName="console" containerID="cri-o://1be61c3ef3e552259e855824e874e96994941a48286d5edbab7713080b636c5e" gracePeriod=15 Sep 04 22:11:48.790692 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:48.790672 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d78c49dbf-4qqfn" Sep 04 22:11:48.858746 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:48.858686 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c26d316e-96e5-4678-a904-71942982a97b-console-oauth-config\") pod \"c26d316e-96e5-4678-a904-71942982a97b\" (UID: \"c26d316e-96e5-4678-a904-71942982a97b\") " Sep 04 22:11:48.858746 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:48.858725 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c26d316e-96e5-4678-a904-71942982a97b-console-serving-cert\") pod \"c26d316e-96e5-4678-a904-71942982a97b\" (UID: \"c26d316e-96e5-4678-a904-71942982a97b\") " Sep 04 22:11:48.858890 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:48.858754 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c26d316e-96e5-4678-a904-71942982a97b-console-config\") pod \"c26d316e-96e5-4678-a904-71942982a97b\" (UID: \"c26d316e-96e5-4678-a904-71942982a97b\") " Sep 04 22:11:48.858890 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:48.858806 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c26d316e-96e5-4678-a904-71942982a97b-service-ca\") pod \"c26d316e-96e5-4678-a904-71942982a97b\" (UID: \"c26d316e-96e5-4678-a904-71942982a97b\") " Sep 04 22:11:48.858965 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:48.858921 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wb67k\" (UniqueName: \"kubernetes.io/projected/c26d316e-96e5-4678-a904-71942982a97b-kube-api-access-wb67k\") pod \"c26d316e-96e5-4678-a904-71942982a97b\" (UID: \"c26d316e-96e5-4678-a904-71942982a97b\") " Sep 04 22:11:48.859012 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:48.858981 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c26d316e-96e5-4678-a904-71942982a97b-oauth-serving-cert\") pod \"c26d316e-96e5-4678-a904-71942982a97b\" (UID: \"c26d316e-96e5-4678-a904-71942982a97b\") " Sep 04 22:11:48.859177 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:48.859142 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c26d316e-96e5-4678-a904-71942982a97b-service-ca" (OuterVolumeSpecName: "service-ca") pod "c26d316e-96e5-4678-a904-71942982a97b" (UID: "c26d316e-96e5-4678-a904-71942982a97b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:11:48.859177 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:48.859151 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c26d316e-96e5-4678-a904-71942982a97b-console-config" (OuterVolumeSpecName: "console-config") pod "c26d316e-96e5-4678-a904-71942982a97b" (UID: "c26d316e-96e5-4678-a904-71942982a97b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:11:48.859339 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:48.859220 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c26d316e-96e5-4678-a904-71942982a97b-console-config\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:11:48.859339 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:48.859238 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c26d316e-96e5-4678-a904-71942982a97b-service-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:11:48.859419 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:48.859387 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c26d316e-96e5-4678-a904-71942982a97b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c26d316e-96e5-4678-a904-71942982a97b" (UID: "c26d316e-96e5-4678-a904-71942982a97b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:11:48.860917 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:48.860887 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c26d316e-96e5-4678-a904-71942982a97b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c26d316e-96e5-4678-a904-71942982a97b" (UID: "c26d316e-96e5-4678-a904-71942982a97b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 04 22:11:48.860999 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:48.860917 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c26d316e-96e5-4678-a904-71942982a97b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c26d316e-96e5-4678-a904-71942982a97b" (UID: "c26d316e-96e5-4678-a904-71942982a97b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 04 22:11:48.860999 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:48.860928 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c26d316e-96e5-4678-a904-71942982a97b-kube-api-access-wb67k" (OuterVolumeSpecName: "kube-api-access-wb67k") pod "c26d316e-96e5-4678-a904-71942982a97b" (UID: "c26d316e-96e5-4678-a904-71942982a97b"). InnerVolumeSpecName "kube-api-access-wb67k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:11:48.959639 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:48.959616 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c26d316e-96e5-4678-a904-71942982a97b-console-oauth-config\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:11:48.959639 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:48.959637 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c26d316e-96e5-4678-a904-71942982a97b-console-serving-cert\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:11:48.959771 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:48.959647 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb67k\" (UniqueName: \"kubernetes.io/projected/c26d316e-96e5-4678-a904-71942982a97b-kube-api-access-wb67k\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:11:48.959771 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:48.959656 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c26d316e-96e5-4678-a904-71942982a97b-oauth-serving-cert\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:11:49.559062 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:49.559026 2574 generic.go:356] "Generic (PLEG): container finished" podID="c26d316e-96e5-4678-a904-71942982a97b" containerID="1be61c3ef3e552259e855824e874e96994941a48286d5edbab7713080b636c5e" exitCode=0 Sep 04 22:11:49.559244 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:49.559071 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d78c49dbf-4qqfn" Sep 04 22:11:49.559244 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:49.559073 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d78c49dbf-4qqfn" event={"ID":"c26d316e-96e5-4678-a904-71942982a97b","Type":"ContainerDied","Data":"1be61c3ef3e552259e855824e874e96994941a48286d5edbab7713080b636c5e"} Sep 04 22:11:49.559244 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:49.559203 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d78c49dbf-4qqfn" event={"ID":"c26d316e-96e5-4678-a904-71942982a97b","Type":"ContainerDied","Data":"fe30cf6dd16819baacb9f7e5a4b870bed128078ce22033b418ba015244cf5df4"} Sep 04 22:11:49.559244 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:49.559237 2574 scope.go:117] "RemoveContainer" containerID="1be61c3ef3e552259e855824e874e96994941a48286d5edbab7713080b636c5e" Sep 04 22:11:49.574396 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:49.574379 2574 scope.go:117] "RemoveContainer" containerID="1be61c3ef3e552259e855824e874e96994941a48286d5edbab7713080b636c5e" Sep 04 22:11:49.574713 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:11:49.574684 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1be61c3ef3e552259e855824e874e96994941a48286d5edbab7713080b636c5e\": container with ID starting with 1be61c3ef3e552259e855824e874e96994941a48286d5edbab7713080b636c5e not found: ID does not exist" containerID="1be61c3ef3e552259e855824e874e96994941a48286d5edbab7713080b636c5e" Sep 04 22:11:49.574765 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:49.574717 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1be61c3ef3e552259e855824e874e96994941a48286d5edbab7713080b636c5e"} err="failed to get container status \"1be61c3ef3e552259e855824e874e96994941a48286d5edbab7713080b636c5e\": rpc error: code = NotFound desc = could not find container \"1be61c3ef3e552259e855824e874e96994941a48286d5edbab7713080b636c5e\": container with ID starting with 1be61c3ef3e552259e855824e874e96994941a48286d5edbab7713080b636c5e not found: ID does not exist" Sep 04 22:11:49.582942 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:49.582919 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d78c49dbf-4qqfn"] Sep 04 22:11:49.585149 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:49.585129 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7d78c49dbf-4qqfn"] Sep 04 22:11:50.712963 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:11:50.712932 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c26d316e-96e5-4678-a904-71942982a97b" path="/var/lib/kubelet/pods/c26d316e-96e5-4678-a904-71942982a97b/volumes" Sep 04 22:12:14.083996 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:14.083965 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-68978fb68d-q58gl"] Sep 04 22:12:14.084366 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:14.084184 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c26d316e-96e5-4678-a904-71942982a97b" containerName="console" Sep 04 22:12:14.084366 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:14.084194 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c26d316e-96e5-4678-a904-71942982a97b" containerName="console" Sep 04 22:12:14.084366 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:14.084241 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c26d316e-96e5-4678-a904-71942982a97b" containerName="console" Sep 04 22:12:14.087102 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:14.087082 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68978fb68d-q58gl" Sep 04 22:12:14.098855 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:14.098831 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68978fb68d-q58gl"] Sep 04 22:12:14.247468 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:14.247433 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpfwz\" (UniqueName: \"kubernetes.io/projected/6b7a5ce0-493e-482b-8243-71e943b0fe41-kube-api-access-bpfwz\") pod \"console-68978fb68d-q58gl\" (UID: \"6b7a5ce0-493e-482b-8243-71e943b0fe41\") " pod="openshift-console/console-68978fb68d-q58gl" Sep 04 22:12:14.247632 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:14.247474 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b7a5ce0-493e-482b-8243-71e943b0fe41-service-ca\") pod \"console-68978fb68d-q58gl\" (UID: \"6b7a5ce0-493e-482b-8243-71e943b0fe41\") " pod="openshift-console/console-68978fb68d-q58gl" Sep 04 22:12:14.247632 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:14.247494 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b7a5ce0-493e-482b-8243-71e943b0fe41-trusted-ca-bundle\") pod \"console-68978fb68d-q58gl\" (UID: \"6b7a5ce0-493e-482b-8243-71e943b0fe41\") " pod="openshift-console/console-68978fb68d-q58gl" Sep 04 22:12:14.247632 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:14.247563 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b7a5ce0-493e-482b-8243-71e943b0fe41-console-config\") pod \"console-68978fb68d-q58gl\" (UID: \"6b7a5ce0-493e-482b-8243-71e943b0fe41\") " pod="openshift-console/console-68978fb68d-q58gl" Sep 04 22:12:14.247632 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:14.247591 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b7a5ce0-493e-482b-8243-71e943b0fe41-console-oauth-config\") pod \"console-68978fb68d-q58gl\" (UID: \"6b7a5ce0-493e-482b-8243-71e943b0fe41\") " pod="openshift-console/console-68978fb68d-q58gl" Sep 04 22:12:14.247785 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:14.247647 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b7a5ce0-493e-482b-8243-71e943b0fe41-oauth-serving-cert\") pod \"console-68978fb68d-q58gl\" (UID: \"6b7a5ce0-493e-482b-8243-71e943b0fe41\") " pod="openshift-console/console-68978fb68d-q58gl" Sep 04 22:12:14.247785 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:14.247681 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b7a5ce0-493e-482b-8243-71e943b0fe41-console-serving-cert\") pod \"console-68978fb68d-q58gl\" (UID: \"6b7a5ce0-493e-482b-8243-71e943b0fe41\") " pod="openshift-console/console-68978fb68d-q58gl" Sep 04 22:12:14.348949 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:14.348866 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpfwz\" (UniqueName: \"kubernetes.io/projected/6b7a5ce0-493e-482b-8243-71e943b0fe41-kube-api-access-bpfwz\") pod \"console-68978fb68d-q58gl\" (UID: \"6b7a5ce0-493e-482b-8243-71e943b0fe41\") " pod="openshift-console/console-68978fb68d-q58gl" Sep 04 22:12:14.348949 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:14.348912 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b7a5ce0-493e-482b-8243-71e943b0fe41-service-ca\") pod \"console-68978fb68d-q58gl\" (UID: \"6b7a5ce0-493e-482b-8243-71e943b0fe41\") " pod="openshift-console/console-68978fb68d-q58gl" Sep 04 22:12:14.348949 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:14.348933 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b7a5ce0-493e-482b-8243-71e943b0fe41-trusted-ca-bundle\") pod \"console-68978fb68d-q58gl\" (UID: \"6b7a5ce0-493e-482b-8243-71e943b0fe41\") " pod="openshift-console/console-68978fb68d-q58gl" Sep 04 22:12:14.349175 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:14.348958 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b7a5ce0-493e-482b-8243-71e943b0fe41-console-config\") pod \"console-68978fb68d-q58gl\" (UID: \"6b7a5ce0-493e-482b-8243-71e943b0fe41\") " pod="openshift-console/console-68978fb68d-q58gl" Sep 04 22:12:14.349175 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:14.348975 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b7a5ce0-493e-482b-8243-71e943b0fe41-console-oauth-config\") pod \"console-68978fb68d-q58gl\" (UID: \"6b7a5ce0-493e-482b-8243-71e943b0fe41\") " pod="openshift-console/console-68978fb68d-q58gl" Sep 04 22:12:14.349175 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:14.349000 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b7a5ce0-493e-482b-8243-71e943b0fe41-oauth-serving-cert\") pod \"console-68978fb68d-q58gl\" (UID: \"6b7a5ce0-493e-482b-8243-71e943b0fe41\") " pod="openshift-console/console-68978fb68d-q58gl" Sep 04 22:12:14.349175 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:14.349018 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b7a5ce0-493e-482b-8243-71e943b0fe41-console-serving-cert\") pod \"console-68978fb68d-q58gl\" (UID: \"6b7a5ce0-493e-482b-8243-71e943b0fe41\") " pod="openshift-console/console-68978fb68d-q58gl" Sep 04 22:12:14.349770 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:14.349744 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b7a5ce0-493e-482b-8243-71e943b0fe41-console-config\") pod \"console-68978fb68d-q58gl\" (UID: \"6b7a5ce0-493e-482b-8243-71e943b0fe41\") " pod="openshift-console/console-68978fb68d-q58gl" Sep 04 22:12:14.349770 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:14.349759 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b7a5ce0-493e-482b-8243-71e943b0fe41-service-ca\") pod \"console-68978fb68d-q58gl\" (UID: \"6b7a5ce0-493e-482b-8243-71e943b0fe41\") " pod="openshift-console/console-68978fb68d-q58gl" Sep 04 22:12:14.349945 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:14.349852 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b7a5ce0-493e-482b-8243-71e943b0fe41-oauth-serving-cert\") pod \"console-68978fb68d-q58gl\" (UID: \"6b7a5ce0-493e-482b-8243-71e943b0fe41\") " pod="openshift-console/console-68978fb68d-q58gl" Sep 04 22:12:14.349945 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:14.349926 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b7a5ce0-493e-482b-8243-71e943b0fe41-trusted-ca-bundle\") pod \"console-68978fb68d-q58gl\" (UID: \"6b7a5ce0-493e-482b-8243-71e943b0fe41\") " pod="openshift-console/console-68978fb68d-q58gl" Sep 04 22:12:14.351496 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:14.351466 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b7a5ce0-493e-482b-8243-71e943b0fe41-console-oauth-config\") pod \"console-68978fb68d-q58gl\" (UID: \"6b7a5ce0-493e-482b-8243-71e943b0fe41\") " pod="openshift-console/console-68978fb68d-q58gl" Sep 04 22:12:14.351610 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:14.351574 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b7a5ce0-493e-482b-8243-71e943b0fe41-console-serving-cert\") pod \"console-68978fb68d-q58gl\" (UID: \"6b7a5ce0-493e-482b-8243-71e943b0fe41\") " pod="openshift-console/console-68978fb68d-q58gl" Sep 04 22:12:14.357619 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:14.357600 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpfwz\" (UniqueName: \"kubernetes.io/projected/6b7a5ce0-493e-482b-8243-71e943b0fe41-kube-api-access-bpfwz\") pod \"console-68978fb68d-q58gl\" (UID: \"6b7a5ce0-493e-482b-8243-71e943b0fe41\") " pod="openshift-console/console-68978fb68d-q58gl" Sep 04 22:12:14.395081 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:14.395055 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68978fb68d-q58gl" Sep 04 22:12:14.666391 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:14.666320 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68978fb68d-q58gl"] Sep 04 22:12:14.669265 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:12:14.669236 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b7a5ce0_493e_482b_8243_71e943b0fe41.slice/crio-16cfcf52e394d500ede947ccd8642797c58f9593d2b3f0ca28f699f988b50ed5 WatchSource:0}: Error finding container 16cfcf52e394d500ede947ccd8642797c58f9593d2b3f0ca28f699f988b50ed5: Status 404 returned error can't find the container with id 16cfcf52e394d500ede947ccd8642797c58f9593d2b3f0ca28f699f988b50ed5 Sep 04 22:12:15.640638 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:15.640599 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68978fb68d-q58gl" event={"ID":"6b7a5ce0-493e-482b-8243-71e943b0fe41","Type":"ContainerStarted","Data":"1d497be00f214c718ac83c9d9f88108df5b2aace9ba98377fd409b047d88eac3"} Sep 04 22:12:15.641007 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:15.640683 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68978fb68d-q58gl" event={"ID":"6b7a5ce0-493e-482b-8243-71e943b0fe41","Type":"ContainerStarted","Data":"16cfcf52e394d500ede947ccd8642797c58f9593d2b3f0ca28f699f988b50ed5"} Sep 04 22:12:15.665038 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:15.664989 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-68978fb68d-q58gl" podStartSLOduration=1.664972842 podStartE2EDuration="1.664972842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:12:15.663058741 +0000 UTC m=+239.427534353" watchObservedRunningTime="2026-09-04 22:12:15.664972842 +0000 UTC m=+239.429448448" Sep 04 22:12:24.396024 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:24.395985 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-68978fb68d-q58gl" Sep 04 22:12:24.396476 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:24.396049 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68978fb68d-q58gl" Sep 04 22:12:24.400578 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:24.400536 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-68978fb68d-q58gl" Sep 04 22:12:24.676981 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:24.676902 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-68978fb68d-q58gl" Sep 04 22:12:24.724166 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:24.724135 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c5dfdb68-8rflk"] Sep 04 22:12:28.649670 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:28.649614 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0972118-6d5c-4d3a-a979-497ab05ca339-metrics-certs\") pod \"network-metrics-daemon-qvwc7\" (UID: \"c0972118-6d5c-4d3a-a979-497ab05ca339\") " pod="openshift-multus/network-metrics-daemon-qvwc7" Sep 04 22:12:28.651985 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:28.651959 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0972118-6d5c-4d3a-a979-497ab05ca339-metrics-certs\") pod \"network-metrics-daemon-qvwc7\" (UID: \"c0972118-6d5c-4d3a-a979-497ab05ca339\") " pod="openshift-multus/network-metrics-daemon-qvwc7" Sep 04 22:12:28.813070 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:28.813030 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-c64pf\"" Sep 04 22:12:28.821132 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:28.821109 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qvwc7" Sep 04 22:12:29.097146 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:29.097114 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qvwc7"] Sep 04 22:12:29.100889 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:12:29.100858 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0972118_6d5c_4d3a_a979_497ab05ca339.slice/crio-ac19c9e5d392d1674fcc5c27a4edfa9457f4018ea98607d5df5e49bba077d667 WatchSource:0}: Error finding container ac19c9e5d392d1674fcc5c27a4edfa9457f4018ea98607d5df5e49bba077d667: Status 404 returned error can't find the container with id ac19c9e5d392d1674fcc5c27a4edfa9457f4018ea98607d5df5e49bba077d667 Sep 04 22:12:29.689944 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:29.689904 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qvwc7" event={"ID":"c0972118-6d5c-4d3a-a979-497ab05ca339","Type":"ContainerStarted","Data":"ac19c9e5d392d1674fcc5c27a4edfa9457f4018ea98607d5df5e49bba077d667"} Sep 04 22:12:30.694668 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:30.694630 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qvwc7" event={"ID":"c0972118-6d5c-4d3a-a979-497ab05ca339","Type":"ContainerStarted","Data":"c4e0674f134799259e86c6a9999adb79937fe8763fe7f001a5db9e5799405a11"} Sep 04 22:12:30.695060 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:30.694685 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qvwc7" event={"ID":"c0972118-6d5c-4d3a-a979-497ab05ca339","Type":"ContainerStarted","Data":"0bf96aa160b154b36fda9aabb982eebfa3c80a5828541d7c4d8fd254c2711ab1"} Sep 04 22:12:30.711881 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:30.711833 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qvwc7" podStartSLOduration=252.665838658 podStartE2EDuration="4m13.711817724s" totalImagesPullingTime="1.045979066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:08:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:12:30.711383119 +0000 UTC m=+254.475858730" watchObservedRunningTime="2026-09-04 22:12:30.711817724 +0000 UTC m=+254.476293368" Sep 04 22:12:49.745880 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:49.745777 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7c5dfdb68-8rflk" podUID="a06b2a76-9cfa-4dc9-9446-3c07b7250fea" containerName="console" containerID="cri-o://d3ee6983a97904627e51b84eba99177b439da6dab342d2e1a1d8a83c3c9abf50" gracePeriod=15 Sep 04 22:12:49.993395 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:49.993369 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c5dfdb68-8rflk" Sep 04 22:12:50.111853 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:50.111820 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-skr2f\" (UniqueName: \"kubernetes.io/projected/a06b2a76-9cfa-4dc9-9446-3c07b7250fea-kube-api-access-skr2f\") pod \"a06b2a76-9cfa-4dc9-9446-3c07b7250fea\" (UID: \"a06b2a76-9cfa-4dc9-9446-3c07b7250fea\") " Sep 04 22:12:50.112067 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:50.111889 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a06b2a76-9cfa-4dc9-9446-3c07b7250fea-oauth-serving-cert\") pod \"a06b2a76-9cfa-4dc9-9446-3c07b7250fea\" (UID: \"a06b2a76-9cfa-4dc9-9446-3c07b7250fea\") " Sep 04 22:12:50.112067 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:50.111923 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a06b2a76-9cfa-4dc9-9446-3c07b7250fea-trusted-ca-bundle\") pod \"a06b2a76-9cfa-4dc9-9446-3c07b7250fea\" (UID: \"a06b2a76-9cfa-4dc9-9446-3c07b7250fea\") " Sep 04 22:12:50.112067 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:50.111960 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a06b2a76-9cfa-4dc9-9446-3c07b7250fea-service-ca\") pod \"a06b2a76-9cfa-4dc9-9446-3c07b7250fea\" (UID: \"a06b2a76-9cfa-4dc9-9446-3c07b7250fea\") " Sep 04 22:12:50.112067 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:50.111981 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a06b2a76-9cfa-4dc9-9446-3c07b7250fea-console-config\") pod \"a06b2a76-9cfa-4dc9-9446-3c07b7250fea\" (UID: \"a06b2a76-9cfa-4dc9-9446-3c07b7250fea\") " Sep 04 22:12:50.112067 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:50.112019 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a06b2a76-9cfa-4dc9-9446-3c07b7250fea-console-serving-cert\") pod \"a06b2a76-9cfa-4dc9-9446-3c07b7250fea\" (UID: \"a06b2a76-9cfa-4dc9-9446-3c07b7250fea\") " Sep 04 22:12:50.112322 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:50.112154 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a06b2a76-9cfa-4dc9-9446-3c07b7250fea-console-oauth-config\") pod \"a06b2a76-9cfa-4dc9-9446-3c07b7250fea\" (UID: \"a06b2a76-9cfa-4dc9-9446-3c07b7250fea\") " Sep 04 22:12:50.112383 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:50.112353 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a06b2a76-9cfa-4dc9-9446-3c07b7250fea-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a06b2a76-9cfa-4dc9-9446-3c07b7250fea" (UID: "a06b2a76-9cfa-4dc9-9446-3c07b7250fea"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:12:50.112438 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:50.112400 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a06b2a76-9cfa-4dc9-9446-3c07b7250fea-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a06b2a76-9cfa-4dc9-9446-3c07b7250fea" (UID: "a06b2a76-9cfa-4dc9-9446-3c07b7250fea"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:12:50.112494 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:50.112426 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a06b2a76-9cfa-4dc9-9446-3c07b7250fea-service-ca" (OuterVolumeSpecName: "service-ca") pod "a06b2a76-9cfa-4dc9-9446-3c07b7250fea" (UID: "a06b2a76-9cfa-4dc9-9446-3c07b7250fea"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:12:50.112572 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:50.112489 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a06b2a76-9cfa-4dc9-9446-3c07b7250fea-console-config" (OuterVolumeSpecName: "console-config") pod "a06b2a76-9cfa-4dc9-9446-3c07b7250fea" (UID: "a06b2a76-9cfa-4dc9-9446-3c07b7250fea"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:12:50.114210 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:50.114173 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a06b2a76-9cfa-4dc9-9446-3c07b7250fea-kube-api-access-skr2f" (OuterVolumeSpecName: "kube-api-access-skr2f") pod "a06b2a76-9cfa-4dc9-9446-3c07b7250fea" (UID: "a06b2a76-9cfa-4dc9-9446-3c07b7250fea"). InnerVolumeSpecName "kube-api-access-skr2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:12:50.114315 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:50.114222 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a06b2a76-9cfa-4dc9-9446-3c07b7250fea-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a06b2a76-9cfa-4dc9-9446-3c07b7250fea" (UID: "a06b2a76-9cfa-4dc9-9446-3c07b7250fea"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 04 22:12:50.114315 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:50.114292 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a06b2a76-9cfa-4dc9-9446-3c07b7250fea-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a06b2a76-9cfa-4dc9-9446-3c07b7250fea" (UID: "a06b2a76-9cfa-4dc9-9446-3c07b7250fea"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 04 22:12:50.213573 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:50.213518 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a06b2a76-9cfa-4dc9-9446-3c07b7250fea-oauth-serving-cert\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:12:50.213573 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:50.213572 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a06b2a76-9cfa-4dc9-9446-3c07b7250fea-trusted-ca-bundle\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:12:50.213573 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:50.213582 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a06b2a76-9cfa-4dc9-9446-3c07b7250fea-service-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:12:50.213795 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:50.213593 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a06b2a76-9cfa-4dc9-9446-3c07b7250fea-console-config\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:12:50.213795 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:50.213600 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a06b2a76-9cfa-4dc9-9446-3c07b7250fea-console-serving-cert\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:12:50.213795 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:50.213608 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a06b2a76-9cfa-4dc9-9446-3c07b7250fea-console-oauth-config\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:12:50.213795 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:50.213616 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skr2f\" (UniqueName: \"kubernetes.io/projected/a06b2a76-9cfa-4dc9-9446-3c07b7250fea-kube-api-access-skr2f\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:12:50.764193 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:50.764155 2574 generic.go:356] "Generic (PLEG): container finished" podID="a06b2a76-9cfa-4dc9-9446-3c07b7250fea" containerID="d3ee6983a97904627e51b84eba99177b439da6dab342d2e1a1d8a83c3c9abf50" exitCode=0 Sep 04 22:12:50.764593 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:50.764204 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c5dfdb68-8rflk" Sep 04 22:12:50.764593 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:50.764245 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c5dfdb68-8rflk" event={"ID":"a06b2a76-9cfa-4dc9-9446-3c07b7250fea","Type":"ContainerDied","Data":"d3ee6983a97904627e51b84eba99177b439da6dab342d2e1a1d8a83c3c9abf50"} Sep 04 22:12:50.764593 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:50.764306 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c5dfdb68-8rflk" event={"ID":"a06b2a76-9cfa-4dc9-9446-3c07b7250fea","Type":"ContainerDied","Data":"5ba628296c150671b506beedd5958aba7b455b8be5a652d6cdaad75d94c7c455"} Sep 04 22:12:50.764593 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:50.764329 2574 scope.go:117] "RemoveContainer" containerID="d3ee6983a97904627e51b84eba99177b439da6dab342d2e1a1d8a83c3c9abf50" Sep 04 22:12:50.772618 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:50.772598 2574 scope.go:117] "RemoveContainer" containerID="d3ee6983a97904627e51b84eba99177b439da6dab342d2e1a1d8a83c3c9abf50" Sep 04 22:12:50.772919 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:12:50.772895 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3ee6983a97904627e51b84eba99177b439da6dab342d2e1a1d8a83c3c9abf50\": container with ID starting with d3ee6983a97904627e51b84eba99177b439da6dab342d2e1a1d8a83c3c9abf50 not found: ID does not exist" containerID="d3ee6983a97904627e51b84eba99177b439da6dab342d2e1a1d8a83c3c9abf50" Sep 04 22:12:50.773006 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:50.772927 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3ee6983a97904627e51b84eba99177b439da6dab342d2e1a1d8a83c3c9abf50"} err="failed to get container status \"d3ee6983a97904627e51b84eba99177b439da6dab342d2e1a1d8a83c3c9abf50\": rpc error: code = NotFound desc = could not find container \"d3ee6983a97904627e51b84eba99177b439da6dab342d2e1a1d8a83c3c9abf50\": container with ID starting with d3ee6983a97904627e51b84eba99177b439da6dab342d2e1a1d8a83c3c9abf50 not found: ID does not exist" Sep 04 22:12:50.784806 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:50.784773 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c5dfdb68-8rflk"] Sep 04 22:12:50.789228 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:50.789198 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c5dfdb68-8rflk"] Sep 04 22:12:52.713659 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:12:52.713626 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a06b2a76-9cfa-4dc9-9446-3c07b7250fea" path="/var/lib/kubelet/pods/a06b2a76-9cfa-4dc9-9446-3c07b7250fea/volumes" Sep 04 22:13:16.657159 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:13:16.657128 2574 kubelet.go:1648] "Image garbage collection succeeded" Sep 04 22:13:43.047693 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:13:43.047656 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d6dfff4f-cjfdl"] Sep 04 22:13:43.050107 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:13:43.047889 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a06b2a76-9cfa-4dc9-9446-3c07b7250fea" containerName="console" Sep 04 22:13:43.050107 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:13:43.047901 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a06b2a76-9cfa-4dc9-9446-3c07b7250fea" containerName="console" Sep 04 22:13:43.050107 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:13:43.047940 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a06b2a76-9cfa-4dc9-9446-3c07b7250fea" containerName="console" Sep 04 22:13:43.051000 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:13:43.050980 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d6dfff4f-cjfdl" Sep 04 22:13:43.063128 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:13:43.063106 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d6dfff4f-cjfdl"] Sep 04 22:13:43.184934 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:13:43.184896 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32c17fdf-2303-4355-bf1f-8ecc1062f7f4-console-config\") pod \"console-6d6dfff4f-cjfdl\" (UID: \"32c17fdf-2303-4355-bf1f-8ecc1062f7f4\") " pod="openshift-console/console-6d6dfff4f-cjfdl" Sep 04 22:13:43.184934 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:13:43.184937 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32c17fdf-2303-4355-bf1f-8ecc1062f7f4-oauth-serving-cert\") pod \"console-6d6dfff4f-cjfdl\" (UID: \"32c17fdf-2303-4355-bf1f-8ecc1062f7f4\") " pod="openshift-console/console-6d6dfff4f-cjfdl" Sep 04 22:13:43.185151 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:13:43.184962 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32c17fdf-2303-4355-bf1f-8ecc1062f7f4-console-oauth-config\") pod \"console-6d6dfff4f-cjfdl\" (UID: \"32c17fdf-2303-4355-bf1f-8ecc1062f7f4\") " pod="openshift-console/console-6d6dfff4f-cjfdl" Sep 04 22:13:43.185151 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:13:43.185010 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32c17fdf-2303-4355-bf1f-8ecc1062f7f4-service-ca\") pod \"console-6d6dfff4f-cjfdl\" (UID: \"32c17fdf-2303-4355-bf1f-8ecc1062f7f4\") " pod="openshift-console/console-6d6dfff4f-cjfdl" Sep 04 22:13:43.185151 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:13:43.185054 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32c17fdf-2303-4355-bf1f-8ecc1062f7f4-trusted-ca-bundle\") pod \"console-6d6dfff4f-cjfdl\" (UID: \"32c17fdf-2303-4355-bf1f-8ecc1062f7f4\") " pod="openshift-console/console-6d6dfff4f-cjfdl" Sep 04 22:13:43.185151 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:13:43.185083 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32c17fdf-2303-4355-bf1f-8ecc1062f7f4-console-serving-cert\") pod \"console-6d6dfff4f-cjfdl\" (UID: \"32c17fdf-2303-4355-bf1f-8ecc1062f7f4\") " pod="openshift-console/console-6d6dfff4f-cjfdl" Sep 04 22:13:43.185151 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:13:43.185105 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92wmc\" (UniqueName: \"kubernetes.io/projected/32c17fdf-2303-4355-bf1f-8ecc1062f7f4-kube-api-access-92wmc\") pod \"console-6d6dfff4f-cjfdl\" (UID: \"32c17fdf-2303-4355-bf1f-8ecc1062f7f4\") " pod="openshift-console/console-6d6dfff4f-cjfdl" Sep 04 22:13:43.286029 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:13:43.285993 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32c17fdf-2303-4355-bf1f-8ecc1062f7f4-console-config\") pod \"console-6d6dfff4f-cjfdl\" (UID: \"32c17fdf-2303-4355-bf1f-8ecc1062f7f4\") " pod="openshift-console/console-6d6dfff4f-cjfdl" Sep 04 22:13:43.286029 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:13:43.286030 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32c17fdf-2303-4355-bf1f-8ecc1062f7f4-oauth-serving-cert\") pod \"console-6d6dfff4f-cjfdl\" (UID: \"32c17fdf-2303-4355-bf1f-8ecc1062f7f4\") " pod="openshift-console/console-6d6dfff4f-cjfdl" Sep 04 22:13:43.286301 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:13:43.286053 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32c17fdf-2303-4355-bf1f-8ecc1062f7f4-console-oauth-config\") pod \"console-6d6dfff4f-cjfdl\" (UID: \"32c17fdf-2303-4355-bf1f-8ecc1062f7f4\") " pod="openshift-console/console-6d6dfff4f-cjfdl" Sep 04 22:13:43.286301 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:13:43.286067 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32c17fdf-2303-4355-bf1f-8ecc1062f7f4-service-ca\") pod \"console-6d6dfff4f-cjfdl\" (UID: \"32c17fdf-2303-4355-bf1f-8ecc1062f7f4\") " pod="openshift-console/console-6d6dfff4f-cjfdl" Sep 04 22:13:43.286301 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:13:43.286087 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32c17fdf-2303-4355-bf1f-8ecc1062f7f4-trusted-ca-bundle\") pod \"console-6d6dfff4f-cjfdl\" (UID: \"32c17fdf-2303-4355-bf1f-8ecc1062f7f4\") " pod="openshift-console/console-6d6dfff4f-cjfdl" Sep 04 22:13:43.286301 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:13:43.286114 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32c17fdf-2303-4355-bf1f-8ecc1062f7f4-console-serving-cert\") pod \"console-6d6dfff4f-cjfdl\" (UID: \"32c17fdf-2303-4355-bf1f-8ecc1062f7f4\") " pod="openshift-console/console-6d6dfff4f-cjfdl" Sep 04 22:13:43.286301 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:13:43.286131 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92wmc\" (UniqueName: \"kubernetes.io/projected/32c17fdf-2303-4355-bf1f-8ecc1062f7f4-kube-api-access-92wmc\") pod \"console-6d6dfff4f-cjfdl\" (UID: \"32c17fdf-2303-4355-bf1f-8ecc1062f7f4\") " pod="openshift-console/console-6d6dfff4f-cjfdl" Sep 04 22:13:43.286910 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:13:43.286882 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32c17fdf-2303-4355-bf1f-8ecc1062f7f4-service-ca\") pod \"console-6d6dfff4f-cjfdl\" (UID: \"32c17fdf-2303-4355-bf1f-8ecc1062f7f4\") " pod="openshift-console/console-6d6dfff4f-cjfdl" Sep 04 22:13:43.287038 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:13:43.286907 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32c17fdf-2303-4355-bf1f-8ecc1062f7f4-oauth-serving-cert\") pod \"console-6d6dfff4f-cjfdl\" (UID: \"32c17fdf-2303-4355-bf1f-8ecc1062f7f4\") " pod="openshift-console/console-6d6dfff4f-cjfdl" Sep 04 22:13:43.287038 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:13:43.286954 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32c17fdf-2303-4355-bf1f-8ecc1062f7f4-console-config\") pod \"console-6d6dfff4f-cjfdl\" (UID: \"32c17fdf-2303-4355-bf1f-8ecc1062f7f4\") " pod="openshift-console/console-6d6dfff4f-cjfdl" Sep 04 22:13:43.287168 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:13:43.287080 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32c17fdf-2303-4355-bf1f-8ecc1062f7f4-trusted-ca-bundle\") pod \"console-6d6dfff4f-cjfdl\" (UID: \"32c17fdf-2303-4355-bf1f-8ecc1062f7f4\") " pod="openshift-console/console-6d6dfff4f-cjfdl" Sep 04 22:13:43.288602 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:13:43.288581 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32c17fdf-2303-4355-bf1f-8ecc1062f7f4-console-serving-cert\") pod \"console-6d6dfff4f-cjfdl\" (UID: \"32c17fdf-2303-4355-bf1f-8ecc1062f7f4\") " pod="openshift-console/console-6d6dfff4f-cjfdl" Sep 04 22:13:43.288775 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:13:43.288760 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32c17fdf-2303-4355-bf1f-8ecc1062f7f4-console-oauth-config\") pod \"console-6d6dfff4f-cjfdl\" (UID: \"32c17fdf-2303-4355-bf1f-8ecc1062f7f4\") " pod="openshift-console/console-6d6dfff4f-cjfdl" Sep 04 22:13:43.294520 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:13:43.294501 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92wmc\" (UniqueName: \"kubernetes.io/projected/32c17fdf-2303-4355-bf1f-8ecc1062f7f4-kube-api-access-92wmc\") pod \"console-6d6dfff4f-cjfdl\" (UID: \"32c17fdf-2303-4355-bf1f-8ecc1062f7f4\") " pod="openshift-console/console-6d6dfff4f-cjfdl" Sep 04 22:13:43.360449 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:13:43.360389 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d6dfff4f-cjfdl" Sep 04 22:13:43.629803 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:13:43.629778 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d6dfff4f-cjfdl"] Sep 04 22:13:43.632213 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:13:43.632184 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32c17fdf_2303_4355_bf1f_8ecc1062f7f4.slice/crio-d223f3e4774f79264965cf3ab15e86c0a77c2823ab1ebf8f65eeb99203ad5ac1 WatchSource:0}: Error finding container d223f3e4774f79264965cf3ab15e86c0a77c2823ab1ebf8f65eeb99203ad5ac1: Status 404 returned error can't find the container with id d223f3e4774f79264965cf3ab15e86c0a77c2823ab1ebf8f65eeb99203ad5ac1 Sep 04 22:13:43.937659 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:13:43.937543 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d6dfff4f-cjfdl" event={"ID":"32c17fdf-2303-4355-bf1f-8ecc1062f7f4","Type":"ContainerStarted","Data":"3772a3f4445460961d455916a1cf92d0632effb559e0d199edf9a7583e272735"} Sep 04 22:13:43.937659 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:13:43.937628 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d6dfff4f-cjfdl" event={"ID":"32c17fdf-2303-4355-bf1f-8ecc1062f7f4","Type":"ContainerStarted","Data":"d223f3e4774f79264965cf3ab15e86c0a77c2823ab1ebf8f65eeb99203ad5ac1"} Sep 04 22:13:43.955862 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:13:43.955810 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d6dfff4f-cjfdl" podStartSLOduration=0.955792536 podStartE2EDuration="955.792536ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:13:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:13:43.955231494 +0000 UTC m=+327.719707109" watchObservedRunningTime="2026-09-04 22:13:43.955792536 +0000 UTC m=+327.720268202" Sep 04 22:13:53.360910 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:13:53.360863 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d6dfff4f-cjfdl" Sep 04 22:13:53.361488 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:13:53.360947 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d6dfff4f-cjfdl" Sep 04 22:13:53.365605 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:13:53.365584 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d6dfff4f-cjfdl" Sep 04 22:13:53.974176 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:13:53.974146 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d6dfff4f-cjfdl" Sep 04 22:13:54.020947 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:13:54.020916 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68978fb68d-q58gl"] Sep 04 22:14:19.042015 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:19.041913 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-68978fb68d-q58gl" podUID="6b7a5ce0-493e-482b-8243-71e943b0fe41" containerName="console" containerID="cri-o://1d497be00f214c718ac83c9d9f88108df5b2aace9ba98377fd409b047d88eac3" gracePeriod=15 Sep 04 22:14:19.279730 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:19.279706 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68978fb68d-q58gl" Sep 04 22:14:19.346107 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:19.346028 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b7a5ce0-493e-482b-8243-71e943b0fe41-trusted-ca-bundle\") pod \"6b7a5ce0-493e-482b-8243-71e943b0fe41\" (UID: \"6b7a5ce0-493e-482b-8243-71e943b0fe41\") " Sep 04 22:14:19.346107 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:19.346071 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bpfwz\" (UniqueName: \"kubernetes.io/projected/6b7a5ce0-493e-482b-8243-71e943b0fe41-kube-api-access-bpfwz\") pod \"6b7a5ce0-493e-482b-8243-71e943b0fe41\" (UID: \"6b7a5ce0-493e-482b-8243-71e943b0fe41\") " Sep 04 22:14:19.346107 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:19.346106 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b7a5ce0-493e-482b-8243-71e943b0fe41-service-ca\") pod \"6b7a5ce0-493e-482b-8243-71e943b0fe41\" (UID: \"6b7a5ce0-493e-482b-8243-71e943b0fe41\") " Sep 04 22:14:19.346373 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:19.346133 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b7a5ce0-493e-482b-8243-71e943b0fe41-console-serving-cert\") pod \"6b7a5ce0-493e-482b-8243-71e943b0fe41\" (UID: \"6b7a5ce0-493e-482b-8243-71e943b0fe41\") " Sep 04 22:14:19.346373 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:19.346163 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b7a5ce0-493e-482b-8243-71e943b0fe41-console-oauth-config\") pod \"6b7a5ce0-493e-482b-8243-71e943b0fe41\" (UID: \"6b7a5ce0-493e-482b-8243-71e943b0fe41\") " Sep 04 22:14:19.346373 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:19.346191 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b7a5ce0-493e-482b-8243-71e943b0fe41-console-config\") pod \"6b7a5ce0-493e-482b-8243-71e943b0fe41\" (UID: \"6b7a5ce0-493e-482b-8243-71e943b0fe41\") " Sep 04 22:14:19.346373 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:19.346226 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b7a5ce0-493e-482b-8243-71e943b0fe41-oauth-serving-cert\") pod \"6b7a5ce0-493e-482b-8243-71e943b0fe41\" (UID: \"6b7a5ce0-493e-482b-8243-71e943b0fe41\") " Sep 04 22:14:19.346659 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:19.346608 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b7a5ce0-493e-482b-8243-71e943b0fe41-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6b7a5ce0-493e-482b-8243-71e943b0fe41" (UID: "6b7a5ce0-493e-482b-8243-71e943b0fe41"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:14:19.346749 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:19.346722 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b7a5ce0-493e-482b-8243-71e943b0fe41-service-ca" (OuterVolumeSpecName: "service-ca") pod "6b7a5ce0-493e-482b-8243-71e943b0fe41" (UID: "6b7a5ce0-493e-482b-8243-71e943b0fe41"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:14:19.346802 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:19.346723 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b7a5ce0-493e-482b-8243-71e943b0fe41-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6b7a5ce0-493e-482b-8243-71e943b0fe41" (UID: "6b7a5ce0-493e-482b-8243-71e943b0fe41"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:14:19.346868 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:19.346842 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b7a5ce0-493e-482b-8243-71e943b0fe41-console-config" (OuterVolumeSpecName: "console-config") pod "6b7a5ce0-493e-482b-8243-71e943b0fe41" (UID: "6b7a5ce0-493e-482b-8243-71e943b0fe41"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:14:19.348243 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:19.348222 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b7a5ce0-493e-482b-8243-71e943b0fe41-kube-api-access-bpfwz" (OuterVolumeSpecName: "kube-api-access-bpfwz") pod "6b7a5ce0-493e-482b-8243-71e943b0fe41" (UID: "6b7a5ce0-493e-482b-8243-71e943b0fe41"). InnerVolumeSpecName "kube-api-access-bpfwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:14:19.348335 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:19.348317 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b7a5ce0-493e-482b-8243-71e943b0fe41-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6b7a5ce0-493e-482b-8243-71e943b0fe41" (UID: "6b7a5ce0-493e-482b-8243-71e943b0fe41"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 04 22:14:19.348377 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:19.348340 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b7a5ce0-493e-482b-8243-71e943b0fe41-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6b7a5ce0-493e-482b-8243-71e943b0fe41" (UID: "6b7a5ce0-493e-482b-8243-71e943b0fe41"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 04 22:14:19.446860 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:19.446827 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6b7a5ce0-493e-482b-8243-71e943b0fe41-oauth-serving-cert\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:14:19.446860 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:19.446854 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b7a5ce0-493e-482b-8243-71e943b0fe41-trusted-ca-bundle\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:14:19.446860 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:19.446864 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpfwz\" (UniqueName: \"kubernetes.io/projected/6b7a5ce0-493e-482b-8243-71e943b0fe41-kube-api-access-bpfwz\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:14:19.447055 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:19.446873 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6b7a5ce0-493e-482b-8243-71e943b0fe41-service-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:14:19.447055 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:19.446883 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b7a5ce0-493e-482b-8243-71e943b0fe41-console-serving-cert\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:14:19.447055 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:19.446891 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6b7a5ce0-493e-482b-8243-71e943b0fe41-console-oauth-config\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:14:19.447055 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:19.446901 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6b7a5ce0-493e-482b-8243-71e943b0fe41-console-config\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:14:20.054212 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:20.054172 2574 generic.go:356] "Generic (PLEG): container finished" podID="6b7a5ce0-493e-482b-8243-71e943b0fe41" containerID="1d497be00f214c718ac83c9d9f88108df5b2aace9ba98377fd409b047d88eac3" exitCode=0 Sep 04 22:14:20.054713 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:20.054221 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68978fb68d-q58gl" Sep 04 22:14:20.054713 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:20.054261 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68978fb68d-q58gl" event={"ID":"6b7a5ce0-493e-482b-8243-71e943b0fe41","Type":"ContainerDied","Data":"1d497be00f214c718ac83c9d9f88108df5b2aace9ba98377fd409b047d88eac3"} Sep 04 22:14:20.054713 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:20.054333 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68978fb68d-q58gl" event={"ID":"6b7a5ce0-493e-482b-8243-71e943b0fe41","Type":"ContainerDied","Data":"16cfcf52e394d500ede947ccd8642797c58f9593d2b3f0ca28f699f988b50ed5"} Sep 04 22:14:20.054713 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:20.054367 2574 scope.go:117] "RemoveContainer" containerID="1d497be00f214c718ac83c9d9f88108df5b2aace9ba98377fd409b047d88eac3" Sep 04 22:14:20.062401 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:20.062382 2574 scope.go:117] "RemoveContainer" containerID="1d497be00f214c718ac83c9d9f88108df5b2aace9ba98377fd409b047d88eac3" Sep 04 22:14:20.062718 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:14:20.062698 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d497be00f214c718ac83c9d9f88108df5b2aace9ba98377fd409b047d88eac3\": container with ID starting with 1d497be00f214c718ac83c9d9f88108df5b2aace9ba98377fd409b047d88eac3 not found: ID does not exist" containerID="1d497be00f214c718ac83c9d9f88108df5b2aace9ba98377fd409b047d88eac3" Sep 04 22:14:20.062772 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:20.062729 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d497be00f214c718ac83c9d9f88108df5b2aace9ba98377fd409b047d88eac3"} err="failed to get container status \"1d497be00f214c718ac83c9d9f88108df5b2aace9ba98377fd409b047d88eac3\": rpc error: code = NotFound desc = could not find container \"1d497be00f214c718ac83c9d9f88108df5b2aace9ba98377fd409b047d88eac3\": container with ID starting with 1d497be00f214c718ac83c9d9f88108df5b2aace9ba98377fd409b047d88eac3 not found: ID does not exist" Sep 04 22:14:20.075839 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:20.075814 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68978fb68d-q58gl"] Sep 04 22:14:20.080520 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:20.080497 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-68978fb68d-q58gl"] Sep 04 22:14:20.713926 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:20.713889 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b7a5ce0-493e-482b-8243-71e943b0fe41" path="/var/lib/kubelet/pods/6b7a5ce0-493e-482b-8243-71e943b0fe41/volumes" Sep 04 22:14:22.377770 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:22.377738 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gmtwr"] Sep 04 22:14:22.378133 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:22.377959 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b7a5ce0-493e-482b-8243-71e943b0fe41" containerName="console" Sep 04 22:14:22.378133 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:22.377970 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b7a5ce0-493e-482b-8243-71e943b0fe41" containerName="console" Sep 04 22:14:22.378133 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:22.378009 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b7a5ce0-493e-482b-8243-71e943b0fe41" containerName="console" Sep 04 22:14:22.383136 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:22.383109 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gmtwr" Sep 04 22:14:22.386748 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:22.386724 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 04 22:14:22.386853 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:22.386791 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 04 22:14:22.387632 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:22.387619 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-pxxpr\"" Sep 04 22:14:22.403545 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:22.403519 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gmtwr"] Sep 04 22:14:22.469641 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:22.469604 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dzbg\" (UniqueName: \"kubernetes.io/projected/476c1e87-97fa-40bc-bbf8-4ec164422bb1-kube-api-access-5dzbg\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gmtwr\" (UID: \"476c1e87-97fa-40bc-bbf8-4ec164422bb1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gmtwr" Sep 04 22:14:22.469641 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:22.469648 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/476c1e87-97fa-40bc-bbf8-4ec164422bb1-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gmtwr\" (UID: \"476c1e87-97fa-40bc-bbf8-4ec164422bb1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gmtwr" Sep 04 22:14:22.469874 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:22.469669 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/476c1e87-97fa-40bc-bbf8-4ec164422bb1-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gmtwr\" (UID: \"476c1e87-97fa-40bc-bbf8-4ec164422bb1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gmtwr" Sep 04 22:14:22.570001 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:22.569961 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/476c1e87-97fa-40bc-bbf8-4ec164422bb1-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gmtwr\" (UID: \"476c1e87-97fa-40bc-bbf8-4ec164422bb1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gmtwr" Sep 04 22:14:22.570207 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:22.570008 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/476c1e87-97fa-40bc-bbf8-4ec164422bb1-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gmtwr\" (UID: \"476c1e87-97fa-40bc-bbf8-4ec164422bb1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gmtwr" Sep 04 22:14:22.570207 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:22.570096 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dzbg\" (UniqueName: \"kubernetes.io/projected/476c1e87-97fa-40bc-bbf8-4ec164422bb1-kube-api-access-5dzbg\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gmtwr\" (UID: \"476c1e87-97fa-40bc-bbf8-4ec164422bb1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gmtwr" Sep 04 22:14:22.570352 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:22.570327 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/476c1e87-97fa-40bc-bbf8-4ec164422bb1-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gmtwr\" (UID: \"476c1e87-97fa-40bc-bbf8-4ec164422bb1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gmtwr" Sep 04 22:14:22.570409 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:22.570359 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/476c1e87-97fa-40bc-bbf8-4ec164422bb1-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gmtwr\" (UID: \"476c1e87-97fa-40bc-bbf8-4ec164422bb1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gmtwr" Sep 04 22:14:22.579975 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:22.579949 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dzbg\" (UniqueName: \"kubernetes.io/projected/476c1e87-97fa-40bc-bbf8-4ec164422bb1-kube-api-access-5dzbg\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gmtwr\" (UID: \"476c1e87-97fa-40bc-bbf8-4ec164422bb1\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gmtwr" Sep 04 22:14:22.691866 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:22.691780 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gmtwr" Sep 04 22:14:22.966343 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:22.966267 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gmtwr"] Sep 04 22:14:22.969617 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:14:22.969579 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod476c1e87_97fa_40bc_bbf8_4ec164422bb1.slice/crio-c292e6bcde58c61c4a88d9fd9c0b2ed1fa6d1fe38e96769d181757a81b3b8669 WatchSource:0}: Error finding container c292e6bcde58c61c4a88d9fd9c0b2ed1fa6d1fe38e96769d181757a81b3b8669: Status 404 returned error can't find the container with id c292e6bcde58c61c4a88d9fd9c0b2ed1fa6d1fe38e96769d181757a81b3b8669 Sep 04 22:14:22.971569 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:22.971537 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 04 22:14:23.066787 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:23.066751 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gmtwr" event={"ID":"476c1e87-97fa-40bc-bbf8-4ec164422bb1","Type":"ContainerStarted","Data":"c292e6bcde58c61c4a88d9fd9c0b2ed1fa6d1fe38e96769d181757a81b3b8669"} Sep 04 22:14:30.094361 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:30.094322 2574 generic.go:356] "Generic (PLEG): container finished" podID="476c1e87-97fa-40bc-bbf8-4ec164422bb1" containerID="fa077cc62be1e4dd81a18bf42842410c9ccb9cd9524e08a9faca04c9dfb73098" exitCode=0 Sep 04 22:14:30.094875 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:30.094373 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gmtwr" event={"ID":"476c1e87-97fa-40bc-bbf8-4ec164422bb1","Type":"ContainerDied","Data":"fa077cc62be1e4dd81a18bf42842410c9ccb9cd9524e08a9faca04c9dfb73098"} Sep 04 22:14:33.107253 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:33.107217 2574 generic.go:356] "Generic (PLEG): container finished" podID="476c1e87-97fa-40bc-bbf8-4ec164422bb1" containerID="10200d666702707b6872648cd7a0ec8c76f1a5618f0af63a78f556df62fb15c2" exitCode=0 Sep 04 22:14:33.107651 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:33.107264 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gmtwr" event={"ID":"476c1e87-97fa-40bc-bbf8-4ec164422bb1","Type":"ContainerDied","Data":"10200d666702707b6872648cd7a0ec8c76f1a5618f0af63a78f556df62fb15c2"} Sep 04 22:14:43.146202 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:43.146169 2574 generic.go:356] "Generic (PLEG): container finished" podID="476c1e87-97fa-40bc-bbf8-4ec164422bb1" containerID="ebd3cc36d17407fe11de3907f84d22ca7b01f5baa8730a5bcad99856986bd0be" exitCode=0 Sep 04 22:14:43.146645 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:43.146259 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gmtwr" event={"ID":"476c1e87-97fa-40bc-bbf8-4ec164422bb1","Type":"ContainerDied","Data":"ebd3cc36d17407fe11de3907f84d22ca7b01f5baa8730a5bcad99856986bd0be"} Sep 04 22:14:44.273338 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:44.273311 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gmtwr" Sep 04 22:14:44.341629 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:44.341596 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/476c1e87-97fa-40bc-bbf8-4ec164422bb1-util\") pod \"476c1e87-97fa-40bc-bbf8-4ec164422bb1\" (UID: \"476c1e87-97fa-40bc-bbf8-4ec164422bb1\") " Sep 04 22:14:44.341771 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:44.341646 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dzbg\" (UniqueName: \"kubernetes.io/projected/476c1e87-97fa-40bc-bbf8-4ec164422bb1-kube-api-access-5dzbg\") pod \"476c1e87-97fa-40bc-bbf8-4ec164422bb1\" (UID: \"476c1e87-97fa-40bc-bbf8-4ec164422bb1\") " Sep 04 22:14:44.341771 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:44.341676 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/476c1e87-97fa-40bc-bbf8-4ec164422bb1-bundle\") pod \"476c1e87-97fa-40bc-bbf8-4ec164422bb1\" (UID: \"476c1e87-97fa-40bc-bbf8-4ec164422bb1\") " Sep 04 22:14:44.342352 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:44.342326 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/476c1e87-97fa-40bc-bbf8-4ec164422bb1-bundle" (OuterVolumeSpecName: "bundle") pod "476c1e87-97fa-40bc-bbf8-4ec164422bb1" (UID: "476c1e87-97fa-40bc-bbf8-4ec164422bb1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 04 22:14:44.343857 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:44.343829 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/476c1e87-97fa-40bc-bbf8-4ec164422bb1-kube-api-access-5dzbg" (OuterVolumeSpecName: "kube-api-access-5dzbg") pod "476c1e87-97fa-40bc-bbf8-4ec164422bb1" (UID: "476c1e87-97fa-40bc-bbf8-4ec164422bb1"). InnerVolumeSpecName "kube-api-access-5dzbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:14:44.346490 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:44.346470 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/476c1e87-97fa-40bc-bbf8-4ec164422bb1-util" (OuterVolumeSpecName: "util") pod "476c1e87-97fa-40bc-bbf8-4ec164422bb1" (UID: "476c1e87-97fa-40bc-bbf8-4ec164422bb1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 04 22:14:44.442565 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:44.442467 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/476c1e87-97fa-40bc-bbf8-4ec164422bb1-util\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:14:44.442565 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:44.442495 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dzbg\" (UniqueName: \"kubernetes.io/projected/476c1e87-97fa-40bc-bbf8-4ec164422bb1-kube-api-access-5dzbg\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:14:44.442565 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:44.442510 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/476c1e87-97fa-40bc-bbf8-4ec164422bb1-bundle\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:14:45.155189 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:45.155153 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gmtwr" event={"ID":"476c1e87-97fa-40bc-bbf8-4ec164422bb1","Type":"ContainerDied","Data":"c292e6bcde58c61c4a88d9fd9c0b2ed1fa6d1fe38e96769d181757a81b3b8669"} Sep 04 22:14:45.155355 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:45.155193 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gmtwr" Sep 04 22:14:45.155355 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:45.155230 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c292e6bcde58c61c4a88d9fd9c0b2ed1fa6d1fe38e96769d181757a81b3b8669" Sep 04 22:14:50.518944 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:50.518912 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p95vt"] Sep 04 22:14:50.519399 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:50.519135 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="476c1e87-97fa-40bc-bbf8-4ec164422bb1" containerName="pull" Sep 04 22:14:50.519399 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:50.519147 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="476c1e87-97fa-40bc-bbf8-4ec164422bb1" containerName="pull" Sep 04 22:14:50.519399 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:50.519158 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="476c1e87-97fa-40bc-bbf8-4ec164422bb1" containerName="extract" Sep 04 22:14:50.519399 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:50.519163 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="476c1e87-97fa-40bc-bbf8-4ec164422bb1" containerName="extract" Sep 04 22:14:50.519399 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:50.519173 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="476c1e87-97fa-40bc-bbf8-4ec164422bb1" containerName="util" Sep 04 22:14:50.519399 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:50.519180 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="476c1e87-97fa-40bc-bbf8-4ec164422bb1" containerName="util" Sep 04 22:14:50.519399 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:50.519219 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="476c1e87-97fa-40bc-bbf8-4ec164422bb1" containerName="extract" Sep 04 22:14:50.522350 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:50.522333 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p95vt" Sep 04 22:14:50.525659 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:50.525581 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-c4rrc\"" Sep 04 22:14:50.526350 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:50.526167 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 04 22:14:50.527570 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:50.526652 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 04 22:14:50.540899 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:50.540875 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p95vt"] Sep 04 22:14:50.691122 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:50.691094 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svc9q\" (UniqueName: \"kubernetes.io/projected/8428144d-1bc3-48da-979f-1ccd9eeefed4-kube-api-access-svc9q\") pod \"cert-manager-operator-controller-manager-55c444bf86-p95vt\" (UID: \"8428144d-1bc3-48da-979f-1ccd9eeefed4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p95vt" Sep 04 22:14:50.691286 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:50.691144 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8428144d-1bc3-48da-979f-1ccd9eeefed4-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-p95vt\" (UID: \"8428144d-1bc3-48da-979f-1ccd9eeefed4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p95vt" Sep 04 22:14:50.792468 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:50.792384 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8428144d-1bc3-48da-979f-1ccd9eeefed4-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-p95vt\" (UID: \"8428144d-1bc3-48da-979f-1ccd9eeefed4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p95vt" Sep 04 22:14:50.792655 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:50.792491 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svc9q\" (UniqueName: \"kubernetes.io/projected/8428144d-1bc3-48da-979f-1ccd9eeefed4-kube-api-access-svc9q\") pod \"cert-manager-operator-controller-manager-55c444bf86-p95vt\" (UID: \"8428144d-1bc3-48da-979f-1ccd9eeefed4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p95vt" Sep 04 22:14:50.792844 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:50.792818 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8428144d-1bc3-48da-979f-1ccd9eeefed4-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-p95vt\" (UID: \"8428144d-1bc3-48da-979f-1ccd9eeefed4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p95vt" Sep 04 22:14:50.801939 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:50.801915 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svc9q\" (UniqueName: \"kubernetes.io/projected/8428144d-1bc3-48da-979f-1ccd9eeefed4-kube-api-access-svc9q\") pod \"cert-manager-operator-controller-manager-55c444bf86-p95vt\" (UID: \"8428144d-1bc3-48da-979f-1ccd9eeefed4\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p95vt" Sep 04 22:14:50.832865 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:50.832842 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p95vt" Sep 04 22:14:51.119016 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:51.118950 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p95vt"] Sep 04 22:14:51.122454 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:14:51.122421 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8428144d_1bc3_48da_979f_1ccd9eeefed4.slice/crio-40e798ad98d665d9e7f5bc7aa8ac7ea80bef198e85ddf5eb4edbc6bf6641a09a WatchSource:0}: Error finding container 40e798ad98d665d9e7f5bc7aa8ac7ea80bef198e85ddf5eb4edbc6bf6641a09a: Status 404 returned error can't find the container with id 40e798ad98d665d9e7f5bc7aa8ac7ea80bef198e85ddf5eb4edbc6bf6641a09a Sep 04 22:14:51.177115 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:51.177082 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p95vt" event={"ID":"8428144d-1bc3-48da-979f-1ccd9eeefed4","Type":"ContainerStarted","Data":"40e798ad98d665d9e7f5bc7aa8ac7ea80bef198e85ddf5eb4edbc6bf6641a09a"} Sep 04 22:14:53.186328 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:53.186289 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p95vt" event={"ID":"8428144d-1bc3-48da-979f-1ccd9eeefed4","Type":"ContainerStarted","Data":"d9bd527bcb6ecef09ee459e67c2745207416ecbbc72534ca15cfbd51c42a2897"} Sep 04 22:14:54.428620 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:54.428531 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p95vt" podStartSLOduration=2.957765642 podStartE2EDuration="4.428513004s" totalImagesPullingTime="1.470747362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:14:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:14:53.211264062 +0000 UTC m=+396.975739675" watchObservedRunningTime="2026-09-04 22:14:54.428513004 +0000 UTC m=+398.192988616" Sep 04 22:14:54.429664 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:54.429645 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9dt5t"] Sep 04 22:14:54.431685 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:54.431670 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9dt5t" Sep 04 22:14:54.435195 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:54.435172 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 04 22:14:54.435622 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:54.435605 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 04 22:14:54.435710 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:54.435641 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-pxxpr\"" Sep 04 22:14:54.455704 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:54.455674 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9dt5t"] Sep 04 22:14:54.517715 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:54.517685 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3358a975-8b36-4cbb-8df8-1c984cd0e885-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9dt5t\" (UID: \"3358a975-8b36-4cbb-8df8-1c984cd0e885\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9dt5t" Sep 04 22:14:54.517838 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:54.517745 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3358a975-8b36-4cbb-8df8-1c984cd0e885-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9dt5t\" (UID: \"3358a975-8b36-4cbb-8df8-1c984cd0e885\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9dt5t" Sep 04 22:14:54.517838 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:54.517782 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twrcg\" (UniqueName: \"kubernetes.io/projected/3358a975-8b36-4cbb-8df8-1c984cd0e885-kube-api-access-twrcg\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9dt5t\" (UID: \"3358a975-8b36-4cbb-8df8-1c984cd0e885\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9dt5t" Sep 04 22:14:54.618812 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:54.618783 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3358a975-8b36-4cbb-8df8-1c984cd0e885-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9dt5t\" (UID: \"3358a975-8b36-4cbb-8df8-1c984cd0e885\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9dt5t" Sep 04 22:14:54.618932 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:54.618834 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3358a975-8b36-4cbb-8df8-1c984cd0e885-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9dt5t\" (UID: \"3358a975-8b36-4cbb-8df8-1c984cd0e885\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9dt5t" Sep 04 22:14:54.618932 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:54.618859 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twrcg\" (UniqueName: \"kubernetes.io/projected/3358a975-8b36-4cbb-8df8-1c984cd0e885-kube-api-access-twrcg\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9dt5t\" (UID: \"3358a975-8b36-4cbb-8df8-1c984cd0e885\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9dt5t" Sep 04 22:14:54.619275 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:54.619254 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3358a975-8b36-4cbb-8df8-1c984cd0e885-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9dt5t\" (UID: \"3358a975-8b36-4cbb-8df8-1c984cd0e885\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9dt5t" Sep 04 22:14:54.619275 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:54.619267 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3358a975-8b36-4cbb-8df8-1c984cd0e885-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9dt5t\" (UID: \"3358a975-8b36-4cbb-8df8-1c984cd0e885\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9dt5t" Sep 04 22:14:54.629289 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:54.629256 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twrcg\" (UniqueName: \"kubernetes.io/projected/3358a975-8b36-4cbb-8df8-1c984cd0e885-kube-api-access-twrcg\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9dt5t\" (UID: \"3358a975-8b36-4cbb-8df8-1c984cd0e885\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9dt5t" Sep 04 22:14:54.741079 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:54.741005 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9dt5t" Sep 04 22:14:55.023009 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:55.022976 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9dt5t"] Sep 04 22:14:55.026667 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:14:55.026633 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3358a975_8b36_4cbb_8df8_1c984cd0e885.slice/crio-facfd49f63af5c47465fef7e2144440ae5e0a70f606737cfeb50139fd2868b0c WatchSource:0}: Error finding container facfd49f63af5c47465fef7e2144440ae5e0a70f606737cfeb50139fd2868b0c: Status 404 returned error can't find the container with id facfd49f63af5c47465fef7e2144440ae5e0a70f606737cfeb50139fd2868b0c Sep 04 22:14:55.195116 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:55.195082 2574 generic.go:356] "Generic (PLEG): container finished" podID="3358a975-8b36-4cbb-8df8-1c984cd0e885" containerID="2e25d9af6b2c2a0c63bb869ee392f878564753ad95fc25a5b384d47ee72345b4" exitCode=0 Sep 04 22:14:55.195250 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:55.195170 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9dt5t" event={"ID":"3358a975-8b36-4cbb-8df8-1c984cd0e885","Type":"ContainerDied","Data":"2e25d9af6b2c2a0c63bb869ee392f878564753ad95fc25a5b384d47ee72345b4"} Sep 04 22:14:55.195250 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:55.195229 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9dt5t" event={"ID":"3358a975-8b36-4cbb-8df8-1c984cd0e885","Type":"ContainerStarted","Data":"facfd49f63af5c47465fef7e2144440ae5e0a70f606737cfeb50139fd2868b0c"} Sep 04 22:14:58.208489 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:58.208453 2574 generic.go:356] "Generic (PLEG): container finished" podID="3358a975-8b36-4cbb-8df8-1c984cd0e885" containerID="e1c0f7ae8bcfd73b2c9acc0c86a8b7584f9691e99dae828b726cc42fdc622a2d" exitCode=0 Sep 04 22:14:58.208912 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:58.208523 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9dt5t" event={"ID":"3358a975-8b36-4cbb-8df8-1c984cd0e885","Type":"ContainerDied","Data":"e1c0f7ae8bcfd73b2c9acc0c86a8b7584f9691e99dae828b726cc42fdc622a2d"} Sep 04 22:14:59.214925 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:59.214886 2574 generic.go:356] "Generic (PLEG): container finished" podID="3358a975-8b36-4cbb-8df8-1c984cd0e885" containerID="36828c59a3228b3276e292e42dbae93ed78225def5d70e75c48608636dfab504" exitCode=0 Sep 04 22:14:59.215351 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:14:59.214939 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9dt5t" event={"ID":"3358a975-8b36-4cbb-8df8-1c984cd0e885","Type":"ContainerDied","Data":"36828c59a3228b3276e292e42dbae93ed78225def5d70e75c48608636dfab504"} Sep 04 22:15:00.341821 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:00.341797 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9dt5t" Sep 04 22:15:00.463956 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:00.463920 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twrcg\" (UniqueName: \"kubernetes.io/projected/3358a975-8b36-4cbb-8df8-1c984cd0e885-kube-api-access-twrcg\") pod \"3358a975-8b36-4cbb-8df8-1c984cd0e885\" (UID: \"3358a975-8b36-4cbb-8df8-1c984cd0e885\") " Sep 04 22:15:00.463956 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:00.463965 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3358a975-8b36-4cbb-8df8-1c984cd0e885-util\") pod \"3358a975-8b36-4cbb-8df8-1c984cd0e885\" (UID: \"3358a975-8b36-4cbb-8df8-1c984cd0e885\") " Sep 04 22:15:00.464163 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:00.464010 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3358a975-8b36-4cbb-8df8-1c984cd0e885-bundle\") pod \"3358a975-8b36-4cbb-8df8-1c984cd0e885\" (UID: \"3358a975-8b36-4cbb-8df8-1c984cd0e885\") " Sep 04 22:15:00.464412 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:00.464392 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3358a975-8b36-4cbb-8df8-1c984cd0e885-bundle" (OuterVolumeSpecName: "bundle") pod "3358a975-8b36-4cbb-8df8-1c984cd0e885" (UID: "3358a975-8b36-4cbb-8df8-1c984cd0e885"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 04 22:15:00.465928 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:00.465904 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3358a975-8b36-4cbb-8df8-1c984cd0e885-kube-api-access-twrcg" (OuterVolumeSpecName: "kube-api-access-twrcg") pod "3358a975-8b36-4cbb-8df8-1c984cd0e885" (UID: "3358a975-8b36-4cbb-8df8-1c984cd0e885"). InnerVolumeSpecName "kube-api-access-twrcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:15:00.491557 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:00.491512 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3358a975-8b36-4cbb-8df8-1c984cd0e885-util" (OuterVolumeSpecName: "util") pod "3358a975-8b36-4cbb-8df8-1c984cd0e885" (UID: "3358a975-8b36-4cbb-8df8-1c984cd0e885"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 04 22:15:00.564611 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:00.564574 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twrcg\" (UniqueName: \"kubernetes.io/projected/3358a975-8b36-4cbb-8df8-1c984cd0e885-kube-api-access-twrcg\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:15:00.564611 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:00.564605 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3358a975-8b36-4cbb-8df8-1c984cd0e885-util\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:15:00.564611 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:00.564617 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3358a975-8b36-4cbb-8df8-1c984cd0e885-bundle\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:15:01.223333 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:01.223307 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9dt5t" Sep 04 22:15:01.223494 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:01.223298 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f9dt5t" event={"ID":"3358a975-8b36-4cbb-8df8-1c984cd0e885","Type":"ContainerDied","Data":"facfd49f63af5c47465fef7e2144440ae5e0a70f606737cfeb50139fd2868b0c"} Sep 04 22:15:01.223494 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:01.223429 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="facfd49f63af5c47465fef7e2144440ae5e0a70f606737cfeb50139fd2868b0c" Sep 04 22:15:02.598715 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:02.598681 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2rr75"] Sep 04 22:15:02.599083 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:02.598906 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3358a975-8b36-4cbb-8df8-1c984cd0e885" containerName="extract" Sep 04 22:15:02.599083 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:02.598917 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3358a975-8b36-4cbb-8df8-1c984cd0e885" containerName="extract" Sep 04 22:15:02.599083 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:02.598927 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3358a975-8b36-4cbb-8df8-1c984cd0e885" containerName="pull" Sep 04 22:15:02.599083 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:02.598932 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3358a975-8b36-4cbb-8df8-1c984cd0e885" containerName="pull" Sep 04 22:15:02.599083 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:02.598945 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3358a975-8b36-4cbb-8df8-1c984cd0e885" containerName="util" Sep 04 22:15:02.599083 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:02.598951 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3358a975-8b36-4cbb-8df8-1c984cd0e885" containerName="util" Sep 04 22:15:02.599083 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:02.598992 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3358a975-8b36-4cbb-8df8-1c984cd0e885" containerName="extract" Sep 04 22:15:02.600682 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:02.600665 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2rr75" Sep 04 22:15:02.603778 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:02.603752 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-zm6t8\"" Sep 04 22:15:02.603899 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:02.603752 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 04 22:15:02.604561 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:02.604538 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 04 22:15:02.614463 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:02.614441 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2rr75"] Sep 04 22:15:02.783147 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:02.783113 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6knc8\" (UniqueName: \"kubernetes.io/projected/a16f2597-c87e-41a5-9e9e-a27210636f24-kube-api-access-6knc8\") pod \"cert-manager-cainjector-559cdd8c87-2rr75\" (UID: \"a16f2597-c87e-41a5-9e9e-a27210636f24\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2rr75" Sep 04 22:15:02.783318 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:02.783155 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a16f2597-c87e-41a5-9e9e-a27210636f24-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2rr75\" (UID: \"a16f2597-c87e-41a5-9e9e-a27210636f24\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2rr75" Sep 04 22:15:02.883744 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:02.883660 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6knc8\" (UniqueName: \"kubernetes.io/projected/a16f2597-c87e-41a5-9e9e-a27210636f24-kube-api-access-6knc8\") pod \"cert-manager-cainjector-559cdd8c87-2rr75\" (UID: \"a16f2597-c87e-41a5-9e9e-a27210636f24\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2rr75" Sep 04 22:15:02.883744 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:02.883707 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a16f2597-c87e-41a5-9e9e-a27210636f24-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2rr75\" (UID: \"a16f2597-c87e-41a5-9e9e-a27210636f24\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2rr75" Sep 04 22:15:02.895092 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:02.895065 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a16f2597-c87e-41a5-9e9e-a27210636f24-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2rr75\" (UID: \"a16f2597-c87e-41a5-9e9e-a27210636f24\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2rr75" Sep 04 22:15:02.895971 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:02.895950 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6knc8\" (UniqueName: \"kubernetes.io/projected/a16f2597-c87e-41a5-9e9e-a27210636f24-kube-api-access-6knc8\") pod \"cert-manager-cainjector-559cdd8c87-2rr75\" (UID: \"a16f2597-c87e-41a5-9e9e-a27210636f24\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2rr75" Sep 04 22:15:02.909606 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:02.909583 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2rr75" Sep 04 22:15:03.178502 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:03.178284 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2rr75"] Sep 04 22:15:03.181055 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:15:03.181030 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda16f2597_c87e_41a5_9e9e_a27210636f24.slice/crio-1ece8fa6e4a464b3a7afc3798521bf4af3483e5907ddb2d0ce1d91e9b472bdb3 WatchSource:0}: Error finding container 1ece8fa6e4a464b3a7afc3798521bf4af3483e5907ddb2d0ce1d91e9b472bdb3: Status 404 returned error can't find the container with id 1ece8fa6e4a464b3a7afc3798521bf4af3483e5907ddb2d0ce1d91e9b472bdb3 Sep 04 22:15:03.233636 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:03.233600 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2rr75" event={"ID":"a16f2597-c87e-41a5-9e9e-a27210636f24","Type":"ContainerStarted","Data":"1ece8fa6e4a464b3a7afc3798521bf4af3483e5907ddb2d0ce1d91e9b472bdb3"} Sep 04 22:15:06.252001 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:06.251970 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2rr75" event={"ID":"a16f2597-c87e-41a5-9e9e-a27210636f24","Type":"ContainerStarted","Data":"29c58c54d383d0b2c3763f376492fc33826aaf48d3a239caefe5f96c121e4b7d"} Sep 04 22:15:06.270938 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:06.270895 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2rr75" podStartSLOduration=1.822498593 podStartE2EDuration="4.270874994s" totalImagesPullingTime="2.448376401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:15:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:15:06.270723958 +0000 UTC m=+410.035199570" watchObservedRunningTime="2026-09-04 22:15:06.270874994 +0000 UTC m=+410.035350607" Sep 04 22:15:08.093261 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:08.093213 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hh5sw"] Sep 04 22:15:08.096591 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:08.096537 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hh5sw" Sep 04 22:15:08.110212 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:08.110173 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-2qfdn\"" Sep 04 22:15:08.110876 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:08.110854 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 04 22:15:08.110943 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:08.110853 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 04 22:15:08.118897 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:08.118852 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hh5sw"] Sep 04 22:15:08.120529 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:08.120494 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1f71386-c859-4117-80ce-9817285e5888-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hh5sw\" (UID: \"b1f71386-c859-4117-80ce-9817285e5888\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hh5sw" Sep 04 22:15:08.120734 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:08.120535 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7fg5\" (UniqueName: \"kubernetes.io/projected/b1f71386-c859-4117-80ce-9817285e5888-kube-api-access-n7fg5\") pod \"openshift-lws-operator-fd8ccff4c-hh5sw\" (UID: \"b1f71386-c859-4117-80ce-9817285e5888\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hh5sw" Sep 04 22:15:08.221107 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:08.221044 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1f71386-c859-4117-80ce-9817285e5888-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hh5sw\" (UID: \"b1f71386-c859-4117-80ce-9817285e5888\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hh5sw" Sep 04 22:15:08.221371 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:08.221187 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7fg5\" (UniqueName: \"kubernetes.io/projected/b1f71386-c859-4117-80ce-9817285e5888-kube-api-access-n7fg5\") pod \"openshift-lws-operator-fd8ccff4c-hh5sw\" (UID: \"b1f71386-c859-4117-80ce-9817285e5888\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hh5sw" Sep 04 22:15:08.221640 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:08.221617 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1f71386-c859-4117-80ce-9817285e5888-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hh5sw\" (UID: \"b1f71386-c859-4117-80ce-9817285e5888\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hh5sw" Sep 04 22:15:08.235931 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:08.235875 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7fg5\" (UniqueName: \"kubernetes.io/projected/b1f71386-c859-4117-80ce-9817285e5888-kube-api-access-n7fg5\") pod \"openshift-lws-operator-fd8ccff4c-hh5sw\" (UID: \"b1f71386-c859-4117-80ce-9817285e5888\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hh5sw" Sep 04 22:15:08.408257 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:08.408133 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hh5sw" Sep 04 22:15:08.695955 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:08.695847 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hh5sw"] Sep 04 22:15:08.701720 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:15:08.701674 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1f71386_c859_4117_80ce_9817285e5888.slice/crio-1b20dcc25d89b3aff081a06a14a5b6cc3febd110ed23c61cefcb75b7af131353 WatchSource:0}: Error finding container 1b20dcc25d89b3aff081a06a14a5b6cc3febd110ed23c61cefcb75b7af131353: Status 404 returned error can't find the container with id 1b20dcc25d89b3aff081a06a14a5b6cc3febd110ed23c61cefcb75b7af131353 Sep 04 22:15:09.266492 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:09.266451 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hh5sw" event={"ID":"b1f71386-c859-4117-80ce-9817285e5888","Type":"ContainerStarted","Data":"1b20dcc25d89b3aff081a06a14a5b6cc3febd110ed23c61cefcb75b7af131353"} Sep 04 22:15:12.279774 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:12.279740 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hh5sw" event={"ID":"b1f71386-c859-4117-80ce-9817285e5888","Type":"ContainerStarted","Data":"a44a349714786a04932e29deef87110d1f712c823fc058d824c15f6a8ad3d98b"} Sep 04 22:15:12.311597 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:12.311523 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hh5sw" podStartSLOduration=1.573615209 podStartE2EDuration="4.311501898s" totalImagesPullingTime="2.737886689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:15:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:15:12.310308181 +0000 UTC m=+416.074783795" watchObservedRunningTime="2026-09-04 22:15:12.311501898 +0000 UTC m=+416.075977518" Sep 04 22:15:14.463823 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:14.463783 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsmk2"] Sep 04 22:15:14.466077 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:14.466060 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsmk2" Sep 04 22:15:14.471337 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:14.471305 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 04 22:15:14.471464 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:14.471364 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 04 22:15:14.471840 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:14.471825 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-pxxpr\"" Sep 04 22:15:14.485121 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:14.485098 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsmk2"] Sep 04 22:15:14.561129 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:14.561093 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/826f3b93-9987-4a62-b32b-b76b0dddc1cb-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsmk2\" (UID: \"826f3b93-9987-4a62-b32b-b76b0dddc1cb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsmk2" Sep 04 22:15:14.561310 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:14.561150 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/826f3b93-9987-4a62-b32b-b76b0dddc1cb-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsmk2\" (UID: \"826f3b93-9987-4a62-b32b-b76b0dddc1cb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsmk2" Sep 04 22:15:14.561310 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:14.561184 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcd86\" (UniqueName: \"kubernetes.io/projected/826f3b93-9987-4a62-b32b-b76b0dddc1cb-kube-api-access-gcd86\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsmk2\" (UID: \"826f3b93-9987-4a62-b32b-b76b0dddc1cb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsmk2" Sep 04 22:15:14.661572 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:14.661510 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/826f3b93-9987-4a62-b32b-b76b0dddc1cb-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsmk2\" (UID: \"826f3b93-9987-4a62-b32b-b76b0dddc1cb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsmk2" Sep 04 22:15:14.661743 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:14.661606 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/826f3b93-9987-4a62-b32b-b76b0dddc1cb-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsmk2\" (UID: \"826f3b93-9987-4a62-b32b-b76b0dddc1cb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsmk2" Sep 04 22:15:14.661743 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:14.661637 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcd86\" (UniqueName: \"kubernetes.io/projected/826f3b93-9987-4a62-b32b-b76b0dddc1cb-kube-api-access-gcd86\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsmk2\" (UID: \"826f3b93-9987-4a62-b32b-b76b0dddc1cb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsmk2" Sep 04 22:15:14.661922 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:14.661901 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/826f3b93-9987-4a62-b32b-b76b0dddc1cb-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsmk2\" (UID: \"826f3b93-9987-4a62-b32b-b76b0dddc1cb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsmk2" Sep 04 22:15:14.661962 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:14.661939 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/826f3b93-9987-4a62-b32b-b76b0dddc1cb-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsmk2\" (UID: \"826f3b93-9987-4a62-b32b-b76b0dddc1cb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsmk2" Sep 04 22:15:14.673997 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:14.673969 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcd86\" (UniqueName: \"kubernetes.io/projected/826f3b93-9987-4a62-b32b-b76b0dddc1cb-kube-api-access-gcd86\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsmk2\" (UID: \"826f3b93-9987-4a62-b32b-b76b0dddc1cb\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsmk2" Sep 04 22:15:14.774451 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:14.774424 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsmk2" Sep 04 22:15:15.060576 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:15.060482 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsmk2"] Sep 04 22:15:15.063404 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:15:15.063370 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod826f3b93_9987_4a62_b32b_b76b0dddc1cb.slice/crio-ebcc112601d3e63e75c725504eed7f1d3b125ff7f9c72819c0cfa1235d6e4880 WatchSource:0}: Error finding container ebcc112601d3e63e75c725504eed7f1d3b125ff7f9c72819c0cfa1235d6e4880: Status 404 returned error can't find the container with id ebcc112601d3e63e75c725504eed7f1d3b125ff7f9c72819c0cfa1235d6e4880 Sep 04 22:15:15.294399 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:15.294366 2574 generic.go:356] "Generic (PLEG): container finished" podID="826f3b93-9987-4a62-b32b-b76b0dddc1cb" containerID="d29c22e085415e22c220cddb3b206a4afe0d458f8229452920c94cdabda72830" exitCode=0 Sep 04 22:15:15.294589 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:15.294412 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsmk2" event={"ID":"826f3b93-9987-4a62-b32b-b76b0dddc1cb","Type":"ContainerDied","Data":"d29c22e085415e22c220cddb3b206a4afe0d458f8229452920c94cdabda72830"} Sep 04 22:15:15.294589 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:15.294448 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsmk2" event={"ID":"826f3b93-9987-4a62-b32b-b76b0dddc1cb","Type":"ContainerStarted","Data":"ebcc112601d3e63e75c725504eed7f1d3b125ff7f9c72819c0cfa1235d6e4880"} Sep 04 22:15:16.300846 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:16.300811 2574 generic.go:356] "Generic (PLEG): container finished" podID="826f3b93-9987-4a62-b32b-b76b0dddc1cb" containerID="4af658286c2b5bf7759df8ac825f874e86479a25127ad8feecff6e8406e036bc" exitCode=0 Sep 04 22:15:16.301251 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:16.300897 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsmk2" event={"ID":"826f3b93-9987-4a62-b32b-b76b0dddc1cb","Type":"ContainerDied","Data":"4af658286c2b5bf7759df8ac825f874e86479a25127ad8feecff6e8406e036bc"} Sep 04 22:15:17.307484 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:17.307451 2574 generic.go:356] "Generic (PLEG): container finished" podID="826f3b93-9987-4a62-b32b-b76b0dddc1cb" containerID="0d3fd62b86e28f97ebdb62118f9d39de7c9760fac365209a374297fecc3c40ee" exitCode=0 Sep 04 22:15:17.307873 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:17.307501 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsmk2" event={"ID":"826f3b93-9987-4a62-b32b-b76b0dddc1cb","Type":"ContainerDied","Data":"0d3fd62b86e28f97ebdb62118f9d39de7c9760fac365209a374297fecc3c40ee"} Sep 04 22:15:18.438399 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:18.438365 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsmk2" Sep 04 22:15:18.486737 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:18.486683 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gcd86\" (UniqueName: \"kubernetes.io/projected/826f3b93-9987-4a62-b32b-b76b0dddc1cb-kube-api-access-gcd86\") pod \"826f3b93-9987-4a62-b32b-b76b0dddc1cb\" (UID: \"826f3b93-9987-4a62-b32b-b76b0dddc1cb\") " Sep 04 22:15:18.486951 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:18.486757 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/826f3b93-9987-4a62-b32b-b76b0dddc1cb-bundle\") pod \"826f3b93-9987-4a62-b32b-b76b0dddc1cb\" (UID: \"826f3b93-9987-4a62-b32b-b76b0dddc1cb\") " Sep 04 22:15:18.486951 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:18.486773 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/826f3b93-9987-4a62-b32b-b76b0dddc1cb-util\") pod \"826f3b93-9987-4a62-b32b-b76b0dddc1cb\" (UID: \"826f3b93-9987-4a62-b32b-b76b0dddc1cb\") " Sep 04 22:15:18.487639 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:18.487600 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/826f3b93-9987-4a62-b32b-b76b0dddc1cb-bundle" (OuterVolumeSpecName: "bundle") pod "826f3b93-9987-4a62-b32b-b76b0dddc1cb" (UID: "826f3b93-9987-4a62-b32b-b76b0dddc1cb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 04 22:15:18.489426 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:18.489396 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/826f3b93-9987-4a62-b32b-b76b0dddc1cb-kube-api-access-gcd86" (OuterVolumeSpecName: "kube-api-access-gcd86") pod "826f3b93-9987-4a62-b32b-b76b0dddc1cb" (UID: "826f3b93-9987-4a62-b32b-b76b0dddc1cb"). InnerVolumeSpecName "kube-api-access-gcd86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:15:18.495159 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:18.495116 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/826f3b93-9987-4a62-b32b-b76b0dddc1cb-util" (OuterVolumeSpecName: "util") pod "826f3b93-9987-4a62-b32b-b76b0dddc1cb" (UID: "826f3b93-9987-4a62-b32b-b76b0dddc1cb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 04 22:15:18.588084 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:18.587956 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcd86\" (UniqueName: \"kubernetes.io/projected/826f3b93-9987-4a62-b32b-b76b0dddc1cb-kube-api-access-gcd86\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:15:18.588084 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:18.588001 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/826f3b93-9987-4a62-b32b-b76b0dddc1cb-bundle\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:15:18.588084 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:18.588012 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/826f3b93-9987-4a62-b32b-b76b0dddc1cb-util\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:15:19.317920 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:19.317890 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsmk2" Sep 04 22:15:19.317920 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:19.317899 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrsmk2" event={"ID":"826f3b93-9987-4a62-b32b-b76b0dddc1cb","Type":"ContainerDied","Data":"ebcc112601d3e63e75c725504eed7f1d3b125ff7f9c72819c0cfa1235d6e4880"} Sep 04 22:15:19.318130 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:19.317943 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebcc112601d3e63e75c725504eed7f1d3b125ff7f9c72819c0cfa1235d6e4880" Sep 04 22:15:25.810134 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:25.810097 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0drtp2"] Sep 04 22:15:25.810545 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:25.810333 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="826f3b93-9987-4a62-b32b-b76b0dddc1cb" containerName="pull" Sep 04 22:15:25.810545 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:25.810345 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="826f3b93-9987-4a62-b32b-b76b0dddc1cb" containerName="pull" Sep 04 22:15:25.810545 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:25.810368 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="826f3b93-9987-4a62-b32b-b76b0dddc1cb" containerName="extract" Sep 04 22:15:25.810545 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:25.810374 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="826f3b93-9987-4a62-b32b-b76b0dddc1cb" containerName="extract" Sep 04 22:15:25.810545 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:25.810381 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="826f3b93-9987-4a62-b32b-b76b0dddc1cb" containerName="util" Sep 04 22:15:25.810545 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:25.810386 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="826f3b93-9987-4a62-b32b-b76b0dddc1cb" containerName="util" Sep 04 22:15:25.810545 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:25.810425 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="826f3b93-9987-4a62-b32b-b76b0dddc1cb" containerName="extract" Sep 04 22:15:25.812653 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:25.812635 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0drtp2" Sep 04 22:15:25.815323 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:25.815304 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-pxxpr\"" Sep 04 22:15:25.815447 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:25.815360 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 04 22:15:25.815447 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:25.815410 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 04 22:15:25.824045 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:25.824024 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0drtp2"] Sep 04 22:15:25.838489 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:25.838455 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt5q9\" (UniqueName: \"kubernetes.io/projected/aef8196c-b1f4-4609-9a1b-efd94ec1b4ce-kube-api-access-nt5q9\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0drtp2\" (UID: \"aef8196c-b1f4-4609-9a1b-efd94ec1b4ce\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0drtp2" Sep 04 22:15:25.838637 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:25.838499 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aef8196c-b1f4-4609-9a1b-efd94ec1b4ce-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0drtp2\" (UID: \"aef8196c-b1f4-4609-9a1b-efd94ec1b4ce\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0drtp2" Sep 04 22:15:25.838637 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:25.838516 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aef8196c-b1f4-4609-9a1b-efd94ec1b4ce-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0drtp2\" (UID: \"aef8196c-b1f4-4609-9a1b-efd94ec1b4ce\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0drtp2" Sep 04 22:15:25.939400 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:25.939366 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt5q9\" (UniqueName: \"kubernetes.io/projected/aef8196c-b1f4-4609-9a1b-efd94ec1b4ce-kube-api-access-nt5q9\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0drtp2\" (UID: \"aef8196c-b1f4-4609-9a1b-efd94ec1b4ce\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0drtp2" Sep 04 22:15:25.939400 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:25.939411 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aef8196c-b1f4-4609-9a1b-efd94ec1b4ce-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0drtp2\" (UID: \"aef8196c-b1f4-4609-9a1b-efd94ec1b4ce\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0drtp2" Sep 04 22:15:25.939619 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:25.939428 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aef8196c-b1f4-4609-9a1b-efd94ec1b4ce-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0drtp2\" (UID: \"aef8196c-b1f4-4609-9a1b-efd94ec1b4ce\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0drtp2" Sep 04 22:15:25.939766 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:25.939751 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aef8196c-b1f4-4609-9a1b-efd94ec1b4ce-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0drtp2\" (UID: \"aef8196c-b1f4-4609-9a1b-efd94ec1b4ce\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0drtp2" Sep 04 22:15:25.939803 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:25.939781 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aef8196c-b1f4-4609-9a1b-efd94ec1b4ce-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0drtp2\" (UID: \"aef8196c-b1f4-4609-9a1b-efd94ec1b4ce\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0drtp2" Sep 04 22:15:25.949770 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:25.949746 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt5q9\" (UniqueName: \"kubernetes.io/projected/aef8196c-b1f4-4609-9a1b-efd94ec1b4ce-kube-api-access-nt5q9\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0drtp2\" (UID: \"aef8196c-b1f4-4609-9a1b-efd94ec1b4ce\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0drtp2" Sep 04 22:15:26.121823 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:26.121734 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0drtp2" Sep 04 22:15:26.403795 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:26.403725 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0drtp2"] Sep 04 22:15:26.406443 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:15:26.406410 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaef8196c_b1f4_4609_9a1b_efd94ec1b4ce.slice/crio-2222df5d08581052033aae75f49d4ca1ef816cfae29024652e6110938a3f4933 WatchSource:0}: Error finding container 2222df5d08581052033aae75f49d4ca1ef816cfae29024652e6110938a3f4933: Status 404 returned error can't find the container with id 2222df5d08581052033aae75f49d4ca1ef816cfae29024652e6110938a3f4933 Sep 04 22:15:27.351053 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:27.351018 2574 generic.go:356] "Generic (PLEG): container finished" podID="aef8196c-b1f4-4609-9a1b-efd94ec1b4ce" containerID="0ebbdd49a5a6157e02598fe0bb1f76d2abeb803ea423044f04d8a0437b019023" exitCode=0 Sep 04 22:15:27.351423 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:27.351103 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0drtp2" event={"ID":"aef8196c-b1f4-4609-9a1b-efd94ec1b4ce","Type":"ContainerDied","Data":"0ebbdd49a5a6157e02598fe0bb1f76d2abeb803ea423044f04d8a0437b019023"} Sep 04 22:15:27.351423 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:27.351194 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0drtp2" event={"ID":"aef8196c-b1f4-4609-9a1b-efd94ec1b4ce","Type":"ContainerStarted","Data":"2222df5d08581052033aae75f49d4ca1ef816cfae29024652e6110938a3f4933"} Sep 04 22:15:27.690078 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:27.689999 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cd686f5d7-hxv2l"] Sep 04 22:15:27.692159 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:27.692144 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-hxv2l" Sep 04 22:15:27.694979 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:27.694951 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-kvsxk\"" Sep 04 22:15:27.695246 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:27.695226 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 04 22:15:27.695460 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:27.695446 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 04 22:15:27.695527 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:27.695511 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 04 22:15:27.698104 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:27.698086 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 04 22:15:27.723385 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:27.723357 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cd686f5d7-hxv2l"] Sep 04 22:15:27.751096 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:27.751065 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67333fdd-f1ff-41b9-bb3a-504f1d3fb547-tmp\") pod \"opendatahub-operator-controller-manager-cd686f5d7-hxv2l\" (UID: \"67333fdd-f1ff-41b9-bb3a-504f1d3fb547\") " pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-hxv2l" Sep 04 22:15:27.751247 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:27.751102 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67333fdd-f1ff-41b9-bb3a-504f1d3fb547-webhook-cert\") pod \"opendatahub-operator-controller-manager-cd686f5d7-hxv2l\" (UID: \"67333fdd-f1ff-41b9-bb3a-504f1d3fb547\") " pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-hxv2l" Sep 04 22:15:27.751247 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:27.751177 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4cj4\" (UniqueName: \"kubernetes.io/projected/67333fdd-f1ff-41b9-bb3a-504f1d3fb547-kube-api-access-l4cj4\") pod \"opendatahub-operator-controller-manager-cd686f5d7-hxv2l\" (UID: \"67333fdd-f1ff-41b9-bb3a-504f1d3fb547\") " pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-hxv2l" Sep 04 22:15:27.751321 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:27.751260 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/67333fdd-f1ff-41b9-bb3a-504f1d3fb547-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cd686f5d7-hxv2l\" (UID: \"67333fdd-f1ff-41b9-bb3a-504f1d3fb547\") " pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-hxv2l" Sep 04 22:15:27.751321 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:27.751303 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/67333fdd-f1ff-41b9-bb3a-504f1d3fb547-manifests\") pod \"opendatahub-operator-controller-manager-cd686f5d7-hxv2l\" (UID: \"67333fdd-f1ff-41b9-bb3a-504f1d3fb547\") " pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-hxv2l" Sep 04 22:15:27.852330 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:27.852298 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4cj4\" (UniqueName: \"kubernetes.io/projected/67333fdd-f1ff-41b9-bb3a-504f1d3fb547-kube-api-access-l4cj4\") pod \"opendatahub-operator-controller-manager-cd686f5d7-hxv2l\" (UID: \"67333fdd-f1ff-41b9-bb3a-504f1d3fb547\") " pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-hxv2l" Sep 04 22:15:27.852492 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:27.852353 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/67333fdd-f1ff-41b9-bb3a-504f1d3fb547-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cd686f5d7-hxv2l\" (UID: \"67333fdd-f1ff-41b9-bb3a-504f1d3fb547\") " pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-hxv2l" Sep 04 22:15:27.852492 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:27.852380 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/67333fdd-f1ff-41b9-bb3a-504f1d3fb547-manifests\") pod \"opendatahub-operator-controller-manager-cd686f5d7-hxv2l\" (UID: \"67333fdd-f1ff-41b9-bb3a-504f1d3fb547\") " pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-hxv2l" Sep 04 22:15:27.852492 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:27.852396 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67333fdd-f1ff-41b9-bb3a-504f1d3fb547-tmp\") pod \"opendatahub-operator-controller-manager-cd686f5d7-hxv2l\" (UID: \"67333fdd-f1ff-41b9-bb3a-504f1d3fb547\") " pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-hxv2l" Sep 04 22:15:27.852492 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:27.852412 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67333fdd-f1ff-41b9-bb3a-504f1d3fb547-webhook-cert\") pod \"opendatahub-operator-controller-manager-cd686f5d7-hxv2l\" (UID: \"67333fdd-f1ff-41b9-bb3a-504f1d3fb547\") " pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-hxv2l" Sep 04 22:15:27.852850 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:27.852822 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/67333fdd-f1ff-41b9-bb3a-504f1d3fb547-manifests\") pod \"opendatahub-operator-controller-manager-cd686f5d7-hxv2l\" (UID: \"67333fdd-f1ff-41b9-bb3a-504f1d3fb547\") " pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-hxv2l" Sep 04 22:15:27.854770 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:27.854741 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67333fdd-f1ff-41b9-bb3a-504f1d3fb547-webhook-cert\") pod \"opendatahub-operator-controller-manager-cd686f5d7-hxv2l\" (UID: \"67333fdd-f1ff-41b9-bb3a-504f1d3fb547\") " pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-hxv2l" Sep 04 22:15:27.854865 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:27.854773 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/67333fdd-f1ff-41b9-bb3a-504f1d3fb547-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cd686f5d7-hxv2l\" (UID: \"67333fdd-f1ff-41b9-bb3a-504f1d3fb547\") " pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-hxv2l" Sep 04 22:15:27.855112 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:27.855091 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67333fdd-f1ff-41b9-bb3a-504f1d3fb547-tmp\") pod \"opendatahub-operator-controller-manager-cd686f5d7-hxv2l\" (UID: \"67333fdd-f1ff-41b9-bb3a-504f1d3fb547\") " pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-hxv2l" Sep 04 22:15:27.868357 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:27.868330 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4cj4\" (UniqueName: \"kubernetes.io/projected/67333fdd-f1ff-41b9-bb3a-504f1d3fb547-kube-api-access-l4cj4\") pod \"opendatahub-operator-controller-manager-cd686f5d7-hxv2l\" (UID: \"67333fdd-f1ff-41b9-bb3a-504f1d3fb547\") " pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-hxv2l" Sep 04 22:15:28.002275 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:28.002248 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-hxv2l" Sep 04 22:15:28.287716 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:28.287685 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cd686f5d7-hxv2l"] Sep 04 22:15:28.290858 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:15:28.290825 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67333fdd_f1ff_41b9_bb3a_504f1d3fb547.slice/crio-610094def094af3db5729e0ded76a60aa074b401b121a2be2410dc9dbd920622 WatchSource:0}: Error finding container 610094def094af3db5729e0ded76a60aa074b401b121a2be2410dc9dbd920622: Status 404 returned error can't find the container with id 610094def094af3db5729e0ded76a60aa074b401b121a2be2410dc9dbd920622 Sep 04 22:15:28.357475 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:28.357445 2574 generic.go:356] "Generic (PLEG): container finished" podID="aef8196c-b1f4-4609-9a1b-efd94ec1b4ce" containerID="f70d081e66ecf1761ea722aa551de1f7fe916b0384b962bf8b23a0906de2b049" exitCode=0 Sep 04 22:15:28.357873 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:28.357534 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0drtp2" event={"ID":"aef8196c-b1f4-4609-9a1b-efd94ec1b4ce","Type":"ContainerDied","Data":"f70d081e66ecf1761ea722aa551de1f7fe916b0384b962bf8b23a0906de2b049"} Sep 04 22:15:28.359007 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:28.358979 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-hxv2l" event={"ID":"67333fdd-f1ff-41b9-bb3a-504f1d3fb547","Type":"ContainerStarted","Data":"610094def094af3db5729e0ded76a60aa074b401b121a2be2410dc9dbd920622"} Sep 04 22:15:29.366221 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:29.366174 2574 generic.go:356] "Generic (PLEG): container finished" podID="aef8196c-b1f4-4609-9a1b-efd94ec1b4ce" containerID="a1bd33af5bb3bc09912894fd3c8690deefcf8d81055f86131b39b7e0526990d4" exitCode=0 Sep 04 22:15:29.366674 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:29.366269 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0drtp2" event={"ID":"aef8196c-b1f4-4609-9a1b-efd94ec1b4ce","Type":"ContainerDied","Data":"a1bd33af5bb3bc09912894fd3c8690deefcf8d81055f86131b39b7e0526990d4"} Sep 04 22:15:31.090949 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:31.090925 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0drtp2" Sep 04 22:15:31.176019 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:31.175997 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nt5q9\" (UniqueName: \"kubernetes.io/projected/aef8196c-b1f4-4609-9a1b-efd94ec1b4ce-kube-api-access-nt5q9\") pod \"aef8196c-b1f4-4609-9a1b-efd94ec1b4ce\" (UID: \"aef8196c-b1f4-4609-9a1b-efd94ec1b4ce\") " Sep 04 22:15:31.176128 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:31.176041 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aef8196c-b1f4-4609-9a1b-efd94ec1b4ce-util\") pod \"aef8196c-b1f4-4609-9a1b-efd94ec1b4ce\" (UID: \"aef8196c-b1f4-4609-9a1b-efd94ec1b4ce\") " Sep 04 22:15:31.177859 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:31.177819 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aef8196c-b1f4-4609-9a1b-efd94ec1b4ce-kube-api-access-nt5q9" (OuterVolumeSpecName: "kube-api-access-nt5q9") pod "aef8196c-b1f4-4609-9a1b-efd94ec1b4ce" (UID: "aef8196c-b1f4-4609-9a1b-efd94ec1b4ce"). InnerVolumeSpecName "kube-api-access-nt5q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:15:31.181568 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:31.181519 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aef8196c-b1f4-4609-9a1b-efd94ec1b4ce-util" (OuterVolumeSpecName: "util") pod "aef8196c-b1f4-4609-9a1b-efd94ec1b4ce" (UID: "aef8196c-b1f4-4609-9a1b-efd94ec1b4ce"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 04 22:15:31.277192 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:31.277156 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aef8196c-b1f4-4609-9a1b-efd94ec1b4ce-bundle\") pod \"aef8196c-b1f4-4609-9a1b-efd94ec1b4ce\" (UID: \"aef8196c-b1f4-4609-9a1b-efd94ec1b4ce\") " Sep 04 22:15:31.277338 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:31.277310 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aef8196c-b1f4-4609-9a1b-efd94ec1b4ce-util\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:15:31.277338 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:31.277328 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt5q9\" (UniqueName: \"kubernetes.io/projected/aef8196c-b1f4-4609-9a1b-efd94ec1b4ce-kube-api-access-nt5q9\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:15:31.277877 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:31.277854 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aef8196c-b1f4-4609-9a1b-efd94ec1b4ce-bundle" (OuterVolumeSpecName: "bundle") pod "aef8196c-b1f4-4609-9a1b-efd94ec1b4ce" (UID: "aef8196c-b1f4-4609-9a1b-efd94ec1b4ce"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 04 22:15:31.375789 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:31.375715 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0drtp2" Sep 04 22:15:31.375948 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:31.375713 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0drtp2" event={"ID":"aef8196c-b1f4-4609-9a1b-efd94ec1b4ce","Type":"ContainerDied","Data":"2222df5d08581052033aae75f49d4ca1ef816cfae29024652e6110938a3f4933"} Sep 04 22:15:31.375948 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:31.375848 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2222df5d08581052033aae75f49d4ca1ef816cfae29024652e6110938a3f4933" Sep 04 22:15:31.377680 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:31.377657 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aef8196c-b1f4-4609-9a1b-efd94ec1b4ce-bundle\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:15:31.377789 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:31.377679 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-hxv2l" event={"ID":"67333fdd-f1ff-41b9-bb3a-504f1d3fb547","Type":"ContainerStarted","Data":"6dc6fa2595cb4f2b8549ab0a3a0793e6b03168b7489a05cc69cb0fd2c56b7547"} Sep 04 22:15:32.383575 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:32.383520 2574 generic.go:356] "Generic (PLEG): container finished" podID="67333fdd-f1ff-41b9-bb3a-504f1d3fb547" containerID="6dc6fa2595cb4f2b8549ab0a3a0793e6b03168b7489a05cc69cb0fd2c56b7547" exitCode=0 Sep 04 22:15:32.383937 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:32.383591 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-hxv2l" event={"ID":"67333fdd-f1ff-41b9-bb3a-504f1d3fb547","Type":"ContainerDied","Data":"6dc6fa2595cb4f2b8549ab0a3a0793e6b03168b7489a05cc69cb0fd2c56b7547"} Sep 04 22:15:33.388996 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:33.388961 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-hxv2l" event={"ID":"67333fdd-f1ff-41b9-bb3a-504f1d3fb547","Type":"ContainerStarted","Data":"262346e01f2160e3a7aed7476d17575642a5e53044e760e4b09bc5b31217174b"} Sep 04 22:15:33.389400 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:33.389200 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-hxv2l" Sep 04 22:15:33.429705 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:33.429639 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-hxv2l" podStartSLOduration=2.1419386129999998 podStartE2EDuration="6.429618346s" totalImagesPullingTime="3.235197439s" totalInitContainerRuntime="1.052482294s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:15:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:15:33.427734461 +0000 UTC m=+437.192210070" watchObservedRunningTime="2026-09-04 22:15:33.429618346 +0000 UTC m=+437.194093960" Sep 04 22:15:44.395034 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:44.394964 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-hxv2l" Sep 04 22:15:45.741683 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:45.741645 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-868df85875-2qvbh"] Sep 04 22:15:45.742090 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:45.741892 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aef8196c-b1f4-4609-9a1b-efd94ec1b4ce" containerName="pull" Sep 04 22:15:45.742090 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:45.741902 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="aef8196c-b1f4-4609-9a1b-efd94ec1b4ce" containerName="pull" Sep 04 22:15:45.742090 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:45.741911 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aef8196c-b1f4-4609-9a1b-efd94ec1b4ce" containerName="extract" Sep 04 22:15:45.742090 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:45.741916 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="aef8196c-b1f4-4609-9a1b-efd94ec1b4ce" containerName="extract" Sep 04 22:15:45.742090 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:45.741924 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aef8196c-b1f4-4609-9a1b-efd94ec1b4ce" containerName="util" Sep 04 22:15:45.742090 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:45.741930 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="aef8196c-b1f4-4609-9a1b-efd94ec1b4ce" containerName="util" Sep 04 22:15:45.742090 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:45.741991 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="aef8196c-b1f4-4609-9a1b-efd94ec1b4ce" containerName="extract" Sep 04 22:15:45.745963 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:45.745945 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-868df85875-2qvbh" Sep 04 22:15:45.750223 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:45.750198 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-cdg74\"" Sep 04 22:15:45.750355 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:45.750198 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 04 22:15:45.750355 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:45.750199 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 04 22:15:45.750355 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:45.750255 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 04 22:15:45.755103 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:45.755082 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-868df85875-2qvbh"] Sep 04 22:15:45.770073 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:45.770053 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5a157212-e6ef-47fa-9c00-f522c947d76d-metrics-cert\") pod \"lws-controller-manager-868df85875-2qvbh\" (UID: \"5a157212-e6ef-47fa-9c00-f522c947d76d\") " pod="openshift-lws-operator/lws-controller-manager-868df85875-2qvbh" Sep 04 22:15:45.770163 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:45.770081 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdkhf\" (UniqueName: \"kubernetes.io/projected/5a157212-e6ef-47fa-9c00-f522c947d76d-kube-api-access-mdkhf\") pod \"lws-controller-manager-868df85875-2qvbh\" (UID: \"5a157212-e6ef-47fa-9c00-f522c947d76d\") " pod="openshift-lws-operator/lws-controller-manager-868df85875-2qvbh" Sep 04 22:15:45.770163 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:45.770109 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a157212-e6ef-47fa-9c00-f522c947d76d-cert\") pod \"lws-controller-manager-868df85875-2qvbh\" (UID: \"5a157212-e6ef-47fa-9c00-f522c947d76d\") " pod="openshift-lws-operator/lws-controller-manager-868df85875-2qvbh" Sep 04 22:15:45.770233 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:45.770190 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5a157212-e6ef-47fa-9c00-f522c947d76d-manager-config\") pod \"lws-controller-manager-868df85875-2qvbh\" (UID: \"5a157212-e6ef-47fa-9c00-f522c947d76d\") " pod="openshift-lws-operator/lws-controller-manager-868df85875-2qvbh" Sep 04 22:15:45.870514 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:45.870476 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5a157212-e6ef-47fa-9c00-f522c947d76d-manager-config\") pod \"lws-controller-manager-868df85875-2qvbh\" (UID: \"5a157212-e6ef-47fa-9c00-f522c947d76d\") " pod="openshift-lws-operator/lws-controller-manager-868df85875-2qvbh" Sep 04 22:15:45.870710 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:45.870525 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5a157212-e6ef-47fa-9c00-f522c947d76d-metrics-cert\") pod \"lws-controller-manager-868df85875-2qvbh\" (UID: \"5a157212-e6ef-47fa-9c00-f522c947d76d\") " pod="openshift-lws-operator/lws-controller-manager-868df85875-2qvbh" Sep 04 22:15:45.870710 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:45.870565 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdkhf\" (UniqueName: \"kubernetes.io/projected/5a157212-e6ef-47fa-9c00-f522c947d76d-kube-api-access-mdkhf\") pod \"lws-controller-manager-868df85875-2qvbh\" (UID: \"5a157212-e6ef-47fa-9c00-f522c947d76d\") " pod="openshift-lws-operator/lws-controller-manager-868df85875-2qvbh" Sep 04 22:15:45.870710 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:45.870594 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a157212-e6ef-47fa-9c00-f522c947d76d-cert\") pod \"lws-controller-manager-868df85875-2qvbh\" (UID: \"5a157212-e6ef-47fa-9c00-f522c947d76d\") " pod="openshift-lws-operator/lws-controller-manager-868df85875-2qvbh" Sep 04 22:15:45.871146 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:45.871124 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5a157212-e6ef-47fa-9c00-f522c947d76d-manager-config\") pod \"lws-controller-manager-868df85875-2qvbh\" (UID: \"5a157212-e6ef-47fa-9c00-f522c947d76d\") " pod="openshift-lws-operator/lws-controller-manager-868df85875-2qvbh" Sep 04 22:15:45.872941 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:45.872911 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5a157212-e6ef-47fa-9c00-f522c947d76d-cert\") pod \"lws-controller-manager-868df85875-2qvbh\" (UID: \"5a157212-e6ef-47fa-9c00-f522c947d76d\") " pod="openshift-lws-operator/lws-controller-manager-868df85875-2qvbh" Sep 04 22:15:45.873035 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:45.872968 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5a157212-e6ef-47fa-9c00-f522c947d76d-metrics-cert\") pod \"lws-controller-manager-868df85875-2qvbh\" (UID: \"5a157212-e6ef-47fa-9c00-f522c947d76d\") " pod="openshift-lws-operator/lws-controller-manager-868df85875-2qvbh" Sep 04 22:15:45.881610 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:45.881588 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdkhf\" (UniqueName: \"kubernetes.io/projected/5a157212-e6ef-47fa-9c00-f522c947d76d-kube-api-access-mdkhf\") pod \"lws-controller-manager-868df85875-2qvbh\" (UID: \"5a157212-e6ef-47fa-9c00-f522c947d76d\") " pod="openshift-lws-operator/lws-controller-manager-868df85875-2qvbh" Sep 04 22:15:46.055922 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:46.055894 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-868df85875-2qvbh" Sep 04 22:15:46.343168 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:46.343084 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-868df85875-2qvbh"] Sep 04 22:15:46.349883 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:15:46.349853 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a157212_e6ef_47fa_9c00_f522c947d76d.slice/crio-fe0a0d5f49988bd31e37d8a493205fa3dc8f869cac8d0b9c3d5996105c1ca54d WatchSource:0}: Error finding container fe0a0d5f49988bd31e37d8a493205fa3dc8f869cac8d0b9c3d5996105c1ca54d: Status 404 returned error can't find the container with id fe0a0d5f49988bd31e37d8a493205fa3dc8f869cac8d0b9c3d5996105c1ca54d Sep 04 22:15:46.448789 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:46.448757 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-868df85875-2qvbh" event={"ID":"5a157212-e6ef-47fa-9c00-f522c947d76d","Type":"ContainerStarted","Data":"fe0a0d5f49988bd31e37d8a493205fa3dc8f869cac8d0b9c3d5996105c1ca54d"} Sep 04 22:15:48.459948 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:48.459905 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-868df85875-2qvbh" event={"ID":"5a157212-e6ef-47fa-9c00-f522c947d76d","Type":"ContainerStarted","Data":"e56d1efb2dc2c0efa9a8b49e270686bc6b73ca18022ef3f0f5e03e95ac242b18"} Sep 04 22:15:48.460361 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:48.460007 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-868df85875-2qvbh" Sep 04 22:15:48.581295 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:48.581238 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-868df85875-2qvbh" podStartSLOduration=2.142636802 podStartE2EDuration="3.581221425s" totalImagesPullingTime="1.438584623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:15:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:15:48.577197344 +0000 UTC m=+452.341672957" watchObservedRunningTime="2026-09-04 22:15:48.581221425 +0000 UTC m=+452.345697038" Sep 04 22:15:57.392628 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:57.392591 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-glhzp"] Sep 04 22:15:57.397255 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:57.397230 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-glhzp" Sep 04 22:15:57.401137 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:57.401114 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 04 22:15:57.401446 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:57.401422 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-8gbzv\"" Sep 04 22:15:57.414384 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:57.414356 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-glhzp"] Sep 04 22:15:57.446001 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:57.445967 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65d33876-fc29-4b81-883c-9d982a830ef8-cert\") pod \"odh-model-controller-7998c67dc4-glhzp\" (UID: \"65d33876-fc29-4b81-883c-9d982a830ef8\") " pod="opendatahub/odh-model-controller-7998c67dc4-glhzp" Sep 04 22:15:57.446168 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:57.446066 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cltmg\" (UniqueName: \"kubernetes.io/projected/65d33876-fc29-4b81-883c-9d982a830ef8-kube-api-access-cltmg\") pod \"odh-model-controller-7998c67dc4-glhzp\" (UID: \"65d33876-fc29-4b81-883c-9d982a830ef8\") " pod="opendatahub/odh-model-controller-7998c67dc4-glhzp" Sep 04 22:15:57.547363 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:57.547320 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65d33876-fc29-4b81-883c-9d982a830ef8-cert\") pod \"odh-model-controller-7998c67dc4-glhzp\" (UID: \"65d33876-fc29-4b81-883c-9d982a830ef8\") " pod="opendatahub/odh-model-controller-7998c67dc4-glhzp" Sep 04 22:15:57.547534 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:57.547411 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cltmg\" (UniqueName: \"kubernetes.io/projected/65d33876-fc29-4b81-883c-9d982a830ef8-kube-api-access-cltmg\") pod \"odh-model-controller-7998c67dc4-glhzp\" (UID: \"65d33876-fc29-4b81-883c-9d982a830ef8\") " pod="opendatahub/odh-model-controller-7998c67dc4-glhzp" Sep 04 22:15:57.547534 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:15:57.547464 2574 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 04 22:15:57.547644 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:15:57.547544 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65d33876-fc29-4b81-883c-9d982a830ef8-cert podName:65d33876-fc29-4b81-883c-9d982a830ef8 nodeName:}" failed. No retries permitted until 2026-09-04 22:15:58.047526359 +0000 UTC m=+461.812001969 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/65d33876-fc29-4b81-883c-9d982a830ef8-cert") pod "odh-model-controller-7998c67dc4-glhzp" (UID: "65d33876-fc29-4b81-883c-9d982a830ef8") : secret "odh-model-controller-webhook-cert" not found Sep 04 22:15:57.561970 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:57.561938 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cltmg\" (UniqueName: \"kubernetes.io/projected/65d33876-fc29-4b81-883c-9d982a830ef8-kube-api-access-cltmg\") pod \"odh-model-controller-7998c67dc4-glhzp\" (UID: \"65d33876-fc29-4b81-883c-9d982a830ef8\") " pod="opendatahub/odh-model-controller-7998c67dc4-glhzp" Sep 04 22:15:58.051710 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:58.051675 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65d33876-fc29-4b81-883c-9d982a830ef8-cert\") pod \"odh-model-controller-7998c67dc4-glhzp\" (UID: \"65d33876-fc29-4b81-883c-9d982a830ef8\") " pod="opendatahub/odh-model-controller-7998c67dc4-glhzp" Sep 04 22:15:58.054328 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:58.054304 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65d33876-fc29-4b81-883c-9d982a830ef8-cert\") pod \"odh-model-controller-7998c67dc4-glhzp\" (UID: \"65d33876-fc29-4b81-883c-9d982a830ef8\") " pod="opendatahub/odh-model-controller-7998c67dc4-glhzp" Sep 04 22:15:58.309967 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:58.309868 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-glhzp" Sep 04 22:15:58.585921 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:58.584352 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-glhzp"] Sep 04 22:15:58.586892 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:15:58.586859 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65d33876_fc29_4b81_883c_9d982a830ef8.slice/crio-21a474dcdab04dffe9002e1bfaf3e6cb3d46e88b0ecd14dfd835b882e13c1466 WatchSource:0}: Error finding container 21a474dcdab04dffe9002e1bfaf3e6cb3d46e88b0ecd14dfd835b882e13c1466: Status 404 returned error can't find the container with id 21a474dcdab04dffe9002e1bfaf3e6cb3d46e88b0ecd14dfd835b882e13c1466 Sep 04 22:15:59.467061 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:59.467025 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-868df85875-2qvbh" Sep 04 22:15:59.523137 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:15:59.521984 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-glhzp" event={"ID":"65d33876-fc29-4b81-883c-9d982a830ef8","Type":"ContainerStarted","Data":"21a474dcdab04dffe9002e1bfaf3e6cb3d46e88b0ecd14dfd835b882e13c1466"} Sep 04 22:16:02.539664 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:02.539623 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-glhzp" event={"ID":"65d33876-fc29-4b81-883c-9d982a830ef8","Type":"ContainerStarted","Data":"2c6969d35e969c01d1b2299ec14d65f4b43752486cc707b79562e87fcbca6024"} Sep 04 22:16:02.540040 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:02.539743 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-glhzp" Sep 04 22:16:05.764907 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:05.764849 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-glhzp" podStartSLOduration=5.645285611 podStartE2EDuration="8.764831831s" totalImagesPullingTime="3.11954622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:15:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:16:02.560689942 +0000 UTC m=+466.325165558" watchObservedRunningTime="2026-09-04 22:16:05.764831831 +0000 UTC m=+469.529307446" Sep 04 22:16:05.765623 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:05.765607 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-b885p"] Sep 04 22:16:05.771158 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:05.771139 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b885p" Sep 04 22:16:05.774096 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:05.774070 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 04 22:16:05.775861 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:05.775841 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 04 22:16:05.775962 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:05.775865 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 04 22:16:05.775962 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:05.775842 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 04 22:16:05.775962 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:05.775955 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-b4srj\"" Sep 04 22:16:05.786016 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:05.785996 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-b885p"] Sep 04 22:16:05.808813 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:05.808784 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fnsz\" (UniqueName: \"kubernetes.io/projected/25883a31-d829-4bb6-b434-8d525c5f7368-kube-api-access-6fnsz\") pod \"istiod-openshift-gateway-5677ccb49b-b885p\" (UID: \"25883a31-d829-4bb6-b434-8d525c5f7368\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b885p" Sep 04 22:16:05.808924 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:05.808826 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/25883a31-d829-4bb6-b434-8d525c5f7368-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-b885p\" (UID: \"25883a31-d829-4bb6-b434-8d525c5f7368\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b885p" Sep 04 22:16:05.808924 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:05.808842 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/25883a31-d829-4bb6-b434-8d525c5f7368-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-b885p\" (UID: \"25883a31-d829-4bb6-b434-8d525c5f7368\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b885p" Sep 04 22:16:05.809039 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:05.808955 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/25883a31-d829-4bb6-b434-8d525c5f7368-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-b885p\" (UID: \"25883a31-d829-4bb6-b434-8d525c5f7368\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b885p" Sep 04 22:16:05.809039 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:05.809014 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/25883a31-d829-4bb6-b434-8d525c5f7368-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-b885p\" (UID: \"25883a31-d829-4bb6-b434-8d525c5f7368\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b885p" Sep 04 22:16:05.809146 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:05.809058 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25883a31-d829-4bb6-b434-8d525c5f7368-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-b885p\" (UID: \"25883a31-d829-4bb6-b434-8d525c5f7368\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b885p" Sep 04 22:16:05.809146 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:05.809110 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/25883a31-d829-4bb6-b434-8d525c5f7368-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-b885p\" (UID: \"25883a31-d829-4bb6-b434-8d525c5f7368\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b885p" Sep 04 22:16:05.910330 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:05.910291 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/25883a31-d829-4bb6-b434-8d525c5f7368-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-b885p\" (UID: \"25883a31-d829-4bb6-b434-8d525c5f7368\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b885p" Sep 04 22:16:05.910508 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:05.910348 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25883a31-d829-4bb6-b434-8d525c5f7368-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-b885p\" (UID: \"25883a31-d829-4bb6-b434-8d525c5f7368\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b885p" Sep 04 22:16:05.910508 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:05.910381 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/25883a31-d829-4bb6-b434-8d525c5f7368-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-b885p\" (UID: \"25883a31-d829-4bb6-b434-8d525c5f7368\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b885p" Sep 04 22:16:05.910508 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:05.910419 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fnsz\" (UniqueName: \"kubernetes.io/projected/25883a31-d829-4bb6-b434-8d525c5f7368-kube-api-access-6fnsz\") pod \"istiod-openshift-gateway-5677ccb49b-b885p\" (UID: \"25883a31-d829-4bb6-b434-8d525c5f7368\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b885p" Sep 04 22:16:05.910508 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:05.910469 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/25883a31-d829-4bb6-b434-8d525c5f7368-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-b885p\" (UID: \"25883a31-d829-4bb6-b434-8d525c5f7368\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b885p" Sep 04 22:16:05.910778 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:05.910492 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/25883a31-d829-4bb6-b434-8d525c5f7368-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-b885p\" (UID: \"25883a31-d829-4bb6-b434-8d525c5f7368\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b885p" Sep 04 22:16:05.910778 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:05.910587 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/25883a31-d829-4bb6-b434-8d525c5f7368-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-b885p\" (UID: \"25883a31-d829-4bb6-b434-8d525c5f7368\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b885p" Sep 04 22:16:05.911255 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:05.911228 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/25883a31-d829-4bb6-b434-8d525c5f7368-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-b885p\" (UID: \"25883a31-d829-4bb6-b434-8d525c5f7368\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b885p" Sep 04 22:16:05.912888 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:05.912861 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/25883a31-d829-4bb6-b434-8d525c5f7368-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-b885p\" (UID: \"25883a31-d829-4bb6-b434-8d525c5f7368\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b885p" Sep 04 22:16:05.912985 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:05.912961 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/25883a31-d829-4bb6-b434-8d525c5f7368-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-b885p\" (UID: \"25883a31-d829-4bb6-b434-8d525c5f7368\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b885p" Sep 04 22:16:05.913024 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:05.913010 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/25883a31-d829-4bb6-b434-8d525c5f7368-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-b885p\" (UID: \"25883a31-d829-4bb6-b434-8d525c5f7368\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b885p" Sep 04 22:16:05.913144 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:05.913127 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/25883a31-d829-4bb6-b434-8d525c5f7368-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-b885p\" (UID: \"25883a31-d829-4bb6-b434-8d525c5f7368\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b885p" Sep 04 22:16:05.947643 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:05.947610 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-867cfc6f4d-2kp9k"] Sep 04 22:16:05.948096 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:05.948074 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fnsz\" (UniqueName: \"kubernetes.io/projected/25883a31-d829-4bb6-b434-8d525c5f7368-kube-api-access-6fnsz\") pod \"istiod-openshift-gateway-5677ccb49b-b885p\" (UID: \"25883a31-d829-4bb6-b434-8d525c5f7368\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b885p" Sep 04 22:16:05.950408 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:05.950389 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-867cfc6f4d-2kp9k" Sep 04 22:16:05.950929 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:05.950903 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25883a31-d829-4bb6-b434-8d525c5f7368-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-b885p\" (UID: \"25883a31-d829-4bb6-b434-8d525c5f7368\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b885p" Sep 04 22:16:05.992224 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:05.992193 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-867cfc6f4d-2kp9k"] Sep 04 22:16:06.011540 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:06.011509 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/514c0fa6-6c00-4791-9c9f-05f743605730-console-serving-cert\") pod \"console-867cfc6f4d-2kp9k\" (UID: \"514c0fa6-6c00-4791-9c9f-05f743605730\") " pod="openshift-console/console-867cfc6f4d-2kp9k" Sep 04 22:16:06.011689 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:06.011600 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/514c0fa6-6c00-4791-9c9f-05f743605730-oauth-serving-cert\") pod \"console-867cfc6f4d-2kp9k\" (UID: \"514c0fa6-6c00-4791-9c9f-05f743605730\") " pod="openshift-console/console-867cfc6f4d-2kp9k" Sep 04 22:16:06.011689 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:06.011670 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/514c0fa6-6c00-4791-9c9f-05f743605730-service-ca\") pod \"console-867cfc6f4d-2kp9k\" (UID: \"514c0fa6-6c00-4791-9c9f-05f743605730\") " pod="openshift-console/console-867cfc6f4d-2kp9k" Sep 04 22:16:06.011769 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:06.011698 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/514c0fa6-6c00-4791-9c9f-05f743605730-console-config\") pod \"console-867cfc6f4d-2kp9k\" (UID: \"514c0fa6-6c00-4791-9c9f-05f743605730\") " pod="openshift-console/console-867cfc6f4d-2kp9k" Sep 04 22:16:06.011769 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:06.011721 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/514c0fa6-6c00-4791-9c9f-05f743605730-trusted-ca-bundle\") pod \"console-867cfc6f4d-2kp9k\" (UID: \"514c0fa6-6c00-4791-9c9f-05f743605730\") " pod="openshift-console/console-867cfc6f4d-2kp9k" Sep 04 22:16:06.011866 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:06.011776 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/514c0fa6-6c00-4791-9c9f-05f743605730-console-oauth-config\") pod \"console-867cfc6f4d-2kp9k\" (UID: \"514c0fa6-6c00-4791-9c9f-05f743605730\") " pod="openshift-console/console-867cfc6f4d-2kp9k" Sep 04 22:16:06.011866 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:06.011828 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szzpc\" (UniqueName: \"kubernetes.io/projected/514c0fa6-6c00-4791-9c9f-05f743605730-kube-api-access-szzpc\") pod \"console-867cfc6f4d-2kp9k\" (UID: \"514c0fa6-6c00-4791-9c9f-05f743605730\") " pod="openshift-console/console-867cfc6f4d-2kp9k" Sep 04 22:16:06.079769 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:06.079688 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b885p" Sep 04 22:16:06.112938 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:06.112907 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szzpc\" (UniqueName: \"kubernetes.io/projected/514c0fa6-6c00-4791-9c9f-05f743605730-kube-api-access-szzpc\") pod \"console-867cfc6f4d-2kp9k\" (UID: \"514c0fa6-6c00-4791-9c9f-05f743605730\") " pod="openshift-console/console-867cfc6f4d-2kp9k" Sep 04 22:16:06.113098 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:06.112955 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/514c0fa6-6c00-4791-9c9f-05f743605730-console-serving-cert\") pod \"console-867cfc6f4d-2kp9k\" (UID: \"514c0fa6-6c00-4791-9c9f-05f743605730\") " pod="openshift-console/console-867cfc6f4d-2kp9k" Sep 04 22:16:06.113098 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:06.112988 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/514c0fa6-6c00-4791-9c9f-05f743605730-oauth-serving-cert\") pod \"console-867cfc6f4d-2kp9k\" (UID: \"514c0fa6-6c00-4791-9c9f-05f743605730\") " pod="openshift-console/console-867cfc6f4d-2kp9k" Sep 04 22:16:06.113098 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:06.113030 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/514c0fa6-6c00-4791-9c9f-05f743605730-service-ca\") pod \"console-867cfc6f4d-2kp9k\" (UID: \"514c0fa6-6c00-4791-9c9f-05f743605730\") " pod="openshift-console/console-867cfc6f4d-2kp9k" Sep 04 22:16:06.113098 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:06.113056 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/514c0fa6-6c00-4791-9c9f-05f743605730-console-config\") pod \"console-867cfc6f4d-2kp9k\" (UID: \"514c0fa6-6c00-4791-9c9f-05f743605730\") " pod="openshift-console/console-867cfc6f4d-2kp9k" Sep 04 22:16:06.113098 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:06.113086 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/514c0fa6-6c00-4791-9c9f-05f743605730-trusted-ca-bundle\") pod \"console-867cfc6f4d-2kp9k\" (UID: \"514c0fa6-6c00-4791-9c9f-05f743605730\") " pod="openshift-console/console-867cfc6f4d-2kp9k" Sep 04 22:16:06.113377 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:06.113146 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/514c0fa6-6c00-4791-9c9f-05f743605730-console-oauth-config\") pod \"console-867cfc6f4d-2kp9k\" (UID: \"514c0fa6-6c00-4791-9c9f-05f743605730\") " pod="openshift-console/console-867cfc6f4d-2kp9k" Sep 04 22:16:06.113865 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:06.113830 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/514c0fa6-6c00-4791-9c9f-05f743605730-oauth-serving-cert\") pod \"console-867cfc6f4d-2kp9k\" (UID: \"514c0fa6-6c00-4791-9c9f-05f743605730\") " pod="openshift-console/console-867cfc6f4d-2kp9k" Sep 04 22:16:06.114001 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:06.113879 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/514c0fa6-6c00-4791-9c9f-05f743605730-service-ca\") pod \"console-867cfc6f4d-2kp9k\" (UID: \"514c0fa6-6c00-4791-9c9f-05f743605730\") " pod="openshift-console/console-867cfc6f4d-2kp9k" Sep 04 22:16:06.114001 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:06.113889 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/514c0fa6-6c00-4791-9c9f-05f743605730-console-config\") pod \"console-867cfc6f4d-2kp9k\" (UID: \"514c0fa6-6c00-4791-9c9f-05f743605730\") " pod="openshift-console/console-867cfc6f4d-2kp9k" Sep 04 22:16:06.114307 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:06.114283 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/514c0fa6-6c00-4791-9c9f-05f743605730-trusted-ca-bundle\") pod \"console-867cfc6f4d-2kp9k\" (UID: \"514c0fa6-6c00-4791-9c9f-05f743605730\") " pod="openshift-console/console-867cfc6f4d-2kp9k" Sep 04 22:16:06.115615 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:06.115591 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/514c0fa6-6c00-4791-9c9f-05f743605730-console-oauth-config\") pod \"console-867cfc6f4d-2kp9k\" (UID: \"514c0fa6-6c00-4791-9c9f-05f743605730\") " pod="openshift-console/console-867cfc6f4d-2kp9k" Sep 04 22:16:06.115853 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:06.115834 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/514c0fa6-6c00-4791-9c9f-05f743605730-console-serving-cert\") pod \"console-867cfc6f4d-2kp9k\" (UID: \"514c0fa6-6c00-4791-9c9f-05f743605730\") " pod="openshift-console/console-867cfc6f4d-2kp9k" Sep 04 22:16:06.124154 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:06.124133 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szzpc\" (UniqueName: \"kubernetes.io/projected/514c0fa6-6c00-4791-9c9f-05f743605730-kube-api-access-szzpc\") pod \"console-867cfc6f4d-2kp9k\" (UID: \"514c0fa6-6c00-4791-9c9f-05f743605730\") " pod="openshift-console/console-867cfc6f4d-2kp9k" Sep 04 22:16:06.266526 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:06.266485 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-867cfc6f4d-2kp9k" Sep 04 22:16:06.368537 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:06.368452 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-b885p"] Sep 04 22:16:06.371589 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:16:06.371561 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25883a31_d829_4bb6_b434_8d525c5f7368.slice/crio-3fb0786b3973e8b35346bf347c1c5ec3a6ca9d44342c6eb69bd6fadd2d821672 WatchSource:0}: Error finding container 3fb0786b3973e8b35346bf347c1c5ec3a6ca9d44342c6eb69bd6fadd2d821672: Status 404 returned error can't find the container with id 3fb0786b3973e8b35346bf347c1c5ec3a6ca9d44342c6eb69bd6fadd2d821672 Sep 04 22:16:06.558203 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:06.558162 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b885p" event={"ID":"25883a31-d829-4bb6-b434-8d525c5f7368","Type":"ContainerStarted","Data":"3fb0786b3973e8b35346bf347c1c5ec3a6ca9d44342c6eb69bd6fadd2d821672"} Sep 04 22:16:06.578208 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:06.578183 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-867cfc6f4d-2kp9k"] Sep 04 22:16:06.579685 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:16:06.579659 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod514c0fa6_6c00_4791_9c9f_05f743605730.slice/crio-2711f598219a72943cca1f650739f2129674d2c8e595f3b9f8b183712c9e3051 WatchSource:0}: Error finding container 2711f598219a72943cca1f650739f2129674d2c8e595f3b9f8b183712c9e3051: Status 404 returned error can't find the container with id 2711f598219a72943cca1f650739f2129674d2c8e595f3b9f8b183712c9e3051 Sep 04 22:16:07.565598 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:07.565544 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-867cfc6f4d-2kp9k" event={"ID":"514c0fa6-6c00-4791-9c9f-05f743605730","Type":"ContainerStarted","Data":"c155150f07927d71179bc1cd7240de126d0e351e11e7cde2a80e9210efdbde5e"} Sep 04 22:16:07.566095 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:07.565612 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-867cfc6f4d-2kp9k" event={"ID":"514c0fa6-6c00-4791-9c9f-05f743605730","Type":"ContainerStarted","Data":"2711f598219a72943cca1f650739f2129674d2c8e595f3b9f8b183712c9e3051"} Sep 04 22:16:07.645048 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:07.644991 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-867cfc6f4d-2kp9k" podStartSLOduration=2.644975456 podStartE2EDuration="2.644975456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:16:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:16:07.635411902 +0000 UTC m=+471.399887518" watchObservedRunningTime="2026-09-04 22:16:07.644975456 +0000 UTC m=+471.409451069" Sep 04 22:16:08.585344 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:08.585302 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 04 22:16:08.585652 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:08.585374 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 04 22:16:09.582170 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:09.582128 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b885p" event={"ID":"25883a31-d829-4bb6-b434-8d525c5f7368","Type":"ContainerStarted","Data":"8241623d25b1966a91ee6fe750db2fd90081677c425a00e9e13d5fb0fcedcb18"} Sep 04 22:16:09.582369 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:09.582302 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b885p" Sep 04 22:16:09.583982 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:09.583954 2574 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-b885p container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 04 22:16:09.584105 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:09.584003 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b885p" podUID="25883a31-d829-4bb6-b434-8d525c5f7368" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 04 22:16:09.772532 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:09.772458 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b885p" podStartSLOduration=2.561701364 podStartE2EDuration="4.772434399s" totalImagesPullingTime="2.210733035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:16:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:16:09.761440737 +0000 UTC m=+473.525916350" watchObservedRunningTime="2026-09-04 22:16:09.772434399 +0000 UTC m=+473.536910014" Sep 04 22:16:10.587123 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:10.587094 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-b885p" Sep 04 22:16:13.545943 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:13.545915 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-glhzp" Sep 04 22:16:16.267520 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:16.267480 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-867cfc6f4d-2kp9k" Sep 04 22:16:16.267892 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:16.267538 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-867cfc6f4d-2kp9k" Sep 04 22:16:16.272323 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:16.272296 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-867cfc6f4d-2kp9k" Sep 04 22:16:16.616079 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:16.615994 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-867cfc6f4d-2kp9k" Sep 04 22:16:16.739445 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:16.739416 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d6dfff4f-cjfdl"] Sep 04 22:16:41.758559 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:41.758499 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6d6dfff4f-cjfdl" podUID="32c17fdf-2303-4355-bf1f-8ecc1062f7f4" containerName="console" containerID="cri-o://3772a3f4445460961d455916a1cf92d0632effb559e0d199edf9a7583e272735" gracePeriod=15 Sep 04 22:16:41.992630 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:41.992603 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d6dfff4f-cjfdl" Sep 04 22:16:42.118799 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:42.118710 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32c17fdf-2303-4355-bf1f-8ecc1062f7f4-console-oauth-config\") pod \"32c17fdf-2303-4355-bf1f-8ecc1062f7f4\" (UID: \"32c17fdf-2303-4355-bf1f-8ecc1062f7f4\") " Sep 04 22:16:42.118799 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:42.118751 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32c17fdf-2303-4355-bf1f-8ecc1062f7f4-service-ca\") pod \"32c17fdf-2303-4355-bf1f-8ecc1062f7f4\" (UID: \"32c17fdf-2303-4355-bf1f-8ecc1062f7f4\") " Sep 04 22:16:42.118799 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:42.118778 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32c17fdf-2303-4355-bf1f-8ecc1062f7f4-console-serving-cert\") pod \"32c17fdf-2303-4355-bf1f-8ecc1062f7f4\" (UID: \"32c17fdf-2303-4355-bf1f-8ecc1062f7f4\") " Sep 04 22:16:42.118799 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:42.118801 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32c17fdf-2303-4355-bf1f-8ecc1062f7f4-oauth-serving-cert\") pod \"32c17fdf-2303-4355-bf1f-8ecc1062f7f4\" (UID: \"32c17fdf-2303-4355-bf1f-8ecc1062f7f4\") " Sep 04 22:16:42.119087 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:42.118852 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-92wmc\" (UniqueName: \"kubernetes.io/projected/32c17fdf-2303-4355-bf1f-8ecc1062f7f4-kube-api-access-92wmc\") pod \"32c17fdf-2303-4355-bf1f-8ecc1062f7f4\" (UID: \"32c17fdf-2303-4355-bf1f-8ecc1062f7f4\") " Sep 04 22:16:42.119087 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:42.118871 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32c17fdf-2303-4355-bf1f-8ecc1062f7f4-console-config\") pod \"32c17fdf-2303-4355-bf1f-8ecc1062f7f4\" (UID: \"32c17fdf-2303-4355-bf1f-8ecc1062f7f4\") " Sep 04 22:16:42.119087 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:42.118917 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32c17fdf-2303-4355-bf1f-8ecc1062f7f4-trusted-ca-bundle\") pod \"32c17fdf-2303-4355-bf1f-8ecc1062f7f4\" (UID: \"32c17fdf-2303-4355-bf1f-8ecc1062f7f4\") " Sep 04 22:16:42.119318 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:42.119287 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32c17fdf-2303-4355-bf1f-8ecc1062f7f4-console-config" (OuterVolumeSpecName: "console-config") pod "32c17fdf-2303-4355-bf1f-8ecc1062f7f4" (UID: "32c17fdf-2303-4355-bf1f-8ecc1062f7f4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:16:42.119450 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:42.119319 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32c17fdf-2303-4355-bf1f-8ecc1062f7f4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "32c17fdf-2303-4355-bf1f-8ecc1062f7f4" (UID: "32c17fdf-2303-4355-bf1f-8ecc1062f7f4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:16:42.119509 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:42.119451 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32c17fdf-2303-4355-bf1f-8ecc1062f7f4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "32c17fdf-2303-4355-bf1f-8ecc1062f7f4" (UID: "32c17fdf-2303-4355-bf1f-8ecc1062f7f4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:16:42.119509 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:42.119493 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32c17fdf-2303-4355-bf1f-8ecc1062f7f4-service-ca" (OuterVolumeSpecName: "service-ca") pod "32c17fdf-2303-4355-bf1f-8ecc1062f7f4" (UID: "32c17fdf-2303-4355-bf1f-8ecc1062f7f4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:16:42.121058 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:42.121028 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32c17fdf-2303-4355-bf1f-8ecc1062f7f4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "32c17fdf-2303-4355-bf1f-8ecc1062f7f4" (UID: "32c17fdf-2303-4355-bf1f-8ecc1062f7f4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 04 22:16:42.121058 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:42.121028 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32c17fdf-2303-4355-bf1f-8ecc1062f7f4-kube-api-access-92wmc" (OuterVolumeSpecName: "kube-api-access-92wmc") pod "32c17fdf-2303-4355-bf1f-8ecc1062f7f4" (UID: "32c17fdf-2303-4355-bf1f-8ecc1062f7f4"). InnerVolumeSpecName "kube-api-access-92wmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:16:42.121198 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:42.121109 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32c17fdf-2303-4355-bf1f-8ecc1062f7f4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "32c17fdf-2303-4355-bf1f-8ecc1062f7f4" (UID: "32c17fdf-2303-4355-bf1f-8ecc1062f7f4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 04 22:16:42.219878 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:42.219853 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32c17fdf-2303-4355-bf1f-8ecc1062f7f4-console-oauth-config\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:16:42.219878 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:42.219875 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32c17fdf-2303-4355-bf1f-8ecc1062f7f4-service-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:16:42.220018 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:42.219884 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32c17fdf-2303-4355-bf1f-8ecc1062f7f4-console-serving-cert\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:16:42.220018 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:42.219892 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32c17fdf-2303-4355-bf1f-8ecc1062f7f4-oauth-serving-cert\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:16:42.220018 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:42.219900 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92wmc\" (UniqueName: \"kubernetes.io/projected/32c17fdf-2303-4355-bf1f-8ecc1062f7f4-kube-api-access-92wmc\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:16:42.220018 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:42.219909 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32c17fdf-2303-4355-bf1f-8ecc1062f7f4-console-config\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:16:42.220018 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:42.219917 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32c17fdf-2303-4355-bf1f-8ecc1062f7f4-trusted-ca-bundle\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:16:42.737398 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:42.737366 2574 generic.go:356] "Generic (PLEG): container finished" podID="32c17fdf-2303-4355-bf1f-8ecc1062f7f4" containerID="3772a3f4445460961d455916a1cf92d0632effb559e0d199edf9a7583e272735" exitCode=0 Sep 04 22:16:42.737537 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:42.737405 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d6dfff4f-cjfdl" Sep 04 22:16:42.737537 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:42.737419 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d6dfff4f-cjfdl" event={"ID":"32c17fdf-2303-4355-bf1f-8ecc1062f7f4","Type":"ContainerDied","Data":"3772a3f4445460961d455916a1cf92d0632effb559e0d199edf9a7583e272735"} Sep 04 22:16:42.737537 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:42.737459 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d6dfff4f-cjfdl" event={"ID":"32c17fdf-2303-4355-bf1f-8ecc1062f7f4","Type":"ContainerDied","Data":"d223f3e4774f79264965cf3ab15e86c0a77c2823ab1ebf8f65eeb99203ad5ac1"} Sep 04 22:16:42.737537 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:42.737477 2574 scope.go:117] "RemoveContainer" containerID="3772a3f4445460961d455916a1cf92d0632effb559e0d199edf9a7583e272735" Sep 04 22:16:42.745438 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:42.745422 2574 scope.go:117] "RemoveContainer" containerID="3772a3f4445460961d455916a1cf92d0632effb559e0d199edf9a7583e272735" Sep 04 22:16:42.745737 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:16:42.745718 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3772a3f4445460961d455916a1cf92d0632effb559e0d199edf9a7583e272735\": container with ID starting with 3772a3f4445460961d455916a1cf92d0632effb559e0d199edf9a7583e272735 not found: ID does not exist" containerID="3772a3f4445460961d455916a1cf92d0632effb559e0d199edf9a7583e272735" Sep 04 22:16:42.745788 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:42.745743 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3772a3f4445460961d455916a1cf92d0632effb559e0d199edf9a7583e272735"} err="failed to get container status \"3772a3f4445460961d455916a1cf92d0632effb559e0d199edf9a7583e272735\": rpc error: code = NotFound desc = could not find container \"3772a3f4445460961d455916a1cf92d0632effb559e0d199edf9a7583e272735\": container with ID starting with 3772a3f4445460961d455916a1cf92d0632effb559e0d199edf9a7583e272735 not found: ID does not exist" Sep 04 22:16:42.756373 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:42.756348 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d6dfff4f-cjfdl"] Sep 04 22:16:42.762512 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:42.762491 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6d6dfff4f-cjfdl"] Sep 04 22:16:44.714277 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:16:44.714239 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32c17fdf-2303-4355-bf1f-8ecc1062f7f4" path="/var/lib/kubelet/pods/32c17fdf-2303-4355-bf1f-8ecc1062f7f4/volumes" Sep 04 22:17:02.378893 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.378861 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bblfz9r"] Sep 04 22:17:02.379340 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.379132 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32c17fdf-2303-4355-bf1f-8ecc1062f7f4" containerName="console" Sep 04 22:17:02.379340 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.379143 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="32c17fdf-2303-4355-bf1f-8ecc1062f7f4" containerName="console" Sep 04 22:17:02.379340 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.379212 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="32c17fdf-2303-4355-bf1f-8ecc1062f7f4" containerName="console" Sep 04 22:17:02.383616 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.383597 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bblfz9r" Sep 04 22:17:02.386214 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.386192 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-pxxpr\"" Sep 04 22:17:02.386318 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.386244 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 04 22:17:02.386318 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.386244 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 04 22:17:02.391908 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.391888 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bblfz9r"] Sep 04 22:17:02.468470 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.468440 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a026g67"] Sep 04 22:17:02.470850 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.470827 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/146651f4-fda6-4f58-915c-a3031b517cfb-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bblfz9r\" (UID: \"146651f4-fda6-4f58-915c-a3031b517cfb\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bblfz9r" Sep 04 22:17:02.470919 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.470859 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/146651f4-fda6-4f58-915c-a3031b517cfb-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bblfz9r\" (UID: \"146651f4-fda6-4f58-915c-a3031b517cfb\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bblfz9r" Sep 04 22:17:02.471050 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.471020 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kdl4\" (UniqueName: \"kubernetes.io/projected/146651f4-fda6-4f58-915c-a3031b517cfb-kube-api-access-8kdl4\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bblfz9r\" (UID: \"146651f4-fda6-4f58-915c-a3031b517cfb\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bblfz9r" Sep 04 22:17:02.471626 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.471611 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a026g67" Sep 04 22:17:02.482612 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.482590 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a026g67"] Sep 04 22:17:02.572423 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.572393 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e66bae1-5543-47ab-8d1c-68fd4ccbffc2-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a026g67\" (UID: \"4e66bae1-5543-47ab-8d1c-68fd4ccbffc2\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a026g67" Sep 04 22:17:02.572611 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.572451 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kdl4\" (UniqueName: \"kubernetes.io/projected/146651f4-fda6-4f58-915c-a3031b517cfb-kube-api-access-8kdl4\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bblfz9r\" (UID: \"146651f4-fda6-4f58-915c-a3031b517cfb\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bblfz9r" Sep 04 22:17:02.572611 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.572514 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e66bae1-5543-47ab-8d1c-68fd4ccbffc2-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a026g67\" (UID: \"4e66bae1-5543-47ab-8d1c-68fd4ccbffc2\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a026g67" Sep 04 22:17:02.572611 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.572580 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crvpz\" (UniqueName: \"kubernetes.io/projected/4e66bae1-5543-47ab-8d1c-68fd4ccbffc2-kube-api-access-crvpz\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a026g67\" (UID: \"4e66bae1-5543-47ab-8d1c-68fd4ccbffc2\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a026g67" Sep 04 22:17:02.572611 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.572605 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/146651f4-fda6-4f58-915c-a3031b517cfb-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bblfz9r\" (UID: \"146651f4-fda6-4f58-915c-a3031b517cfb\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bblfz9r" Sep 04 22:17:02.572826 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.572623 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/146651f4-fda6-4f58-915c-a3031b517cfb-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bblfz9r\" (UID: \"146651f4-fda6-4f58-915c-a3031b517cfb\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bblfz9r" Sep 04 22:17:02.573028 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.573008 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/146651f4-fda6-4f58-915c-a3031b517cfb-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bblfz9r\" (UID: \"146651f4-fda6-4f58-915c-a3031b517cfb\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bblfz9r" Sep 04 22:17:02.573095 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.573029 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/146651f4-fda6-4f58-915c-a3031b517cfb-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bblfz9r\" (UID: \"146651f4-fda6-4f58-915c-a3031b517cfb\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bblfz9r" Sep 04 22:17:02.575930 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.575906 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c54x8k"] Sep 04 22:17:02.579280 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.579263 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c54x8k" Sep 04 22:17:02.583360 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.583340 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kdl4\" (UniqueName: \"kubernetes.io/projected/146651f4-fda6-4f58-915c-a3031b517cfb-kube-api-access-8kdl4\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bblfz9r\" (UID: \"146651f4-fda6-4f58-915c-a3031b517cfb\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bblfz9r" Sep 04 22:17:02.591185 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.591161 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c54x8k"] Sep 04 22:17:02.673314 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.673229 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4d4829c-5ae8-493f-93e4-fb441fbdc995-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c54x8k\" (UID: \"a4d4829c-5ae8-493f-93e4-fb441fbdc995\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c54x8k" Sep 04 22:17:02.673314 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.673277 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w57mk\" (UniqueName: \"kubernetes.io/projected/a4d4829c-5ae8-493f-93e4-fb441fbdc995-kube-api-access-w57mk\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c54x8k\" (UID: \"a4d4829c-5ae8-493f-93e4-fb441fbdc995\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c54x8k" Sep 04 22:17:02.673536 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.673398 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e66bae1-5543-47ab-8d1c-68fd4ccbffc2-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a026g67\" (UID: \"4e66bae1-5543-47ab-8d1c-68fd4ccbffc2\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a026g67" Sep 04 22:17:02.673536 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.673437 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crvpz\" (UniqueName: \"kubernetes.io/projected/4e66bae1-5543-47ab-8d1c-68fd4ccbffc2-kube-api-access-crvpz\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a026g67\" (UID: \"4e66bae1-5543-47ab-8d1c-68fd4ccbffc2\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a026g67" Sep 04 22:17:02.673536 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.673470 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4d4829c-5ae8-493f-93e4-fb441fbdc995-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c54x8k\" (UID: \"a4d4829c-5ae8-493f-93e4-fb441fbdc995\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c54x8k" Sep 04 22:17:02.673536 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.673504 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e66bae1-5543-47ab-8d1c-68fd4ccbffc2-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a026g67\" (UID: \"4e66bae1-5543-47ab-8d1c-68fd4ccbffc2\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a026g67" Sep 04 22:17:02.673870 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.673852 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e66bae1-5543-47ab-8d1c-68fd4ccbffc2-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a026g67\" (UID: \"4e66bae1-5543-47ab-8d1c-68fd4ccbffc2\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a026g67" Sep 04 22:17:02.673946 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.673862 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e66bae1-5543-47ab-8d1c-68fd4ccbffc2-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a026g67\" (UID: \"4e66bae1-5543-47ab-8d1c-68fd4ccbffc2\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a026g67" Sep 04 22:17:02.674046 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.674028 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053gqzvc"] Sep 04 22:17:02.677507 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.677490 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053gqzvc" Sep 04 22:17:02.682131 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.682109 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crvpz\" (UniqueName: \"kubernetes.io/projected/4e66bae1-5543-47ab-8d1c-68fd4ccbffc2-kube-api-access-crvpz\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a026g67\" (UID: \"4e66bae1-5543-47ab-8d1c-68fd4ccbffc2\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a026g67" Sep 04 22:17:02.686664 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.686642 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053gqzvc"] Sep 04 22:17:02.693821 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.693802 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bblfz9r" Sep 04 22:17:02.774380 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.774353 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90252eef-35e0-47b8-a4f0-cc438c183380-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053gqzvc\" (UID: \"90252eef-35e0-47b8-a4f0-cc438c183380\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053gqzvc" Sep 04 22:17:02.774530 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.774385 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90252eef-35e0-47b8-a4f0-cc438c183380-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053gqzvc\" (UID: \"90252eef-35e0-47b8-a4f0-cc438c183380\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053gqzvc" Sep 04 22:17:02.774530 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.774427 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4d4829c-5ae8-493f-93e4-fb441fbdc995-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c54x8k\" (UID: \"a4d4829c-5ae8-493f-93e4-fb441fbdc995\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c54x8k" Sep 04 22:17:02.774530 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.774478 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qzqj\" (UniqueName: \"kubernetes.io/projected/90252eef-35e0-47b8-a4f0-cc438c183380-kube-api-access-7qzqj\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053gqzvc\" (UID: \"90252eef-35e0-47b8-a4f0-cc438c183380\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053gqzvc" Sep 04 22:17:02.774530 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.774522 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4d4829c-5ae8-493f-93e4-fb441fbdc995-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c54x8k\" (UID: \"a4d4829c-5ae8-493f-93e4-fb441fbdc995\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c54x8k" Sep 04 22:17:02.774791 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.774569 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w57mk\" (UniqueName: \"kubernetes.io/projected/a4d4829c-5ae8-493f-93e4-fb441fbdc995-kube-api-access-w57mk\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c54x8k\" (UID: \"a4d4829c-5ae8-493f-93e4-fb441fbdc995\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c54x8k" Sep 04 22:17:02.774863 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.774842 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4d4829c-5ae8-493f-93e4-fb441fbdc995-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c54x8k\" (UID: \"a4d4829c-5ae8-493f-93e4-fb441fbdc995\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c54x8k" Sep 04 22:17:02.774928 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.774866 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4d4829c-5ae8-493f-93e4-fb441fbdc995-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c54x8k\" (UID: \"a4d4829c-5ae8-493f-93e4-fb441fbdc995\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c54x8k" Sep 04 22:17:02.780287 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.780253 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a026g67" Sep 04 22:17:02.784086 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.784066 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w57mk\" (UniqueName: \"kubernetes.io/projected/a4d4829c-5ae8-493f-93e4-fb441fbdc995-kube-api-access-w57mk\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c54x8k\" (UID: \"a4d4829c-5ae8-493f-93e4-fb441fbdc995\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c54x8k" Sep 04 22:17:02.875180 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.875150 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qzqj\" (UniqueName: \"kubernetes.io/projected/90252eef-35e0-47b8-a4f0-cc438c183380-kube-api-access-7qzqj\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053gqzvc\" (UID: \"90252eef-35e0-47b8-a4f0-cc438c183380\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053gqzvc" Sep 04 22:17:02.875322 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.875221 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90252eef-35e0-47b8-a4f0-cc438c183380-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053gqzvc\" (UID: \"90252eef-35e0-47b8-a4f0-cc438c183380\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053gqzvc" Sep 04 22:17:02.875420 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.875393 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90252eef-35e0-47b8-a4f0-cc438c183380-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053gqzvc\" (UID: \"90252eef-35e0-47b8-a4f0-cc438c183380\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053gqzvc" Sep 04 22:17:02.875601 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.875580 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90252eef-35e0-47b8-a4f0-cc438c183380-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053gqzvc\" (UID: \"90252eef-35e0-47b8-a4f0-cc438c183380\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053gqzvc" Sep 04 22:17:02.875703 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.875689 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90252eef-35e0-47b8-a4f0-cc438c183380-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053gqzvc\" (UID: \"90252eef-35e0-47b8-a4f0-cc438c183380\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053gqzvc" Sep 04 22:17:02.884881 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.884861 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qzqj\" (UniqueName: \"kubernetes.io/projected/90252eef-35e0-47b8-a4f0-cc438c183380-kube-api-access-7qzqj\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053gqzvc\" (UID: \"90252eef-35e0-47b8-a4f0-cc438c183380\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053gqzvc" Sep 04 22:17:02.894662 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.894646 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c54x8k" Sep 04 22:17:02.972320 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.972287 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bblfz9r"] Sep 04 22:17:02.974725 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:17:02.974684 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146651f4_fda6_4f58_915c_a3031b517cfb.slice/crio-2e3bd6b09677fe9c374bf7606a266f384566e92a87d6d97182c89ff11ecc9c0a WatchSource:0}: Error finding container 2e3bd6b09677fe9c374bf7606a266f384566e92a87d6d97182c89ff11ecc9c0a: Status 404 returned error can't find the container with id 2e3bd6b09677fe9c374bf7606a266f384566e92a87d6d97182c89ff11ecc9c0a Sep 04 22:17:02.994466 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:02.994444 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053gqzvc" Sep 04 22:17:03.059366 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:03.059333 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a026g67"] Sep 04 22:17:03.059856 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:17:03.059828 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e66bae1_5543_47ab_8d1c_68fd4ccbffc2.slice/crio-228caa9ab99784f24470be5724a357f3d3fef073c32dff7b00ed95b8c4c294b2 WatchSource:0}: Error finding container 228caa9ab99784f24470be5724a357f3d3fef073c32dff7b00ed95b8c4c294b2: Status 404 returned error can't find the container with id 228caa9ab99784f24470be5724a357f3d3fef073c32dff7b00ed95b8c4c294b2 Sep 04 22:17:03.177956 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:03.177920 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c54x8k"] Sep 04 22:17:03.178843 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:17:03.178818 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4d4829c_5ae8_493f_93e4_fb441fbdc995.slice/crio-1d0ff81ce5cb78a4cfc26c626d4aa30cd1f12adc570cde16992fb19efa7a076a WatchSource:0}: Error finding container 1d0ff81ce5cb78a4cfc26c626d4aa30cd1f12adc570cde16992fb19efa7a076a: Status 404 returned error can't find the container with id 1d0ff81ce5cb78a4cfc26c626d4aa30cd1f12adc570cde16992fb19efa7a076a Sep 04 22:17:03.268007 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:03.267984 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053gqzvc"] Sep 04 22:17:03.269587 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:17:03.269535 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90252eef_35e0_47b8_a4f0_cc438c183380.slice/crio-229a51f4f7cf3c8b07465cfb9601ca5ceca4f8ea9b2133941aa268138153f816 WatchSource:0}: Error finding container 229a51f4f7cf3c8b07465cfb9601ca5ceca4f8ea9b2133941aa268138153f816: Status 404 returned error can't find the container with id 229a51f4f7cf3c8b07465cfb9601ca5ceca4f8ea9b2133941aa268138153f816 Sep 04 22:17:03.832667 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:03.832636 2574 generic.go:356] "Generic (PLEG): container finished" podID="146651f4-fda6-4f58-915c-a3031b517cfb" containerID="b75c6805d6408e36168ed56c9bcd6ffd7252c5dbdb08c908805589956b447335" exitCode=0 Sep 04 22:17:03.833100 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:03.832723 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bblfz9r" event={"ID":"146651f4-fda6-4f58-915c-a3031b517cfb","Type":"ContainerDied","Data":"b75c6805d6408e36168ed56c9bcd6ffd7252c5dbdb08c908805589956b447335"} Sep 04 22:17:03.833100 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:03.832775 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bblfz9r" event={"ID":"146651f4-fda6-4f58-915c-a3031b517cfb","Type":"ContainerStarted","Data":"2e3bd6b09677fe9c374bf7606a266f384566e92a87d6d97182c89ff11ecc9c0a"} Sep 04 22:17:03.834422 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:03.834397 2574 generic.go:356] "Generic (PLEG): container finished" podID="90252eef-35e0-47b8-a4f0-cc438c183380" containerID="fdc83b8a9b91327ae11d68fe2aa246f5088ce8c637c4e985f91b88afede97f97" exitCode=0 Sep 04 22:17:03.834521 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:03.834472 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053gqzvc" event={"ID":"90252eef-35e0-47b8-a4f0-cc438c183380","Type":"ContainerDied","Data":"fdc83b8a9b91327ae11d68fe2aa246f5088ce8c637c4e985f91b88afede97f97"} Sep 04 22:17:03.834607 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:03.834521 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053gqzvc" event={"ID":"90252eef-35e0-47b8-a4f0-cc438c183380","Type":"ContainerStarted","Data":"229a51f4f7cf3c8b07465cfb9601ca5ceca4f8ea9b2133941aa268138153f816"} Sep 04 22:17:03.836020 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:03.836003 2574 generic.go:356] "Generic (PLEG): container finished" podID="4e66bae1-5543-47ab-8d1c-68fd4ccbffc2" containerID="f92114169097b9736c8b378b0651142a801e3e640d9e620dd24a935383fb5c74" exitCode=0 Sep 04 22:17:03.836101 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:03.836082 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a026g67" event={"ID":"4e66bae1-5543-47ab-8d1c-68fd4ccbffc2","Type":"ContainerDied","Data":"f92114169097b9736c8b378b0651142a801e3e640d9e620dd24a935383fb5c74"} Sep 04 22:17:03.836145 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:03.836126 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a026g67" event={"ID":"4e66bae1-5543-47ab-8d1c-68fd4ccbffc2","Type":"ContainerStarted","Data":"228caa9ab99784f24470be5724a357f3d3fef073c32dff7b00ed95b8c4c294b2"} Sep 04 22:17:03.837903 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:03.837886 2574 generic.go:356] "Generic (PLEG): container finished" podID="a4d4829c-5ae8-493f-93e4-fb441fbdc995" containerID="a379b32340ccff423f2b0d511ac540eba7377986b42c5e2c78c310c04cc78417" exitCode=0 Sep 04 22:17:03.837980 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:03.837913 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c54x8k" event={"ID":"a4d4829c-5ae8-493f-93e4-fb441fbdc995","Type":"ContainerDied","Data":"a379b32340ccff423f2b0d511ac540eba7377986b42c5e2c78c310c04cc78417"} Sep 04 22:17:03.837980 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:03.837946 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c54x8k" event={"ID":"a4d4829c-5ae8-493f-93e4-fb441fbdc995","Type":"ContainerStarted","Data":"1d0ff81ce5cb78a4cfc26c626d4aa30cd1f12adc570cde16992fb19efa7a076a"} Sep 04 22:17:04.844480 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:04.844447 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a026g67" event={"ID":"4e66bae1-5543-47ab-8d1c-68fd4ccbffc2","Type":"ContainerStarted","Data":"6d639edcc32f62c745f5dde94ff86548062262d56644792b02b9a7c3bc5bf21d"} Sep 04 22:17:05.850640 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:05.850606 2574 generic.go:356] "Generic (PLEG): container finished" podID="a4d4829c-5ae8-493f-93e4-fb441fbdc995" containerID="c93160af1ee9523feb8ce4beb7e16548124f18b67fcae0da0a104cc647ce4472" exitCode=0 Sep 04 22:17:05.851049 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:05.850694 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c54x8k" event={"ID":"a4d4829c-5ae8-493f-93e4-fb441fbdc995","Type":"ContainerDied","Data":"c93160af1ee9523feb8ce4beb7e16548124f18b67fcae0da0a104cc647ce4472"} Sep 04 22:17:05.852632 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:05.852613 2574 generic.go:356] "Generic (PLEG): container finished" podID="90252eef-35e0-47b8-a4f0-cc438c183380" containerID="f1f7fabcb0878fdde56536034ca24bdb7c6347e3d82e94da72cf8382d79c26b7" exitCode=0 Sep 04 22:17:05.852716 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:05.852662 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053gqzvc" event={"ID":"90252eef-35e0-47b8-a4f0-cc438c183380","Type":"ContainerDied","Data":"f1f7fabcb0878fdde56536034ca24bdb7c6347e3d82e94da72cf8382d79c26b7"} Sep 04 22:17:05.854602 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:05.854582 2574 generic.go:356] "Generic (PLEG): container finished" podID="4e66bae1-5543-47ab-8d1c-68fd4ccbffc2" containerID="6d639edcc32f62c745f5dde94ff86548062262d56644792b02b9a7c3bc5bf21d" exitCode=0 Sep 04 22:17:05.854689 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:05.854629 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a026g67" event={"ID":"4e66bae1-5543-47ab-8d1c-68fd4ccbffc2","Type":"ContainerDied","Data":"6d639edcc32f62c745f5dde94ff86548062262d56644792b02b9a7c3bc5bf21d"} Sep 04 22:17:06.860664 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:06.860630 2574 generic.go:356] "Generic (PLEG): container finished" podID="146651f4-fda6-4f58-915c-a3031b517cfb" containerID="504f4e9b6ccd9d810df0abb6ca97c46070288d02c14cc353ebac81b4aa9c1e4b" exitCode=0 Sep 04 22:17:06.861037 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:06.860719 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bblfz9r" event={"ID":"146651f4-fda6-4f58-915c-a3031b517cfb","Type":"ContainerDied","Data":"504f4e9b6ccd9d810df0abb6ca97c46070288d02c14cc353ebac81b4aa9c1e4b"} Sep 04 22:17:06.862891 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:06.862867 2574 generic.go:356] "Generic (PLEG): container finished" podID="90252eef-35e0-47b8-a4f0-cc438c183380" containerID="5d29c7009d695844c91221dc6f6fadcb373639a25ae6b2f1b68b55fb9b2a99a5" exitCode=0 Sep 04 22:17:06.862985 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:06.862954 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053gqzvc" event={"ID":"90252eef-35e0-47b8-a4f0-cc438c183380","Type":"ContainerDied","Data":"5d29c7009d695844c91221dc6f6fadcb373639a25ae6b2f1b68b55fb9b2a99a5"} Sep 04 22:17:06.865026 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:06.864997 2574 generic.go:356] "Generic (PLEG): container finished" podID="4e66bae1-5543-47ab-8d1c-68fd4ccbffc2" containerID="dda86c4f0572a56ac4b618b274bcff914f989e9f1cf2a91f418a53db0fce169a" exitCode=0 Sep 04 22:17:06.865131 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:06.865072 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a026g67" event={"ID":"4e66bae1-5543-47ab-8d1c-68fd4ccbffc2","Type":"ContainerDied","Data":"dda86c4f0572a56ac4b618b274bcff914f989e9f1cf2a91f418a53db0fce169a"} Sep 04 22:17:06.867271 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:06.867250 2574 generic.go:356] "Generic (PLEG): container finished" podID="a4d4829c-5ae8-493f-93e4-fb441fbdc995" containerID="cdfa99ac390b86c82ae458d0022ce8e686f357d8fc37aca26645b13024a9cebd" exitCode=0 Sep 04 22:17:06.867348 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:06.867298 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c54x8k" event={"ID":"a4d4829c-5ae8-493f-93e4-fb441fbdc995","Type":"ContainerDied","Data":"cdfa99ac390b86c82ae458d0022ce8e686f357d8fc37aca26645b13024a9cebd"} Sep 04 22:17:07.874288 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:07.874258 2574 generic.go:356] "Generic (PLEG): container finished" podID="146651f4-fda6-4f58-915c-a3031b517cfb" containerID="45690a1e9793a1986dc590c5aeccc010293108d0492ae774612f32761c598be1" exitCode=0 Sep 04 22:17:07.874741 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:07.874329 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bblfz9r" event={"ID":"146651f4-fda6-4f58-915c-a3031b517cfb","Type":"ContainerDied","Data":"45690a1e9793a1986dc590c5aeccc010293108d0492ae774612f32761c598be1"} Sep 04 22:17:08.014910 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:08.014889 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c54x8k" Sep 04 22:17:08.048502 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:08.048477 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053gqzvc" Sep 04 22:17:08.051615 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:08.051595 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a026g67" Sep 04 22:17:08.119764 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:08.119737 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crvpz\" (UniqueName: \"kubernetes.io/projected/4e66bae1-5543-47ab-8d1c-68fd4ccbffc2-kube-api-access-crvpz\") pod \"4e66bae1-5543-47ab-8d1c-68fd4ccbffc2\" (UID: \"4e66bae1-5543-47ab-8d1c-68fd4ccbffc2\") " Sep 04 22:17:08.119898 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:08.119770 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90252eef-35e0-47b8-a4f0-cc438c183380-bundle\") pod \"90252eef-35e0-47b8-a4f0-cc438c183380\" (UID: \"90252eef-35e0-47b8-a4f0-cc438c183380\") " Sep 04 22:17:08.119898 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:08.119802 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4d4829c-5ae8-493f-93e4-fb441fbdc995-util\") pod \"a4d4829c-5ae8-493f-93e4-fb441fbdc995\" (UID: \"a4d4829c-5ae8-493f-93e4-fb441fbdc995\") " Sep 04 22:17:08.119898 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:08.119832 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90252eef-35e0-47b8-a4f0-cc438c183380-util\") pod \"90252eef-35e0-47b8-a4f0-cc438c183380\" (UID: \"90252eef-35e0-47b8-a4f0-cc438c183380\") " Sep 04 22:17:08.119898 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:08.119859 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qzqj\" (UniqueName: \"kubernetes.io/projected/90252eef-35e0-47b8-a4f0-cc438c183380-kube-api-access-7qzqj\") pod \"90252eef-35e0-47b8-a4f0-cc438c183380\" (UID: \"90252eef-35e0-47b8-a4f0-cc438c183380\") " Sep 04 22:17:08.119898 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:08.119894 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e66bae1-5543-47ab-8d1c-68fd4ccbffc2-bundle\") pod \"4e66bae1-5543-47ab-8d1c-68fd4ccbffc2\" (UID: \"4e66bae1-5543-47ab-8d1c-68fd4ccbffc2\") " Sep 04 22:17:08.120149 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:08.119956 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e66bae1-5543-47ab-8d1c-68fd4ccbffc2-util\") pod \"4e66bae1-5543-47ab-8d1c-68fd4ccbffc2\" (UID: \"4e66bae1-5543-47ab-8d1c-68fd4ccbffc2\") " Sep 04 22:17:08.120149 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:08.119979 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4d4829c-5ae8-493f-93e4-fb441fbdc995-bundle\") pod \"a4d4829c-5ae8-493f-93e4-fb441fbdc995\" (UID: \"a4d4829c-5ae8-493f-93e4-fb441fbdc995\") " Sep 04 22:17:08.120149 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:08.120014 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w57mk\" (UniqueName: \"kubernetes.io/projected/a4d4829c-5ae8-493f-93e4-fb441fbdc995-kube-api-access-w57mk\") pod \"a4d4829c-5ae8-493f-93e4-fb441fbdc995\" (UID: \"a4d4829c-5ae8-493f-93e4-fb441fbdc995\") " Sep 04 22:17:08.120718 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:08.120686 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90252eef-35e0-47b8-a4f0-cc438c183380-bundle" (OuterVolumeSpecName: "bundle") pod "90252eef-35e0-47b8-a4f0-cc438c183380" (UID: "90252eef-35e0-47b8-a4f0-cc438c183380"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 04 22:17:08.120872 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:08.120847 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e66bae1-5543-47ab-8d1c-68fd4ccbffc2-bundle" (OuterVolumeSpecName: "bundle") pod "4e66bae1-5543-47ab-8d1c-68fd4ccbffc2" (UID: "4e66bae1-5543-47ab-8d1c-68fd4ccbffc2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 04 22:17:08.121421 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:08.121384 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d4829c-5ae8-493f-93e4-fb441fbdc995-bundle" (OuterVolumeSpecName: "bundle") pod "a4d4829c-5ae8-493f-93e4-fb441fbdc995" (UID: "a4d4829c-5ae8-493f-93e4-fb441fbdc995"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 04 22:17:08.122404 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:08.122375 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4d4829c-5ae8-493f-93e4-fb441fbdc995-kube-api-access-w57mk" (OuterVolumeSpecName: "kube-api-access-w57mk") pod "a4d4829c-5ae8-493f-93e4-fb441fbdc995" (UID: "a4d4829c-5ae8-493f-93e4-fb441fbdc995"). InnerVolumeSpecName "kube-api-access-w57mk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:17:08.122404 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:08.122385 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e66bae1-5543-47ab-8d1c-68fd4ccbffc2-kube-api-access-crvpz" (OuterVolumeSpecName: "kube-api-access-crvpz") pod "4e66bae1-5543-47ab-8d1c-68fd4ccbffc2" (UID: "4e66bae1-5543-47ab-8d1c-68fd4ccbffc2"). InnerVolumeSpecName "kube-api-access-crvpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:17:08.123333 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:08.123304 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90252eef-35e0-47b8-a4f0-cc438c183380-kube-api-access-7qzqj" (OuterVolumeSpecName: "kube-api-access-7qzqj") pod "90252eef-35e0-47b8-a4f0-cc438c183380" (UID: "90252eef-35e0-47b8-a4f0-cc438c183380"). InnerVolumeSpecName "kube-api-access-7qzqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:17:08.126119 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:08.126066 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d4829c-5ae8-493f-93e4-fb441fbdc995-util" (OuterVolumeSpecName: "util") pod "a4d4829c-5ae8-493f-93e4-fb441fbdc995" (UID: "a4d4829c-5ae8-493f-93e4-fb441fbdc995"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 04 22:17:08.126390 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:08.126368 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90252eef-35e0-47b8-a4f0-cc438c183380-util" (OuterVolumeSpecName: "util") pod "90252eef-35e0-47b8-a4f0-cc438c183380" (UID: "90252eef-35e0-47b8-a4f0-cc438c183380"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 04 22:17:08.128919 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:08.128869 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e66bae1-5543-47ab-8d1c-68fd4ccbffc2-util" (OuterVolumeSpecName: "util") pod "4e66bae1-5543-47ab-8d1c-68fd4ccbffc2" (UID: "4e66bae1-5543-47ab-8d1c-68fd4ccbffc2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 04 22:17:08.221306 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:08.221282 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4e66bae1-5543-47ab-8d1c-68fd4ccbffc2-util\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:17:08.221306 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:08.221302 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4d4829c-5ae8-493f-93e4-fb441fbdc995-bundle\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:17:08.221423 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:08.221312 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w57mk\" (UniqueName: \"kubernetes.io/projected/a4d4829c-5ae8-493f-93e4-fb441fbdc995-kube-api-access-w57mk\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:17:08.221423 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:08.221321 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crvpz\" (UniqueName: \"kubernetes.io/projected/4e66bae1-5543-47ab-8d1c-68fd4ccbffc2-kube-api-access-crvpz\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:17:08.221423 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:08.221330 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90252eef-35e0-47b8-a4f0-cc438c183380-bundle\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:17:08.221423 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:08.221338 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4d4829c-5ae8-493f-93e4-fb441fbdc995-util\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:17:08.221423 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:08.221346 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90252eef-35e0-47b8-a4f0-cc438c183380-util\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:17:08.221423 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:08.221354 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qzqj\" (UniqueName: \"kubernetes.io/projected/90252eef-35e0-47b8-a4f0-cc438c183380-kube-api-access-7qzqj\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:17:08.221423 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:08.221362 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4e66bae1-5543-47ab-8d1c-68fd4ccbffc2-bundle\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:17:08.885521 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:08.885485 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053gqzvc" event={"ID":"90252eef-35e0-47b8-a4f0-cc438c183380","Type":"ContainerDied","Data":"229a51f4f7cf3c8b07465cfb9601ca5ceca4f8ea9b2133941aa268138153f816"} Sep 04 22:17:08.885870 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:08.885500 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053gqzvc" Sep 04 22:17:08.885870 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:08.885585 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="229a51f4f7cf3c8b07465cfb9601ca5ceca4f8ea9b2133941aa268138153f816" Sep 04 22:17:08.887630 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:08.887608 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a026g67" Sep 04 22:17:08.887812 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:08.887596 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a026g67" event={"ID":"4e66bae1-5543-47ab-8d1c-68fd4ccbffc2","Type":"ContainerDied","Data":"228caa9ab99784f24470be5724a357f3d3fef073c32dff7b00ed95b8c4c294b2"} Sep 04 22:17:08.887812 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:08.887802 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="228caa9ab99784f24470be5724a357f3d3fef073c32dff7b00ed95b8c4c294b2" Sep 04 22:17:08.889868 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:08.889850 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c54x8k" Sep 04 22:17:08.889933 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:08.889862 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c54x8k" event={"ID":"a4d4829c-5ae8-493f-93e4-fb441fbdc995","Type":"ContainerDied","Data":"1d0ff81ce5cb78a4cfc26c626d4aa30cd1f12adc570cde16992fb19efa7a076a"} Sep 04 22:17:08.889933 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:08.889895 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d0ff81ce5cb78a4cfc26c626d4aa30cd1f12adc570cde16992fb19efa7a076a" Sep 04 22:17:09.009425 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:09.009402 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bblfz9r" Sep 04 22:17:09.128708 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:09.128638 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kdl4\" (UniqueName: \"kubernetes.io/projected/146651f4-fda6-4f58-915c-a3031b517cfb-kube-api-access-8kdl4\") pod \"146651f4-fda6-4f58-915c-a3031b517cfb\" (UID: \"146651f4-fda6-4f58-915c-a3031b517cfb\") " Sep 04 22:17:09.128708 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:09.128695 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/146651f4-fda6-4f58-915c-a3031b517cfb-bundle\") pod \"146651f4-fda6-4f58-915c-a3031b517cfb\" (UID: \"146651f4-fda6-4f58-915c-a3031b517cfb\") " Sep 04 22:17:09.128894 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:09.128713 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/146651f4-fda6-4f58-915c-a3031b517cfb-util\") pod \"146651f4-fda6-4f58-915c-a3031b517cfb\" (UID: \"146651f4-fda6-4f58-915c-a3031b517cfb\") " Sep 04 22:17:09.129214 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:09.129188 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146651f4-fda6-4f58-915c-a3031b517cfb-bundle" (OuterVolumeSpecName: "bundle") pod "146651f4-fda6-4f58-915c-a3031b517cfb" (UID: "146651f4-fda6-4f58-915c-a3031b517cfb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 04 22:17:09.130806 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:09.130778 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/146651f4-fda6-4f58-915c-a3031b517cfb-kube-api-access-8kdl4" (OuterVolumeSpecName: "kube-api-access-8kdl4") pod "146651f4-fda6-4f58-915c-a3031b517cfb" (UID: "146651f4-fda6-4f58-915c-a3031b517cfb"). InnerVolumeSpecName "kube-api-access-8kdl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:17:09.133206 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:09.133169 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146651f4-fda6-4f58-915c-a3031b517cfb-util" (OuterVolumeSpecName: "util") pod "146651f4-fda6-4f58-915c-a3031b517cfb" (UID: "146651f4-fda6-4f58-915c-a3031b517cfb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 04 22:17:09.230026 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:09.229984 2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/146651f4-fda6-4f58-915c-a3031b517cfb-bundle\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:17:09.230026 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:09.230021 2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/146651f4-fda6-4f58-915c-a3031b517cfb-util\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:17:09.230026 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:09.230030 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kdl4\" (UniqueName: \"kubernetes.io/projected/146651f4-fda6-4f58-915c-a3031b517cfb-kube-api-access-8kdl4\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:17:09.896388 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:09.896352 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bblfz9r" event={"ID":"146651f4-fda6-4f58-915c-a3031b517cfb","Type":"ContainerDied","Data":"2e3bd6b09677fe9c374bf7606a266f384566e92a87d6d97182c89ff11ecc9c0a"} Sep 04 22:17:09.896761 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:09.896407 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e3bd6b09677fe9c374bf7606a266f384566e92a87d6d97182c89ff11ecc9c0a" Sep 04 22:17:09.896761 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:17:09.896365 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bblfz9r" Sep 04 22:19:04.947198 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:04.947159 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7d46c4fdb8-258vt"] Sep 04 22:19:04.947674 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:04.947486 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90252eef-35e0-47b8-a4f0-cc438c183380" containerName="util" Sep 04 22:19:04.947674 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:04.947499 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="90252eef-35e0-47b8-a4f0-cc438c183380" containerName="util" Sep 04 22:19:04.947674 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:04.947510 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90252eef-35e0-47b8-a4f0-cc438c183380" containerName="pull" Sep 04 22:19:04.947674 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:04.947515 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="90252eef-35e0-47b8-a4f0-cc438c183380" containerName="pull" Sep 04 22:19:04.947674 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:04.947522 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e66bae1-5543-47ab-8d1c-68fd4ccbffc2" containerName="util" Sep 04 22:19:04.947674 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:04.947527 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e66bae1-5543-47ab-8d1c-68fd4ccbffc2" containerName="util" Sep 04 22:19:04.947674 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:04.947534 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e66bae1-5543-47ab-8d1c-68fd4ccbffc2" containerName="pull" Sep 04 22:19:04.947674 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:04.947539 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e66bae1-5543-47ab-8d1c-68fd4ccbffc2" containerName="pull" Sep 04 22:19:04.947674 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:04.947560 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="146651f4-fda6-4f58-915c-a3031b517cfb" containerName="extract" Sep 04 22:19:04.947674 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:04.947565 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="146651f4-fda6-4f58-915c-a3031b517cfb" containerName="extract" Sep 04 22:19:04.947674 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:04.947575 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4d4829c-5ae8-493f-93e4-fb441fbdc995" containerName="util" Sep 04 22:19:04.947674 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:04.947580 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4d4829c-5ae8-493f-93e4-fb441fbdc995" containerName="util" Sep 04 22:19:04.947674 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:04.947586 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4d4829c-5ae8-493f-93e4-fb441fbdc995" containerName="pull" Sep 04 22:19:04.947674 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:04.947591 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4d4829c-5ae8-493f-93e4-fb441fbdc995" containerName="pull" Sep 04 22:19:04.947674 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:04.947595 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="146651f4-fda6-4f58-915c-a3031b517cfb" containerName="pull" Sep 04 22:19:04.947674 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:04.947600 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="146651f4-fda6-4f58-915c-a3031b517cfb" containerName="pull" Sep 04 22:19:04.947674 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:04.947606 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4d4829c-5ae8-493f-93e4-fb441fbdc995" containerName="extract" Sep 04 22:19:04.947674 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:04.947611 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4d4829c-5ae8-493f-93e4-fb441fbdc995" containerName="extract" Sep 04 22:19:04.947674 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:04.947616 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e66bae1-5543-47ab-8d1c-68fd4ccbffc2" containerName="extract" Sep 04 22:19:04.947674 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:04.947621 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e66bae1-5543-47ab-8d1c-68fd4ccbffc2" containerName="extract" Sep 04 22:19:04.947674 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:04.947630 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90252eef-35e0-47b8-a4f0-cc438c183380" containerName="extract" Sep 04 22:19:04.947674 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:04.947634 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="90252eef-35e0-47b8-a4f0-cc438c183380" containerName="extract" Sep 04 22:19:04.947674 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:04.947643 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="146651f4-fda6-4f58-915c-a3031b517cfb" containerName="util" Sep 04 22:19:04.947674 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:04.947647 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="146651f4-fda6-4f58-915c-a3031b517cfb" containerName="util" Sep 04 22:19:04.948451 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:04.947691 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e66bae1-5543-47ab-8d1c-68fd4ccbffc2" containerName="extract" Sep 04 22:19:04.948451 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:04.947698 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4d4829c-5ae8-493f-93e4-fb441fbdc995" containerName="extract" Sep 04 22:19:04.948451 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:04.947705 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="90252eef-35e0-47b8-a4f0-cc438c183380" containerName="extract" Sep 04 22:19:04.948451 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:04.947712 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="146651f4-fda6-4f58-915c-a3031b517cfb" containerName="extract" Sep 04 22:19:04.950385 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:04.950370 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d46c4fdb8-258vt" Sep 04 22:19:04.952775 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:04.952743 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 04 22:19:04.952904 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:04.952885 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 04 22:19:04.953606 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:04.953589 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 04 22:19:04.953724 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:04.953592 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2nskt\"" Sep 04 22:19:04.964234 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:04.964212 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d46c4fdb8-258vt"] Sep 04 22:19:05.100576 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:05.100531 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4dcc57fc-2886-47f5-a1db-d9c251ff8559-webhook-cert\") pod \"maas-controller-7d46c4fdb8-258vt\" (UID: \"4dcc57fc-2886-47f5-a1db-d9c251ff8559\") " pod="opendatahub/maas-controller-7d46c4fdb8-258vt" Sep 04 22:19:05.100576 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:05.100579 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dcc57fc-2886-47f5-a1db-d9c251ff8559-metrics-certs\") pod \"maas-controller-7d46c4fdb8-258vt\" (UID: \"4dcc57fc-2886-47f5-a1db-d9c251ff8559\") " pod="opendatahub/maas-controller-7d46c4fdb8-258vt" Sep 04 22:19:05.100781 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:05.100599 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l6pl\" (UniqueName: \"kubernetes.io/projected/4dcc57fc-2886-47f5-a1db-d9c251ff8559-kube-api-access-5l6pl\") pod \"maas-controller-7d46c4fdb8-258vt\" (UID: \"4dcc57fc-2886-47f5-a1db-d9c251ff8559\") " pod="opendatahub/maas-controller-7d46c4fdb8-258vt" Sep 04 22:19:05.201334 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:05.201246 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4dcc57fc-2886-47f5-a1db-d9c251ff8559-webhook-cert\") pod \"maas-controller-7d46c4fdb8-258vt\" (UID: \"4dcc57fc-2886-47f5-a1db-d9c251ff8559\") " pod="opendatahub/maas-controller-7d46c4fdb8-258vt" Sep 04 22:19:05.201334 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:05.201283 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dcc57fc-2886-47f5-a1db-d9c251ff8559-metrics-certs\") pod \"maas-controller-7d46c4fdb8-258vt\" (UID: \"4dcc57fc-2886-47f5-a1db-d9c251ff8559\") " pod="opendatahub/maas-controller-7d46c4fdb8-258vt" Sep 04 22:19:05.201334 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:05.201299 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l6pl\" (UniqueName: \"kubernetes.io/projected/4dcc57fc-2886-47f5-a1db-d9c251ff8559-kube-api-access-5l6pl\") pod \"maas-controller-7d46c4fdb8-258vt\" (UID: \"4dcc57fc-2886-47f5-a1db-d9c251ff8559\") " pod="opendatahub/maas-controller-7d46c4fdb8-258vt" Sep 04 22:19:05.201594 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:19:05.201411 2574 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Sep 04 22:19:05.201594 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:19:05.201490 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dcc57fc-2886-47f5-a1db-d9c251ff8559-webhook-cert podName:4dcc57fc-2886-47f5-a1db-d9c251ff8559 nodeName:}" failed. No retries permitted until 2026-09-04 22:19:05.701470079 +0000 UTC m=+649.465945686 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/4dcc57fc-2886-47f5-a1db-d9c251ff8559-webhook-cert") pod "maas-controller-7d46c4fdb8-258vt" (UID: "4dcc57fc-2886-47f5-a1db-d9c251ff8559") : secret "maas-controller-webhook-cert" not found Sep 04 22:19:05.203738 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:05.203711 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dcc57fc-2886-47f5-a1db-d9c251ff8559-metrics-certs\") pod \"maas-controller-7d46c4fdb8-258vt\" (UID: \"4dcc57fc-2886-47f5-a1db-d9c251ff8559\") " pod="opendatahub/maas-controller-7d46c4fdb8-258vt" Sep 04 22:19:05.210342 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:05.210320 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l6pl\" (UniqueName: \"kubernetes.io/projected/4dcc57fc-2886-47f5-a1db-d9c251ff8559-kube-api-access-5l6pl\") pod \"maas-controller-7d46c4fdb8-258vt\" (UID: \"4dcc57fc-2886-47f5-a1db-d9c251ff8559\") " pod="opendatahub/maas-controller-7d46c4fdb8-258vt" Sep 04 22:19:05.706148 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:05.706093 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4dcc57fc-2886-47f5-a1db-d9c251ff8559-webhook-cert\") pod \"maas-controller-7d46c4fdb8-258vt\" (UID: \"4dcc57fc-2886-47f5-a1db-d9c251ff8559\") " pod="opendatahub/maas-controller-7d46c4fdb8-258vt" Sep 04 22:19:05.708459 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:05.708436 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4dcc57fc-2886-47f5-a1db-d9c251ff8559-webhook-cert\") pod \"maas-controller-7d46c4fdb8-258vt\" (UID: \"4dcc57fc-2886-47f5-a1db-d9c251ff8559\") " pod="opendatahub/maas-controller-7d46c4fdb8-258vt" Sep 04 22:19:05.860428 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:05.860378 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d46c4fdb8-258vt" Sep 04 22:19:06.132052 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:06.132024 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d46c4fdb8-258vt"] Sep 04 22:19:06.137027 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:19:06.136941 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dcc57fc_2886_47f5_a1db_d9c251ff8559.slice/crio-4a584a2be63a6950b465529d237aef2d876dbd4c02c3451a63539d1adf8c3bc5 WatchSource:0}: Error finding container 4a584a2be63a6950b465529d237aef2d876dbd4c02c3451a63539d1adf8c3bc5: Status 404 returned error can't find the container with id 4a584a2be63a6950b465529d237aef2d876dbd4c02c3451a63539d1adf8c3bc5 Sep 04 22:19:06.430269 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:06.430177 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d46c4fdb8-258vt" event={"ID":"4dcc57fc-2886-47f5-a1db-d9c251ff8559","Type":"ContainerStarted","Data":"4a584a2be63a6950b465529d237aef2d876dbd4c02c3451a63539d1adf8c3bc5"} Sep 04 22:19:09.447625 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:09.447584 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d46c4fdb8-258vt" event={"ID":"4dcc57fc-2886-47f5-a1db-d9c251ff8559","Type":"ContainerStarted","Data":"d191677f6ed94f7e4b2c6deb0cb5aa02ea995bd3027e19151d5b567d46fce198"} Sep 04 22:19:09.448049 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:09.447708 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7d46c4fdb8-258vt" Sep 04 22:19:09.467139 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:09.467083 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7d46c4fdb8-258vt" podStartSLOduration=3.070829834 podStartE2EDuration="5.467065997s" totalImagesPullingTime="2.396236163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:19:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:19:09.466606744 +0000 UTC m=+653.231082362" watchObservedRunningTime="2026-09-04 22:19:09.467065997 +0000 UTC m=+653.231541611" Sep 04 22:19:20.421440 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:20.421402 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7d46c4fdb8-258vt"] Sep 04 22:19:20.421936 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:20.421650 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7d46c4fdb8-258vt" podUID="4dcc57fc-2886-47f5-a1db-d9c251ff8559" containerName="manager" containerID="cri-o://d191677f6ed94f7e4b2c6deb0cb5aa02ea995bd3027e19151d5b567d46fce198" gracePeriod=30 Sep 04 22:19:20.423442 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:20.423420 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7d46c4fdb8-258vt" Sep 04 22:19:20.660845 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:20.660818 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d46c4fdb8-258vt" Sep 04 22:19:20.724976 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:20.724910 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dcc57fc-2886-47f5-a1db-d9c251ff8559-metrics-certs\") pod \"4dcc57fc-2886-47f5-a1db-d9c251ff8559\" (UID: \"4dcc57fc-2886-47f5-a1db-d9c251ff8559\") " Sep 04 22:19:20.724976 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:20.724964 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5l6pl\" (UniqueName: \"kubernetes.io/projected/4dcc57fc-2886-47f5-a1db-d9c251ff8559-kube-api-access-5l6pl\") pod \"4dcc57fc-2886-47f5-a1db-d9c251ff8559\" (UID: \"4dcc57fc-2886-47f5-a1db-d9c251ff8559\") " Sep 04 22:19:20.725172 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:20.724997 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4dcc57fc-2886-47f5-a1db-d9c251ff8559-webhook-cert\") pod \"4dcc57fc-2886-47f5-a1db-d9c251ff8559\" (UID: \"4dcc57fc-2886-47f5-a1db-d9c251ff8559\") " Sep 04 22:19:20.727151 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:20.727113 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dcc57fc-2886-47f5-a1db-d9c251ff8559-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "4dcc57fc-2886-47f5-a1db-d9c251ff8559" (UID: "4dcc57fc-2886-47f5-a1db-d9c251ff8559"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 04 22:19:20.727151 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:20.727145 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dcc57fc-2886-47f5-a1db-d9c251ff8559-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4dcc57fc-2886-47f5-a1db-d9c251ff8559" (UID: "4dcc57fc-2886-47f5-a1db-d9c251ff8559"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 04 22:19:20.727320 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:20.727167 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dcc57fc-2886-47f5-a1db-d9c251ff8559-kube-api-access-5l6pl" (OuterVolumeSpecName: "kube-api-access-5l6pl") pod "4dcc57fc-2886-47f5-a1db-d9c251ff8559" (UID: "4dcc57fc-2886-47f5-a1db-d9c251ff8559"). InnerVolumeSpecName "kube-api-access-5l6pl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:19:20.826315 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:20.826282 2574 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4dcc57fc-2886-47f5-a1db-d9c251ff8559-webhook-cert\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:19:20.826315 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:20.826307 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dcc57fc-2886-47f5-a1db-d9c251ff8559-metrics-certs\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:19:20.826315 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:20.826316 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l6pl\" (UniqueName: \"kubernetes.io/projected/4dcc57fc-2886-47f5-a1db-d9c251ff8559-kube-api-access-5l6pl\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:19:21.507057 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:21.507019 2574 generic.go:356] "Generic (PLEG): container finished" podID="4dcc57fc-2886-47f5-a1db-d9c251ff8559" containerID="d191677f6ed94f7e4b2c6deb0cb5aa02ea995bd3027e19151d5b567d46fce198" exitCode=0 Sep 04 22:19:21.507466 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:21.507065 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d46c4fdb8-258vt" Sep 04 22:19:21.507466 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:21.507102 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d46c4fdb8-258vt" event={"ID":"4dcc57fc-2886-47f5-a1db-d9c251ff8559","Type":"ContainerDied","Data":"d191677f6ed94f7e4b2c6deb0cb5aa02ea995bd3027e19151d5b567d46fce198"} Sep 04 22:19:21.507466 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:21.507159 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d46c4fdb8-258vt" event={"ID":"4dcc57fc-2886-47f5-a1db-d9c251ff8559","Type":"ContainerDied","Data":"4a584a2be63a6950b465529d237aef2d876dbd4c02c3451a63539d1adf8c3bc5"} Sep 04 22:19:21.507466 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:21.507184 2574 scope.go:117] "RemoveContainer" containerID="d191677f6ed94f7e4b2c6deb0cb5aa02ea995bd3027e19151d5b567d46fce198" Sep 04 22:19:21.516320 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:21.516299 2574 scope.go:117] "RemoveContainer" containerID="d191677f6ed94f7e4b2c6deb0cb5aa02ea995bd3027e19151d5b567d46fce198" Sep 04 22:19:21.516597 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:19:21.516580 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d191677f6ed94f7e4b2c6deb0cb5aa02ea995bd3027e19151d5b567d46fce198\": container with ID starting with d191677f6ed94f7e4b2c6deb0cb5aa02ea995bd3027e19151d5b567d46fce198 not found: ID does not exist" containerID="d191677f6ed94f7e4b2c6deb0cb5aa02ea995bd3027e19151d5b567d46fce198" Sep 04 22:19:21.516649 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:21.516604 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d191677f6ed94f7e4b2c6deb0cb5aa02ea995bd3027e19151d5b567d46fce198"} err="failed to get container status \"d191677f6ed94f7e4b2c6deb0cb5aa02ea995bd3027e19151d5b567d46fce198\": rpc error: code = NotFound desc = could not find container \"d191677f6ed94f7e4b2c6deb0cb5aa02ea995bd3027e19151d5b567d46fce198\": container with ID starting with d191677f6ed94f7e4b2c6deb0cb5aa02ea995bd3027e19151d5b567d46fce198 not found: ID does not exist" Sep 04 22:19:21.530158 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:21.530137 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7d46c4fdb8-258vt"] Sep 04 22:19:21.537315 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:21.537291 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7d46c4fdb8-258vt"] Sep 04 22:19:22.715121 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:22.715085 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4dcc57fc-2886-47f5-a1db-d9c251ff8559" path="/var/lib/kubelet/pods/4dcc57fc-2886-47f5-a1db-d9c251ff8559/volumes" Sep 04 22:19:42.907474 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:42.907431 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-xj6cg"] Sep 04 22:19:42.907945 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:42.907749 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dcc57fc-2886-47f5-a1db-d9c251ff8559" containerName="manager" Sep 04 22:19:42.907945 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:42.907764 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dcc57fc-2886-47f5-a1db-d9c251ff8559" containerName="manager" Sep 04 22:19:42.907945 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:42.907826 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dcc57fc-2886-47f5-a1db-d9c251ff8559" containerName="manager" Sep 04 22:19:42.913381 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:42.913356 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-xj6cg" Sep 04 22:19:42.915928 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:42.915900 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-sf5sm\"" Sep 04 22:19:42.916066 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:42.915932 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 04 22:19:42.920682 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:42.920653 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-xj6cg"] Sep 04 22:19:42.988454 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:42.988416 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7d9w\" (UniqueName: \"kubernetes.io/projected/8e12172f-2c9e-4eaa-b780-076e4c686406-kube-api-access-x7d9w\") pod \"payload-processing-5b54b7fbb7-xj6cg\" (UID: \"8e12172f-2c9e-4eaa-b780-076e4c686406\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-xj6cg" Sep 04 22:19:42.988643 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:42.988486 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e12172f-2c9e-4eaa-b780-076e4c686406-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-xj6cg\" (UID: \"8e12172f-2c9e-4eaa-b780-076e4c686406\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-xj6cg" Sep 04 22:19:43.089672 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:43.089635 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e12172f-2c9e-4eaa-b780-076e4c686406-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-xj6cg\" (UID: \"8e12172f-2c9e-4eaa-b780-076e4c686406\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-xj6cg" Sep 04 22:19:43.089845 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:43.089703 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7d9w\" (UniqueName: \"kubernetes.io/projected/8e12172f-2c9e-4eaa-b780-076e4c686406-kube-api-access-x7d9w\") pod \"payload-processing-5b54b7fbb7-xj6cg\" (UID: \"8e12172f-2c9e-4eaa-b780-076e4c686406\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-xj6cg" Sep 04 22:19:43.090300 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:43.090278 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e12172f-2c9e-4eaa-b780-076e4c686406-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-xj6cg\" (UID: \"8e12172f-2c9e-4eaa-b780-076e4c686406\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-xj6cg" Sep 04 22:19:43.099675 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:43.099650 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7d9w\" (UniqueName: \"kubernetes.io/projected/8e12172f-2c9e-4eaa-b780-076e4c686406-kube-api-access-x7d9w\") pod \"payload-processing-5b54b7fbb7-xj6cg\" (UID: \"8e12172f-2c9e-4eaa-b780-076e4c686406\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-xj6cg" Sep 04 22:19:43.224817 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:43.224742 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-xj6cg" Sep 04 22:19:43.493827 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:43.493802 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-xj6cg"] Sep 04 22:19:43.496017 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:19:43.495988 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e12172f_2c9e_4eaa_b780_076e4c686406.slice/crio-f0863c541c551737ecb66399689dbb3c175afdc74c7478c00948e93bf71d69f3 WatchSource:0}: Error finding container f0863c541c551737ecb66399689dbb3c175afdc74c7478c00948e93bf71d69f3: Status 404 returned error can't find the container with id f0863c541c551737ecb66399689dbb3c175afdc74c7478c00948e93bf71d69f3 Sep 04 22:19:43.498043 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:43.498027 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 04 22:19:43.607196 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:43.607161 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-xj6cg" event={"ID":"8e12172f-2c9e-4eaa-b780-076e4c686406","Type":"ContainerStarted","Data":"f0863c541c551737ecb66399689dbb3c175afdc74c7478c00948e93bf71d69f3"} Sep 04 22:19:45.619330 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:45.619289 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-xj6cg" event={"ID":"8e12172f-2c9e-4eaa-b780-076e4c686406","Type":"ContainerStarted","Data":"dd223fc2a4951b674ab1b8c80028db8c725256a315d3ecf9c3f0073833444859"} Sep 04 22:19:45.619725 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:45.619386 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-xj6cg" Sep 04 22:19:45.637208 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:45.637160 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-xj6cg" podStartSLOduration=1.892557139 podStartE2EDuration="3.637145327s" totalImagesPullingTime="1.744588188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:19:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:19:45.635761223 +0000 UTC m=+689.400236837" watchObservedRunningTime="2026-09-04 22:19:45.637145327 +0000 UTC m=+689.401620940" Sep 04 22:19:56.625733 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:19:56.625699 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-xj6cg" Sep 04 22:20:18.584706 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:18.584668 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-gdkld"] Sep 04 22:20:18.591153 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:18.591130 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gdkld" Sep 04 22:20:18.594671 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:18.594645 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tsxvl\"" Sep 04 22:20:18.594671 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:18.594664 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 04 22:20:18.594854 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:18.594708 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 04 22:20:18.594854 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:18.594644 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 04 22:20:18.599901 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:18.599875 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-gdkld"] Sep 04 22:20:18.771453 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:18.771425 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd62n\" (UniqueName: \"kubernetes.io/projected/1ac44222-4d72-4c15-b2a5-e6db9e022242-kube-api-access-wd62n\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gdkld\" (UID: \"1ac44222-4d72-4c15-b2a5-e6db9e022242\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gdkld" Sep 04 22:20:18.771453 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:18.771454 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ac44222-4d72-4c15-b2a5-e6db9e022242-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gdkld\" (UID: \"1ac44222-4d72-4c15-b2a5-e6db9e022242\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gdkld" Sep 04 22:20:18.771686 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:18.771488 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ac44222-4d72-4c15-b2a5-e6db9e022242-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gdkld\" (UID: \"1ac44222-4d72-4c15-b2a5-e6db9e022242\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gdkld" Sep 04 22:20:18.771686 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:18.771579 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ac44222-4d72-4c15-b2a5-e6db9e022242-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gdkld\" (UID: \"1ac44222-4d72-4c15-b2a5-e6db9e022242\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gdkld" Sep 04 22:20:18.771686 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:18.771615 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ac44222-4d72-4c15-b2a5-e6db9e022242-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gdkld\" (UID: \"1ac44222-4d72-4c15-b2a5-e6db9e022242\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gdkld" Sep 04 22:20:18.771686 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:18.771631 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ac44222-4d72-4c15-b2a5-e6db9e022242-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gdkld\" (UID: \"1ac44222-4d72-4c15-b2a5-e6db9e022242\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gdkld" Sep 04 22:20:18.872819 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:18.872728 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ac44222-4d72-4c15-b2a5-e6db9e022242-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gdkld\" (UID: \"1ac44222-4d72-4c15-b2a5-e6db9e022242\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gdkld" Sep 04 22:20:18.872819 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:18.872789 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ac44222-4d72-4c15-b2a5-e6db9e022242-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gdkld\" (UID: \"1ac44222-4d72-4c15-b2a5-e6db9e022242\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gdkld" Sep 04 22:20:18.873026 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:18.872962 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ac44222-4d72-4c15-b2a5-e6db9e022242-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gdkld\" (UID: \"1ac44222-4d72-4c15-b2a5-e6db9e022242\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gdkld" Sep 04 22:20:18.873085 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:18.873042 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd62n\" (UniqueName: \"kubernetes.io/projected/1ac44222-4d72-4c15-b2a5-e6db9e022242-kube-api-access-wd62n\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gdkld\" (UID: \"1ac44222-4d72-4c15-b2a5-e6db9e022242\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gdkld" Sep 04 22:20:18.873085 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:18.873072 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ac44222-4d72-4c15-b2a5-e6db9e022242-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gdkld\" (UID: \"1ac44222-4d72-4c15-b2a5-e6db9e022242\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gdkld" Sep 04 22:20:18.873176 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:18.873138 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ac44222-4d72-4c15-b2a5-e6db9e022242-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gdkld\" (UID: \"1ac44222-4d72-4c15-b2a5-e6db9e022242\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gdkld" Sep 04 22:20:18.873235 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:18.873184 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ac44222-4d72-4c15-b2a5-e6db9e022242-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gdkld\" (UID: \"1ac44222-4d72-4c15-b2a5-e6db9e022242\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gdkld" Sep 04 22:20:18.873287 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:18.873234 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ac44222-4d72-4c15-b2a5-e6db9e022242-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gdkld\" (UID: \"1ac44222-4d72-4c15-b2a5-e6db9e022242\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gdkld" Sep 04 22:20:18.873335 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:18.873315 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ac44222-4d72-4c15-b2a5-e6db9e022242-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gdkld\" (UID: \"1ac44222-4d72-4c15-b2a5-e6db9e022242\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gdkld" Sep 04 22:20:18.875262 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:18.875233 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ac44222-4d72-4c15-b2a5-e6db9e022242-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gdkld\" (UID: \"1ac44222-4d72-4c15-b2a5-e6db9e022242\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gdkld" Sep 04 22:20:18.875672 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:18.875653 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ac44222-4d72-4c15-b2a5-e6db9e022242-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gdkld\" (UID: \"1ac44222-4d72-4c15-b2a5-e6db9e022242\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gdkld" Sep 04 22:20:18.881385 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:18.881362 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd62n\" (UniqueName: \"kubernetes.io/projected/1ac44222-4d72-4c15-b2a5-e6db9e022242-kube-api-access-wd62n\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gdkld\" (UID: \"1ac44222-4d72-4c15-b2a5-e6db9e022242\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gdkld" Sep 04 22:20:18.903107 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:18.903084 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gdkld" Sep 04 22:20:19.173975 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:19.173951 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-gdkld"] Sep 04 22:20:19.176199 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:20:19.176166 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ac44222_4d72_4c15_b2a5_e6db9e022242.slice/crio-ebc941267aca6cb7a8229d0a66bef3919e363bfdad959eee381f0bc333f7fd51 WatchSource:0}: Error finding container ebc941267aca6cb7a8229d0a66bef3919e363bfdad959eee381f0bc333f7fd51: Status 404 returned error can't find the container with id ebc941267aca6cb7a8229d0a66bef3919e363bfdad959eee381f0bc333f7fd51 Sep 04 22:20:19.777689 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:19.777656 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gdkld" event={"ID":"1ac44222-4d72-4c15-b2a5-e6db9e022242","Type":"ContainerStarted","Data":"ebc941267aca6cb7a8229d0a66bef3919e363bfdad959eee381f0bc333f7fd51"} Sep 04 22:20:20.784202 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:20.784166 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gdkld" event={"ID":"1ac44222-4d72-4c15-b2a5-e6db9e022242","Type":"ContainerStarted","Data":"446694beb0dbce8e7e643218b68b3edfbc8130012d57ca04e62c8e90a7c7439c"} Sep 04 22:20:20.804365 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:20.804316 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gdkld" podStartSLOduration=1.868091992 podStartE2EDuration="2.804301718s" totalImagesPullingTime="936.209726ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:20:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:20:20.802201333 +0000 UTC m=+724.566676946" watchObservedRunningTime="2026-09-04 22:20:20.804301718 +0000 UTC m=+724.568777331" Sep 04 22:20:28.903420 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:28.903382 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gdkld" Sep 04 22:20:28.903872 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:28.903452 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gdkld" Sep 04 22:20:28.915924 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:28.915902 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gdkld" Sep 04 22:20:29.837155 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:29.837121 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gdkld" Sep 04 22:20:30.701772 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:30.701740 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6"] Sep 04 22:20:30.705043 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:30.705021 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6" Sep 04 22:20:30.712627 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:30.712606 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 04 22:20:30.718028 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:30.718005 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6"] Sep 04 22:20:30.770137 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:30.770108 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ded97bf-a59e-4609-bed2-616814538e8c-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6\" (UID: \"7ded97bf-a59e-4609-bed2-616814538e8c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6" Sep 04 22:20:30.770289 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:30.770149 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ded97bf-a59e-4609-bed2-616814538e8c-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6\" (UID: \"7ded97bf-a59e-4609-bed2-616814538e8c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6" Sep 04 22:20:30.770289 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:30.770168 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ded97bf-a59e-4609-bed2-616814538e8c-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6\" (UID: \"7ded97bf-a59e-4609-bed2-616814538e8c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6" Sep 04 22:20:30.770289 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:30.770183 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r77k8\" (UniqueName: \"kubernetes.io/projected/7ded97bf-a59e-4609-bed2-616814538e8c-kube-api-access-r77k8\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6\" (UID: \"7ded97bf-a59e-4609-bed2-616814538e8c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6" Sep 04 22:20:30.770289 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:30.770245 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ded97bf-a59e-4609-bed2-616814538e8c-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6\" (UID: \"7ded97bf-a59e-4609-bed2-616814538e8c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6" Sep 04 22:20:30.770424 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:30.770320 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ded97bf-a59e-4609-bed2-616814538e8c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6\" (UID: \"7ded97bf-a59e-4609-bed2-616814538e8c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6" Sep 04 22:20:30.871645 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:30.871619 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ded97bf-a59e-4609-bed2-616814538e8c-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6\" (UID: \"7ded97bf-a59e-4609-bed2-616814538e8c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6" Sep 04 22:20:30.871803 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:30.871669 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ded97bf-a59e-4609-bed2-616814538e8c-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6\" (UID: \"7ded97bf-a59e-4609-bed2-616814538e8c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6" Sep 04 22:20:30.871803 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:30.871697 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ded97bf-a59e-4609-bed2-616814538e8c-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6\" (UID: \"7ded97bf-a59e-4609-bed2-616814538e8c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6" Sep 04 22:20:30.871803 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:30.871719 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r77k8\" (UniqueName: \"kubernetes.io/projected/7ded97bf-a59e-4609-bed2-616814538e8c-kube-api-access-r77k8\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6\" (UID: \"7ded97bf-a59e-4609-bed2-616814538e8c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6" Sep 04 22:20:30.871803 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:30.871758 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ded97bf-a59e-4609-bed2-616814538e8c-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6\" (UID: \"7ded97bf-a59e-4609-bed2-616814538e8c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6" Sep 04 22:20:30.871983 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:30.871839 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ded97bf-a59e-4609-bed2-616814538e8c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6\" (UID: \"7ded97bf-a59e-4609-bed2-616814538e8c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6" Sep 04 22:20:30.872111 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:30.872088 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ded97bf-a59e-4609-bed2-616814538e8c-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6\" (UID: \"7ded97bf-a59e-4609-bed2-616814538e8c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6" Sep 04 22:20:30.872337 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:30.872315 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ded97bf-a59e-4609-bed2-616814538e8c-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6\" (UID: \"7ded97bf-a59e-4609-bed2-616814538e8c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6" Sep 04 22:20:30.872433 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:30.872405 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ded97bf-a59e-4609-bed2-616814538e8c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6\" (UID: \"7ded97bf-a59e-4609-bed2-616814538e8c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6" Sep 04 22:20:30.874031 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:30.874002 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ded97bf-a59e-4609-bed2-616814538e8c-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6\" (UID: \"7ded97bf-a59e-4609-bed2-616814538e8c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6" Sep 04 22:20:30.874114 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:30.874093 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ded97bf-a59e-4609-bed2-616814538e8c-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6\" (UID: \"7ded97bf-a59e-4609-bed2-616814538e8c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6" Sep 04 22:20:30.880445 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:30.880421 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r77k8\" (UniqueName: \"kubernetes.io/projected/7ded97bf-a59e-4609-bed2-616814538e8c-kube-api-access-r77k8\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6\" (UID: \"7ded97bf-a59e-4609-bed2-616814538e8c\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6" Sep 04 22:20:31.016531 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:31.016504 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6" Sep 04 22:20:31.287802 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:31.287770 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6"] Sep 04 22:20:31.289306 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:20:31.289280 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ded97bf_a59e_4609_bed2_616814538e8c.slice/crio-0550b810ed2b8e32bb5c1e7c739a24922dcf9e378a79394b6040f13efd9a08aa WatchSource:0}: Error finding container 0550b810ed2b8e32bb5c1e7c739a24922dcf9e378a79394b6040f13efd9a08aa: Status 404 returned error can't find the container with id 0550b810ed2b8e32bb5c1e7c739a24922dcf9e378a79394b6040f13efd9a08aa Sep 04 22:20:31.836034 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:31.835997 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6" event={"ID":"7ded97bf-a59e-4609-bed2-616814538e8c","Type":"ContainerStarted","Data":"43f5b98afc2375d1fdcccf388ba84ae0ae5f14f9d9dd9800eca15ee4a554fca3"} Sep 04 22:20:31.836419 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:31.836050 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6" event={"ID":"7ded97bf-a59e-4609-bed2-616814538e8c","Type":"ContainerStarted","Data":"0550b810ed2b8e32bb5c1e7c739a24922dcf9e378a79394b6040f13efd9a08aa"} Sep 04 22:20:31.868856 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:31.868809 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6" podStartSLOduration=1.690208457 podStartE2EDuration="1.868793238s" totalImagesPullingTime="178.584781ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:20:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:20:31.868253455 +0000 UTC m=+735.632729094" watchObservedRunningTime="2026-09-04 22:20:31.868793238 +0000 UTC m=+735.633268851" Sep 04 22:20:41.017589 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:41.017543 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6" Sep 04 22:20:41.017998 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:41.017617 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6" Sep 04 22:20:41.029982 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:41.029958 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6" Sep 04 22:20:41.892983 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:41.892955 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6" Sep 04 22:20:59.801495 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:59.801462 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6"] Sep 04 22:20:59.805479 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:59.805459 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6" Sep 04 22:20:59.808032 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:59.808008 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Sep 04 22:20:59.814802 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:59.814780 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6"] Sep 04 22:20:59.908853 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:59.908825 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvm98\" (UniqueName: \"kubernetes.io/projected/7caf7d48-54df-4902-861d-3bd25adc8de0-kube-api-access-hvm98\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6\" (UID: \"7caf7d48-54df-4902-861d-3bd25adc8de0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6" Sep 04 22:20:59.909069 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:59.908860 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7caf7d48-54df-4902-861d-3bd25adc8de0-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6\" (UID: \"7caf7d48-54df-4902-861d-3bd25adc8de0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6" Sep 04 22:20:59.909069 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:59.908918 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7caf7d48-54df-4902-861d-3bd25adc8de0-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6\" (UID: \"7caf7d48-54df-4902-861d-3bd25adc8de0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6" Sep 04 22:20:59.909069 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:59.909009 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7caf7d48-54df-4902-861d-3bd25adc8de0-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6\" (UID: \"7caf7d48-54df-4902-861d-3bd25adc8de0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6" Sep 04 22:20:59.909069 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:59.909044 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7caf7d48-54df-4902-861d-3bd25adc8de0-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6\" (UID: \"7caf7d48-54df-4902-861d-3bd25adc8de0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6" Sep 04 22:20:59.909069 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:20:59.909061 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7caf7d48-54df-4902-861d-3bd25adc8de0-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6\" (UID: \"7caf7d48-54df-4902-861d-3bd25adc8de0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6" Sep 04 22:21:00.010309 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:21:00.010280 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7caf7d48-54df-4902-861d-3bd25adc8de0-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6\" (UID: \"7caf7d48-54df-4902-861d-3bd25adc8de0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6" Sep 04 22:21:00.010309 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:21:00.010311 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7caf7d48-54df-4902-861d-3bd25adc8de0-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6\" (UID: \"7caf7d48-54df-4902-861d-3bd25adc8de0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6" Sep 04 22:21:00.010473 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:21:00.010344 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvm98\" (UniqueName: \"kubernetes.io/projected/7caf7d48-54df-4902-861d-3bd25adc8de0-kube-api-access-hvm98\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6\" (UID: \"7caf7d48-54df-4902-861d-3bd25adc8de0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6" Sep 04 22:21:00.010473 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:21:00.010382 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7caf7d48-54df-4902-861d-3bd25adc8de0-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6\" (UID: \"7caf7d48-54df-4902-861d-3bd25adc8de0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6" Sep 04 22:21:00.010473 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:21:00.010417 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7caf7d48-54df-4902-861d-3bd25adc8de0-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6\" (UID: \"7caf7d48-54df-4902-861d-3bd25adc8de0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6" Sep 04 22:21:00.010473 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:21:00.010460 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7caf7d48-54df-4902-861d-3bd25adc8de0-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6\" (UID: \"7caf7d48-54df-4902-861d-3bd25adc8de0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6" Sep 04 22:21:00.010792 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:21:00.010765 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7caf7d48-54df-4902-861d-3bd25adc8de0-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6\" (UID: \"7caf7d48-54df-4902-861d-3bd25adc8de0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6" Sep 04 22:21:00.010884 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:21:00.010827 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7caf7d48-54df-4902-861d-3bd25adc8de0-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6\" (UID: \"7caf7d48-54df-4902-861d-3bd25adc8de0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6" Sep 04 22:21:00.010884 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:21:00.010858 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7caf7d48-54df-4902-861d-3bd25adc8de0-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6\" (UID: \"7caf7d48-54df-4902-861d-3bd25adc8de0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6" Sep 04 22:21:00.012692 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:21:00.012674 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7caf7d48-54df-4902-861d-3bd25adc8de0-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6\" (UID: \"7caf7d48-54df-4902-861d-3bd25adc8de0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6" Sep 04 22:21:00.012961 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:21:00.012944 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7caf7d48-54df-4902-861d-3bd25adc8de0-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6\" (UID: \"7caf7d48-54df-4902-861d-3bd25adc8de0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6" Sep 04 22:21:00.020650 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:21:00.020631 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvm98\" (UniqueName: \"kubernetes.io/projected/7caf7d48-54df-4902-861d-3bd25adc8de0-kube-api-access-hvm98\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6\" (UID: \"7caf7d48-54df-4902-861d-3bd25adc8de0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6" Sep 04 22:21:00.116922 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:21:00.116857 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6" Sep 04 22:21:00.391467 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:21:00.391443 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6"] Sep 04 22:21:00.393375 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:21:00.393350 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7caf7d48_54df_4902_861d_3bd25adc8de0.slice/crio-f8aceccee28b4598f35deeb2e8368bfe3c13a83fbcdf9e1b4a806f58e3ff2c10 WatchSource:0}: Error finding container f8aceccee28b4598f35deeb2e8368bfe3c13a83fbcdf9e1b4a806f58e3ff2c10: Status 404 returned error can't find the container with id f8aceccee28b4598f35deeb2e8368bfe3c13a83fbcdf9e1b4a806f58e3ff2c10 Sep 04 22:21:00.971919 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:21:00.971887 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6" event={"ID":"7caf7d48-54df-4902-861d-3bd25adc8de0","Type":"ContainerStarted","Data":"639d806b67d4de09810dde3bcb73ffdb767d92c58484c66cca7a9d20565022fb"} Sep 04 22:21:00.972285 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:21:00.971937 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6" event={"ID":"7caf7d48-54df-4902-861d-3bd25adc8de0","Type":"ContainerStarted","Data":"f8aceccee28b4598f35deeb2e8368bfe3c13a83fbcdf9e1b4a806f58e3ff2c10"} Sep 04 22:21:00.993216 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:21:00.993171 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6" podStartSLOduration=1.819417535 podStartE2EDuration="1.993156067s" totalImagesPullingTime="173.738532ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:20:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:21:00.991338673 +0000 UTC m=+764.755814290" watchObservedRunningTime="2026-09-04 22:21:00.993156067 +0000 UTC m=+764.757631680" Sep 04 22:21:10.117231 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:21:10.117176 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6" Sep 04 22:21:10.117672 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:21:10.117272 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6" Sep 04 22:21:10.129878 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:21:10.129854 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6" Sep 04 22:21:11.031125 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:21:11.031095 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6" Sep 04 22:22:48.613659 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:48.613624 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d9f7ae-56b97d44c8-pgnhl"] Sep 04 22:22:48.616785 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:48.616763 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d9f7ae-56b97d44c8-pgnhl" Sep 04 22:22:48.618984 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:48.618964 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d9f7ae\"" Sep 04 22:22:48.627430 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:48.627405 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d9f7ae-56b97d44c8-pgnhl"] Sep 04 22:22:48.633871 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:48.633846 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz9vt\" (UniqueName: \"kubernetes.io/projected/6b920f63-12eb-47ba-8ade-f32eb154b0a8-kube-api-access-cz9vt\") pod \"payload-pre-processing-e2e-inf-a-d9f7ae-56b97d44c8-pgnhl\" (UID: \"6b920f63-12eb-47ba-8ade-f32eb154b0a8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d9f7ae-56b97d44c8-pgnhl" Sep 04 22:22:48.633978 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:48.633927 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b920f63-12eb-47ba-8ade-f32eb154b0a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d9f7ae-56b97d44c8-pgnhl\" (UID: \"6b920f63-12eb-47ba-8ade-f32eb154b0a8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d9f7ae-56b97d44c8-pgnhl" Sep 04 22:22:48.637201 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:48.637182 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d9f7ae-648c996649-ts2sr"] Sep 04 22:22:48.640229 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:48.640215 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d9f7ae-648c996649-ts2sr" Sep 04 22:22:48.643033 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:48.642999 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d9f7ae-dockercfg-kbvkb\"" Sep 04 22:22:48.650412 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:48.650394 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d9f7ae-648c996649-ts2sr"] Sep 04 22:22:48.734688 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:48.734656 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b920f63-12eb-47ba-8ade-f32eb154b0a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d9f7ae-56b97d44c8-pgnhl\" (UID: \"6b920f63-12eb-47ba-8ade-f32eb154b0a8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d9f7ae-56b97d44c8-pgnhl" Sep 04 22:22:48.734835 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:48.734713 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db497\" (UniqueName: \"kubernetes.io/projected/5d1fbc5c-436f-4698-a943-391d39241a02-kube-api-access-db497\") pod \"payload-processing-e2e-inf-a-d9f7ae-648c996649-ts2sr\" (UID: \"5d1fbc5c-436f-4698-a943-391d39241a02\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d9f7ae-648c996649-ts2sr" Sep 04 22:22:48.734835 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:48.734733 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz9vt\" (UniqueName: \"kubernetes.io/projected/6b920f63-12eb-47ba-8ade-f32eb154b0a8-kube-api-access-cz9vt\") pod \"payload-pre-processing-e2e-inf-a-d9f7ae-56b97d44c8-pgnhl\" (UID: \"6b920f63-12eb-47ba-8ade-f32eb154b0a8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d9f7ae-56b97d44c8-pgnhl" Sep 04 22:22:48.734835 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:48.734754 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d1fbc5c-436f-4698-a943-391d39241a02-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d9f7ae-648c996649-ts2sr\" (UID: \"5d1fbc5c-436f-4698-a943-391d39241a02\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d9f7ae-648c996649-ts2sr" Sep 04 22:22:48.735343 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:48.735325 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b920f63-12eb-47ba-8ade-f32eb154b0a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d9f7ae-56b97d44c8-pgnhl\" (UID: \"6b920f63-12eb-47ba-8ade-f32eb154b0a8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d9f7ae-56b97d44c8-pgnhl" Sep 04 22:22:48.746742 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:48.746708 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz9vt\" (UniqueName: \"kubernetes.io/projected/6b920f63-12eb-47ba-8ade-f32eb154b0a8-kube-api-access-cz9vt\") pod \"payload-pre-processing-e2e-inf-a-d9f7ae-56b97d44c8-pgnhl\" (UID: \"6b920f63-12eb-47ba-8ade-f32eb154b0a8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d9f7ae-56b97d44c8-pgnhl" Sep 04 22:22:48.835423 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:48.835392 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db497\" (UniqueName: \"kubernetes.io/projected/5d1fbc5c-436f-4698-a943-391d39241a02-kube-api-access-db497\") pod \"payload-processing-e2e-inf-a-d9f7ae-648c996649-ts2sr\" (UID: \"5d1fbc5c-436f-4698-a943-391d39241a02\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d9f7ae-648c996649-ts2sr" Sep 04 22:22:48.835595 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:48.835427 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d1fbc5c-436f-4698-a943-391d39241a02-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d9f7ae-648c996649-ts2sr\" (UID: \"5d1fbc5c-436f-4698-a943-391d39241a02\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d9f7ae-648c996649-ts2sr" Sep 04 22:22:48.836110 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:48.836092 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d1fbc5c-436f-4698-a943-391d39241a02-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d9f7ae-648c996649-ts2sr\" (UID: \"5d1fbc5c-436f-4698-a943-391d39241a02\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d9f7ae-648c996649-ts2sr" Sep 04 22:22:48.849702 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:48.849681 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-db497\" (UniqueName: \"kubernetes.io/projected/5d1fbc5c-436f-4698-a943-391d39241a02-kube-api-access-db497\") pod \"payload-processing-e2e-inf-a-d9f7ae-648c996649-ts2sr\" (UID: \"5d1fbc5c-436f-4698-a943-391d39241a02\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d9f7ae-648c996649-ts2sr" Sep 04 22:22:48.927396 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:48.927332 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d9f7ae-56b97d44c8-pgnhl" Sep 04 22:22:48.954207 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:48.954166 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d9f7ae-648c996649-ts2sr" Sep 04 22:22:49.206581 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:49.206544 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d9f7ae-56b97d44c8-pgnhl"] Sep 04 22:22:49.209069 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:22:49.209040 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b920f63_12eb_47ba_8ade_f32eb154b0a8.slice/crio-8ce5ffcec1ef0867637874166479f5b326dac61f8170564f388d976c6862c801 WatchSource:0}: Error finding container 8ce5ffcec1ef0867637874166479f5b326dac61f8170564f388d976c6862c801: Status 404 returned error can't find the container with id 8ce5ffcec1ef0867637874166479f5b326dac61f8170564f388d976c6862c801 Sep 04 22:22:49.236699 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:49.236676 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d9f7ae-648c996649-ts2sr"] Sep 04 22:22:49.246639 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:22:49.246616 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d1fbc5c_436f_4698_a943_391d39241a02.slice/crio-835bd6896acd5424e3f3fb4dab59443fc055c53cbc7e04a11843d769b40bdf94 WatchSource:0}: Error finding container 835bd6896acd5424e3f3fb4dab59443fc055c53cbc7e04a11843d769b40bdf94: Status 404 returned error can't find the container with id 835bd6896acd5424e3f3fb4dab59443fc055c53cbc7e04a11843d769b40bdf94 Sep 04 22:22:49.492400 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:49.492304 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d9f7ae-56b97d44c8-pgnhl" event={"ID":"6b920f63-12eb-47ba-8ade-f32eb154b0a8","Type":"ContainerStarted","Data":"2bc41b494c676a7d39e5e2e0f118c7f2721de0e04e517c3ae195382b1fe4b0ee"} Sep 04 22:22:49.492400 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:49.492377 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d9f7ae-56b97d44c8-pgnhl" event={"ID":"6b920f63-12eb-47ba-8ade-f32eb154b0a8","Type":"ContainerStarted","Data":"8ce5ffcec1ef0867637874166479f5b326dac61f8170564f388d976c6862c801"} Sep 04 22:22:49.492623 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:49.492414 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d9f7ae-56b97d44c8-pgnhl" Sep 04 22:22:49.494172 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:49.494145 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d9f7ae-648c996649-ts2sr" event={"ID":"5d1fbc5c-436f-4698-a943-391d39241a02","Type":"ContainerStarted","Data":"a19ce1c65d7d9e1d76583bde92c7f4faa87aca6bae5dff18905713206a07bd51"} Sep 04 22:22:49.494300 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:49.494187 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d9f7ae-648c996649-ts2sr" event={"ID":"5d1fbc5c-436f-4698-a943-391d39241a02","Type":"ContainerStarted","Data":"835bd6896acd5424e3f3fb4dab59443fc055c53cbc7e04a11843d769b40bdf94"} Sep 04 22:22:49.494300 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:49.494208 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d9f7ae-648c996649-ts2sr" Sep 04 22:22:49.511453 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:49.511405 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d9f7ae-56b97d44c8-pgnhl" podStartSLOduration=1.51138759 podStartE2EDuration="1.51138759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:22:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:22:49.509238661 +0000 UTC m=+873.273714274" watchObservedRunningTime="2026-09-04 22:22:49.51138759 +0000 UTC m=+873.275863207" Sep 04 22:22:49.527772 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:49.527729 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-d9f7ae-648c996649-ts2sr" podStartSLOduration=1.527717233 podStartE2EDuration="1.527717233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:22:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:22:49.526377219 +0000 UTC m=+873.290852834" watchObservedRunningTime="2026-09-04 22:22:49.527717233 +0000 UTC m=+873.292192862" Sep 04 22:22:57.063041 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:57.063009 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c561f9fe-86c968f68f-k2dlt"] Sep 04 22:22:57.067668 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:57.067646 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c561f9fe-86c968f68f-k2dlt" Sep 04 22:22:57.070072 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:57.070054 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c561f9fe-dockercfg-m7xz4\"" Sep 04 22:22:57.070148 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:57.070117 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c561f9fe\"" Sep 04 22:22:57.076954 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:57.076929 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c561f9fe-86c968f68f-k2dlt"] Sep 04 22:22:57.114844 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:57.114818 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00a0cc2e-8458-40d7-8116-8062af094b42-plugins-config-volume\") pod \"payload-processing-e2e-ait-c561f9fe-86c968f68f-k2dlt\" (UID: \"00a0cc2e-8458-40d7-8116-8062af094b42\") " pod="openshift-ingress/payload-processing-e2e-ait-c561f9fe-86c968f68f-k2dlt" Sep 04 22:22:57.114844 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:57.114845 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nm58\" (UniqueName: \"kubernetes.io/projected/00a0cc2e-8458-40d7-8116-8062af094b42-kube-api-access-5nm58\") pod \"payload-processing-e2e-ait-c561f9fe-86c968f68f-k2dlt\" (UID: \"00a0cc2e-8458-40d7-8116-8062af094b42\") " pod="openshift-ingress/payload-processing-e2e-ait-c561f9fe-86c968f68f-k2dlt" Sep 04 22:22:57.216053 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:57.216016 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00a0cc2e-8458-40d7-8116-8062af094b42-plugins-config-volume\") pod \"payload-processing-e2e-ait-c561f9fe-86c968f68f-k2dlt\" (UID: \"00a0cc2e-8458-40d7-8116-8062af094b42\") " pod="openshift-ingress/payload-processing-e2e-ait-c561f9fe-86c968f68f-k2dlt" Sep 04 22:22:57.216053 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:57.216059 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nm58\" (UniqueName: \"kubernetes.io/projected/00a0cc2e-8458-40d7-8116-8062af094b42-kube-api-access-5nm58\") pod \"payload-processing-e2e-ait-c561f9fe-86c968f68f-k2dlt\" (UID: \"00a0cc2e-8458-40d7-8116-8062af094b42\") " pod="openshift-ingress/payload-processing-e2e-ait-c561f9fe-86c968f68f-k2dlt" Sep 04 22:22:57.216767 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:57.216745 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00a0cc2e-8458-40d7-8116-8062af094b42-plugins-config-volume\") pod \"payload-processing-e2e-ait-c561f9fe-86c968f68f-k2dlt\" (UID: \"00a0cc2e-8458-40d7-8116-8062af094b42\") " pod="openshift-ingress/payload-processing-e2e-ait-c561f9fe-86c968f68f-k2dlt" Sep 04 22:22:57.225018 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:57.224996 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nm58\" (UniqueName: \"kubernetes.io/projected/00a0cc2e-8458-40d7-8116-8062af094b42-kube-api-access-5nm58\") pod \"payload-processing-e2e-ait-c561f9fe-86c968f68f-k2dlt\" (UID: \"00a0cc2e-8458-40d7-8116-8062af094b42\") " pod="openshift-ingress/payload-processing-e2e-ait-c561f9fe-86c968f68f-k2dlt" Sep 04 22:22:57.379408 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:57.379341 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c561f9fe-86c968f68f-k2dlt" Sep 04 22:22:57.652574 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:57.652537 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c561f9fe-86c968f68f-k2dlt"] Sep 04 22:22:57.654830 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:22:57.654803 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00a0cc2e_8458_40d7_8116_8062af094b42.slice/crio-cd5d343a8f6a396e2b9554f753a2ba0a112b572fc97e1b0500084e1dd41241cc WatchSource:0}: Error finding container cd5d343a8f6a396e2b9554f753a2ba0a112b572fc97e1b0500084e1dd41241cc: Status 404 returned error can't find the container with id cd5d343a8f6a396e2b9554f753a2ba0a112b572fc97e1b0500084e1dd41241cc Sep 04 22:22:58.542367 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:58.542324 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c561f9fe-86c968f68f-k2dlt" event={"ID":"00a0cc2e-8458-40d7-8116-8062af094b42","Type":"ContainerStarted","Data":"a3ebb46caeacbbaf94af0cf4ffd04c3409d64394ee6ba1fa3fcd642cc013feef"} Sep 04 22:22:58.542790 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:58.542394 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c561f9fe-86c968f68f-k2dlt" event={"ID":"00a0cc2e-8458-40d7-8116-8062af094b42","Type":"ContainerStarted","Data":"cd5d343a8f6a396e2b9554f753a2ba0a112b572fc97e1b0500084e1dd41241cc"} Sep 04 22:22:58.542790 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:58.542437 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-c561f9fe-86c968f68f-k2dlt" Sep 04 22:22:58.561888 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:22:58.561845 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-c561f9fe-86c968f68f-k2dlt" podStartSLOduration=1.5618293840000002 podStartE2EDuration="1.561829384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:22:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:22:58.558923715 +0000 UTC m=+882.323399331" watchObservedRunningTime="2026-09-04 22:22:58.561829384 +0000 UTC m=+882.326304997" Sep 04 22:23:00.502787 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:00.502732 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d9f7ae-56b97d44c8-pgnhl" Sep 04 22:23:00.503241 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:00.502826 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d9f7ae-648c996649-ts2sr" Sep 04 22:23:01.008578 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:01.008532 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a8d845-776d46797c-pv87c"] Sep 04 22:23:01.012102 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:01.012084 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a8d845-776d46797c-pv87c" Sep 04 22:23:01.014651 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:01.014628 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-a8d845-dockercfg-d5h98\"" Sep 04 22:23:01.014758 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:01.014629 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-a8d845\"" Sep 04 22:23:01.025321 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:01.025293 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a8d845-776d46797c-pv87c"] Sep 04 22:23:01.069266 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:01.069238 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a8d845-6469bc984b-p2zd9"] Sep 04 22:23:01.072584 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:01.072569 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-a8d845-6469bc984b-p2zd9" Sep 04 22:23:01.081787 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:01.081763 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a8d845-6469bc984b-p2zd9"] Sep 04 22:23:01.146526 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:01.146495 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c01200a-3f15-4223-aaa3-7f7ccbd1427c-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-a8d845-6469bc984b-p2zd9\" (UID: \"6c01200a-3f15-4223-aaa3-7f7ccbd1427c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a8d845-6469bc984b-p2zd9" Sep 04 22:23:01.146694 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:01.146538 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hgfr\" (UniqueName: \"kubernetes.io/projected/6c01200a-3f15-4223-aaa3-7f7ccbd1427c-kube-api-access-5hgfr\") pod \"payload-pre-processing-e2e-autores-a8d845-6469bc984b-p2zd9\" (UID: \"6c01200a-3f15-4223-aaa3-7f7ccbd1427c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a8d845-6469bc984b-p2zd9" Sep 04 22:23:01.146694 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:01.146626 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad508a5b-b62f-4d57-b7b9-c490dbefa055-plugins-config-volume\") pod \"payload-processing-e2e-autores-a8d845-776d46797c-pv87c\" (UID: \"ad508a5b-b62f-4d57-b7b9-c490dbefa055\") " pod="openshift-ingress/payload-processing-e2e-autores-a8d845-776d46797c-pv87c" Sep 04 22:23:01.146781 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:01.146737 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqssw\" (UniqueName: \"kubernetes.io/projected/ad508a5b-b62f-4d57-b7b9-c490dbefa055-kube-api-access-fqssw\") pod \"payload-processing-e2e-autores-a8d845-776d46797c-pv87c\" (UID: \"ad508a5b-b62f-4d57-b7b9-c490dbefa055\") " pod="openshift-ingress/payload-processing-e2e-autores-a8d845-776d46797c-pv87c" Sep 04 22:23:01.247359 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:01.247324 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hgfr\" (UniqueName: \"kubernetes.io/projected/6c01200a-3f15-4223-aaa3-7f7ccbd1427c-kube-api-access-5hgfr\") pod \"payload-pre-processing-e2e-autores-a8d845-6469bc984b-p2zd9\" (UID: \"6c01200a-3f15-4223-aaa3-7f7ccbd1427c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a8d845-6469bc984b-p2zd9" Sep 04 22:23:01.247656 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:01.247631 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad508a5b-b62f-4d57-b7b9-c490dbefa055-plugins-config-volume\") pod \"payload-processing-e2e-autores-a8d845-776d46797c-pv87c\" (UID: \"ad508a5b-b62f-4d57-b7b9-c490dbefa055\") " pod="openshift-ingress/payload-processing-e2e-autores-a8d845-776d46797c-pv87c" Sep 04 22:23:01.247845 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:01.247831 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqssw\" (UniqueName: \"kubernetes.io/projected/ad508a5b-b62f-4d57-b7b9-c490dbefa055-kube-api-access-fqssw\") pod \"payload-processing-e2e-autores-a8d845-776d46797c-pv87c\" (UID: \"ad508a5b-b62f-4d57-b7b9-c490dbefa055\") " pod="openshift-ingress/payload-processing-e2e-autores-a8d845-776d46797c-pv87c" Sep 04 22:23:01.247975 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:01.247962 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c01200a-3f15-4223-aaa3-7f7ccbd1427c-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-a8d845-6469bc984b-p2zd9\" (UID: \"6c01200a-3f15-4223-aaa3-7f7ccbd1427c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a8d845-6469bc984b-p2zd9" Sep 04 22:23:01.248901 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:01.248875 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c01200a-3f15-4223-aaa3-7f7ccbd1427c-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-a8d845-6469bc984b-p2zd9\" (UID: \"6c01200a-3f15-4223-aaa3-7f7ccbd1427c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a8d845-6469bc984b-p2zd9" Sep 04 22:23:01.249499 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:01.249473 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad508a5b-b62f-4d57-b7b9-c490dbefa055-plugins-config-volume\") pod \"payload-processing-e2e-autores-a8d845-776d46797c-pv87c\" (UID: \"ad508a5b-b62f-4d57-b7b9-c490dbefa055\") " pod="openshift-ingress/payload-processing-e2e-autores-a8d845-776d46797c-pv87c" Sep 04 22:23:01.257533 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:01.257507 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hgfr\" (UniqueName: \"kubernetes.io/projected/6c01200a-3f15-4223-aaa3-7f7ccbd1427c-kube-api-access-5hgfr\") pod \"payload-pre-processing-e2e-autores-a8d845-6469bc984b-p2zd9\" (UID: \"6c01200a-3f15-4223-aaa3-7f7ccbd1427c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a8d845-6469bc984b-p2zd9" Sep 04 22:23:01.257633 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:01.257543 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqssw\" (UniqueName: \"kubernetes.io/projected/ad508a5b-b62f-4d57-b7b9-c490dbefa055-kube-api-access-fqssw\") pod \"payload-processing-e2e-autores-a8d845-776d46797c-pv87c\" (UID: \"ad508a5b-b62f-4d57-b7b9-c490dbefa055\") " pod="openshift-ingress/payload-processing-e2e-autores-a8d845-776d46797c-pv87c" Sep 04 22:23:01.323696 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:01.323641 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a8d845-776d46797c-pv87c" Sep 04 22:23:01.383937 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:01.383907 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-a8d845-6469bc984b-p2zd9" Sep 04 22:23:01.597153 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:01.597115 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a8d845-776d46797c-pv87c"] Sep 04 22:23:01.598314 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:23:01.598288 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad508a5b_b62f_4d57_b7b9_c490dbefa055.slice/crio-6435423ff927a061bd73847467ddfccca7e83bc3f15658e9dc736dc54d45d312 WatchSource:0}: Error finding container 6435423ff927a061bd73847467ddfccca7e83bc3f15658e9dc736dc54d45d312: Status 404 returned error can't find the container with id 6435423ff927a061bd73847467ddfccca7e83bc3f15658e9dc736dc54d45d312 Sep 04 22:23:01.664797 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:01.664772 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a8d845-6469bc984b-p2zd9"] Sep 04 22:23:01.665848 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:23:01.665826 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c01200a_3f15_4223_aaa3_7f7ccbd1427c.slice/crio-3afb0e519074df6413b4cc53a711212ed82e949cb3bf3a3099be83aa393040e5 WatchSource:0}: Error finding container 3afb0e519074df6413b4cc53a711212ed82e949cb3bf3a3099be83aa393040e5: Status 404 returned error can't find the container with id 3afb0e519074df6413b4cc53a711212ed82e949cb3bf3a3099be83aa393040e5 Sep 04 22:23:02.564498 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:02.564459 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a8d845-776d46797c-pv87c" event={"ID":"ad508a5b-b62f-4d57-b7b9-c490dbefa055","Type":"ContainerStarted","Data":"8548f7466b9ea6773aadad5704764439e1328702eacc6cd799ccd31b9035ad3f"} Sep 04 22:23:02.564704 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:02.564580 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a8d845-776d46797c-pv87c" event={"ID":"ad508a5b-b62f-4d57-b7b9-c490dbefa055","Type":"ContainerStarted","Data":"6435423ff927a061bd73847467ddfccca7e83bc3f15658e9dc736dc54d45d312"} Sep 04 22:23:02.564704 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:02.564613 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-a8d845-776d46797c-pv87c" Sep 04 22:23:02.566140 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:02.566113 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-a8d845-6469bc984b-p2zd9" event={"ID":"6c01200a-3f15-4223-aaa3-7f7ccbd1427c","Type":"ContainerStarted","Data":"9efa8a6f2576ba06a6c1f5b3c07e46b228185695465197f96becee6d6277c353"} Sep 04 22:23:02.566236 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:02.566156 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-a8d845-6469bc984b-p2zd9" event={"ID":"6c01200a-3f15-4223-aaa3-7f7ccbd1427c","Type":"ContainerStarted","Data":"3afb0e519074df6413b4cc53a711212ed82e949cb3bf3a3099be83aa393040e5"} Sep 04 22:23:02.566236 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:02.566201 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-a8d845-6469bc984b-p2zd9" Sep 04 22:23:02.584282 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:02.584239 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-a8d845-776d46797c-pv87c" podStartSLOduration=2.584225053 podStartE2EDuration="2.584225053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:23:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:23:02.58202158 +0000 UTC m=+886.346497195" watchObservedRunningTime="2026-09-04 22:23:02.584225053 +0000 UTC m=+886.348700719" Sep 04 22:23:02.599358 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:02.599315 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-a8d845-6469bc984b-p2zd9" podStartSLOduration=1.599302158 podStartE2EDuration="1.599302158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:23:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:23:02.598026946 +0000 UTC m=+886.362502607" watchObservedRunningTime="2026-09-04 22:23:02.599302158 +0000 UTC m=+886.363777819" Sep 04 22:23:09.549427 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:09.549338 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-c561f9fe-86c968f68f-k2dlt" Sep 04 22:23:13.573347 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:13.573317 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-a8d845-776d46797c-pv87c" Sep 04 22:23:13.573744 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:13.573387 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-a8d845-6469bc984b-p2zd9" Sep 04 22:23:20.360125 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:20.360077 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c561f9fe-pj2wz"] Sep 04 22:23:20.364050 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:20.364025 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c561f9fe-pj2wz" Sep 04 22:23:20.366889 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:20.366865 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 04 22:23:20.366989 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:20.366875 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hc7x4\"" Sep 04 22:23:20.374509 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:20.374487 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c561f9fe-pj2wz"] Sep 04 22:23:20.505492 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:20.505464 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0af424ca-b429-4e63-b168-dd328f1971c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c561f9fe-pj2wz\" (UID: \"0af424ca-b429-4e63-b168-dd328f1971c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c561f9fe-pj2wz" Sep 04 22:23:20.606038 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:20.606008 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0af424ca-b429-4e63-b168-dd328f1971c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c561f9fe-pj2wz\" (UID: \"0af424ca-b429-4e63-b168-dd328f1971c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c561f9fe-pj2wz" Sep 04 22:23:20.606602 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:20.606585 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0af424ca-b429-4e63-b168-dd328f1971c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c561f9fe-pj2wz\" (UID: \"0af424ca-b429-4e63-b168-dd328f1971c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c561f9fe-pj2wz" Sep 04 22:23:20.675352 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:20.675291 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c561f9fe-pj2wz" Sep 04 22:23:20.948234 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:20.948191 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c561f9fe-pj2wz"] Sep 04 22:23:20.949574 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:23:20.949532 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0af424ca_b429_4e63_b168_dd328f1971c3.slice/crio-358a1fa602d563faf6c30af88cc9910940ab8bb6c3842903a653c76b41372e39 WatchSource:0}: Error finding container 358a1fa602d563faf6c30af88cc9910940ab8bb6c3842903a653c76b41372e39: Status 404 returned error can't find the container with id 358a1fa602d563faf6c30af88cc9910940ab8bb6c3842903a653c76b41372e39 Sep 04 22:23:21.662110 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:21.662075 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c561f9fe-pj2wz" event={"ID":"0af424ca-b429-4e63-b168-dd328f1971c3","Type":"ContainerStarted","Data":"358a1fa602d563faf6c30af88cc9910940ab8bb6c3842903a653c76b41372e39"} Sep 04 22:23:22.371686 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:22.371651 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fecd36-8557dfccfd-gr2gt"] Sep 04 22:23:22.375317 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:22.375289 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fecd36-8557dfccfd-gr2gt" Sep 04 22:23:22.377770 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:22.377747 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-fecd36\"" Sep 04 22:23:22.390186 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:22.390124 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fecd36-8557dfccfd-gr2gt"] Sep 04 22:23:22.493422 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:22.493389 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fecd36-795b599dfc-5nhnb"] Sep 04 22:23:22.500485 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:22.500461 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fecd36-795b599dfc-5nhnb" Sep 04 22:23:22.503015 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:22.502989 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-fecd36-dockercfg-dx469\"" Sep 04 22:23:22.507702 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:22.507674 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fecd36-795b599dfc-5nhnb"] Sep 04 22:23:22.523377 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:22.523352 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fbpq\" (UniqueName: \"kubernetes.io/projected/e5ec485f-1099-45ab-8af3-c0878cac4233-kube-api-access-4fbpq\") pod \"payload-pre-processing-e2e-inf-b-fecd36-8557dfccfd-gr2gt\" (UID: \"e5ec485f-1099-45ab-8af3-c0878cac4233\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fecd36-8557dfccfd-gr2gt" Sep 04 22:23:22.523538 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:22.523413 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5ec485f-1099-45ab-8af3-c0878cac4233-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-fecd36-8557dfccfd-gr2gt\" (UID: \"e5ec485f-1099-45ab-8af3-c0878cac4233\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fecd36-8557dfccfd-gr2gt" Sep 04 22:23:22.624746 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:22.624661 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd8hb\" (UniqueName: \"kubernetes.io/projected/2f64ef12-518c-4f18-adc4-99b085c05d37-kube-api-access-rd8hb\") pod \"payload-processing-e2e-inf-b-fecd36-795b599dfc-5nhnb\" (UID: \"2f64ef12-518c-4f18-adc4-99b085c05d37\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fecd36-795b599dfc-5nhnb" Sep 04 22:23:22.624746 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:22.624736 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fbpq\" (UniqueName: \"kubernetes.io/projected/e5ec485f-1099-45ab-8af3-c0878cac4233-kube-api-access-4fbpq\") pod \"payload-pre-processing-e2e-inf-b-fecd36-8557dfccfd-gr2gt\" (UID: \"e5ec485f-1099-45ab-8af3-c0878cac4233\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fecd36-8557dfccfd-gr2gt" Sep 04 22:23:22.624949 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:22.624803 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5ec485f-1099-45ab-8af3-c0878cac4233-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-fecd36-8557dfccfd-gr2gt\" (UID: \"e5ec485f-1099-45ab-8af3-c0878cac4233\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fecd36-8557dfccfd-gr2gt" Sep 04 22:23:22.624949 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:22.624845 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f64ef12-518c-4f18-adc4-99b085c05d37-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-fecd36-795b599dfc-5nhnb\" (UID: \"2f64ef12-518c-4f18-adc4-99b085c05d37\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fecd36-795b599dfc-5nhnb" Sep 04 22:23:22.625507 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:22.625483 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5ec485f-1099-45ab-8af3-c0878cac4233-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-fecd36-8557dfccfd-gr2gt\" (UID: \"e5ec485f-1099-45ab-8af3-c0878cac4233\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fecd36-8557dfccfd-gr2gt" Sep 04 22:23:22.634068 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:22.634042 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fbpq\" (UniqueName: \"kubernetes.io/projected/e5ec485f-1099-45ab-8af3-c0878cac4233-kube-api-access-4fbpq\") pod \"payload-pre-processing-e2e-inf-b-fecd36-8557dfccfd-gr2gt\" (UID: \"e5ec485f-1099-45ab-8af3-c0878cac4233\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fecd36-8557dfccfd-gr2gt" Sep 04 22:23:22.687142 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:22.687111 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fecd36-8557dfccfd-gr2gt" Sep 04 22:23:22.725836 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:22.725807 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f64ef12-518c-4f18-adc4-99b085c05d37-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-fecd36-795b599dfc-5nhnb\" (UID: \"2f64ef12-518c-4f18-adc4-99b085c05d37\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fecd36-795b599dfc-5nhnb" Sep 04 22:23:22.725973 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:22.725885 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd8hb\" (UniqueName: \"kubernetes.io/projected/2f64ef12-518c-4f18-adc4-99b085c05d37-kube-api-access-rd8hb\") pod \"payload-processing-e2e-inf-b-fecd36-795b599dfc-5nhnb\" (UID: \"2f64ef12-518c-4f18-adc4-99b085c05d37\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fecd36-795b599dfc-5nhnb" Sep 04 22:23:22.726498 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:22.726477 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f64ef12-518c-4f18-adc4-99b085c05d37-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-fecd36-795b599dfc-5nhnb\" (UID: \"2f64ef12-518c-4f18-adc4-99b085c05d37\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fecd36-795b599dfc-5nhnb" Sep 04 22:23:22.735023 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:22.735001 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd8hb\" (UniqueName: \"kubernetes.io/projected/2f64ef12-518c-4f18-adc4-99b085c05d37-kube-api-access-rd8hb\") pod \"payload-processing-e2e-inf-b-fecd36-795b599dfc-5nhnb\" (UID: \"2f64ef12-518c-4f18-adc4-99b085c05d37\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fecd36-795b599dfc-5nhnb" Sep 04 22:23:22.815841 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:22.815808 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fecd36-795b599dfc-5nhnb" Sep 04 22:23:23.346055 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:23.346028 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fecd36-8557dfccfd-gr2gt"] Sep 04 22:23:23.347688 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:23:23.347663 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f64ef12_518c_4f18_adc4_99b085c05d37.slice/crio-eb52a229f11db329ef3989b1700581a0e5c4ef4de42e0b0d49cc8deebbde2f53 WatchSource:0}: Error finding container eb52a229f11db329ef3989b1700581a0e5c4ef4de42e0b0d49cc8deebbde2f53: Status 404 returned error can't find the container with id eb52a229f11db329ef3989b1700581a0e5c4ef4de42e0b0d49cc8deebbde2f53 Sep 04 22:23:23.348633 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:23:23.348522 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ec485f_1099_45ab_8af3_c0878cac4233.slice/crio-ee8562b3ea9cdf01a72603b635286491dcf80e9abec275ef7bb81c18f6410193 WatchSource:0}: Error finding container ee8562b3ea9cdf01a72603b635286491dcf80e9abec275ef7bb81c18f6410193: Status 404 returned error can't find the container with id ee8562b3ea9cdf01a72603b635286491dcf80e9abec275ef7bb81c18f6410193 Sep 04 22:23:23.349338 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:23.349310 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fecd36-795b599dfc-5nhnb"] Sep 04 22:23:23.677573 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:23.677478 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c561f9fe-pj2wz" event={"ID":"0af424ca-b429-4e63-b168-dd328f1971c3","Type":"ContainerStarted","Data":"cb398f6d928644fedbccad354b5c6b16a702d2c9e2cf132c8db965b6453f8ca4"} Sep 04 22:23:23.679319 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:23.679243 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fecd36-8557dfccfd-gr2gt" event={"ID":"e5ec485f-1099-45ab-8af3-c0878cac4233","Type":"ContainerStarted","Data":"8292474bf7752f32525e95f81978ebb0a6ca5c8f6fa236073ac94f1cf3125fac"} Sep 04 22:23:23.679319 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:23.679302 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fecd36-8557dfccfd-gr2gt" Sep 04 22:23:23.679511 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:23.679325 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fecd36-8557dfccfd-gr2gt" event={"ID":"e5ec485f-1099-45ab-8af3-c0878cac4233","Type":"ContainerStarted","Data":"ee8562b3ea9cdf01a72603b635286491dcf80e9abec275ef7bb81c18f6410193"} Sep 04 22:23:23.681890 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:23.681715 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fecd36-795b599dfc-5nhnb" event={"ID":"2f64ef12-518c-4f18-adc4-99b085c05d37","Type":"ContainerStarted","Data":"7e4f3f17c1515f3de81b371e5b87004fb9bb17d8495afbae876916cdaec98d69"} Sep 04 22:23:23.681890 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:23.681813 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-fecd36-795b599dfc-5nhnb" Sep 04 22:23:23.681890 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:23.681846 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fecd36-795b599dfc-5nhnb" event={"ID":"2f64ef12-518c-4f18-adc4-99b085c05d37","Type":"ContainerStarted","Data":"eb52a229f11db329ef3989b1700581a0e5c4ef4de42e0b0d49cc8deebbde2f53"} Sep 04 22:23:23.696696 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:23.696649 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c561f9fe-pj2wz" podStartSLOduration=1.5097342660000002 podStartE2EDuration="3.696637973s" totalImagesPullingTime="2.186903707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:23:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:23:23.693723775 +0000 UTC m=+907.458199388" watchObservedRunningTime="2026-09-04 22:23:23.696637973 +0000 UTC m=+907.461113586" Sep 04 22:23:23.712771 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:23.712714 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fecd36-8557dfccfd-gr2gt" podStartSLOduration=1.712697637 podStartE2EDuration="1.712697637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:23:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:23:23.710076129 +0000 UTC m=+907.474551745" watchObservedRunningTime="2026-09-04 22:23:23.712697637 +0000 UTC m=+907.477173251" Sep 04 22:23:23.727732 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:23.727684 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-fecd36-795b599dfc-5nhnb" podStartSLOduration=1.727667742 podStartE2EDuration="1.727667742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:23:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:23:23.726734375 +0000 UTC m=+907.491209989" watchObservedRunningTime="2026-09-04 22:23:23.727667742 +0000 UTC m=+907.492143353" Sep 04 22:23:24.688654 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:24.688618 2574 generic.go:356] "Generic (PLEG): container finished" podID="0af424ca-b429-4e63-b168-dd328f1971c3" containerID="cb398f6d928644fedbccad354b5c6b16a702d2c9e2cf132c8db965b6453f8ca4" exitCode=0 Sep 04 22:23:24.688837 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:24.688706 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c561f9fe-pj2wz" event={"ID":"0af424ca-b429-4e63-b168-dd328f1971c3","Type":"ContainerDied","Data":"cb398f6d928644fedbccad354b5c6b16a702d2c9e2cf132c8db965b6453f8ca4"} Sep 04 22:23:25.824346 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:25.824324 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c561f9fe-pj2wz" Sep 04 22:23:25.955246 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:25.955171 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0af424ca-b429-4e63-b168-dd328f1971c3-maas-api-service-ca\") pod \"0af424ca-b429-4e63-b168-dd328f1971c3\" (UID: \"0af424ca-b429-4e63-b168-dd328f1971c3\") " Sep 04 22:23:25.955497 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:25.955476 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0af424ca-b429-4e63-b168-dd328f1971c3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0af424ca-b429-4e63-b168-dd328f1971c3" (UID: "0af424ca-b429-4e63-b168-dd328f1971c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:23:26.056220 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:26.056187 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0af424ca-b429-4e63-b168-dd328f1971c3-maas-api-service-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:23:26.699827 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:26.699790 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c561f9fe-pj2wz" event={"ID":"0af424ca-b429-4e63-b168-dd328f1971c3","Type":"ContainerDied","Data":"358a1fa602d563faf6c30af88cc9910940ab8bb6c3842903a653c76b41372e39"} Sep 04 22:23:26.699827 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:26.699817 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c561f9fe-pj2wz" Sep 04 22:23:26.700024 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:26.699850 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="358a1fa602d563faf6c30af88cc9910940ab8bb6c3842903a653c76b41372e39" Sep 04 22:23:34.690193 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:34.690157 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fecd36-8557dfccfd-gr2gt" Sep 04 22:23:34.690586 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:34.690246 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-fecd36-795b599dfc-5nhnb" Sep 04 22:23:37.117897 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:37.117861 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c561f9fe-86c968f68f-k2dlt"] Sep 04 22:23:37.118410 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:37.118072 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-c561f9fe-86c968f68f-k2dlt" podUID="00a0cc2e-8458-40d7-8116-8062af094b42" containerName="payload-processing" containerID="cri-o://a3ebb46caeacbbaf94af0cf4ffd04c3409d64394ee6ba1fa3fcd642cc013feef" gracePeriod=30 Sep 04 22:23:37.309200 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:37.309161 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c561f9fe-pj2wz"] Sep 04 22:23:37.311608 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:37.311579 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c561f9fe-pj2wz"] Sep 04 22:23:37.366990 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:37.366967 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c561f9fe-86c968f68f-k2dlt" Sep 04 22:23:37.448945 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:37.448874 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nm58\" (UniqueName: \"kubernetes.io/projected/00a0cc2e-8458-40d7-8116-8062af094b42-kube-api-access-5nm58\") pod \"00a0cc2e-8458-40d7-8116-8062af094b42\" (UID: \"00a0cc2e-8458-40d7-8116-8062af094b42\") " Sep 04 22:23:37.449150 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:37.448959 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00a0cc2e-8458-40d7-8116-8062af094b42-plugins-config-volume\") pod \"00a0cc2e-8458-40d7-8116-8062af094b42\" (UID: \"00a0cc2e-8458-40d7-8116-8062af094b42\") " Sep 04 22:23:37.449353 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:37.449321 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00a0cc2e-8458-40d7-8116-8062af094b42-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "00a0cc2e-8458-40d7-8116-8062af094b42" (UID: "00a0cc2e-8458-40d7-8116-8062af094b42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:23:37.452218 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:37.452192 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00a0cc2e-8458-40d7-8116-8062af094b42-kube-api-access-5nm58" (OuterVolumeSpecName: "kube-api-access-5nm58") pod "00a0cc2e-8458-40d7-8116-8062af094b42" (UID: "00a0cc2e-8458-40d7-8116-8062af094b42"). InnerVolumeSpecName "kube-api-access-5nm58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:23:37.550441 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:37.550415 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nm58\" (UniqueName: \"kubernetes.io/projected/00a0cc2e-8458-40d7-8116-8062af094b42-kube-api-access-5nm58\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:23:37.550441 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:37.550440 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00a0cc2e-8458-40d7-8116-8062af094b42-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:23:37.758101 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:37.758070 2574 generic.go:356] "Generic (PLEG): container finished" podID="00a0cc2e-8458-40d7-8116-8062af094b42" containerID="a3ebb46caeacbbaf94af0cf4ffd04c3409d64394ee6ba1fa3fcd642cc013feef" exitCode=0 Sep 04 22:23:37.758403 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:37.758112 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c561f9fe-86c968f68f-k2dlt" Sep 04 22:23:37.758403 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:37.758159 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c561f9fe-86c968f68f-k2dlt" event={"ID":"00a0cc2e-8458-40d7-8116-8062af094b42","Type":"ContainerDied","Data":"a3ebb46caeacbbaf94af0cf4ffd04c3409d64394ee6ba1fa3fcd642cc013feef"} Sep 04 22:23:37.758403 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:37.758227 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c561f9fe-86c968f68f-k2dlt" event={"ID":"00a0cc2e-8458-40d7-8116-8062af094b42","Type":"ContainerDied","Data":"cd5d343a8f6a396e2b9554f753a2ba0a112b572fc97e1b0500084e1dd41241cc"} Sep 04 22:23:37.758403 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:37.758262 2574 scope.go:117] "RemoveContainer" containerID="a3ebb46caeacbbaf94af0cf4ffd04c3409d64394ee6ba1fa3fcd642cc013feef" Sep 04 22:23:37.767559 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:37.767532 2574 scope.go:117] "RemoveContainer" containerID="a3ebb46caeacbbaf94af0cf4ffd04c3409d64394ee6ba1fa3fcd642cc013feef" Sep 04 22:23:37.767823 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:23:37.767804 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3ebb46caeacbbaf94af0cf4ffd04c3409d64394ee6ba1fa3fcd642cc013feef\": container with ID starting with a3ebb46caeacbbaf94af0cf4ffd04c3409d64394ee6ba1fa3fcd642cc013feef not found: ID does not exist" containerID="a3ebb46caeacbbaf94af0cf4ffd04c3409d64394ee6ba1fa3fcd642cc013feef" Sep 04 22:23:37.767881 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:37.767832 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3ebb46caeacbbaf94af0cf4ffd04c3409d64394ee6ba1fa3fcd642cc013feef"} err="failed to get container status \"a3ebb46caeacbbaf94af0cf4ffd04c3409d64394ee6ba1fa3fcd642cc013feef\": rpc error: code = NotFound desc = could not find container \"a3ebb46caeacbbaf94af0cf4ffd04c3409d64394ee6ba1fa3fcd642cc013feef\": container with ID starting with a3ebb46caeacbbaf94af0cf4ffd04c3409d64394ee6ba1fa3fcd642cc013feef not found: ID does not exist" Sep 04 22:23:37.781752 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:37.781728 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c561f9fe-86c968f68f-k2dlt"] Sep 04 22:23:37.786803 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:37.786782 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c561f9fe-86c968f68f-k2dlt"] Sep 04 22:23:38.715330 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:38.715294 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00a0cc2e-8458-40d7-8116-8062af094b42" path="/var/lib/kubelet/pods/00a0cc2e-8458-40d7-8116-8062af094b42/volumes" Sep 04 22:23:38.715716 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:38.715632 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0af424ca-b429-4e63-b168-dd328f1971c3" path="/var/lib/kubelet/pods/0af424ca-b429-4e63-b168-dd328f1971c3/volumes" Sep 04 22:23:41.688104 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:41.688070 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-bb137412-75b6ff7f78-46gjh"] Sep 04 22:23:41.688458 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:41.688396 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00a0cc2e-8458-40d7-8116-8062af094b42" containerName="payload-processing" Sep 04 22:23:41.688458 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:41.688408 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="00a0cc2e-8458-40d7-8116-8062af094b42" containerName="payload-processing" Sep 04 22:23:41.688458 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:41.688419 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0af424ca-b429-4e63-b168-dd328f1971c3" containerName="revoke-keys" Sep 04 22:23:41.688458 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:41.688424 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0af424ca-b429-4e63-b168-dd328f1971c3" containerName="revoke-keys" Sep 04 22:23:41.688637 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:41.688480 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="00a0cc2e-8458-40d7-8116-8062af094b42" containerName="payload-processing" Sep 04 22:23:41.688637 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:41.688492 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0af424ca-b429-4e63-b168-dd328f1971c3" containerName="revoke-keys" Sep 04 22:23:41.692691 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:41.692675 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb137412-75b6ff7f78-46gjh" Sep 04 22:23:41.695888 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:41.695858 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-bb137412\"" Sep 04 22:23:41.704963 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:41.704942 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-bb137412-75b6ff7f78-46gjh"] Sep 04 22:23:41.707623 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:41.707601 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-bb137412-57f86c67d-pn2fj"] Sep 04 22:23:41.710761 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:41.710738 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-bb137412-57f86c67d-pn2fj" Sep 04 22:23:41.713048 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:41.713028 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-bb137412-dockercfg-zrsfj\"" Sep 04 22:23:41.724343 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:41.724320 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-bb137412-57f86c67d-pn2fj"] Sep 04 22:23:41.786929 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:41.786900 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2gqz\" (UniqueName: \"kubernetes.io/projected/15fcf374-0dbe-4c9c-af62-fb2b7bc20919-kube-api-access-k2gqz\") pod \"payload-processing-e2e-migrate-bb137412-57f86c67d-pn2fj\" (UID: \"15fcf374-0dbe-4c9c-af62-fb2b7bc20919\") " pod="openshift-ingress/payload-processing-e2e-migrate-bb137412-57f86c67d-pn2fj" Sep 04 22:23:41.787071 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:41.786948 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsglm\" (UniqueName: \"kubernetes.io/projected/52fb505b-6089-4557-9682-e39f2f176607-kube-api-access-zsglm\") pod \"payload-pre-processing-e2e-migrate-bb137412-75b6ff7f78-46gjh\" (UID: \"52fb505b-6089-4557-9682-e39f2f176607\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb137412-75b6ff7f78-46gjh" Sep 04 22:23:41.787071 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:41.787004 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52fb505b-6089-4557-9682-e39f2f176607-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-bb137412-75b6ff7f78-46gjh\" (UID: \"52fb505b-6089-4557-9682-e39f2f176607\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb137412-75b6ff7f78-46gjh" Sep 04 22:23:41.787071 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:41.787031 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15fcf374-0dbe-4c9c-af62-fb2b7bc20919-plugins-config-volume\") pod \"payload-processing-e2e-migrate-bb137412-57f86c67d-pn2fj\" (UID: \"15fcf374-0dbe-4c9c-af62-fb2b7bc20919\") " pod="openshift-ingress/payload-processing-e2e-migrate-bb137412-57f86c67d-pn2fj" Sep 04 22:23:41.887787 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:41.887731 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2gqz\" (UniqueName: \"kubernetes.io/projected/15fcf374-0dbe-4c9c-af62-fb2b7bc20919-kube-api-access-k2gqz\") pod \"payload-processing-e2e-migrate-bb137412-57f86c67d-pn2fj\" (UID: \"15fcf374-0dbe-4c9c-af62-fb2b7bc20919\") " pod="openshift-ingress/payload-processing-e2e-migrate-bb137412-57f86c67d-pn2fj" Sep 04 22:23:41.887969 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:41.887837 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsglm\" (UniqueName: \"kubernetes.io/projected/52fb505b-6089-4557-9682-e39f2f176607-kube-api-access-zsglm\") pod \"payload-pre-processing-e2e-migrate-bb137412-75b6ff7f78-46gjh\" (UID: \"52fb505b-6089-4557-9682-e39f2f176607\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb137412-75b6ff7f78-46gjh" Sep 04 22:23:41.887969 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:41.887896 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52fb505b-6089-4557-9682-e39f2f176607-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-bb137412-75b6ff7f78-46gjh\" (UID: \"52fb505b-6089-4557-9682-e39f2f176607\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb137412-75b6ff7f78-46gjh" Sep 04 22:23:41.887969 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:41.887942 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15fcf374-0dbe-4c9c-af62-fb2b7bc20919-plugins-config-volume\") pod \"payload-processing-e2e-migrate-bb137412-57f86c67d-pn2fj\" (UID: \"15fcf374-0dbe-4c9c-af62-fb2b7bc20919\") " pod="openshift-ingress/payload-processing-e2e-migrate-bb137412-57f86c67d-pn2fj" Sep 04 22:23:41.888763 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:41.888735 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52fb505b-6089-4557-9682-e39f2f176607-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-bb137412-75b6ff7f78-46gjh\" (UID: \"52fb505b-6089-4557-9682-e39f2f176607\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb137412-75b6ff7f78-46gjh" Sep 04 22:23:41.888870 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:41.888832 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15fcf374-0dbe-4c9c-af62-fb2b7bc20919-plugins-config-volume\") pod \"payload-processing-e2e-migrate-bb137412-57f86c67d-pn2fj\" (UID: \"15fcf374-0dbe-4c9c-af62-fb2b7bc20919\") " pod="openshift-ingress/payload-processing-e2e-migrate-bb137412-57f86c67d-pn2fj" Sep 04 22:23:41.897409 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:41.897384 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2gqz\" (UniqueName: \"kubernetes.io/projected/15fcf374-0dbe-4c9c-af62-fb2b7bc20919-kube-api-access-k2gqz\") pod \"payload-processing-e2e-migrate-bb137412-57f86c67d-pn2fj\" (UID: \"15fcf374-0dbe-4c9c-af62-fb2b7bc20919\") " pod="openshift-ingress/payload-processing-e2e-migrate-bb137412-57f86c67d-pn2fj" Sep 04 22:23:41.897505 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:41.897392 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsglm\" (UniqueName: \"kubernetes.io/projected/52fb505b-6089-4557-9682-e39f2f176607-kube-api-access-zsglm\") pod \"payload-pre-processing-e2e-migrate-bb137412-75b6ff7f78-46gjh\" (UID: \"52fb505b-6089-4557-9682-e39f2f176607\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb137412-75b6ff7f78-46gjh" Sep 04 22:23:42.004233 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:42.004197 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb137412-75b6ff7f78-46gjh" Sep 04 22:23:42.021364 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:42.021341 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-bb137412-57f86c67d-pn2fj" Sep 04 22:23:42.279512 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:42.279485 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-bb137412-75b6ff7f78-46gjh"] Sep 04 22:23:42.281557 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:23:42.281518 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52fb505b_6089_4557_9682_e39f2f176607.slice/crio-6d337a48ab1178b81f17e0a960214442e187474942c013e9dc31dc090143060c WatchSource:0}: Error finding container 6d337a48ab1178b81f17e0a960214442e187474942c013e9dc31dc090143060c: Status 404 returned error can't find the container with id 6d337a48ab1178b81f17e0a960214442e187474942c013e9dc31dc090143060c Sep 04 22:23:42.305010 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:42.304986 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-bb137412-57f86c67d-pn2fj"] Sep 04 22:23:42.312616 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:23:42.312592 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15fcf374_0dbe_4c9c_af62_fb2b7bc20919.slice/crio-99a4d983c250f1adf9d577dbda9f00b499de99abcf8b41e70a9dd8d0c3c290ca WatchSource:0}: Error finding container 99a4d983c250f1adf9d577dbda9f00b499de99abcf8b41e70a9dd8d0c3c290ca: Status 404 returned error can't find the container with id 99a4d983c250f1adf9d577dbda9f00b499de99abcf8b41e70a9dd8d0c3c290ca Sep 04 22:23:42.787529 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:42.787495 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-bb137412-57f86c67d-pn2fj" event={"ID":"15fcf374-0dbe-4c9c-af62-fb2b7bc20919","Type":"ContainerStarted","Data":"989e4bdf413858e32dfa5eb5599bd92a6b5e3957b3a1ca9d3138b5d4141daf44"} Sep 04 22:23:42.787988 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:42.787574 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-bb137412-57f86c67d-pn2fj" event={"ID":"15fcf374-0dbe-4c9c-af62-fb2b7bc20919","Type":"ContainerStarted","Data":"99a4d983c250f1adf9d577dbda9f00b499de99abcf8b41e70a9dd8d0c3c290ca"} Sep 04 22:23:42.787988 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:42.787728 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-bb137412-57f86c67d-pn2fj" Sep 04 22:23:42.789389 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:42.789363 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb137412-75b6ff7f78-46gjh" event={"ID":"52fb505b-6089-4557-9682-e39f2f176607","Type":"ContainerStarted","Data":"29571c934a693dc4549b8be3f7f0a9c43ab473fa06a6a4704e739180c5a0f5ef"} Sep 04 22:23:42.789667 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:42.789403 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb137412-75b6ff7f78-46gjh" event={"ID":"52fb505b-6089-4557-9682-e39f2f176607","Type":"ContainerStarted","Data":"6d337a48ab1178b81f17e0a960214442e187474942c013e9dc31dc090143060c"} Sep 04 22:23:42.789667 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:42.789489 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb137412-75b6ff7f78-46gjh" Sep 04 22:23:42.805880 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:42.805839 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-bb137412-57f86c67d-pn2fj" podStartSLOduration=1.805825885 podStartE2EDuration="1.805825885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:23:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:23:42.804145803 +0000 UTC m=+926.568621420" watchObservedRunningTime="2026-09-04 22:23:42.805825885 +0000 UTC m=+926.570301498" Sep 04 22:23:42.821235 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:42.821192 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb137412-75b6ff7f78-46gjh" podStartSLOduration=1.821175263 podStartE2EDuration="1.821175263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:23:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:23:42.820700388 +0000 UTC m=+926.585176002" watchObservedRunningTime="2026-09-04 22:23:42.821175263 +0000 UTC m=+926.585650882" Sep 04 22:23:53.796896 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:53.796862 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-bb137412-57f86c67d-pn2fj" Sep 04 22:23:53.797404 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:23:53.796940 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb137412-75b6ff7f78-46gjh" Sep 04 22:24:05.652086 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:05.652056 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb137412-l9sg6"] Sep 04 22:24:05.655444 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:05.655428 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb137412-l9sg6" Sep 04 22:24:05.657890 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:05.657871 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hc7x4\"" Sep 04 22:24:05.657993 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:05.657870 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 04 22:24:05.663143 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:05.663112 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb137412-l9sg6"] Sep 04 22:24:05.779441 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:05.779413 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba16a970-61a4-4b03-b53e-531499f4c969-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bb137412-l9sg6\" (UID: \"ba16a970-61a4-4b03-b53e-531499f4c969\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb137412-l9sg6" Sep 04 22:24:05.880669 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:05.880642 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba16a970-61a4-4b03-b53e-531499f4c969-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bb137412-l9sg6\" (UID: \"ba16a970-61a4-4b03-b53e-531499f4c969\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb137412-l9sg6" Sep 04 22:24:05.881192 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:05.881174 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba16a970-61a4-4b03-b53e-531499f4c969-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-bb137412-l9sg6\" (UID: \"ba16a970-61a4-4b03-b53e-531499f4c969\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb137412-l9sg6" Sep 04 22:24:05.967338 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:05.967279 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb137412-l9sg6" Sep 04 22:24:06.236892 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:06.236817 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb137412-l9sg6"] Sep 04 22:24:06.239499 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:24:06.239474 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba16a970_61a4_4b03_b53e_531499f4c969.slice/crio-3a11b2d63834a36afd6901a69640f50137432016dac3768779e1155ba38a4b12 WatchSource:0}: Error finding container 3a11b2d63834a36afd6901a69640f50137432016dac3768779e1155ba38a4b12: Status 404 returned error can't find the container with id 3a11b2d63834a36afd6901a69640f50137432016dac3768779e1155ba38a4b12 Sep 04 22:24:06.912015 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:06.911983 2574 generic.go:356] "Generic (PLEG): container finished" podID="ba16a970-61a4-4b03-b53e-531499f4c969" containerID="52928bf308cbabac4a0447a63209eac0ffa3f8279e632379024f2d428b074fe4" exitCode=0 Sep 04 22:24:06.912352 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:06.912057 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb137412-l9sg6" event={"ID":"ba16a970-61a4-4b03-b53e-531499f4c969","Type":"ContainerDied","Data":"52928bf308cbabac4a0447a63209eac0ffa3f8279e632379024f2d428b074fe4"} Sep 04 22:24:06.912352 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:06.912122 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb137412-l9sg6" event={"ID":"ba16a970-61a4-4b03-b53e-531499f4c969","Type":"ContainerStarted","Data":"3a11b2d63834a36afd6901a69640f50137432016dac3768779e1155ba38a4b12"} Sep 04 22:24:08.041630 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:08.041609 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb137412-l9sg6" Sep 04 22:24:08.099730 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:08.099703 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba16a970-61a4-4b03-b53e-531499f4c969-maas-api-service-ca\") pod \"ba16a970-61a4-4b03-b53e-531499f4c969\" (UID: \"ba16a970-61a4-4b03-b53e-531499f4c969\") " Sep 04 22:24:08.100004 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:08.099982 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba16a970-61a4-4b03-b53e-531499f4c969-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ba16a970-61a4-4b03-b53e-531499f4c969" (UID: "ba16a970-61a4-4b03-b53e-531499f4c969"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:24:08.201279 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:08.201221 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba16a970-61a4-4b03-b53e-531499f4c969-maas-api-service-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:24:08.923098 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:08.923067 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb137412-l9sg6" Sep 04 22:24:08.923244 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:08.923095 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb137412-l9sg6" event={"ID":"ba16a970-61a4-4b03-b53e-531499f4c969","Type":"ContainerDied","Data":"3a11b2d63834a36afd6901a69640f50137432016dac3768779e1155ba38a4b12"} Sep 04 22:24:08.923244 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:08.923144 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a11b2d63834a36afd6901a69640f50137432016dac3768779e1155ba38a4b12" Sep 04 22:24:12.421074 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:12.421044 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-bb137412-57f86c67d-pn2fj"] Sep 04 22:24:12.421461 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:12.421274 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-bb137412-57f86c67d-pn2fj" podUID="15fcf374-0dbe-4c9c-af62-fb2b7bc20919" containerName="payload-processing" containerID="cri-o://989e4bdf413858e32dfa5eb5599bd92a6b5e3957b3a1ca9d3138b5d4141daf44" gracePeriod=30 Sep 04 22:24:12.428254 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:12.428217 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-bb137412-75b6ff7f78-46gjh"] Sep 04 22:24:12.429891 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:12.428791 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb137412-75b6ff7f78-46gjh" podUID="52fb505b-6089-4557-9682-e39f2f176607" containerName="payload-pre-processing" containerID="cri-o://29571c934a693dc4549b8be3f7f0a9c43ab473fa06a6a4704e739180c5a0f5ef" gracePeriod=30 Sep 04 22:24:12.616783 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:12.616753 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb137412-l9sg6"] Sep 04 22:24:12.620007 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:12.619984 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-bb137412-l9sg6"] Sep 04 22:24:12.701063 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:12.701040 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-bb137412-57f86c67d-pn2fj" Sep 04 22:24:12.704823 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:12.704801 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb137412-75b6ff7f78-46gjh" Sep 04 22:24:12.716401 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:12.716345 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba16a970-61a4-4b03-b53e-531499f4c969" path="/var/lib/kubelet/pods/ba16a970-61a4-4b03-b53e-531499f4c969/volumes" Sep 04 22:24:12.839386 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:12.839317 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsglm\" (UniqueName: \"kubernetes.io/projected/52fb505b-6089-4557-9682-e39f2f176607-kube-api-access-zsglm\") pod \"52fb505b-6089-4557-9682-e39f2f176607\" (UID: \"52fb505b-6089-4557-9682-e39f2f176607\") " Sep 04 22:24:12.839386 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:12.839354 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2gqz\" (UniqueName: \"kubernetes.io/projected/15fcf374-0dbe-4c9c-af62-fb2b7bc20919-kube-api-access-k2gqz\") pod \"15fcf374-0dbe-4c9c-af62-fb2b7bc20919\" (UID: \"15fcf374-0dbe-4c9c-af62-fb2b7bc20919\") " Sep 04 22:24:12.839628 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:12.839608 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15fcf374-0dbe-4c9c-af62-fb2b7bc20919-plugins-config-volume\") pod \"15fcf374-0dbe-4c9c-af62-fb2b7bc20919\" (UID: \"15fcf374-0dbe-4c9c-af62-fb2b7bc20919\") " Sep 04 22:24:12.839686 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:12.839667 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52fb505b-6089-4557-9682-e39f2f176607-plugins-config-volume\") pod \"52fb505b-6089-4557-9682-e39f2f176607\" (UID: \"52fb505b-6089-4557-9682-e39f2f176607\") " Sep 04 22:24:12.840099 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:12.840062 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15fcf374-0dbe-4c9c-af62-fb2b7bc20919-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "15fcf374-0dbe-4c9c-af62-fb2b7bc20919" (UID: "15fcf374-0dbe-4c9c-af62-fb2b7bc20919"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:24:12.840264 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:12.840074 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52fb505b-6089-4557-9682-e39f2f176607-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "52fb505b-6089-4557-9682-e39f2f176607" (UID: "52fb505b-6089-4557-9682-e39f2f176607"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:24:12.841736 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:12.841712 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52fb505b-6089-4557-9682-e39f2f176607-kube-api-access-zsglm" (OuterVolumeSpecName: "kube-api-access-zsglm") pod "52fb505b-6089-4557-9682-e39f2f176607" (UID: "52fb505b-6089-4557-9682-e39f2f176607"). InnerVolumeSpecName "kube-api-access-zsglm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:24:12.841809 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:12.841719 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15fcf374-0dbe-4c9c-af62-fb2b7bc20919-kube-api-access-k2gqz" (OuterVolumeSpecName: "kube-api-access-k2gqz") pod "15fcf374-0dbe-4c9c-af62-fb2b7bc20919" (UID: "15fcf374-0dbe-4c9c-af62-fb2b7bc20919"). InnerVolumeSpecName "kube-api-access-k2gqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:24:12.941038 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:12.940956 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15fcf374-0dbe-4c9c-af62-fb2b7bc20919-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:24:12.941038 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:12.940985 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52fb505b-6089-4557-9682-e39f2f176607-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:24:12.941038 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:12.940999 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsglm\" (UniqueName: \"kubernetes.io/projected/52fb505b-6089-4557-9682-e39f2f176607-kube-api-access-zsglm\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:24:12.941038 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:12.941011 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2gqz\" (UniqueName: \"kubernetes.io/projected/15fcf374-0dbe-4c9c-af62-fb2b7bc20919-kube-api-access-k2gqz\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:24:12.946300 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:12.946266 2574 generic.go:356] "Generic (PLEG): container finished" podID="52fb505b-6089-4557-9682-e39f2f176607" containerID="29571c934a693dc4549b8be3f7f0a9c43ab473fa06a6a4704e739180c5a0f5ef" exitCode=0 Sep 04 22:24:12.946426 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:12.946331 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb137412-75b6ff7f78-46gjh" Sep 04 22:24:12.946426 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:12.946355 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb137412-75b6ff7f78-46gjh" event={"ID":"52fb505b-6089-4557-9682-e39f2f176607","Type":"ContainerDied","Data":"29571c934a693dc4549b8be3f7f0a9c43ab473fa06a6a4704e739180c5a0f5ef"} Sep 04 22:24:12.946568 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:12.946449 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bb137412-75b6ff7f78-46gjh" event={"ID":"52fb505b-6089-4557-9682-e39f2f176607","Type":"ContainerDied","Data":"6d337a48ab1178b81f17e0a960214442e187474942c013e9dc31dc090143060c"} Sep 04 22:24:12.946568 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:12.946482 2574 scope.go:117] "RemoveContainer" containerID="29571c934a693dc4549b8be3f7f0a9c43ab473fa06a6a4704e739180c5a0f5ef" Sep 04 22:24:12.947805 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:12.947787 2574 generic.go:356] "Generic (PLEG): container finished" podID="15fcf374-0dbe-4c9c-af62-fb2b7bc20919" containerID="989e4bdf413858e32dfa5eb5599bd92a6b5e3957b3a1ca9d3138b5d4141daf44" exitCode=0 Sep 04 22:24:12.947900 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:12.947846 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-bb137412-57f86c67d-pn2fj" Sep 04 22:24:12.947900 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:12.947865 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-bb137412-57f86c67d-pn2fj" event={"ID":"15fcf374-0dbe-4c9c-af62-fb2b7bc20919","Type":"ContainerDied","Data":"989e4bdf413858e32dfa5eb5599bd92a6b5e3957b3a1ca9d3138b5d4141daf44"} Sep 04 22:24:12.948023 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:12.947908 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-bb137412-57f86c67d-pn2fj" event={"ID":"15fcf374-0dbe-4c9c-af62-fb2b7bc20919","Type":"ContainerDied","Data":"99a4d983c250f1adf9d577dbda9f00b499de99abcf8b41e70a9dd8d0c3c290ca"} Sep 04 22:24:12.955490 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:12.955472 2574 scope.go:117] "RemoveContainer" containerID="29571c934a693dc4549b8be3f7f0a9c43ab473fa06a6a4704e739180c5a0f5ef" Sep 04 22:24:12.955806 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:24:12.955780 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29571c934a693dc4549b8be3f7f0a9c43ab473fa06a6a4704e739180c5a0f5ef\": container with ID starting with 29571c934a693dc4549b8be3f7f0a9c43ab473fa06a6a4704e739180c5a0f5ef not found: ID does not exist" containerID="29571c934a693dc4549b8be3f7f0a9c43ab473fa06a6a4704e739180c5a0f5ef" Sep 04 22:24:12.955896 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:12.955816 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29571c934a693dc4549b8be3f7f0a9c43ab473fa06a6a4704e739180c5a0f5ef"} err="failed to get container status \"29571c934a693dc4549b8be3f7f0a9c43ab473fa06a6a4704e739180c5a0f5ef\": rpc error: code = NotFound desc = could not find container \"29571c934a693dc4549b8be3f7f0a9c43ab473fa06a6a4704e739180c5a0f5ef\": container with ID starting with 29571c934a693dc4549b8be3f7f0a9c43ab473fa06a6a4704e739180c5a0f5ef not found: ID does not exist" Sep 04 22:24:12.955896 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:12.955837 2574 scope.go:117] "RemoveContainer" containerID="989e4bdf413858e32dfa5eb5599bd92a6b5e3957b3a1ca9d3138b5d4141daf44" Sep 04 22:24:12.964682 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:12.964624 2574 scope.go:117] "RemoveContainer" containerID="989e4bdf413858e32dfa5eb5599bd92a6b5e3957b3a1ca9d3138b5d4141daf44" Sep 04 22:24:12.964883 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:24:12.964867 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"989e4bdf413858e32dfa5eb5599bd92a6b5e3957b3a1ca9d3138b5d4141daf44\": container with ID starting with 989e4bdf413858e32dfa5eb5599bd92a6b5e3957b3a1ca9d3138b5d4141daf44 not found: ID does not exist" containerID="989e4bdf413858e32dfa5eb5599bd92a6b5e3957b3a1ca9d3138b5d4141daf44" Sep 04 22:24:12.964931 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:12.964889 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"989e4bdf413858e32dfa5eb5599bd92a6b5e3957b3a1ca9d3138b5d4141daf44"} err="failed to get container status \"989e4bdf413858e32dfa5eb5599bd92a6b5e3957b3a1ca9d3138b5d4141daf44\": rpc error: code = NotFound desc = could not find container \"989e4bdf413858e32dfa5eb5599bd92a6b5e3957b3a1ca9d3138b5d4141daf44\": container with ID starting with 989e4bdf413858e32dfa5eb5599bd92a6b5e3957b3a1ca9d3138b5d4141daf44 not found: ID does not exist" Sep 04 22:24:12.973796 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:12.973775 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-bb137412-57f86c67d-pn2fj"] Sep 04 22:24:12.977946 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:12.977926 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-bb137412-57f86c67d-pn2fj"] Sep 04 22:24:12.989831 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:12.989808 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-bb137412-75b6ff7f78-46gjh"] Sep 04 22:24:12.994459 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:12.994438 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-bb137412-75b6ff7f78-46gjh"] Sep 04 22:24:14.715454 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:14.715419 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15fcf374-0dbe-4c9c-af62-fb2b7bc20919" path="/var/lib/kubelet/pods/15fcf374-0dbe-4c9c-af62-fb2b7bc20919/volumes" Sep 04 22:24:14.715815 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:14.715780 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52fb505b-6089-4557-9682-e39f2f176607" path="/var/lib/kubelet/pods/52fb505b-6089-4557-9682-e39f2f176607/volumes" Sep 04 22:24:18.296185 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:18.296155 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-20c0448a-5b6dccf46-pf9bf"] Sep 04 22:24:18.296658 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:18.296634 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15fcf374-0dbe-4c9c-af62-fb2b7bc20919" containerName="payload-processing" Sep 04 22:24:18.296658 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:18.296655 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="15fcf374-0dbe-4c9c-af62-fb2b7bc20919" containerName="payload-processing" Sep 04 22:24:18.296658 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:18.296669 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba16a970-61a4-4b03-b53e-531499f4c969" containerName="revoke-keys" Sep 04 22:24:18.296658 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:18.296677 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba16a970-61a4-4b03-b53e-531499f4c969" containerName="revoke-keys" Sep 04 22:24:18.296964 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:18.296707 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52fb505b-6089-4557-9682-e39f2f176607" containerName="payload-pre-processing" Sep 04 22:24:18.296964 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:18.296716 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="52fb505b-6089-4557-9682-e39f2f176607" containerName="payload-pre-processing" Sep 04 22:24:18.296964 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:18.296795 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba16a970-61a4-4b03-b53e-531499f4c969" containerName="revoke-keys" Sep 04 22:24:18.296964 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:18.296811 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="52fb505b-6089-4557-9682-e39f2f176607" containerName="payload-pre-processing" Sep 04 22:24:18.296964 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:18.296820 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="15fcf374-0dbe-4c9c-af62-fb2b7bc20919" containerName="payload-processing" Sep 04 22:24:18.301263 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:18.301243 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-20c0448a-5b6dccf46-pf9bf" Sep 04 22:24:18.305120 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:18.305102 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-20c0448a\"" Sep 04 22:24:18.306917 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:18.306899 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-20c0448a-dockercfg-qt692\"" Sep 04 22:24:18.313814 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:18.313790 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-20c0448a-5b6dccf46-pf9bf"] Sep 04 22:24:18.388080 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:18.388053 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a193180f-f681-4187-919b-d1ba47f07ce2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-20c0448a-5b6dccf46-pf9bf\" (UID: \"a193180f-f681-4187-919b-d1ba47f07ce2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-20c0448a-5b6dccf46-pf9bf" Sep 04 22:24:18.388185 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:18.388111 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpddk\" (UniqueName: \"kubernetes.io/projected/a193180f-f681-4187-919b-d1ba47f07ce2-kube-api-access-tpddk\") pod \"payload-pre-processing-e2e-ait-20c0448a-5b6dccf46-pf9bf\" (UID: \"a193180f-f681-4187-919b-d1ba47f07ce2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-20c0448a-5b6dccf46-pf9bf" Sep 04 22:24:18.489176 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:18.489144 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpddk\" (UniqueName: \"kubernetes.io/projected/a193180f-f681-4187-919b-d1ba47f07ce2-kube-api-access-tpddk\") pod \"payload-pre-processing-e2e-ait-20c0448a-5b6dccf46-pf9bf\" (UID: \"a193180f-f681-4187-919b-d1ba47f07ce2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-20c0448a-5b6dccf46-pf9bf" Sep 04 22:24:18.489332 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:18.489226 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a193180f-f681-4187-919b-d1ba47f07ce2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-20c0448a-5b6dccf46-pf9bf\" (UID: \"a193180f-f681-4187-919b-d1ba47f07ce2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-20c0448a-5b6dccf46-pf9bf" Sep 04 22:24:18.489919 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:18.489900 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a193180f-f681-4187-919b-d1ba47f07ce2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-20c0448a-5b6dccf46-pf9bf\" (UID: \"a193180f-f681-4187-919b-d1ba47f07ce2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-20c0448a-5b6dccf46-pf9bf" Sep 04 22:24:18.497416 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:18.497395 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpddk\" (UniqueName: \"kubernetes.io/projected/a193180f-f681-4187-919b-d1ba47f07ce2-kube-api-access-tpddk\") pod \"payload-pre-processing-e2e-ait-20c0448a-5b6dccf46-pf9bf\" (UID: \"a193180f-f681-4187-919b-d1ba47f07ce2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-20c0448a-5b6dccf46-pf9bf" Sep 04 22:24:18.612174 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:18.612096 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-20c0448a-5b6dccf46-pf9bf" Sep 04 22:24:18.880505 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:18.880472 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-20c0448a-5b6dccf46-pf9bf"] Sep 04 22:24:18.882261 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:24:18.882232 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda193180f_f681_4187_919b_d1ba47f07ce2.slice/crio-bb3d6a496e43806e0be77ca93737aa7750aa7d3c38418530a98a9926fb5780ed WatchSource:0}: Error finding container bb3d6a496e43806e0be77ca93737aa7750aa7d3c38418530a98a9926fb5780ed: Status 404 returned error can't find the container with id bb3d6a496e43806e0be77ca93737aa7750aa7d3c38418530a98a9926fb5780ed Sep 04 22:24:18.980375 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:18.980344 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-20c0448a-5b6dccf46-pf9bf" event={"ID":"a193180f-f681-4187-919b-d1ba47f07ce2","Type":"ContainerStarted","Data":"d9b466060c64c84285773068133df928e72790e7196fab82109d0d939d33305f"} Sep 04 22:24:18.980532 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:18.980397 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-20c0448a-5b6dccf46-pf9bf" event={"ID":"a193180f-f681-4187-919b-d1ba47f07ce2","Type":"ContainerStarted","Data":"bb3d6a496e43806e0be77ca93737aa7750aa7d3c38418530a98a9926fb5780ed"} Sep 04 22:24:18.980532 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:18.980485 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-20c0448a-5b6dccf46-pf9bf" Sep 04 22:24:18.997289 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:18.997246 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-20c0448a-5b6dccf46-pf9bf" podStartSLOduration=0.997232377 podStartE2EDuration="997.232377ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:24:18.996262503 +0000 UTC m=+962.760738128" watchObservedRunningTime="2026-09-04 22:24:18.997232377 +0000 UTC m=+962.761707987" Sep 04 22:24:29.987538 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:29.987505 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-20c0448a-5b6dccf46-pf9bf" Sep 04 22:24:42.087612 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:42.087518 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20c0448a-q5dnl"] Sep 04 22:24:42.091203 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:42.091188 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20c0448a-q5dnl" Sep 04 22:24:42.093594 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:42.093570 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 04 22:24:42.093711 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:42.093610 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hc7x4\"" Sep 04 22:24:42.099710 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:42.099686 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20c0448a-q5dnl"] Sep 04 22:24:42.166795 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:42.166772 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b0405b1-2663-409c-8c78-cd2f5c975a59-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-20c0448a-q5dnl\" (UID: \"1b0405b1-2663-409c-8c78-cd2f5c975a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20c0448a-q5dnl" Sep 04 22:24:42.267479 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:42.267448 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b0405b1-2663-409c-8c78-cd2f5c975a59-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-20c0448a-q5dnl\" (UID: \"1b0405b1-2663-409c-8c78-cd2f5c975a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20c0448a-q5dnl" Sep 04 22:24:42.268027 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:42.268010 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b0405b1-2663-409c-8c78-cd2f5c975a59-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-20c0448a-q5dnl\" (UID: \"1b0405b1-2663-409c-8c78-cd2f5c975a59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20c0448a-q5dnl" Sep 04 22:24:42.401802 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:42.401743 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20c0448a-q5dnl" Sep 04 22:24:42.669748 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:42.669721 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20c0448a-q5dnl"] Sep 04 22:24:42.671119 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:24:42.671094 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b0405b1_2663_409c_8c78_cd2f5c975a59.slice/crio-af8aa347dd0a0177e033fe6dd05b4c356dc07dbead36ed4eec5070f07aaefe4f WatchSource:0}: Error finding container af8aa347dd0a0177e033fe6dd05b4c356dc07dbead36ed4eec5070f07aaefe4f: Status 404 returned error can't find the container with id af8aa347dd0a0177e033fe6dd05b4c356dc07dbead36ed4eec5070f07aaefe4f Sep 04 22:24:43.102731 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:43.102695 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20c0448a-q5dnl" event={"ID":"1b0405b1-2663-409c-8c78-cd2f5c975a59","Type":"ContainerStarted","Data":"b282440f064ebdfbb8ac4a9a12094869ebf73d5bd64fd53958f5526234f2dd30"} Sep 04 22:24:43.103118 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:43.102749 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20c0448a-q5dnl" event={"ID":"1b0405b1-2663-409c-8c78-cd2f5c975a59","Type":"ContainerStarted","Data":"af8aa347dd0a0177e033fe6dd05b4c356dc07dbead36ed4eec5070f07aaefe4f"} Sep 04 22:24:43.120308 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:43.120260 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20c0448a-q5dnl" podStartSLOduration=1.120245187 podStartE2EDuration="1.120245187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:24:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:24:43.119043183 +0000 UTC m=+986.883518787" watchObservedRunningTime="2026-09-04 22:24:43.120245187 +0000 UTC m=+986.884720798" Sep 04 22:24:44.110193 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:44.110160 2574 generic.go:356] "Generic (PLEG): container finished" podID="1b0405b1-2663-409c-8c78-cd2f5c975a59" containerID="b282440f064ebdfbb8ac4a9a12094869ebf73d5bd64fd53958f5526234f2dd30" exitCode=0 Sep 04 22:24:44.110584 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:44.110247 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20c0448a-q5dnl" event={"ID":"1b0405b1-2663-409c-8c78-cd2f5c975a59","Type":"ContainerDied","Data":"b282440f064ebdfbb8ac4a9a12094869ebf73d5bd64fd53958f5526234f2dd30"} Sep 04 22:24:45.245437 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:45.245413 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20c0448a-q5dnl" Sep 04 22:24:45.288326 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:45.288300 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b0405b1-2663-409c-8c78-cd2f5c975a59-maas-api-service-ca\") pod \"1b0405b1-2663-409c-8c78-cd2f5c975a59\" (UID: \"1b0405b1-2663-409c-8c78-cd2f5c975a59\") " Sep 04 22:24:45.288637 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:45.288617 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b0405b1-2663-409c-8c78-cd2f5c975a59-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1b0405b1-2663-409c-8c78-cd2f5c975a59" (UID: "1b0405b1-2663-409c-8c78-cd2f5c975a59"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:24:45.388831 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:45.388774 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b0405b1-2663-409c-8c78-cd2f5c975a59-maas-api-service-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:24:46.121284 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:46.121258 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20c0448a-q5dnl" Sep 04 22:24:46.121482 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:46.121254 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20c0448a-q5dnl" event={"ID":"1b0405b1-2663-409c-8c78-cd2f5c975a59","Type":"ContainerDied","Data":"af8aa347dd0a0177e033fe6dd05b4c356dc07dbead36ed4eec5070f07aaefe4f"} Sep 04 22:24:46.121482 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:46.121380 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af8aa347dd0a0177e033fe6dd05b4c356dc07dbead36ed4eec5070f07aaefe4f" Sep 04 22:24:48.198648 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:48.198617 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-20c0448a-5b6dccf46-pf9bf"] Sep 04 22:24:48.199024 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:48.198827 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-20c0448a-5b6dccf46-pf9bf" podUID="a193180f-f681-4187-919b-d1ba47f07ce2" containerName="payload-pre-processing" containerID="cri-o://d9b466060c64c84285773068133df928e72790e7196fab82109d0d939d33305f" gracePeriod=30 Sep 04 22:24:48.379501 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:48.379469 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20c0448a-q5dnl"] Sep 04 22:24:48.382664 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:48.382636 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20c0448a-q5dnl"] Sep 04 22:24:48.435943 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:48.435921 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-20c0448a-5b6dccf46-pf9bf" Sep 04 22:24:48.511635 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:48.511601 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpddk\" (UniqueName: \"kubernetes.io/projected/a193180f-f681-4187-919b-d1ba47f07ce2-kube-api-access-tpddk\") pod \"a193180f-f681-4187-919b-d1ba47f07ce2\" (UID: \"a193180f-f681-4187-919b-d1ba47f07ce2\") " Sep 04 22:24:48.511846 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:48.511674 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a193180f-f681-4187-919b-d1ba47f07ce2-plugins-config-volume\") pod \"a193180f-f681-4187-919b-d1ba47f07ce2\" (UID: \"a193180f-f681-4187-919b-d1ba47f07ce2\") " Sep 04 22:24:48.512403 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:48.512365 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a193180f-f681-4187-919b-d1ba47f07ce2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a193180f-f681-4187-919b-d1ba47f07ce2" (UID: "a193180f-f681-4187-919b-d1ba47f07ce2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:24:48.514806 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:48.514777 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a193180f-f681-4187-919b-d1ba47f07ce2-kube-api-access-tpddk" (OuterVolumeSpecName: "kube-api-access-tpddk") pod "a193180f-f681-4187-919b-d1ba47f07ce2" (UID: "a193180f-f681-4187-919b-d1ba47f07ce2"). InnerVolumeSpecName "kube-api-access-tpddk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:24:48.613216 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:48.613189 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpddk\" (UniqueName: \"kubernetes.io/projected/a193180f-f681-4187-919b-d1ba47f07ce2-kube-api-access-tpddk\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:24:48.613216 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:48.613215 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a193180f-f681-4187-919b-d1ba47f07ce2-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:24:48.714211 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:48.714182 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b0405b1-2663-409c-8c78-cd2f5c975a59" path="/var/lib/kubelet/pods/1b0405b1-2663-409c-8c78-cd2f5c975a59/volumes" Sep 04 22:24:49.139231 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:49.139202 2574 generic.go:356] "Generic (PLEG): container finished" podID="a193180f-f681-4187-919b-d1ba47f07ce2" containerID="d9b466060c64c84285773068133df928e72790e7196fab82109d0d939d33305f" exitCode=0 Sep 04 22:24:49.139406 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:49.139245 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-20c0448a-5b6dccf46-pf9bf" Sep 04 22:24:49.139406 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:49.139235 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-20c0448a-5b6dccf46-pf9bf" event={"ID":"a193180f-f681-4187-919b-d1ba47f07ce2","Type":"ContainerDied","Data":"d9b466060c64c84285773068133df928e72790e7196fab82109d0d939d33305f"} Sep 04 22:24:49.139406 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:49.139380 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-20c0448a-5b6dccf46-pf9bf" event={"ID":"a193180f-f681-4187-919b-d1ba47f07ce2","Type":"ContainerDied","Data":"bb3d6a496e43806e0be77ca93737aa7750aa7d3c38418530a98a9926fb5780ed"} Sep 04 22:24:49.139511 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:49.139417 2574 scope.go:117] "RemoveContainer" containerID="d9b466060c64c84285773068133df928e72790e7196fab82109d0d939d33305f" Sep 04 22:24:49.148173 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:49.148143 2574 scope.go:117] "RemoveContainer" containerID="d9b466060c64c84285773068133df928e72790e7196fab82109d0d939d33305f" Sep 04 22:24:49.148472 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:24:49.148453 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9b466060c64c84285773068133df928e72790e7196fab82109d0d939d33305f\": container with ID starting with d9b466060c64c84285773068133df928e72790e7196fab82109d0d939d33305f not found: ID does not exist" containerID="d9b466060c64c84285773068133df928e72790e7196fab82109d0d939d33305f" Sep 04 22:24:49.148565 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:49.148480 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9b466060c64c84285773068133df928e72790e7196fab82109d0d939d33305f"} err="failed to get container status \"d9b466060c64c84285773068133df928e72790e7196fab82109d0d939d33305f\": rpc error: code = NotFound desc = could not find container \"d9b466060c64c84285773068133df928e72790e7196fab82109d0d939d33305f\": container with ID starting with d9b466060c64c84285773068133df928e72790e7196fab82109d0d939d33305f not found: ID does not exist" Sep 04 22:24:49.161517 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:49.161482 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-20c0448a-5b6dccf46-pf9bf"] Sep 04 22:24:49.166684 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:49.166658 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-20c0448a-5b6dccf46-pf9bf"] Sep 04 22:24:50.715137 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:50.715101 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a193180f-f681-4187-919b-d1ba47f07ce2" path="/var/lib/kubelet/pods/a193180f-f681-4187-919b-d1ba47f07ce2/volumes" Sep 04 22:24:54.739794 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:54.739762 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-25fea002-5d85fc9ffc-lsdp8"] Sep 04 22:24:54.740145 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:54.740087 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a193180f-f681-4187-919b-d1ba47f07ce2" containerName="payload-pre-processing" Sep 04 22:24:54.740145 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:54.740098 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a193180f-f681-4187-919b-d1ba47f07ce2" containerName="payload-pre-processing" Sep 04 22:24:54.740145 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:54.740118 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b0405b1-2663-409c-8c78-cd2f5c975a59" containerName="revoke-keys" Sep 04 22:24:54.740145 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:54.740125 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b0405b1-2663-409c-8c78-cd2f5c975a59" containerName="revoke-keys" Sep 04 22:24:54.740272 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:54.740187 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b0405b1-2663-409c-8c78-cd2f5c975a59" containerName="revoke-keys" Sep 04 22:24:54.740272 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:54.740198 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a193180f-f681-4187-919b-d1ba47f07ce2" containerName="payload-pre-processing" Sep 04 22:24:54.744509 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:54.744490 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-25fea002-5d85fc9ffc-lsdp8" Sep 04 22:24:54.746808 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:54.746786 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-25fea002-dockercfg-rfc8x\"" Sep 04 22:24:54.746932 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:54.746787 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-25fea002\"" Sep 04 22:24:54.757580 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:54.757560 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-25fea002-5d85fc9ffc-lsdp8"] Sep 04 22:24:54.763433 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:54.763413 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4bbcc72-218f-42fd-b422-c2afa187c815-plugins-config-volume\") pod \"payload-processing-e2e-derive-25fea002-5d85fc9ffc-lsdp8\" (UID: \"c4bbcc72-218f-42fd-b422-c2afa187c815\") " pod="openshift-ingress/payload-processing-e2e-derive-25fea002-5d85fc9ffc-lsdp8" Sep 04 22:24:54.763560 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:54.763463 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99l6l\" (UniqueName: \"kubernetes.io/projected/c4bbcc72-218f-42fd-b422-c2afa187c815-kube-api-access-99l6l\") pod \"payload-processing-e2e-derive-25fea002-5d85fc9ffc-lsdp8\" (UID: \"c4bbcc72-218f-42fd-b422-c2afa187c815\") " pod="openshift-ingress/payload-processing-e2e-derive-25fea002-5d85fc9ffc-lsdp8" Sep 04 22:24:54.864567 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:54.864529 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4bbcc72-218f-42fd-b422-c2afa187c815-plugins-config-volume\") pod \"payload-processing-e2e-derive-25fea002-5d85fc9ffc-lsdp8\" (UID: \"c4bbcc72-218f-42fd-b422-c2afa187c815\") " pod="openshift-ingress/payload-processing-e2e-derive-25fea002-5d85fc9ffc-lsdp8" Sep 04 22:24:54.864719 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:54.864615 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99l6l\" (UniqueName: \"kubernetes.io/projected/c4bbcc72-218f-42fd-b422-c2afa187c815-kube-api-access-99l6l\") pod \"payload-processing-e2e-derive-25fea002-5d85fc9ffc-lsdp8\" (UID: \"c4bbcc72-218f-42fd-b422-c2afa187c815\") " pod="openshift-ingress/payload-processing-e2e-derive-25fea002-5d85fc9ffc-lsdp8" Sep 04 22:24:54.865230 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:54.865195 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4bbcc72-218f-42fd-b422-c2afa187c815-plugins-config-volume\") pod \"payload-processing-e2e-derive-25fea002-5d85fc9ffc-lsdp8\" (UID: \"c4bbcc72-218f-42fd-b422-c2afa187c815\") " pod="openshift-ingress/payload-processing-e2e-derive-25fea002-5d85fc9ffc-lsdp8" Sep 04 22:24:54.875448 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:54.875425 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99l6l\" (UniqueName: \"kubernetes.io/projected/c4bbcc72-218f-42fd-b422-c2afa187c815-kube-api-access-99l6l\") pod \"payload-processing-e2e-derive-25fea002-5d85fc9ffc-lsdp8\" (UID: \"c4bbcc72-218f-42fd-b422-c2afa187c815\") " pod="openshift-ingress/payload-processing-e2e-derive-25fea002-5d85fc9ffc-lsdp8" Sep 04 22:24:55.055831 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:55.055801 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-25fea002-5d85fc9ffc-lsdp8" Sep 04 22:24:55.334818 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:55.334785 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-25fea002-5d85fc9ffc-lsdp8"] Sep 04 22:24:56.177930 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:56.177887 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-25fea002-5d85fc9ffc-lsdp8" event={"ID":"c4bbcc72-218f-42fd-b422-c2afa187c815","Type":"ContainerStarted","Data":"1897032d0bb92b6713e23cc3cc5d62136cf5b3b70a533410a0646a3c208116e3"} Sep 04 22:24:56.178338 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:56.178053 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-25fea002-5d85fc9ffc-lsdp8" Sep 04 22:24:56.178338 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:56.178106 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-25fea002-5d85fc9ffc-lsdp8" event={"ID":"c4bbcc72-218f-42fd-b422-c2afa187c815","Type":"ContainerStarted","Data":"550ed7c019be7b11aed31d58c39149b91b7fdbfb066385227f591d2a571e68bb"} Sep 04 22:24:56.199385 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:24:56.199342 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-25fea002-5d85fc9ffc-lsdp8" podStartSLOduration=2.199328682 podStartE2EDuration="2.199328682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:24:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:24:56.196711998 +0000 UTC m=+999.961187610" watchObservedRunningTime="2026-09-04 22:24:56.199328682 +0000 UTC m=+999.963804351" Sep 04 22:25:07.184702 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:07.184674 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-25fea002-5d85fc9ffc-lsdp8" Sep 04 22:25:07.398009 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:07.397972 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a8d845-b8ktn"] Sep 04 22:25:07.401648 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:07.401631 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a8d845-b8ktn" Sep 04 22:25:07.404130 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:07.404108 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 04 22:25:07.404233 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:07.404159 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hc7x4\"" Sep 04 22:25:07.411902 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:07.411877 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a8d845-b8ktn"] Sep 04 22:25:07.466418 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:07.466347 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd85365d-678e-4eb2-be9d-a1bb9378e219-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-a8d845-b8ktn\" (UID: \"cd85365d-678e-4eb2-be9d-a1bb9378e219\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a8d845-b8ktn" Sep 04 22:25:07.566952 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:07.566918 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd85365d-678e-4eb2-be9d-a1bb9378e219-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-a8d845-b8ktn\" (UID: \"cd85365d-678e-4eb2-be9d-a1bb9378e219\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a8d845-b8ktn" Sep 04 22:25:07.567519 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:07.567499 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd85365d-678e-4eb2-be9d-a1bb9378e219-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-a8d845-b8ktn\" (UID: \"cd85365d-678e-4eb2-be9d-a1bb9378e219\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a8d845-b8ktn" Sep 04 22:25:07.712652 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:07.712616 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a8d845-b8ktn" Sep 04 22:25:07.982263 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:07.982239 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a8d845-b8ktn"] Sep 04 22:25:07.983858 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:25:07.983830 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd85365d_678e_4eb2_be9d_a1bb9378e219.slice/crio-f098a012649dc597bf529256e09507db77aaaa4496a700543e5ffa09bcf9d383 WatchSource:0}: Error finding container f098a012649dc597bf529256e09507db77aaaa4496a700543e5ffa09bcf9d383: Status 404 returned error can't find the container with id f098a012649dc597bf529256e09507db77aaaa4496a700543e5ffa09bcf9d383 Sep 04 22:25:08.243211 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:08.243182 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a8d845-b8ktn" event={"ID":"cd85365d-678e-4eb2-be9d-a1bb9378e219","Type":"ContainerStarted","Data":"f098a012649dc597bf529256e09507db77aaaa4496a700543e5ffa09bcf9d383"} Sep 04 22:25:09.249156 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:09.249122 2574 generic.go:356] "Generic (PLEG): container finished" podID="cd85365d-678e-4eb2-be9d-a1bb9378e219" containerID="d6dc6560178fb697fab2ffac91e1c3e6ca9a5b7a91b84dc66047b6c01e6904c1" exitCode=0 Sep 04 22:25:09.249515 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:09.249210 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a8d845-b8ktn" event={"ID":"cd85365d-678e-4eb2-be9d-a1bb9378e219","Type":"ContainerDied","Data":"d6dc6560178fb697fab2ffac91e1c3e6ca9a5b7a91b84dc66047b6c01e6904c1"} Sep 04 22:25:10.378866 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:10.378840 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a8d845-b8ktn" Sep 04 22:25:10.491987 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:10.491960 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd85365d-678e-4eb2-be9d-a1bb9378e219-maas-api-service-ca\") pod \"cd85365d-678e-4eb2-be9d-a1bb9378e219\" (UID: \"cd85365d-678e-4eb2-be9d-a1bb9378e219\") " Sep 04 22:25:10.492303 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:10.492280 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd85365d-678e-4eb2-be9d-a1bb9378e219-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cd85365d-678e-4eb2-be9d-a1bb9378e219" (UID: "cd85365d-678e-4eb2-be9d-a1bb9378e219"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:25:10.593231 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:10.593167 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd85365d-678e-4eb2-be9d-a1bb9378e219-maas-api-service-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:25:11.261031 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:11.260999 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a8d845-b8ktn" event={"ID":"cd85365d-678e-4eb2-be9d-a1bb9378e219","Type":"ContainerDied","Data":"f098a012649dc597bf529256e09507db77aaaa4496a700543e5ffa09bcf9d383"} Sep 04 22:25:11.261031 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:11.261033 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a8d845-b8ktn" Sep 04 22:25:11.261251 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:11.261055 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f098a012649dc597bf529256e09507db77aaaa4496a700543e5ffa09bcf9d383" Sep 04 22:25:17.510367 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:17.510328 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-25fea002-k8glb"] Sep 04 22:25:17.510883 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:17.510863 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd85365d-678e-4eb2-be9d-a1bb9378e219" containerName="revoke-keys" Sep 04 22:25:17.510929 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:17.510887 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd85365d-678e-4eb2-be9d-a1bb9378e219" containerName="revoke-keys" Sep 04 22:25:17.511007 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:17.510996 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd85365d-678e-4eb2-be9d-a1bb9378e219" containerName="revoke-keys" Sep 04 22:25:17.515156 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:17.515139 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-25fea002-k8glb" Sep 04 22:25:17.517511 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:17.517487 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 04 22:25:17.517620 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:17.517527 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hc7x4\"" Sep 04 22:25:17.522217 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:17.522190 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-25fea002-k8glb"] Sep 04 22:25:17.652694 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:17.652664 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4f0c7d1-7714-49b3-8aa5-342c1a1fb25f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-25fea002-k8glb\" (UID: \"b4f0c7d1-7714-49b3-8aa5-342c1a1fb25f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-25fea002-k8glb" Sep 04 22:25:17.753787 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:17.753753 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4f0c7d1-7714-49b3-8aa5-342c1a1fb25f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-25fea002-k8glb\" (UID: \"b4f0c7d1-7714-49b3-8aa5-342c1a1fb25f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-25fea002-k8glb" Sep 04 22:25:17.754334 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:17.754314 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4f0c7d1-7714-49b3-8aa5-342c1a1fb25f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-25fea002-k8glb\" (UID: \"b4f0c7d1-7714-49b3-8aa5-342c1a1fb25f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-25fea002-k8glb" Sep 04 22:25:17.825595 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:17.825530 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-25fea002-k8glb" Sep 04 22:25:18.094666 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:18.094637 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-25fea002-k8glb"] Sep 04 22:25:18.095918 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:25:18.095891 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4f0c7d1_7714_49b3_8aa5_342c1a1fb25f.slice/crio-5adc530d356418630960e28417d901f789e8b8d4e87445a8f81318ea0851c227 WatchSource:0}: Error finding container 5adc530d356418630960e28417d901f789e8b8d4e87445a8f81318ea0851c227: Status 404 returned error can't find the container with id 5adc530d356418630960e28417d901f789e8b8d4e87445a8f81318ea0851c227 Sep 04 22:25:18.302708 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:18.302676 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-25fea002-k8glb" event={"ID":"b4f0c7d1-7714-49b3-8aa5-342c1a1fb25f","Type":"ContainerStarted","Data":"5adc530d356418630960e28417d901f789e8b8d4e87445a8f81318ea0851c227"} Sep 04 22:25:19.310362 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:19.310326 2574 generic.go:356] "Generic (PLEG): container finished" podID="b4f0c7d1-7714-49b3-8aa5-342c1a1fb25f" containerID="6ad373034975bf051aa09057cca01427c436cbc9722b65073417542189732bb4" exitCode=0 Sep 04 22:25:19.310768 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:19.310409 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-25fea002-k8glb" event={"ID":"b4f0c7d1-7714-49b3-8aa5-342c1a1fb25f","Type":"ContainerDied","Data":"6ad373034975bf051aa09057cca01427c436cbc9722b65073417542189732bb4"} Sep 04 22:25:19.359051 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:19.359020 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a8d845-776d46797c-pv87c"] Sep 04 22:25:19.359302 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:19.359273 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-a8d845-776d46797c-pv87c" podUID="ad508a5b-b62f-4d57-b7b9-c490dbefa055" containerName="payload-processing" containerID="cri-o://8548f7466b9ea6773aadad5704764439e1328702eacc6cd799ccd31b9035ad3f" gracePeriod=30 Sep 04 22:25:19.366710 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:19.366683 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a8d845-6469bc984b-p2zd9"] Sep 04 22:25:19.366938 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:19.366892 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-a8d845-6469bc984b-p2zd9" podUID="6c01200a-3f15-4223-aaa3-7f7ccbd1427c" containerName="payload-pre-processing" containerID="cri-o://9efa8a6f2576ba06a6c1f5b3c07e46b228185695465197f96becee6d6277c353" gracePeriod=30 Sep 04 22:25:19.555822 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:19.555792 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a8d845-b8ktn"] Sep 04 22:25:19.560917 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:19.560844 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a8d845-b8ktn"] Sep 04 22:25:19.622034 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:19.622010 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a8d845-776d46797c-pv87c" Sep 04 22:25:19.632181 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:19.632163 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-a8d845-6469bc984b-p2zd9" Sep 04 22:25:19.773382 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:19.773353 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad508a5b-b62f-4d57-b7b9-c490dbefa055-plugins-config-volume\") pod \"ad508a5b-b62f-4d57-b7b9-c490dbefa055\" (UID: \"ad508a5b-b62f-4d57-b7b9-c490dbefa055\") " Sep 04 22:25:19.773382 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:19.773390 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c01200a-3f15-4223-aaa3-7f7ccbd1427c-plugins-config-volume\") pod \"6c01200a-3f15-4223-aaa3-7f7ccbd1427c\" (UID: \"6c01200a-3f15-4223-aaa3-7f7ccbd1427c\") " Sep 04 22:25:19.773642 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:19.773413 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqssw\" (UniqueName: \"kubernetes.io/projected/ad508a5b-b62f-4d57-b7b9-c490dbefa055-kube-api-access-fqssw\") pod \"ad508a5b-b62f-4d57-b7b9-c490dbefa055\" (UID: \"ad508a5b-b62f-4d57-b7b9-c490dbefa055\") " Sep 04 22:25:19.773642 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:19.773539 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hgfr\" (UniqueName: \"kubernetes.io/projected/6c01200a-3f15-4223-aaa3-7f7ccbd1427c-kube-api-access-5hgfr\") pod \"6c01200a-3f15-4223-aaa3-7f7ccbd1427c\" (UID: \"6c01200a-3f15-4223-aaa3-7f7ccbd1427c\") " Sep 04 22:25:19.773832 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:19.773809 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c01200a-3f15-4223-aaa3-7f7ccbd1427c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6c01200a-3f15-4223-aaa3-7f7ccbd1427c" (UID: "6c01200a-3f15-4223-aaa3-7f7ccbd1427c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:25:19.773894 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:19.773811 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad508a5b-b62f-4d57-b7b9-c490dbefa055-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ad508a5b-b62f-4d57-b7b9-c490dbefa055" (UID: "ad508a5b-b62f-4d57-b7b9-c490dbefa055"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:25:19.775435 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:19.775414 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c01200a-3f15-4223-aaa3-7f7ccbd1427c-kube-api-access-5hgfr" (OuterVolumeSpecName: "kube-api-access-5hgfr") pod "6c01200a-3f15-4223-aaa3-7f7ccbd1427c" (UID: "6c01200a-3f15-4223-aaa3-7f7ccbd1427c"). InnerVolumeSpecName "kube-api-access-5hgfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:25:19.775480 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:19.775455 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad508a5b-b62f-4d57-b7b9-c490dbefa055-kube-api-access-fqssw" (OuterVolumeSpecName: "kube-api-access-fqssw") pod "ad508a5b-b62f-4d57-b7b9-c490dbefa055" (UID: "ad508a5b-b62f-4d57-b7b9-c490dbefa055"). InnerVolumeSpecName "kube-api-access-fqssw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:25:19.874935 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:19.874863 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c01200a-3f15-4223-aaa3-7f7ccbd1427c-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:25:19.874935 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:19.874889 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqssw\" (UniqueName: \"kubernetes.io/projected/ad508a5b-b62f-4d57-b7b9-c490dbefa055-kube-api-access-fqssw\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:25:19.874935 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:19.874906 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hgfr\" (UniqueName: \"kubernetes.io/projected/6c01200a-3f15-4223-aaa3-7f7ccbd1427c-kube-api-access-5hgfr\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:25:19.874935 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:19.874919 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad508a5b-b62f-4d57-b7b9-c490dbefa055-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:25:20.317535 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:20.317495 2574 generic.go:356] "Generic (PLEG): container finished" podID="6c01200a-3f15-4223-aaa3-7f7ccbd1427c" containerID="9efa8a6f2576ba06a6c1f5b3c07e46b228185695465197f96becee6d6277c353" exitCode=0 Sep 04 22:25:20.317976 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:20.317579 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-a8d845-6469bc984b-p2zd9" Sep 04 22:25:20.317976 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:20.317583 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-a8d845-6469bc984b-p2zd9" event={"ID":"6c01200a-3f15-4223-aaa3-7f7ccbd1427c","Type":"ContainerDied","Data":"9efa8a6f2576ba06a6c1f5b3c07e46b228185695465197f96becee6d6277c353"} Sep 04 22:25:20.317976 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:20.317664 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-a8d845-6469bc984b-p2zd9" event={"ID":"6c01200a-3f15-4223-aaa3-7f7ccbd1427c","Type":"ContainerDied","Data":"3afb0e519074df6413b4cc53a711212ed82e949cb3bf3a3099be83aa393040e5"} Sep 04 22:25:20.317976 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:20.317686 2574 scope.go:117] "RemoveContainer" containerID="9efa8a6f2576ba06a6c1f5b3c07e46b228185695465197f96becee6d6277c353" Sep 04 22:25:20.319123 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:20.319100 2574 generic.go:356] "Generic (PLEG): container finished" podID="ad508a5b-b62f-4d57-b7b9-c490dbefa055" containerID="8548f7466b9ea6773aadad5704764439e1328702eacc6cd799ccd31b9035ad3f" exitCode=0 Sep 04 22:25:20.319231 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:20.319204 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a8d845-776d46797c-pv87c" event={"ID":"ad508a5b-b62f-4d57-b7b9-c490dbefa055","Type":"ContainerDied","Data":"8548f7466b9ea6773aadad5704764439e1328702eacc6cd799ccd31b9035ad3f"} Sep 04 22:25:20.319288 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:20.319246 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a8d845-776d46797c-pv87c" event={"ID":"ad508a5b-b62f-4d57-b7b9-c490dbefa055","Type":"ContainerDied","Data":"6435423ff927a061bd73847467ddfccca7e83bc3f15658e9dc736dc54d45d312"} Sep 04 22:25:20.319391 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:20.319372 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a8d845-776d46797c-pv87c" Sep 04 22:25:20.328796 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:20.328777 2574 scope.go:117] "RemoveContainer" containerID="9efa8a6f2576ba06a6c1f5b3c07e46b228185695465197f96becee6d6277c353" Sep 04 22:25:20.329101 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:25:20.329080 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9efa8a6f2576ba06a6c1f5b3c07e46b228185695465197f96becee6d6277c353\": container with ID starting with 9efa8a6f2576ba06a6c1f5b3c07e46b228185695465197f96becee6d6277c353 not found: ID does not exist" containerID="9efa8a6f2576ba06a6c1f5b3c07e46b228185695465197f96becee6d6277c353" Sep 04 22:25:20.329180 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:20.329108 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9efa8a6f2576ba06a6c1f5b3c07e46b228185695465197f96becee6d6277c353"} err="failed to get container status \"9efa8a6f2576ba06a6c1f5b3c07e46b228185695465197f96becee6d6277c353\": rpc error: code = NotFound desc = could not find container \"9efa8a6f2576ba06a6c1f5b3c07e46b228185695465197f96becee6d6277c353\": container with ID starting with 9efa8a6f2576ba06a6c1f5b3c07e46b228185695465197f96becee6d6277c353 not found: ID does not exist" Sep 04 22:25:20.329180 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:20.329124 2574 scope.go:117] "RemoveContainer" containerID="8548f7466b9ea6773aadad5704764439e1328702eacc6cd799ccd31b9035ad3f" Sep 04 22:25:20.337651 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:20.337634 2574 scope.go:117] "RemoveContainer" containerID="8548f7466b9ea6773aadad5704764439e1328702eacc6cd799ccd31b9035ad3f" Sep 04 22:25:20.337935 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:25:20.337915 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8548f7466b9ea6773aadad5704764439e1328702eacc6cd799ccd31b9035ad3f\": container with ID starting with 8548f7466b9ea6773aadad5704764439e1328702eacc6cd799ccd31b9035ad3f not found: ID does not exist" containerID="8548f7466b9ea6773aadad5704764439e1328702eacc6cd799ccd31b9035ad3f" Sep 04 22:25:20.338014 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:20.337941 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8548f7466b9ea6773aadad5704764439e1328702eacc6cd799ccd31b9035ad3f"} err="failed to get container status \"8548f7466b9ea6773aadad5704764439e1328702eacc6cd799ccd31b9035ad3f\": rpc error: code = NotFound desc = could not find container \"8548f7466b9ea6773aadad5704764439e1328702eacc6cd799ccd31b9035ad3f\": container with ID starting with 8548f7466b9ea6773aadad5704764439e1328702eacc6cd799ccd31b9035ad3f not found: ID does not exist" Sep 04 22:25:20.345512 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:20.345483 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a8d845-776d46797c-pv87c"] Sep 04 22:25:20.349870 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:20.349851 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a8d845-776d46797c-pv87c"] Sep 04 22:25:20.360302 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:20.360276 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a8d845-6469bc984b-p2zd9"] Sep 04 22:25:20.369569 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:20.369526 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a8d845-6469bc984b-p2zd9"] Sep 04 22:25:20.448415 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:20.448393 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-25fea002-k8glb" Sep 04 22:25:20.580821 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:20.580751 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4f0c7d1-7714-49b3-8aa5-342c1a1fb25f-maas-api-service-ca\") pod \"b4f0c7d1-7714-49b3-8aa5-342c1a1fb25f\" (UID: \"b4f0c7d1-7714-49b3-8aa5-342c1a1fb25f\") " Sep 04 22:25:20.581111 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:20.581091 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4f0c7d1-7714-49b3-8aa5-342c1a1fb25f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b4f0c7d1-7714-49b3-8aa5-342c1a1fb25f" (UID: "b4f0c7d1-7714-49b3-8aa5-342c1a1fb25f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:25:20.681916 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:20.681889 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4f0c7d1-7714-49b3-8aa5-342c1a1fb25f-maas-api-service-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:25:20.714346 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:20.714319 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c01200a-3f15-4223-aaa3-7f7ccbd1427c" path="/var/lib/kubelet/pods/6c01200a-3f15-4223-aaa3-7f7ccbd1427c/volumes" Sep 04 22:25:20.714674 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:20.714661 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad508a5b-b62f-4d57-b7b9-c490dbefa055" path="/var/lib/kubelet/pods/ad508a5b-b62f-4d57-b7b9-c490dbefa055/volumes" Sep 04 22:25:20.714951 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:20.714939 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd85365d-678e-4eb2-be9d-a1bb9378e219" path="/var/lib/kubelet/pods/cd85365d-678e-4eb2-be9d-a1bb9378e219/volumes" Sep 04 22:25:21.329998 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:21.329960 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-25fea002-k8glb" event={"ID":"b4f0c7d1-7714-49b3-8aa5-342c1a1fb25f","Type":"ContainerDied","Data":"5adc530d356418630960e28417d901f789e8b8d4e87445a8f81318ea0851c227"} Sep 04 22:25:21.330361 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:21.330016 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5adc530d356418630960e28417d901f789e8b8d4e87445a8f81318ea0851c227" Sep 04 22:25:21.330361 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:21.329977 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-25fea002-k8glb" Sep 04 22:25:24.381927 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:24.381896 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-25fea002-5d85fc9ffc-lsdp8"] Sep 04 22:25:24.382351 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:24.382104 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-25fea002-5d85fc9ffc-lsdp8" podUID="c4bbcc72-218f-42fd-b422-c2afa187c815" containerName="payload-processing" containerID="cri-o://1897032d0bb92b6713e23cc3cc5d62136cf5b3b70a533410a0646a3c208116e3" gracePeriod=30 Sep 04 22:25:24.571788 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:24.571750 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-25fea002-k8glb"] Sep 04 22:25:24.574203 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:24.574175 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-25fea002-k8glb"] Sep 04 22:25:24.631055 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:24.631031 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-25fea002-5d85fc9ffc-lsdp8" Sep 04 22:25:24.712127 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:24.712049 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4bbcc72-218f-42fd-b422-c2afa187c815-plugins-config-volume\") pod \"c4bbcc72-218f-42fd-b422-c2afa187c815\" (UID: \"c4bbcc72-218f-42fd-b422-c2afa187c815\") " Sep 04 22:25:24.712127 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:24.712112 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99l6l\" (UniqueName: \"kubernetes.io/projected/c4bbcc72-218f-42fd-b422-c2afa187c815-kube-api-access-99l6l\") pod \"c4bbcc72-218f-42fd-b422-c2afa187c815\" (UID: \"c4bbcc72-218f-42fd-b422-c2afa187c815\") " Sep 04 22:25:24.712518 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:24.712492 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4bbcc72-218f-42fd-b422-c2afa187c815-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c4bbcc72-218f-42fd-b422-c2afa187c815" (UID: "c4bbcc72-218f-42fd-b422-c2afa187c815"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:25:24.714201 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:24.714175 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4bbcc72-218f-42fd-b422-c2afa187c815-kube-api-access-99l6l" (OuterVolumeSpecName: "kube-api-access-99l6l") pod "c4bbcc72-218f-42fd-b422-c2afa187c815" (UID: "c4bbcc72-218f-42fd-b422-c2afa187c815"). InnerVolumeSpecName "kube-api-access-99l6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:25:24.715090 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:24.715065 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4f0c7d1-7714-49b3-8aa5-342c1a1fb25f" path="/var/lib/kubelet/pods/b4f0c7d1-7714-49b3-8aa5-342c1a1fb25f/volumes" Sep 04 22:25:24.813025 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:24.812994 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4bbcc72-218f-42fd-b422-c2afa187c815-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:25:24.813025 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:24.813026 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99l6l\" (UniqueName: \"kubernetes.io/projected/c4bbcc72-218f-42fd-b422-c2afa187c815-kube-api-access-99l6l\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:25:25.352880 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:25.352844 2574 generic.go:356] "Generic (PLEG): container finished" podID="c4bbcc72-218f-42fd-b422-c2afa187c815" containerID="1897032d0bb92b6713e23cc3cc5d62136cf5b3b70a533410a0646a3c208116e3" exitCode=0 Sep 04 22:25:25.353042 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:25.352891 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-25fea002-5d85fc9ffc-lsdp8" Sep 04 22:25:25.353042 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:25.352918 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-25fea002-5d85fc9ffc-lsdp8" event={"ID":"c4bbcc72-218f-42fd-b422-c2afa187c815","Type":"ContainerDied","Data":"1897032d0bb92b6713e23cc3cc5d62136cf5b3b70a533410a0646a3c208116e3"} Sep 04 22:25:25.353042 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:25.352964 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-25fea002-5d85fc9ffc-lsdp8" event={"ID":"c4bbcc72-218f-42fd-b422-c2afa187c815","Type":"ContainerDied","Data":"550ed7c019be7b11aed31d58c39149b91b7fdbfb066385227f591d2a571e68bb"} Sep 04 22:25:25.353042 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:25.352987 2574 scope.go:117] "RemoveContainer" containerID="1897032d0bb92b6713e23cc3cc5d62136cf5b3b70a533410a0646a3c208116e3" Sep 04 22:25:25.361435 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:25.361418 2574 scope.go:117] "RemoveContainer" containerID="1897032d0bb92b6713e23cc3cc5d62136cf5b3b70a533410a0646a3c208116e3" Sep 04 22:25:25.361717 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:25:25.361698 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1897032d0bb92b6713e23cc3cc5d62136cf5b3b70a533410a0646a3c208116e3\": container with ID starting with 1897032d0bb92b6713e23cc3cc5d62136cf5b3b70a533410a0646a3c208116e3 not found: ID does not exist" containerID="1897032d0bb92b6713e23cc3cc5d62136cf5b3b70a533410a0646a3c208116e3" Sep 04 22:25:25.361764 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:25.361724 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1897032d0bb92b6713e23cc3cc5d62136cf5b3b70a533410a0646a3c208116e3"} err="failed to get container status \"1897032d0bb92b6713e23cc3cc5d62136cf5b3b70a533410a0646a3c208116e3\": rpc error: code = NotFound desc = could not find container \"1897032d0bb92b6713e23cc3cc5d62136cf5b3b70a533410a0646a3c208116e3\": container with ID starting with 1897032d0bb92b6713e23cc3cc5d62136cf5b3b70a533410a0646a3c208116e3 not found: ID does not exist" Sep 04 22:25:25.371610 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:25.371591 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-25fea002-5d85fc9ffc-lsdp8"] Sep 04 22:25:25.376224 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:25.376204 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-25fea002-5d85fc9ffc-lsdp8"] Sep 04 22:25:26.715272 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:25:26.715239 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4bbcc72-218f-42fd-b422-c2afa187c815" path="/var/lib/kubelet/pods/c4bbcc72-218f-42fd-b422-c2afa187c815/volumes" Sep 04 22:26:18.455256 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:18.455219 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b91726-7c4fb4db4b-ds7vd"] Sep 04 22:26:18.455818 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:18.455695 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4f0c7d1-7714-49b3-8aa5-342c1a1fb25f" containerName="revoke-keys" Sep 04 22:26:18.455818 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:18.455714 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4f0c7d1-7714-49b3-8aa5-342c1a1fb25f" containerName="revoke-keys" Sep 04 22:26:18.455818 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:18.455743 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c01200a-3f15-4223-aaa3-7f7ccbd1427c" containerName="payload-pre-processing" Sep 04 22:26:18.455818 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:18.455752 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c01200a-3f15-4223-aaa3-7f7ccbd1427c" containerName="payload-pre-processing" Sep 04 22:26:18.455818 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:18.455769 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4bbcc72-218f-42fd-b422-c2afa187c815" containerName="payload-processing" Sep 04 22:26:18.455818 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:18.455778 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4bbcc72-218f-42fd-b422-c2afa187c815" containerName="payload-processing" Sep 04 22:26:18.455818 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:18.455798 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad508a5b-b62f-4d57-b7b9-c490dbefa055" containerName="payload-processing" Sep 04 22:26:18.455818 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:18.455807 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad508a5b-b62f-4d57-b7b9-c490dbefa055" containerName="payload-processing" Sep 04 22:26:18.456238 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:18.455891 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad508a5b-b62f-4d57-b7b9-c490dbefa055" containerName="payload-processing" Sep 04 22:26:18.456238 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:18.455908 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c01200a-3f15-4223-aaa3-7f7ccbd1427c" containerName="payload-pre-processing" Sep 04 22:26:18.456238 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:18.455918 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4bbcc72-218f-42fd-b422-c2afa187c815" containerName="payload-processing" Sep 04 22:26:18.456238 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:18.455929 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4f0c7d1-7714-49b3-8aa5-342c1a1fb25f" containerName="revoke-keys" Sep 04 22:26:18.460286 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:18.460267 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b91726-7c4fb4db4b-ds7vd" Sep 04 22:26:18.462769 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:18.462747 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-b91726\"" Sep 04 22:26:18.467888 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:18.467866 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b91726-7c4fb4db4b-ds7vd"] Sep 04 22:26:18.484271 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:18.484244 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b91726-54c9fc98f7-h72w7"] Sep 04 22:26:18.487497 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:18.487477 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b91726-54c9fc98f7-h72w7" Sep 04 22:26:18.489809 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:18.489793 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-b91726-dockercfg-d285d\"" Sep 04 22:26:18.500479 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:18.500458 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b91726-54c9fc98f7-h72w7"] Sep 04 22:26:18.537199 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:18.537168 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79e9aa33-2ef0-41cd-a493-2c7223450768-plugins-config-volume\") pod \"payload-processing-e2e-explicit-b91726-54c9fc98f7-h72w7\" (UID: \"79e9aa33-2ef0-41cd-a493-2c7223450768\") " pod="openshift-ingress/payload-processing-e2e-explicit-b91726-54c9fc98f7-h72w7" Sep 04 22:26:18.537328 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:18.537204 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbnpf\" (UniqueName: \"kubernetes.io/projected/79e9aa33-2ef0-41cd-a493-2c7223450768-kube-api-access-xbnpf\") pod \"payload-processing-e2e-explicit-b91726-54c9fc98f7-h72w7\" (UID: \"79e9aa33-2ef0-41cd-a493-2c7223450768\") " pod="openshift-ingress/payload-processing-e2e-explicit-b91726-54c9fc98f7-h72w7" Sep 04 22:26:18.537328 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:18.537286 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17b0b736-24b5-4e85-b858-c4968c2e9b04-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-b91726-7c4fb4db4b-ds7vd\" (UID: \"17b0b736-24b5-4e85-b858-c4968c2e9b04\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b91726-7c4fb4db4b-ds7vd" Sep 04 22:26:18.537443 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:18.537373 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5rrt\" (UniqueName: \"kubernetes.io/projected/17b0b736-24b5-4e85-b858-c4968c2e9b04-kube-api-access-m5rrt\") pod \"payload-pre-processing-e2e-explicit-b91726-7c4fb4db4b-ds7vd\" (UID: \"17b0b736-24b5-4e85-b858-c4968c2e9b04\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b91726-7c4fb4db4b-ds7vd" Sep 04 22:26:18.638125 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:18.638096 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79e9aa33-2ef0-41cd-a493-2c7223450768-plugins-config-volume\") pod \"payload-processing-e2e-explicit-b91726-54c9fc98f7-h72w7\" (UID: \"79e9aa33-2ef0-41cd-a493-2c7223450768\") " pod="openshift-ingress/payload-processing-e2e-explicit-b91726-54c9fc98f7-h72w7" Sep 04 22:26:18.638125 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:18.638130 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbnpf\" (UniqueName: \"kubernetes.io/projected/79e9aa33-2ef0-41cd-a493-2c7223450768-kube-api-access-xbnpf\") pod \"payload-processing-e2e-explicit-b91726-54c9fc98f7-h72w7\" (UID: \"79e9aa33-2ef0-41cd-a493-2c7223450768\") " pod="openshift-ingress/payload-processing-e2e-explicit-b91726-54c9fc98f7-h72w7" Sep 04 22:26:18.638403 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:18.638173 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17b0b736-24b5-4e85-b858-c4968c2e9b04-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-b91726-7c4fb4db4b-ds7vd\" (UID: \"17b0b736-24b5-4e85-b858-c4968c2e9b04\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b91726-7c4fb4db4b-ds7vd" Sep 04 22:26:18.638403 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:18.638227 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5rrt\" (UniqueName: \"kubernetes.io/projected/17b0b736-24b5-4e85-b858-c4968c2e9b04-kube-api-access-m5rrt\") pod \"payload-pre-processing-e2e-explicit-b91726-7c4fb4db4b-ds7vd\" (UID: \"17b0b736-24b5-4e85-b858-c4968c2e9b04\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b91726-7c4fb4db4b-ds7vd" Sep 04 22:26:18.638882 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:18.638862 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79e9aa33-2ef0-41cd-a493-2c7223450768-plugins-config-volume\") pod \"payload-processing-e2e-explicit-b91726-54c9fc98f7-h72w7\" (UID: \"79e9aa33-2ef0-41cd-a493-2c7223450768\") " pod="openshift-ingress/payload-processing-e2e-explicit-b91726-54c9fc98f7-h72w7" Sep 04 22:26:18.638930 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:18.638878 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17b0b736-24b5-4e85-b858-c4968c2e9b04-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-b91726-7c4fb4db4b-ds7vd\" (UID: \"17b0b736-24b5-4e85-b858-c4968c2e9b04\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b91726-7c4fb4db4b-ds7vd" Sep 04 22:26:18.646784 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:18.646759 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbnpf\" (UniqueName: \"kubernetes.io/projected/79e9aa33-2ef0-41cd-a493-2c7223450768-kube-api-access-xbnpf\") pod \"payload-processing-e2e-explicit-b91726-54c9fc98f7-h72w7\" (UID: \"79e9aa33-2ef0-41cd-a493-2c7223450768\") " pod="openshift-ingress/payload-processing-e2e-explicit-b91726-54c9fc98f7-h72w7" Sep 04 22:26:18.646890 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:18.646816 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5rrt\" (UniqueName: \"kubernetes.io/projected/17b0b736-24b5-4e85-b858-c4968c2e9b04-kube-api-access-m5rrt\") pod \"payload-pre-processing-e2e-explicit-b91726-7c4fb4db4b-ds7vd\" (UID: \"17b0b736-24b5-4e85-b858-c4968c2e9b04\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b91726-7c4fb4db4b-ds7vd" Sep 04 22:26:18.771234 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:18.771208 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b91726-7c4fb4db4b-ds7vd" Sep 04 22:26:18.797264 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:18.797241 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b91726-54c9fc98f7-h72w7" Sep 04 22:26:19.049195 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:19.049169 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b91726-7c4fb4db4b-ds7vd"] Sep 04 22:26:19.051232 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:26:19.051204 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17b0b736_24b5_4e85_b858_c4968c2e9b04.slice/crio-e0f76df3f82d4accf1afaf34d0c8d2406a315ccbaa10b7373411e5977a93b6c3 WatchSource:0}: Error finding container e0f76df3f82d4accf1afaf34d0c8d2406a315ccbaa10b7373411e5977a93b6c3: Status 404 returned error can't find the container with id e0f76df3f82d4accf1afaf34d0c8d2406a315ccbaa10b7373411e5977a93b6c3 Sep 04 22:26:19.076580 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:19.076544 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b91726-54c9fc98f7-h72w7"] Sep 04 22:26:19.087602 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:26:19.087578 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e9aa33_2ef0_41cd_a493_2c7223450768.slice/crio-47c518d80f0fb47115ad1c32990e0b011ee34718d745bd95cd2732443933e5ea WatchSource:0}: Error finding container 47c518d80f0fb47115ad1c32990e0b011ee34718d745bd95cd2732443933e5ea: Status 404 returned error can't find the container with id 47c518d80f0fb47115ad1c32990e0b011ee34718d745bd95cd2732443933e5ea Sep 04 22:26:19.632115 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:19.632079 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b91726-7c4fb4db4b-ds7vd" event={"ID":"17b0b736-24b5-4e85-b858-c4968c2e9b04","Type":"ContainerStarted","Data":"817ca08f89dbeaea522b6e4c3a599b5ccaa6a7fe6f5fd78bb90f1b69b6b49b0a"} Sep 04 22:26:19.632566 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:19.632137 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b91726-7c4fb4db4b-ds7vd" event={"ID":"17b0b736-24b5-4e85-b858-c4968c2e9b04","Type":"ContainerStarted","Data":"e0f76df3f82d4accf1afaf34d0c8d2406a315ccbaa10b7373411e5977a93b6c3"} Sep 04 22:26:19.632566 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:19.632182 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b91726-7c4fb4db4b-ds7vd" Sep 04 22:26:19.633898 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:19.633874 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b91726-54c9fc98f7-h72w7" event={"ID":"79e9aa33-2ef0-41cd-a493-2c7223450768","Type":"ContainerStarted","Data":"73ed7c1c6d965d11a869080bc78c2bcb825cbf1864505c745d369c828a902342"} Sep 04 22:26:19.633997 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:19.633923 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-b91726-54c9fc98f7-h72w7" Sep 04 22:26:19.633997 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:19.633948 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b91726-54c9fc98f7-h72w7" event={"ID":"79e9aa33-2ef0-41cd-a493-2c7223450768","Type":"ContainerStarted","Data":"47c518d80f0fb47115ad1c32990e0b011ee34718d745bd95cd2732443933e5ea"} Sep 04 22:26:19.648420 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:19.648375 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b91726-7c4fb4db4b-ds7vd" podStartSLOduration=1.648359531 podStartE2EDuration="1.648359531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:26:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:26:19.647725738 +0000 UTC m=+1083.412201353" watchObservedRunningTime="2026-09-04 22:26:19.648359531 +0000 UTC m=+1083.412835143" Sep 04 22:26:19.671826 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:19.671782 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-b91726-54c9fc98f7-h72w7" podStartSLOduration=1.671768954 podStartE2EDuration="1.671768954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:26:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:26:19.670269335 +0000 UTC m=+1083.434744948" watchObservedRunningTime="2026-09-04 22:26:19.671768954 +0000 UTC m=+1083.436244565" Sep 04 22:26:30.641458 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:30.641428 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b91726-7c4fb4db4b-ds7vd" Sep 04 22:26:30.641945 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:26:30.641591 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-b91726-54c9fc98f7-h72w7" Sep 04 22:27:07.768524 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:07.768489 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-84624e97-7d76d9cbf4-nxvt2"] Sep 04 22:27:07.772100 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:07.772077 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-84624e97-7d76d9cbf4-nxvt2" Sep 04 22:27:07.774522 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:07.774500 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-84624e97\"" Sep 04 22:27:07.774648 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:07.774567 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-84624e97-dockercfg-zrxh5\"" Sep 04 22:27:07.784854 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:07.784823 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-84624e97-7d76d9cbf4-nxvt2"] Sep 04 22:27:07.801916 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:07.801887 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-84624e97-b64ff55bf-hcwlf"] Sep 04 22:27:07.805288 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:07.805264 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-84624e97-b64ff55bf-hcwlf" Sep 04 22:27:07.817249 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:07.817222 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-84624e97-b64ff55bf-hcwlf"] Sep 04 22:27:07.851671 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:07.851637 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4kjt\" (UniqueName: \"kubernetes.io/projected/714f5646-7ac5-4c59-855c-f0dc9ee82833-kube-api-access-f4kjt\") pod \"payload-pre-processing-e2e-mt-84624e97-7d76d9cbf4-nxvt2\" (UID: \"714f5646-7ac5-4c59-855c-f0dc9ee82833\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-84624e97-7d76d9cbf4-nxvt2" Sep 04 22:27:07.851849 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:07.851690 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f04e7dfa-e9fb-48b4-bf1d-3a0e5d45f75a-plugins-config-volume\") pod \"payload-processing-e2e-mt-84624e97-b64ff55bf-hcwlf\" (UID: \"f04e7dfa-e9fb-48b4-bf1d-3a0e5d45f75a\") " pod="openshift-ingress/payload-processing-e2e-mt-84624e97-b64ff55bf-hcwlf" Sep 04 22:27:07.851849 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:07.851759 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/714f5646-7ac5-4c59-855c-f0dc9ee82833-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-84624e97-7d76d9cbf4-nxvt2\" (UID: \"714f5646-7ac5-4c59-855c-f0dc9ee82833\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-84624e97-7d76d9cbf4-nxvt2" Sep 04 22:27:07.851849 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:07.851808 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fccp\" (UniqueName: \"kubernetes.io/projected/f04e7dfa-e9fb-48b4-bf1d-3a0e5d45f75a-kube-api-access-5fccp\") pod \"payload-processing-e2e-mt-84624e97-b64ff55bf-hcwlf\" (UID: \"f04e7dfa-e9fb-48b4-bf1d-3a0e5d45f75a\") " pod="openshift-ingress/payload-processing-e2e-mt-84624e97-b64ff55bf-hcwlf" Sep 04 22:27:07.952897 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:07.952863 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f04e7dfa-e9fb-48b4-bf1d-3a0e5d45f75a-plugins-config-volume\") pod \"payload-processing-e2e-mt-84624e97-b64ff55bf-hcwlf\" (UID: \"f04e7dfa-e9fb-48b4-bf1d-3a0e5d45f75a\") " pod="openshift-ingress/payload-processing-e2e-mt-84624e97-b64ff55bf-hcwlf" Sep 04 22:27:07.952897 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:07.952906 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/714f5646-7ac5-4c59-855c-f0dc9ee82833-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-84624e97-7d76d9cbf4-nxvt2\" (UID: \"714f5646-7ac5-4c59-855c-f0dc9ee82833\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-84624e97-7d76d9cbf4-nxvt2" Sep 04 22:27:07.953168 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:07.952936 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fccp\" (UniqueName: \"kubernetes.io/projected/f04e7dfa-e9fb-48b4-bf1d-3a0e5d45f75a-kube-api-access-5fccp\") pod \"payload-processing-e2e-mt-84624e97-b64ff55bf-hcwlf\" (UID: \"f04e7dfa-e9fb-48b4-bf1d-3a0e5d45f75a\") " pod="openshift-ingress/payload-processing-e2e-mt-84624e97-b64ff55bf-hcwlf" Sep 04 22:27:07.953168 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:07.953018 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4kjt\" (UniqueName: \"kubernetes.io/projected/714f5646-7ac5-4c59-855c-f0dc9ee82833-kube-api-access-f4kjt\") pod \"payload-pre-processing-e2e-mt-84624e97-7d76d9cbf4-nxvt2\" (UID: \"714f5646-7ac5-4c59-855c-f0dc9ee82833\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-84624e97-7d76d9cbf4-nxvt2" Sep 04 22:27:07.953694 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:07.953672 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f04e7dfa-e9fb-48b4-bf1d-3a0e5d45f75a-plugins-config-volume\") pod \"payload-processing-e2e-mt-84624e97-b64ff55bf-hcwlf\" (UID: \"f04e7dfa-e9fb-48b4-bf1d-3a0e5d45f75a\") " pod="openshift-ingress/payload-processing-e2e-mt-84624e97-b64ff55bf-hcwlf" Sep 04 22:27:07.953786 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:07.953671 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/714f5646-7ac5-4c59-855c-f0dc9ee82833-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-84624e97-7d76d9cbf4-nxvt2\" (UID: \"714f5646-7ac5-4c59-855c-f0dc9ee82833\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-84624e97-7d76d9cbf4-nxvt2" Sep 04 22:27:07.961155 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:07.961128 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fccp\" (UniqueName: \"kubernetes.io/projected/f04e7dfa-e9fb-48b4-bf1d-3a0e5d45f75a-kube-api-access-5fccp\") pod \"payload-processing-e2e-mt-84624e97-b64ff55bf-hcwlf\" (UID: \"f04e7dfa-e9fb-48b4-bf1d-3a0e5d45f75a\") " pod="openshift-ingress/payload-processing-e2e-mt-84624e97-b64ff55bf-hcwlf" Sep 04 22:27:07.961265 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:07.961159 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4kjt\" (UniqueName: \"kubernetes.io/projected/714f5646-7ac5-4c59-855c-f0dc9ee82833-kube-api-access-f4kjt\") pod \"payload-pre-processing-e2e-mt-84624e97-7d76d9cbf4-nxvt2\" (UID: \"714f5646-7ac5-4c59-855c-f0dc9ee82833\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-84624e97-7d76d9cbf4-nxvt2" Sep 04 22:27:08.083424 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:08.083327 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-84624e97-7d76d9cbf4-nxvt2" Sep 04 22:27:08.115851 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:08.115818 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-84624e97-b64ff55bf-hcwlf" Sep 04 22:27:08.358679 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:08.358654 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-84624e97-7d76d9cbf4-nxvt2"] Sep 04 22:27:08.359979 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:27:08.359955 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod714f5646_7ac5_4c59_855c_f0dc9ee82833.slice/crio-183d80dabeb874079ff7e268d1b358a0db872e9dd301d4738dcaaa098adcd36d WatchSource:0}: Error finding container 183d80dabeb874079ff7e268d1b358a0db872e9dd301d4738dcaaa098adcd36d: Status 404 returned error can't find the container with id 183d80dabeb874079ff7e268d1b358a0db872e9dd301d4738dcaaa098adcd36d Sep 04 22:27:08.402492 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:08.402465 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-84624e97-b64ff55bf-hcwlf"] Sep 04 22:27:08.404109 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:27:08.404089 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf04e7dfa_e9fb_48b4_bf1d_3a0e5d45f75a.slice/crio-5bff90223e834c5fdd2341b19a478f54db39327c56c57bbf7c597796c18fd852 WatchSource:0}: Error finding container 5bff90223e834c5fdd2341b19a478f54db39327c56c57bbf7c597796c18fd852: Status 404 returned error can't find the container with id 5bff90223e834c5fdd2341b19a478f54db39327c56c57bbf7c597796c18fd852 Sep 04 22:27:08.896855 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:08.896821 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-84624e97-7d76d9cbf4-nxvt2" event={"ID":"714f5646-7ac5-4c59-855c-f0dc9ee82833","Type":"ContainerStarted","Data":"db0ae305c0c4b0fe8e2afec308d3e6a5ad5fe64b64ded04ac1d98c146ad4d531"} Sep 04 22:27:08.897330 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:08.896884 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-84624e97-7d76d9cbf4-nxvt2" Sep 04 22:27:08.897330 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:08.896910 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-84624e97-7d76d9cbf4-nxvt2" event={"ID":"714f5646-7ac5-4c59-855c-f0dc9ee82833","Type":"ContainerStarted","Data":"183d80dabeb874079ff7e268d1b358a0db872e9dd301d4738dcaaa098adcd36d"} Sep 04 22:27:08.898483 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:08.898459 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-84624e97-b64ff55bf-hcwlf" event={"ID":"f04e7dfa-e9fb-48b4-bf1d-3a0e5d45f75a","Type":"ContainerStarted","Data":"6913fbbd5f72db6bfa3a356d2c1e756c57ac3b26b48c2c4e492780533385201c"} Sep 04 22:27:08.898567 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:08.898512 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-84624e97-b64ff55bf-hcwlf" event={"ID":"f04e7dfa-e9fb-48b4-bf1d-3a0e5d45f75a","Type":"ContainerStarted","Data":"5bff90223e834c5fdd2341b19a478f54db39327c56c57bbf7c597796c18fd852"} Sep 04 22:27:08.898567 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:08.898558 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-84624e97-b64ff55bf-hcwlf" Sep 04 22:27:08.914655 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:08.914611 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-84624e97-7d76d9cbf4-nxvt2" podStartSLOduration=1.914598594 podStartE2EDuration="1.914598594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:27:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:27:08.912128344 +0000 UTC m=+1132.676603960" watchObservedRunningTime="2026-09-04 22:27:08.914598594 +0000 UTC m=+1132.679074207" Sep 04 22:27:08.928885 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:08.928842 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-84624e97-b64ff55bf-hcwlf" podStartSLOduration=1.9288313540000002 podStartE2EDuration="1.928831354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:27:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:27:08.927598159 +0000 UTC m=+1132.692073790" watchObservedRunningTime="2026-09-04 22:27:08.928831354 +0000 UTC m=+1132.693306966" Sep 04 22:27:19.906266 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:19.906235 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-84624e97-b64ff55bf-hcwlf" Sep 04 22:27:19.906656 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:19.906348 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-84624e97-7d76d9cbf4-nxvt2" Sep 04 22:27:38.757464 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:38.757429 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84624e97-gkrv6"] Sep 04 22:27:38.760805 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:38.760786 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84624e97-gkrv6" Sep 04 22:27:38.764348 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:38.764328 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 04 22:27:38.764441 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:38.764369 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hc7x4\"" Sep 04 22:27:38.770078 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:38.770053 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84624e97-gkrv6"] Sep 04 22:27:38.940199 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:38.940156 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65ffe5b4-11c0-4cb0-a1df-b68ca7b32f0f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-84624e97-gkrv6\" (UID: \"65ffe5b4-11c0-4cb0-a1df-b68ca7b32f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84624e97-gkrv6" Sep 04 22:27:39.041636 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:39.041519 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65ffe5b4-11c0-4cb0-a1df-b68ca7b32f0f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-84624e97-gkrv6\" (UID: \"65ffe5b4-11c0-4cb0-a1df-b68ca7b32f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84624e97-gkrv6" Sep 04 22:27:39.042171 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:39.042146 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65ffe5b4-11c0-4cb0-a1df-b68ca7b32f0f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-84624e97-gkrv6\" (UID: \"65ffe5b4-11c0-4cb0-a1df-b68ca7b32f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84624e97-gkrv6" Sep 04 22:27:39.071545 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:39.071514 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84624e97-gkrv6" Sep 04 22:27:39.342740 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:39.342705 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84624e97-gkrv6"] Sep 04 22:27:39.344046 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:27:39.344017 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65ffe5b4_11c0_4cb0_a1df_b68ca7b32f0f.slice/crio-ac80ad2d43e6dffe45ee73a94a242757dce3f4c9e22f7f0f68dfae88d08eb386 WatchSource:0}: Error finding container ac80ad2d43e6dffe45ee73a94a242757dce3f4c9e22f7f0f68dfae88d08eb386: Status 404 returned error can't find the container with id ac80ad2d43e6dffe45ee73a94a242757dce3f4c9e22f7f0f68dfae88d08eb386 Sep 04 22:27:40.072093 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:40.072062 2574 generic.go:356] "Generic (PLEG): container finished" podID="65ffe5b4-11c0-4cb0-a1df-b68ca7b32f0f" containerID="3f1ac0a7420107fcddfa8271bf5b0966179ea1736c25087178fd094416f64c11" exitCode=0 Sep 04 22:27:40.072492 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:40.072148 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84624e97-gkrv6" event={"ID":"65ffe5b4-11c0-4cb0-a1df-b68ca7b32f0f","Type":"ContainerDied","Data":"3f1ac0a7420107fcddfa8271bf5b0966179ea1736c25087178fd094416f64c11"} Sep 04 22:27:40.072492 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:40.072304 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84624e97-gkrv6" event={"ID":"65ffe5b4-11c0-4cb0-a1df-b68ca7b32f0f","Type":"ContainerStarted","Data":"ac80ad2d43e6dffe45ee73a94a242757dce3f4c9e22f7f0f68dfae88d08eb386"} Sep 04 22:27:41.208862 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:41.208838 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84624e97-gkrv6" Sep 04 22:27:41.260803 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:41.260774 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65ffe5b4-11c0-4cb0-a1df-b68ca7b32f0f-maas-api-service-ca\") pod \"65ffe5b4-11c0-4cb0-a1df-b68ca7b32f0f\" (UID: \"65ffe5b4-11c0-4cb0-a1df-b68ca7b32f0f\") " Sep 04 22:27:41.261108 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:41.261082 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65ffe5b4-11c0-4cb0-a1df-b68ca7b32f0f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "65ffe5b4-11c0-4cb0-a1df-b68ca7b32f0f" (UID: "65ffe5b4-11c0-4cb0-a1df-b68ca7b32f0f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:27:41.361859 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:41.361769 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65ffe5b4-11c0-4cb0-a1df-b68ca7b32f0f-maas-api-service-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:27:42.084776 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:42.084747 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84624e97-gkrv6" Sep 04 22:27:42.084945 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:42.084781 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84624e97-gkrv6" event={"ID":"65ffe5b4-11c0-4cb0-a1df-b68ca7b32f0f","Type":"ContainerDied","Data":"ac80ad2d43e6dffe45ee73a94a242757dce3f4c9e22f7f0f68dfae88d08eb386"} Sep 04 22:27:42.084945 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:42.084828 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac80ad2d43e6dffe45ee73a94a242757dce3f4c9e22f7f0f68dfae88d08eb386" Sep 04 22:27:44.606837 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:44.606800 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b91726-269hk"] Sep 04 22:27:44.607350 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:44.607331 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65ffe5b4-11c0-4cb0-a1df-b68ca7b32f0f" containerName="revoke-keys" Sep 04 22:27:44.607405 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:44.607353 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="65ffe5b4-11c0-4cb0-a1df-b68ca7b32f0f" containerName="revoke-keys" Sep 04 22:27:44.607466 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:44.607453 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="65ffe5b4-11c0-4cb0-a1df-b68ca7b32f0f" containerName="revoke-keys" Sep 04 22:27:44.610622 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:44.610600 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b91726-269hk" Sep 04 22:27:44.612862 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:44.612841 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hc7x4\"" Sep 04 22:27:44.612979 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:44.612841 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 04 22:27:44.617846 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:44.617822 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b91726-269hk"] Sep 04 22:27:44.688977 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:44.688941 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3461b23-9695-43ab-871f-bb4ff991d36f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-b91726-269hk\" (UID: \"f3461b23-9695-43ab-871f-bb4ff991d36f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b91726-269hk" Sep 04 22:27:44.790229 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:44.790187 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3461b23-9695-43ab-871f-bb4ff991d36f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-b91726-269hk\" (UID: \"f3461b23-9695-43ab-871f-bb4ff991d36f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b91726-269hk" Sep 04 22:27:44.790846 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:44.790827 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3461b23-9695-43ab-871f-bb4ff991d36f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-b91726-269hk\" (UID: \"f3461b23-9695-43ab-871f-bb4ff991d36f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b91726-269hk" Sep 04 22:27:44.922002 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:44.921916 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b91726-269hk" Sep 04 22:27:45.196823 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:45.196799 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b91726-269hk"] Sep 04 22:27:45.198441 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:27:45.198416 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3461b23_9695_43ab_871f_bb4ff991d36f.slice/crio-210e7bbe254d9b2a752f77e03a1ae96321ab97e3138d50ccc91749ead6414001 WatchSource:0}: Error finding container 210e7bbe254d9b2a752f77e03a1ae96321ab97e3138d50ccc91749ead6414001: Status 404 returned error can't find the container with id 210e7bbe254d9b2a752f77e03a1ae96321ab97e3138d50ccc91749ead6414001 Sep 04 22:27:46.107027 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:46.106988 2574 generic.go:356] "Generic (PLEG): container finished" podID="f3461b23-9695-43ab-871f-bb4ff991d36f" containerID="48fca6171c402cd0754a7807d822e2b2d81c3b65fcf53e1e02f307bd859503c3" exitCode=0 Sep 04 22:27:46.107398 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:46.107077 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b91726-269hk" event={"ID":"f3461b23-9695-43ab-871f-bb4ff991d36f","Type":"ContainerDied","Data":"48fca6171c402cd0754a7807d822e2b2d81c3b65fcf53e1e02f307bd859503c3"} Sep 04 22:27:46.107398 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:46.107132 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b91726-269hk" event={"ID":"f3461b23-9695-43ab-871f-bb4ff991d36f","Type":"ContainerStarted","Data":"210e7bbe254d9b2a752f77e03a1ae96321ab97e3138d50ccc91749ead6414001"} Sep 04 22:27:47.250212 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:47.250185 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b91726-269hk" Sep 04 22:27:47.310789 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:47.310754 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3461b23-9695-43ab-871f-bb4ff991d36f-maas-api-service-ca\") pod \"f3461b23-9695-43ab-871f-bb4ff991d36f\" (UID: \"f3461b23-9695-43ab-871f-bb4ff991d36f\") " Sep 04 22:27:47.311108 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:47.311084 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3461b23-9695-43ab-871f-bb4ff991d36f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f3461b23-9695-43ab-871f-bb4ff991d36f" (UID: "f3461b23-9695-43ab-871f-bb4ff991d36f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:27:47.411716 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:47.411628 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3461b23-9695-43ab-871f-bb4ff991d36f-maas-api-service-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:27:48.120700 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:48.120668 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b91726-269hk" Sep 04 22:27:48.120874 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:48.120705 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b91726-269hk" event={"ID":"f3461b23-9695-43ab-871f-bb4ff991d36f","Type":"ContainerDied","Data":"210e7bbe254d9b2a752f77e03a1ae96321ab97e3138d50ccc91749ead6414001"} Sep 04 22:27:48.120874 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:48.120754 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="210e7bbe254d9b2a752f77e03a1ae96321ab97e3138d50ccc91749ead6414001" Sep 04 22:27:49.702213 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:49.702174 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-84624e97-b64ff55bf-hcwlf"] Sep 04 22:27:49.702693 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:49.702392 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-84624e97-b64ff55bf-hcwlf" podUID="f04e7dfa-e9fb-48b4-bf1d-3a0e5d45f75a" containerName="payload-processing" containerID="cri-o://6913fbbd5f72db6bfa3a356d2c1e756c57ac3b26b48c2c4e492780533385201c" gracePeriod=30 Sep 04 22:27:49.708273 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:49.708246 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-84624e97-7d76d9cbf4-nxvt2"] Sep 04 22:27:49.708770 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:49.708740 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-84624e97-7d76d9cbf4-nxvt2" podUID="714f5646-7ac5-4c59-855c-f0dc9ee82833" containerName="payload-pre-processing" containerID="cri-o://db0ae305c0c4b0fe8e2afec308d3e6a5ad5fe64b64ded04ac1d98c146ad4d531" gracePeriod=30 Sep 04 22:27:49.886691 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:49.886660 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84624e97-gkrv6"] Sep 04 22:27:49.890229 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:49.890202 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-84624e97-gkrv6"] Sep 04 22:27:49.904773 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:49.904737 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-84624e97-7d76d9cbf4-nxvt2" podUID="714f5646-7ac5-4c59-855c-f0dc9ee82833" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.56:9004: connect: connection refused" Sep 04 22:27:49.989814 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:49.989791 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-84624e97-b64ff55bf-hcwlf" Sep 04 22:27:49.992672 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:49.992651 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-84624e97-7d76d9cbf4-nxvt2" Sep 04 22:27:50.033787 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:50.033760 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/714f5646-7ac5-4c59-855c-f0dc9ee82833-plugins-config-volume\") pod \"714f5646-7ac5-4c59-855c-f0dc9ee82833\" (UID: \"714f5646-7ac5-4c59-855c-f0dc9ee82833\") " Sep 04 22:27:50.033945 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:50.033813 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5fccp\" (UniqueName: \"kubernetes.io/projected/f04e7dfa-e9fb-48b4-bf1d-3a0e5d45f75a-kube-api-access-5fccp\") pod \"f04e7dfa-e9fb-48b4-bf1d-3a0e5d45f75a\" (UID: \"f04e7dfa-e9fb-48b4-bf1d-3a0e5d45f75a\") " Sep 04 22:27:50.033945 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:50.033839 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4kjt\" (UniqueName: \"kubernetes.io/projected/714f5646-7ac5-4c59-855c-f0dc9ee82833-kube-api-access-f4kjt\") pod \"714f5646-7ac5-4c59-855c-f0dc9ee82833\" (UID: \"714f5646-7ac5-4c59-855c-f0dc9ee82833\") " Sep 04 22:27:50.034053 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:50.033953 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f04e7dfa-e9fb-48b4-bf1d-3a0e5d45f75a-plugins-config-volume\") pod \"f04e7dfa-e9fb-48b4-bf1d-3a0e5d45f75a\" (UID: \"f04e7dfa-e9fb-48b4-bf1d-3a0e5d45f75a\") " Sep 04 22:27:50.034262 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:50.034236 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/714f5646-7ac5-4c59-855c-f0dc9ee82833-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "714f5646-7ac5-4c59-855c-f0dc9ee82833" (UID: "714f5646-7ac5-4c59-855c-f0dc9ee82833"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:27:50.034388 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:50.034360 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f04e7dfa-e9fb-48b4-bf1d-3a0e5d45f75a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f04e7dfa-e9fb-48b4-bf1d-3a0e5d45f75a" (UID: "f04e7dfa-e9fb-48b4-bf1d-3a0e5d45f75a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:27:50.035982 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:50.035950 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f04e7dfa-e9fb-48b4-bf1d-3a0e5d45f75a-kube-api-access-5fccp" (OuterVolumeSpecName: "kube-api-access-5fccp") pod "f04e7dfa-e9fb-48b4-bf1d-3a0e5d45f75a" (UID: "f04e7dfa-e9fb-48b4-bf1d-3a0e5d45f75a"). InnerVolumeSpecName "kube-api-access-5fccp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:27:50.035982 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:50.035963 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/714f5646-7ac5-4c59-855c-f0dc9ee82833-kube-api-access-f4kjt" (OuterVolumeSpecName: "kube-api-access-f4kjt") pod "714f5646-7ac5-4c59-855c-f0dc9ee82833" (UID: "714f5646-7ac5-4c59-855c-f0dc9ee82833"). InnerVolumeSpecName "kube-api-access-f4kjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:27:50.133260 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:50.133228 2574 generic.go:356] "Generic (PLEG): container finished" podID="f04e7dfa-e9fb-48b4-bf1d-3a0e5d45f75a" containerID="6913fbbd5f72db6bfa3a356d2c1e756c57ac3b26b48c2c4e492780533385201c" exitCode=0 Sep 04 22:27:50.133416 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:50.133272 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-84624e97-b64ff55bf-hcwlf" Sep 04 22:27:50.133416 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:50.133319 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-84624e97-b64ff55bf-hcwlf" event={"ID":"f04e7dfa-e9fb-48b4-bf1d-3a0e5d45f75a","Type":"ContainerDied","Data":"6913fbbd5f72db6bfa3a356d2c1e756c57ac3b26b48c2c4e492780533385201c"} Sep 04 22:27:50.133416 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:50.133377 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-84624e97-b64ff55bf-hcwlf" event={"ID":"f04e7dfa-e9fb-48b4-bf1d-3a0e5d45f75a","Type":"ContainerDied","Data":"5bff90223e834c5fdd2341b19a478f54db39327c56c57bbf7c597796c18fd852"} Sep 04 22:27:50.133416 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:50.133406 2574 scope.go:117] "RemoveContainer" containerID="6913fbbd5f72db6bfa3a356d2c1e756c57ac3b26b48c2c4e492780533385201c" Sep 04 22:27:50.134747 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:50.134703 2574 generic.go:356] "Generic (PLEG): container finished" podID="714f5646-7ac5-4c59-855c-f0dc9ee82833" containerID="db0ae305c0c4b0fe8e2afec308d3e6a5ad5fe64b64ded04ac1d98c146ad4d531" exitCode=0 Sep 04 22:27:50.134866 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:50.134747 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fccp\" (UniqueName: \"kubernetes.io/projected/f04e7dfa-e9fb-48b4-bf1d-3a0e5d45f75a-kube-api-access-5fccp\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:27:50.134866 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:50.134757 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-84624e97-7d76d9cbf4-nxvt2" Sep 04 22:27:50.134866 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:50.134777 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4kjt\" (UniqueName: \"kubernetes.io/projected/714f5646-7ac5-4c59-855c-f0dc9ee82833-kube-api-access-f4kjt\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:27:50.134866 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:50.134794 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f04e7dfa-e9fb-48b4-bf1d-3a0e5d45f75a-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:27:50.134866 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:50.134808 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/714f5646-7ac5-4c59-855c-f0dc9ee82833-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:27:50.134866 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:50.134740 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-84624e97-7d76d9cbf4-nxvt2" event={"ID":"714f5646-7ac5-4c59-855c-f0dc9ee82833","Type":"ContainerDied","Data":"db0ae305c0c4b0fe8e2afec308d3e6a5ad5fe64b64ded04ac1d98c146ad4d531"} Sep 04 22:27:50.134866 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:50.134856 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-84624e97-7d76d9cbf4-nxvt2" event={"ID":"714f5646-7ac5-4c59-855c-f0dc9ee82833","Type":"ContainerDied","Data":"183d80dabeb874079ff7e268d1b358a0db872e9dd301d4738dcaaa098adcd36d"} Sep 04 22:27:50.142860 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:50.142837 2574 scope.go:117] "RemoveContainer" containerID="6913fbbd5f72db6bfa3a356d2c1e756c57ac3b26b48c2c4e492780533385201c" Sep 04 22:27:50.143134 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:27:50.143118 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6913fbbd5f72db6bfa3a356d2c1e756c57ac3b26b48c2c4e492780533385201c\": container with ID starting with 6913fbbd5f72db6bfa3a356d2c1e756c57ac3b26b48c2c4e492780533385201c not found: ID does not exist" containerID="6913fbbd5f72db6bfa3a356d2c1e756c57ac3b26b48c2c4e492780533385201c" Sep 04 22:27:50.143208 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:50.143142 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6913fbbd5f72db6bfa3a356d2c1e756c57ac3b26b48c2c4e492780533385201c"} err="failed to get container status \"6913fbbd5f72db6bfa3a356d2c1e756c57ac3b26b48c2c4e492780533385201c\": rpc error: code = NotFound desc = could not find container \"6913fbbd5f72db6bfa3a356d2c1e756c57ac3b26b48c2c4e492780533385201c\": container with ID starting with 6913fbbd5f72db6bfa3a356d2c1e756c57ac3b26b48c2c4e492780533385201c not found: ID does not exist" Sep 04 22:27:50.143208 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:50.143159 2574 scope.go:117] "RemoveContainer" containerID="db0ae305c0c4b0fe8e2afec308d3e6a5ad5fe64b64ded04ac1d98c146ad4d531" Sep 04 22:27:50.151018 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:50.151000 2574 scope.go:117] "RemoveContainer" containerID="db0ae305c0c4b0fe8e2afec308d3e6a5ad5fe64b64ded04ac1d98c146ad4d531" Sep 04 22:27:50.151275 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:27:50.151255 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db0ae305c0c4b0fe8e2afec308d3e6a5ad5fe64b64ded04ac1d98c146ad4d531\": container with ID starting with db0ae305c0c4b0fe8e2afec308d3e6a5ad5fe64b64ded04ac1d98c146ad4d531 not found: ID does not exist" containerID="db0ae305c0c4b0fe8e2afec308d3e6a5ad5fe64b64ded04ac1d98c146ad4d531" Sep 04 22:27:50.151339 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:50.151286 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db0ae305c0c4b0fe8e2afec308d3e6a5ad5fe64b64ded04ac1d98c146ad4d531"} err="failed to get container status \"db0ae305c0c4b0fe8e2afec308d3e6a5ad5fe64b64ded04ac1d98c146ad4d531\": rpc error: code = NotFound desc = could not find container \"db0ae305c0c4b0fe8e2afec308d3e6a5ad5fe64b64ded04ac1d98c146ad4d531\": container with ID starting with db0ae305c0c4b0fe8e2afec308d3e6a5ad5fe64b64ded04ac1d98c146ad4d531 not found: ID does not exist" Sep 04 22:27:50.163980 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:50.162274 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-84624e97-b64ff55bf-hcwlf"] Sep 04 22:27:50.166864 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:50.166837 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-84624e97-b64ff55bf-hcwlf"] Sep 04 22:27:50.180210 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:50.180189 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-84624e97-7d76d9cbf4-nxvt2"] Sep 04 22:27:50.182446 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:50.182426 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-84624e97-7d76d9cbf4-nxvt2"] Sep 04 22:27:50.715624 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:50.715591 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65ffe5b4-11c0-4cb0-a1df-b68ca7b32f0f" path="/var/lib/kubelet/pods/65ffe5b4-11c0-4cb0-a1df-b68ca7b32f0f/volumes" Sep 04 22:27:50.716114 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:50.716094 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="714f5646-7ac5-4c59-855c-f0dc9ee82833" path="/var/lib/kubelet/pods/714f5646-7ac5-4c59-855c-f0dc9ee82833/volumes" Sep 04 22:27:50.716465 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:50.716448 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f04e7dfa-e9fb-48b4-bf1d-3a0e5d45f75a" path="/var/lib/kubelet/pods/f04e7dfa-e9fb-48b4-bf1d-3a0e5d45f75a/volumes" Sep 04 22:27:50.904624 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:50.904581 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-84624e97-b64ff55bf-hcwlf" podUID="f04e7dfa-e9fb-48b4-bf1d-3a0e5d45f75a" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.57:9004: i/o timeout" Sep 04 22:27:55.497167 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:55.497129 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b91726-54c9fc98f7-h72w7"] Sep 04 22:27:55.497664 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:55.497381 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-b91726-54c9fc98f7-h72w7" podUID="79e9aa33-2ef0-41cd-a493-2c7223450768" containerName="payload-processing" containerID="cri-o://73ed7c1c6d965d11a869080bc78c2bcb825cbf1864505c745d369c828a902342" gracePeriod=30 Sep 04 22:27:55.507443 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:55.507413 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b91726-7c4fb4db4b-ds7vd"] Sep 04 22:27:55.508395 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:55.508107 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b91726-7c4fb4db4b-ds7vd" podUID="17b0b736-24b5-4e85-b858-c4968c2e9b04" containerName="payload-pre-processing" containerID="cri-o://817ca08f89dbeaea522b6e4c3a599b5ccaa6a7fe6f5fd78bb90f1b69b6b49b0a" gracePeriod=30 Sep 04 22:27:55.680594 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:55.680511 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b91726-269hk"] Sep 04 22:27:55.684147 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:55.684114 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b91726-269hk"] Sep 04 22:27:55.788633 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:55.788609 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b91726-54c9fc98f7-h72w7" Sep 04 22:27:55.791468 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:55.791443 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b91726-7c4fb4db4b-ds7vd" Sep 04 22:27:55.890266 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:55.890171 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79e9aa33-2ef0-41cd-a493-2c7223450768-plugins-config-volume\") pod \"79e9aa33-2ef0-41cd-a493-2c7223450768\" (UID: \"79e9aa33-2ef0-41cd-a493-2c7223450768\") " Sep 04 22:27:55.890266 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:55.890207 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbnpf\" (UniqueName: \"kubernetes.io/projected/79e9aa33-2ef0-41cd-a493-2c7223450768-kube-api-access-xbnpf\") pod \"79e9aa33-2ef0-41cd-a493-2c7223450768\" (UID: \"79e9aa33-2ef0-41cd-a493-2c7223450768\") " Sep 04 22:27:55.890266 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:55.890259 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17b0b736-24b5-4e85-b858-c4968c2e9b04-plugins-config-volume\") pod \"17b0b736-24b5-4e85-b858-c4968c2e9b04\" (UID: \"17b0b736-24b5-4e85-b858-c4968c2e9b04\") " Sep 04 22:27:55.890601 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:55.890322 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5rrt\" (UniqueName: \"kubernetes.io/projected/17b0b736-24b5-4e85-b858-c4968c2e9b04-kube-api-access-m5rrt\") pod \"17b0b736-24b5-4e85-b858-c4968c2e9b04\" (UID: \"17b0b736-24b5-4e85-b858-c4968c2e9b04\") " Sep 04 22:27:55.890673 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:55.890630 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17b0b736-24b5-4e85-b858-c4968c2e9b04-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "17b0b736-24b5-4e85-b858-c4968c2e9b04" (UID: "17b0b736-24b5-4e85-b858-c4968c2e9b04"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:27:55.890673 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:55.890651 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79e9aa33-2ef0-41cd-a493-2c7223450768-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "79e9aa33-2ef0-41cd-a493-2c7223450768" (UID: "79e9aa33-2ef0-41cd-a493-2c7223450768"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:27:55.892472 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:55.892450 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e9aa33-2ef0-41cd-a493-2c7223450768-kube-api-access-xbnpf" (OuterVolumeSpecName: "kube-api-access-xbnpf") pod "79e9aa33-2ef0-41cd-a493-2c7223450768" (UID: "79e9aa33-2ef0-41cd-a493-2c7223450768"). InnerVolumeSpecName "kube-api-access-xbnpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:27:55.892576 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:55.892498 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17b0b736-24b5-4e85-b858-c4968c2e9b04-kube-api-access-m5rrt" (OuterVolumeSpecName: "kube-api-access-m5rrt") pod "17b0b736-24b5-4e85-b858-c4968c2e9b04" (UID: "17b0b736-24b5-4e85-b858-c4968c2e9b04"). InnerVolumeSpecName "kube-api-access-m5rrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:27:55.991151 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:55.991114 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79e9aa33-2ef0-41cd-a493-2c7223450768-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:27:55.991151 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:55.991145 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbnpf\" (UniqueName: \"kubernetes.io/projected/79e9aa33-2ef0-41cd-a493-2c7223450768-kube-api-access-xbnpf\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:27:55.991151 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:55.991155 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17b0b736-24b5-4e85-b858-c4968c2e9b04-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:27:55.991379 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:55.991165 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5rrt\" (UniqueName: \"kubernetes.io/projected/17b0b736-24b5-4e85-b858-c4968c2e9b04-kube-api-access-m5rrt\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:27:56.170180 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:56.170091 2574 generic.go:356] "Generic (PLEG): container finished" podID="17b0b736-24b5-4e85-b858-c4968c2e9b04" containerID="817ca08f89dbeaea522b6e4c3a599b5ccaa6a7fe6f5fd78bb90f1b69b6b49b0a" exitCode=0 Sep 04 22:27:56.170180 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:56.170135 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b91726-7c4fb4db4b-ds7vd" Sep 04 22:27:56.170396 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:56.170190 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b91726-7c4fb4db4b-ds7vd" event={"ID":"17b0b736-24b5-4e85-b858-c4968c2e9b04","Type":"ContainerDied","Data":"817ca08f89dbeaea522b6e4c3a599b5ccaa6a7fe6f5fd78bb90f1b69b6b49b0a"} Sep 04 22:27:56.170396 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:56.170235 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b91726-7c4fb4db4b-ds7vd" event={"ID":"17b0b736-24b5-4e85-b858-c4968c2e9b04","Type":"ContainerDied","Data":"e0f76df3f82d4accf1afaf34d0c8d2406a315ccbaa10b7373411e5977a93b6c3"} Sep 04 22:27:56.170396 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:56.170258 2574 scope.go:117] "RemoveContainer" containerID="817ca08f89dbeaea522b6e4c3a599b5ccaa6a7fe6f5fd78bb90f1b69b6b49b0a" Sep 04 22:27:56.171725 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:56.171700 2574 generic.go:356] "Generic (PLEG): container finished" podID="79e9aa33-2ef0-41cd-a493-2c7223450768" containerID="73ed7c1c6d965d11a869080bc78c2bcb825cbf1864505c745d369c828a902342" exitCode=0 Sep 04 22:27:56.171825 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:56.171742 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b91726-54c9fc98f7-h72w7" Sep 04 22:27:56.171889 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:56.171819 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b91726-54c9fc98f7-h72w7" event={"ID":"79e9aa33-2ef0-41cd-a493-2c7223450768","Type":"ContainerDied","Data":"73ed7c1c6d965d11a869080bc78c2bcb825cbf1864505c745d369c828a902342"} Sep 04 22:27:56.171889 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:56.171865 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b91726-54c9fc98f7-h72w7" event={"ID":"79e9aa33-2ef0-41cd-a493-2c7223450768","Type":"ContainerDied","Data":"47c518d80f0fb47115ad1c32990e0b011ee34718d745bd95cd2732443933e5ea"} Sep 04 22:27:56.182761 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:56.182737 2574 scope.go:117] "RemoveContainer" containerID="817ca08f89dbeaea522b6e4c3a599b5ccaa6a7fe6f5fd78bb90f1b69b6b49b0a" Sep 04 22:27:56.183384 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:27:56.183364 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"817ca08f89dbeaea522b6e4c3a599b5ccaa6a7fe6f5fd78bb90f1b69b6b49b0a\": container with ID starting with 817ca08f89dbeaea522b6e4c3a599b5ccaa6a7fe6f5fd78bb90f1b69b6b49b0a not found: ID does not exist" containerID="817ca08f89dbeaea522b6e4c3a599b5ccaa6a7fe6f5fd78bb90f1b69b6b49b0a" Sep 04 22:27:56.183457 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:56.183394 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"817ca08f89dbeaea522b6e4c3a599b5ccaa6a7fe6f5fd78bb90f1b69b6b49b0a"} err="failed to get container status \"817ca08f89dbeaea522b6e4c3a599b5ccaa6a7fe6f5fd78bb90f1b69b6b49b0a\": rpc error: code = NotFound desc = could not find container \"817ca08f89dbeaea522b6e4c3a599b5ccaa6a7fe6f5fd78bb90f1b69b6b49b0a\": container with ID starting with 817ca08f89dbeaea522b6e4c3a599b5ccaa6a7fe6f5fd78bb90f1b69b6b49b0a not found: ID does not exist" Sep 04 22:27:56.183457 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:56.183420 2574 scope.go:117] "RemoveContainer" containerID="73ed7c1c6d965d11a869080bc78c2bcb825cbf1864505c745d369c828a902342" Sep 04 22:27:56.191978 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:56.191955 2574 scope.go:117] "RemoveContainer" containerID="73ed7c1c6d965d11a869080bc78c2bcb825cbf1864505c745d369c828a902342" Sep 04 22:27:56.192266 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:27:56.192246 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73ed7c1c6d965d11a869080bc78c2bcb825cbf1864505c745d369c828a902342\": container with ID starting with 73ed7c1c6d965d11a869080bc78c2bcb825cbf1864505c745d369c828a902342 not found: ID does not exist" containerID="73ed7c1c6d965d11a869080bc78c2bcb825cbf1864505c745d369c828a902342" Sep 04 22:27:56.192334 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:56.192274 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73ed7c1c6d965d11a869080bc78c2bcb825cbf1864505c745d369c828a902342"} err="failed to get container status \"73ed7c1c6d965d11a869080bc78c2bcb825cbf1864505c745d369c828a902342\": rpc error: code = NotFound desc = could not find container \"73ed7c1c6d965d11a869080bc78c2bcb825cbf1864505c745d369c828a902342\": container with ID starting with 73ed7c1c6d965d11a869080bc78c2bcb825cbf1864505c745d369c828a902342 not found: ID does not exist" Sep 04 22:27:56.195514 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:56.195489 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b91726-7c4fb4db4b-ds7vd"] Sep 04 22:27:56.201542 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:56.201521 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b91726-7c4fb4db4b-ds7vd"] Sep 04 22:27:56.213995 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:56.213971 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b91726-54c9fc98f7-h72w7"] Sep 04 22:27:56.215878 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:56.215860 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b91726-54c9fc98f7-h72w7"] Sep 04 22:27:56.716148 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:56.716117 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17b0b736-24b5-4e85-b858-c4968c2e9b04" path="/var/lib/kubelet/pods/17b0b736-24b5-4e85-b858-c4968c2e9b04/volumes" Sep 04 22:27:56.716497 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:56.716447 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79e9aa33-2ef0-41cd-a493-2c7223450768" path="/var/lib/kubelet/pods/79e9aa33-2ef0-41cd-a493-2c7223450768/volumes" Sep 04 22:27:56.716741 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:27:56.716729 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3461b23-9695-43ab-871f-bb4ff991d36f" path="/var/lib/kubelet/pods/f3461b23-9695-43ab-871f-bb4ff991d36f/volumes" Sep 04 22:28:04.416054 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:04.416018 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d9f7ae-phz9s"] Sep 04 22:28:04.416416 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:04.416333 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="714f5646-7ac5-4c59-855c-f0dc9ee82833" containerName="payload-pre-processing" Sep 04 22:28:04.416416 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:04.416343 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="714f5646-7ac5-4c59-855c-f0dc9ee82833" containerName="payload-pre-processing" Sep 04 22:28:04.416416 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:04.416351 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f04e7dfa-e9fb-48b4-bf1d-3a0e5d45f75a" containerName="payload-processing" Sep 04 22:28:04.416416 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:04.416358 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f04e7dfa-e9fb-48b4-bf1d-3a0e5d45f75a" containerName="payload-processing" Sep 04 22:28:04.416416 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:04.416371 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17b0b736-24b5-4e85-b858-c4968c2e9b04" containerName="payload-pre-processing" Sep 04 22:28:04.416416 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:04.416377 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="17b0b736-24b5-4e85-b858-c4968c2e9b04" containerName="payload-pre-processing" Sep 04 22:28:04.416416 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:04.416384 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79e9aa33-2ef0-41cd-a493-2c7223450768" containerName="payload-processing" Sep 04 22:28:04.416416 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:04.416390 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="79e9aa33-2ef0-41cd-a493-2c7223450768" containerName="payload-processing" Sep 04 22:28:04.416416 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:04.416405 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3461b23-9695-43ab-871f-bb4ff991d36f" containerName="revoke-keys" Sep 04 22:28:04.416416 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:04.416410 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3461b23-9695-43ab-871f-bb4ff991d36f" containerName="revoke-keys" Sep 04 22:28:04.416754 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:04.416464 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="79e9aa33-2ef0-41cd-a493-2c7223450768" containerName="payload-processing" Sep 04 22:28:04.416754 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:04.416472 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f04e7dfa-e9fb-48b4-bf1d-3a0e5d45f75a" containerName="payload-processing" Sep 04 22:28:04.416754 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:04.416479 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3461b23-9695-43ab-871f-bb4ff991d36f" containerName="revoke-keys" Sep 04 22:28:04.416754 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:04.416488 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="17b0b736-24b5-4e85-b858-c4968c2e9b04" containerName="payload-pre-processing" Sep 04 22:28:04.416754 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:04.416493 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="714f5646-7ac5-4c59-855c-f0dc9ee82833" containerName="payload-pre-processing" Sep 04 22:28:04.420946 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:04.420930 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d9f7ae-phz9s" Sep 04 22:28:04.423292 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:04.423267 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hc7x4\"" Sep 04 22:28:04.423292 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:04.423284 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 04 22:28:04.428093 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:04.427798 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d9f7ae-phz9s"] Sep 04 22:28:04.466287 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:04.466257 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26e8d742-d2ff-47d5-b82c-969a9e0968be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d9f7ae-phz9s\" (UID: \"26e8d742-d2ff-47d5-b82c-969a9e0968be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d9f7ae-phz9s" Sep 04 22:28:04.566989 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:04.566958 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26e8d742-d2ff-47d5-b82c-969a9e0968be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d9f7ae-phz9s\" (UID: \"26e8d742-d2ff-47d5-b82c-969a9e0968be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d9f7ae-phz9s" Sep 04 22:28:04.567524 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:04.567504 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26e8d742-d2ff-47d5-b82c-969a9e0968be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d9f7ae-phz9s\" (UID: \"26e8d742-d2ff-47d5-b82c-969a9e0968be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d9f7ae-phz9s" Sep 04 22:28:04.732061 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:04.731971 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d9f7ae-phz9s" Sep 04 22:28:05.010600 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:05.006843 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d9f7ae-phz9s"] Sep 04 22:28:05.014216 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:28:05.014186 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26e8d742_d2ff_47d5_b82c_969a9e0968be.slice/crio-6ce1c90218030b41e2694220a1d68e6a1766ec6e0deb57ccc17ce79eb5a66e9a WatchSource:0}: Error finding container 6ce1c90218030b41e2694220a1d68e6a1766ec6e0deb57ccc17ce79eb5a66e9a: Status 404 returned error can't find the container with id 6ce1c90218030b41e2694220a1d68e6a1766ec6e0deb57ccc17ce79eb5a66e9a Sep 04 22:28:05.224425 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:05.224391 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d9f7ae-phz9s" event={"ID":"26e8d742-d2ff-47d5-b82c-969a9e0968be","Type":"ContainerStarted","Data":"6ce1c90218030b41e2694220a1d68e6a1766ec6e0deb57ccc17ce79eb5a66e9a"} Sep 04 22:28:06.231300 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:06.231263 2574 generic.go:356] "Generic (PLEG): container finished" podID="26e8d742-d2ff-47d5-b82c-969a9e0968be" containerID="5ffd75a03b61d9bfa2827fc07c8e1e06e953f835cbad037788f10ad8458f232f" exitCode=0 Sep 04 22:28:06.231713 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:06.231350 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d9f7ae-phz9s" event={"ID":"26e8d742-d2ff-47d5-b82c-969a9e0968be","Type":"ContainerDied","Data":"5ffd75a03b61d9bfa2827fc07c8e1e06e953f835cbad037788f10ad8458f232f"} Sep 04 22:28:07.368788 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:07.368761 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d9f7ae-phz9s" Sep 04 22:28:07.491521 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:07.491428 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26e8d742-d2ff-47d5-b82c-969a9e0968be-maas-api-service-ca\") pod \"26e8d742-d2ff-47d5-b82c-969a9e0968be\" (UID: \"26e8d742-d2ff-47d5-b82c-969a9e0968be\") " Sep 04 22:28:07.491833 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:07.491810 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26e8d742-d2ff-47d5-b82c-969a9e0968be-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "26e8d742-d2ff-47d5-b82c-969a9e0968be" (UID: "26e8d742-d2ff-47d5-b82c-969a9e0968be"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:28:07.592859 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:07.592829 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26e8d742-d2ff-47d5-b82c-969a9e0968be-maas-api-service-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:28:08.243426 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:08.243395 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d9f7ae-phz9s" Sep 04 22:28:08.243594 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:08.243429 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d9f7ae-phz9s" event={"ID":"26e8d742-d2ff-47d5-b82c-969a9e0968be","Type":"ContainerDied","Data":"6ce1c90218030b41e2694220a1d68e6a1766ec6e0deb57ccc17ce79eb5a66e9a"} Sep 04 22:28:08.243594 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:08.243470 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ce1c90218030b41e2694220a1d68e6a1766ec6e0deb57ccc17ce79eb5a66e9a" Sep 04 22:28:15.008479 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:15.008449 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d9f7ae-648c996649-ts2sr"] Sep 04 22:28:15.009059 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:15.008702 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-d9f7ae-648c996649-ts2sr" podUID="5d1fbc5c-436f-4698-a943-391d39241a02" containerName="payload-processing" containerID="cri-o://a19ce1c65d7d9e1d76583bde92c7f4faa87aca6bae5dff18905713206a07bd51" gracePeriod=30 Sep 04 22:28:15.019385 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:15.019360 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d9f7ae-56b97d44c8-pgnhl"] Sep 04 22:28:15.019675 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:15.019649 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d9f7ae-56b97d44c8-pgnhl" podUID="6b920f63-12eb-47ba-8ade-f32eb154b0a8" containerName="payload-pre-processing" containerID="cri-o://2bc41b494c676a7d39e5e2e0f118c7f2721de0e04e517c3ae195382b1fe4b0ee" gracePeriod=30 Sep 04 22:28:15.210932 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:15.210846 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d9f7ae-phz9s"] Sep 04 22:28:15.213578 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:15.213541 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d9f7ae-phz9s"] Sep 04 22:28:15.289013 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:15.288980 2574 generic.go:356] "Generic (PLEG): container finished" podID="5d1fbc5c-436f-4698-a943-391d39241a02" containerID="a19ce1c65d7d9e1d76583bde92c7f4faa87aca6bae5dff18905713206a07bd51" exitCode=0 Sep 04 22:28:15.289121 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:15.289055 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d9f7ae-648c996649-ts2sr" event={"ID":"5d1fbc5c-436f-4698-a943-391d39241a02","Type":"ContainerDied","Data":"a19ce1c65d7d9e1d76583bde92c7f4faa87aca6bae5dff18905713206a07bd51"} Sep 04 22:28:15.290749 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:15.290641 2574 generic.go:356] "Generic (PLEG): container finished" podID="6b920f63-12eb-47ba-8ade-f32eb154b0a8" containerID="2bc41b494c676a7d39e5e2e0f118c7f2721de0e04e517c3ae195382b1fe4b0ee" exitCode=0 Sep 04 22:28:15.290749 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:15.290675 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d9f7ae-56b97d44c8-pgnhl" event={"ID":"6b920f63-12eb-47ba-8ade-f32eb154b0a8","Type":"ContainerDied","Data":"2bc41b494c676a7d39e5e2e0f118c7f2721de0e04e517c3ae195382b1fe4b0ee"} Sep 04 22:28:15.306827 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:15.306808 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d9f7ae-56b97d44c8-pgnhl" Sep 04 22:28:15.309602 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:15.309586 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d9f7ae-648c996649-ts2sr" Sep 04 22:28:15.467070 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:15.467035 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d1fbc5c-436f-4698-a943-391d39241a02-plugins-config-volume\") pod \"5d1fbc5c-436f-4698-a943-391d39241a02\" (UID: \"5d1fbc5c-436f-4698-a943-391d39241a02\") " Sep 04 22:28:15.467266 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:15.467084 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cz9vt\" (UniqueName: \"kubernetes.io/projected/6b920f63-12eb-47ba-8ade-f32eb154b0a8-kube-api-access-cz9vt\") pod \"6b920f63-12eb-47ba-8ade-f32eb154b0a8\" (UID: \"6b920f63-12eb-47ba-8ade-f32eb154b0a8\") " Sep 04 22:28:15.467266 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:15.467123 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-db497\" (UniqueName: \"kubernetes.io/projected/5d1fbc5c-436f-4698-a943-391d39241a02-kube-api-access-db497\") pod \"5d1fbc5c-436f-4698-a943-391d39241a02\" (UID: \"5d1fbc5c-436f-4698-a943-391d39241a02\") " Sep 04 22:28:15.467266 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:15.467158 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b920f63-12eb-47ba-8ade-f32eb154b0a8-plugins-config-volume\") pod \"6b920f63-12eb-47ba-8ade-f32eb154b0a8\" (UID: \"6b920f63-12eb-47ba-8ade-f32eb154b0a8\") " Sep 04 22:28:15.467596 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:15.467541 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d1fbc5c-436f-4698-a943-391d39241a02-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5d1fbc5c-436f-4698-a943-391d39241a02" (UID: "5d1fbc5c-436f-4698-a943-391d39241a02"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:28:15.467714 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:15.467624 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b920f63-12eb-47ba-8ade-f32eb154b0a8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6b920f63-12eb-47ba-8ade-f32eb154b0a8" (UID: "6b920f63-12eb-47ba-8ade-f32eb154b0a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:28:15.469187 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:15.469170 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b920f63-12eb-47ba-8ade-f32eb154b0a8-kube-api-access-cz9vt" (OuterVolumeSpecName: "kube-api-access-cz9vt") pod "6b920f63-12eb-47ba-8ade-f32eb154b0a8" (UID: "6b920f63-12eb-47ba-8ade-f32eb154b0a8"). InnerVolumeSpecName "kube-api-access-cz9vt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:28:15.469272 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:15.469257 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d1fbc5c-436f-4698-a943-391d39241a02-kube-api-access-db497" (OuterVolumeSpecName: "kube-api-access-db497") pod "5d1fbc5c-436f-4698-a943-391d39241a02" (UID: "5d1fbc5c-436f-4698-a943-391d39241a02"). InnerVolumeSpecName "kube-api-access-db497". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:28:15.568522 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:15.568438 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d1fbc5c-436f-4698-a943-391d39241a02-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:28:15.568522 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:15.568467 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz9vt\" (UniqueName: \"kubernetes.io/projected/6b920f63-12eb-47ba-8ade-f32eb154b0a8-kube-api-access-cz9vt\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:28:15.568522 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:15.568480 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db497\" (UniqueName: \"kubernetes.io/projected/5d1fbc5c-436f-4698-a943-391d39241a02-kube-api-access-db497\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:28:15.568522 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:15.568489 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b920f63-12eb-47ba-8ade-f32eb154b0a8-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:28:16.297860 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:16.297833 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d9f7ae-56b97d44c8-pgnhl" Sep 04 22:28:16.297860 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:16.297840 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d9f7ae-56b97d44c8-pgnhl" event={"ID":"6b920f63-12eb-47ba-8ade-f32eb154b0a8","Type":"ContainerDied","Data":"8ce5ffcec1ef0867637874166479f5b326dac61f8170564f388d976c6862c801"} Sep 04 22:28:16.298351 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:16.297901 2574 scope.go:117] "RemoveContainer" containerID="2bc41b494c676a7d39e5e2e0f118c7f2721de0e04e517c3ae195382b1fe4b0ee" Sep 04 22:28:16.299387 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:16.299357 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d9f7ae-648c996649-ts2sr" Sep 04 22:28:16.299524 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:16.299394 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d9f7ae-648c996649-ts2sr" event={"ID":"5d1fbc5c-436f-4698-a943-391d39241a02","Type":"ContainerDied","Data":"835bd6896acd5424e3f3fb4dab59443fc055c53cbc7e04a11843d769b40bdf94"} Sep 04 22:28:16.307372 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:16.307349 2574 scope.go:117] "RemoveContainer" containerID="a19ce1c65d7d9e1d76583bde92c7f4faa87aca6bae5dff18905713206a07bd51" Sep 04 22:28:16.323142 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:16.323115 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d9f7ae-648c996649-ts2sr"] Sep 04 22:28:16.330147 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:16.330127 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d9f7ae-648c996649-ts2sr"] Sep 04 22:28:16.341917 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:16.341897 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d9f7ae-56b97d44c8-pgnhl"] Sep 04 22:28:16.349844 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:16.349822 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d9f7ae-56b97d44c8-pgnhl"] Sep 04 22:28:16.715177 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:16.715100 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26e8d742-d2ff-47d5-b82c-969a9e0968be" path="/var/lib/kubelet/pods/26e8d742-d2ff-47d5-b82c-969a9e0968be/volumes" Sep 04 22:28:16.715468 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:16.715456 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d1fbc5c-436f-4698-a943-391d39241a02" path="/var/lib/kubelet/pods/5d1fbc5c-436f-4698-a943-391d39241a02/volumes" Sep 04 22:28:16.715792 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:16.715780 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b920f63-12eb-47ba-8ade-f32eb154b0a8" path="/var/lib/kubelet/pods/6b920f63-12eb-47ba-8ade-f32eb154b0a8/volumes" Sep 04 22:28:18.498157 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:18.498126 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64b20-fb5bd6b54-fpcwb"] Sep 04 22:28:18.498660 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:18.498598 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26e8d742-d2ff-47d5-b82c-969a9e0968be" containerName="revoke-keys" Sep 04 22:28:18.498660 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:18.498617 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="26e8d742-d2ff-47d5-b82c-969a9e0968be" containerName="revoke-keys" Sep 04 22:28:18.498660 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:18.498637 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d1fbc5c-436f-4698-a943-391d39241a02" containerName="payload-processing" Sep 04 22:28:18.498660 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:18.498645 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d1fbc5c-436f-4698-a943-391d39241a02" containerName="payload-processing" Sep 04 22:28:18.498893 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:18.498665 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b920f63-12eb-47ba-8ade-f32eb154b0a8" containerName="payload-pre-processing" Sep 04 22:28:18.498893 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:18.498674 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b920f63-12eb-47ba-8ade-f32eb154b0a8" containerName="payload-pre-processing" Sep 04 22:28:18.498893 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:18.498774 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d1fbc5c-436f-4698-a943-391d39241a02" containerName="payload-processing" Sep 04 22:28:18.498893 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:18.498787 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b920f63-12eb-47ba-8ade-f32eb154b0a8" containerName="payload-pre-processing" Sep 04 22:28:18.498893 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:18.498796 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="26e8d742-d2ff-47d5-b82c-969a9e0968be" containerName="revoke-keys" Sep 04 22:28:18.503349 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:18.503327 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64b20-fb5bd6b54-fpcwb" Sep 04 22:28:18.505718 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:18.505697 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-f64b20-dockercfg-s7ntr\"" Sep 04 22:28:18.505829 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:18.505699 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-f64b20\"" Sep 04 22:28:18.516221 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:18.516198 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64b20-fb5bd6b54-fpcwb"] Sep 04 22:28:18.533047 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:18.533025 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f64b20-768cdb7d5-f5vz6"] Sep 04 22:28:18.536173 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:18.536158 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f64b20-768cdb7d5-f5vz6" Sep 04 22:28:18.545649 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:18.545628 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f64b20-768cdb7d5-f5vz6"] Sep 04 22:28:18.595416 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:18.595384 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9abe2458-1046-4ebd-8c55-108ec7c14c6d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-f64b20-fb5bd6b54-fpcwb\" (UID: \"9abe2458-1046-4ebd-8c55-108ec7c14c6d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64b20-fb5bd6b54-fpcwb" Sep 04 22:28:18.595581 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:18.595426 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4llf\" (UniqueName: \"kubernetes.io/projected/9abe2458-1046-4ebd-8c55-108ec7c14c6d-kube-api-access-w4llf\") pod \"payload-pre-processing-e2e-shared-a-w2-f64b20-fb5bd6b54-fpcwb\" (UID: \"9abe2458-1046-4ebd-8c55-108ec7c14c6d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64b20-fb5bd6b54-fpcwb" Sep 04 22:28:18.696887 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:18.696847 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktvwp\" (UniqueName: \"kubernetes.io/projected/fd097038-b010-4ea0-91c8-8ab9fe80a57e-kube-api-access-ktvwp\") pod \"payload-processing-e2e-shared-a-w2-f64b20-768cdb7d5-f5vz6\" (UID: \"fd097038-b010-4ea0-91c8-8ab9fe80a57e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f64b20-768cdb7d5-f5vz6" Sep 04 22:28:18.697058 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:18.696903 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd097038-b010-4ea0-91c8-8ab9fe80a57e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-f64b20-768cdb7d5-f5vz6\" (UID: \"fd097038-b010-4ea0-91c8-8ab9fe80a57e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f64b20-768cdb7d5-f5vz6" Sep 04 22:28:18.697058 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:18.696960 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9abe2458-1046-4ebd-8c55-108ec7c14c6d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-f64b20-fb5bd6b54-fpcwb\" (UID: \"9abe2458-1046-4ebd-8c55-108ec7c14c6d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64b20-fb5bd6b54-fpcwb" Sep 04 22:28:18.697058 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:18.697005 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4llf\" (UniqueName: \"kubernetes.io/projected/9abe2458-1046-4ebd-8c55-108ec7c14c6d-kube-api-access-w4llf\") pod \"payload-pre-processing-e2e-shared-a-w2-f64b20-fb5bd6b54-fpcwb\" (UID: \"9abe2458-1046-4ebd-8c55-108ec7c14c6d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64b20-fb5bd6b54-fpcwb" Sep 04 22:28:18.697709 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:18.697692 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9abe2458-1046-4ebd-8c55-108ec7c14c6d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-f64b20-fb5bd6b54-fpcwb\" (UID: \"9abe2458-1046-4ebd-8c55-108ec7c14c6d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64b20-fb5bd6b54-fpcwb" Sep 04 22:28:18.706049 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:18.706017 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4llf\" (UniqueName: \"kubernetes.io/projected/9abe2458-1046-4ebd-8c55-108ec7c14c6d-kube-api-access-w4llf\") pod \"payload-pre-processing-e2e-shared-a-w2-f64b20-fb5bd6b54-fpcwb\" (UID: \"9abe2458-1046-4ebd-8c55-108ec7c14c6d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64b20-fb5bd6b54-fpcwb" Sep 04 22:28:18.798104 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:18.798019 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktvwp\" (UniqueName: \"kubernetes.io/projected/fd097038-b010-4ea0-91c8-8ab9fe80a57e-kube-api-access-ktvwp\") pod \"payload-processing-e2e-shared-a-w2-f64b20-768cdb7d5-f5vz6\" (UID: \"fd097038-b010-4ea0-91c8-8ab9fe80a57e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f64b20-768cdb7d5-f5vz6" Sep 04 22:28:18.798104 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:18.798069 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd097038-b010-4ea0-91c8-8ab9fe80a57e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-f64b20-768cdb7d5-f5vz6\" (UID: \"fd097038-b010-4ea0-91c8-8ab9fe80a57e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f64b20-768cdb7d5-f5vz6" Sep 04 22:28:18.798774 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:18.798756 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd097038-b010-4ea0-91c8-8ab9fe80a57e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-f64b20-768cdb7d5-f5vz6\" (UID: \"fd097038-b010-4ea0-91c8-8ab9fe80a57e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f64b20-768cdb7d5-f5vz6" Sep 04 22:28:18.807456 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:18.807437 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktvwp\" (UniqueName: \"kubernetes.io/projected/fd097038-b010-4ea0-91c8-8ab9fe80a57e-kube-api-access-ktvwp\") pod \"payload-processing-e2e-shared-a-w2-f64b20-768cdb7d5-f5vz6\" (UID: \"fd097038-b010-4ea0-91c8-8ab9fe80a57e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f64b20-768cdb7d5-f5vz6" Sep 04 22:28:18.813302 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:18.813282 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64b20-fb5bd6b54-fpcwb" Sep 04 22:28:18.846595 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:18.846565 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f64b20-768cdb7d5-f5vz6" Sep 04 22:28:19.085718 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:19.085685 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64b20-fb5bd6b54-fpcwb"] Sep 04 22:28:19.133494 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:19.133473 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f64b20-768cdb7d5-f5vz6"] Sep 04 22:28:19.134747 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:28:19.134711 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd097038_b010_4ea0_91c8_8ab9fe80a57e.slice/crio-f2411bcd616f1b584ccfae12b734f8c1d74379db539d534e9a4b5abd1758d5dd WatchSource:0}: Error finding container f2411bcd616f1b584ccfae12b734f8c1d74379db539d534e9a4b5abd1758d5dd: Status 404 returned error can't find the container with id f2411bcd616f1b584ccfae12b734f8c1d74379db539d534e9a4b5abd1758d5dd Sep 04 22:28:19.324413 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:19.324355 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f64b20-768cdb7d5-f5vz6" event={"ID":"fd097038-b010-4ea0-91c8-8ab9fe80a57e","Type":"ContainerStarted","Data":"b96b51b92bc911b066dcdce076a97344ebdb6dd1fbc9336c462de5d800eadc86"} Sep 04 22:28:19.324701 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:19.324456 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f64b20-768cdb7d5-f5vz6" event={"ID":"fd097038-b010-4ea0-91c8-8ab9fe80a57e","Type":"ContainerStarted","Data":"f2411bcd616f1b584ccfae12b734f8c1d74379db539d534e9a4b5abd1758d5dd"} Sep 04 22:28:19.324701 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:19.324534 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f64b20-768cdb7d5-f5vz6" Sep 04 22:28:19.326301 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:19.326275 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64b20-fb5bd6b54-fpcwb" event={"ID":"9abe2458-1046-4ebd-8c55-108ec7c14c6d","Type":"ContainerStarted","Data":"cf6f78d15180594b7d850f2f5cef00dff5f3e4dc0f6be7c62ad5657e9e263201"} Sep 04 22:28:19.326418 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:19.326322 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64b20-fb5bd6b54-fpcwb" event={"ID":"9abe2458-1046-4ebd-8c55-108ec7c14c6d","Type":"ContainerStarted","Data":"ceb37ef365b4ac845631b40b2b051b0be3f712cc456b1fa69ee5b80b83c63423"} Sep 04 22:28:19.326418 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:19.326387 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64b20-fb5bd6b54-fpcwb" Sep 04 22:28:19.343950 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:19.343860 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f64b20-768cdb7d5-f5vz6" podStartSLOduration=1.343846828 podStartE2EDuration="1.343846828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:28:19.341020821 +0000 UTC m=+1203.105496447" watchObservedRunningTime="2026-09-04 22:28:19.343846828 +0000 UTC m=+1203.108322441" Sep 04 22:28:19.359335 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:19.359285 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64b20-fb5bd6b54-fpcwb" podStartSLOduration=1.359268222 podStartE2EDuration="1.359268222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:28:19.357340847 +0000 UTC m=+1203.121816460" watchObservedRunningTime="2026-09-04 22:28:19.359268222 +0000 UTC m=+1203.123743839" Sep 04 22:28:23.021322 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:23.021287 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fecd36-r5vvb"] Sep 04 22:28:23.024818 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:23.024800 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fecd36-r5vvb" Sep 04 22:28:23.027311 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:23.027286 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 04 22:28:23.027441 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:23.027289 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hc7x4\"" Sep 04 22:28:23.033027 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:23.033004 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fecd36-r5vvb"] Sep 04 22:28:23.138324 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:23.138289 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a30eded0-076b-4bab-ba06-b584f0e86dbc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fecd36-r5vvb\" (UID: \"a30eded0-076b-4bab-ba06-b584f0e86dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fecd36-r5vvb" Sep 04 22:28:23.239601 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:23.239543 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a30eded0-076b-4bab-ba06-b584f0e86dbc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fecd36-r5vvb\" (UID: \"a30eded0-076b-4bab-ba06-b584f0e86dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fecd36-r5vvb" Sep 04 22:28:23.240147 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:23.240127 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a30eded0-076b-4bab-ba06-b584f0e86dbc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fecd36-r5vvb\" (UID: \"a30eded0-076b-4bab-ba06-b584f0e86dbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fecd36-r5vvb" Sep 04 22:28:23.336974 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:23.336896 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fecd36-r5vvb" Sep 04 22:28:23.610126 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:23.610095 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fecd36-r5vvb"] Sep 04 22:28:23.612152 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:28:23.612122 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda30eded0_076b_4bab_ba06_b584f0e86dbc.slice/crio-5bf6d3efda20a77989074e247fed68c6edfeadd835fb75d6f4d6a5e9036531b5 WatchSource:0}: Error finding container 5bf6d3efda20a77989074e247fed68c6edfeadd835fb75d6f4d6a5e9036531b5: Status 404 returned error can't find the container with id 5bf6d3efda20a77989074e247fed68c6edfeadd835fb75d6f4d6a5e9036531b5 Sep 04 22:28:24.355496 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:24.355458 2574 generic.go:356] "Generic (PLEG): container finished" podID="a30eded0-076b-4bab-ba06-b584f0e86dbc" containerID="bfe71659afa5745b9b29863341babf9a59fe7a5b39edac749d2c97265391f053" exitCode=0 Sep 04 22:28:24.355868 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:24.355516 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fecd36-r5vvb" event={"ID":"a30eded0-076b-4bab-ba06-b584f0e86dbc","Type":"ContainerDied","Data":"bfe71659afa5745b9b29863341babf9a59fe7a5b39edac749d2c97265391f053"} Sep 04 22:28:24.355868 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:24.355632 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fecd36-r5vvb" event={"ID":"a30eded0-076b-4bab-ba06-b584f0e86dbc","Type":"ContainerStarted","Data":"5bf6d3efda20a77989074e247fed68c6edfeadd835fb75d6f4d6a5e9036531b5"} Sep 04 22:28:25.493619 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:25.493594 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fecd36-r5vvb" Sep 04 22:28:25.658570 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:25.658467 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a30eded0-076b-4bab-ba06-b584f0e86dbc-maas-api-service-ca\") pod \"a30eded0-076b-4bab-ba06-b584f0e86dbc\" (UID: \"a30eded0-076b-4bab-ba06-b584f0e86dbc\") " Sep 04 22:28:25.658816 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:25.658794 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a30eded0-076b-4bab-ba06-b584f0e86dbc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a30eded0-076b-4bab-ba06-b584f0e86dbc" (UID: "a30eded0-076b-4bab-ba06-b584f0e86dbc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:28:25.759342 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:25.759307 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a30eded0-076b-4bab-ba06-b584f0e86dbc-maas-api-service-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:28:26.367724 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:26.367693 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fecd36-r5vvb" Sep 04 22:28:26.367902 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:26.367735 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fecd36-r5vvb" event={"ID":"a30eded0-076b-4bab-ba06-b584f0e86dbc","Type":"ContainerDied","Data":"5bf6d3efda20a77989074e247fed68c6edfeadd835fb75d6f4d6a5e9036531b5"} Sep 04 22:28:26.367902 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:26.367781 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5bf6d3efda20a77989074e247fed68c6edfeadd835fb75d6f4d6a5e9036531b5" Sep 04 22:28:30.334222 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:30.334188 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f64b20-768cdb7d5-f5vz6" Sep 04 22:28:30.334605 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:30.334261 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64b20-fb5bd6b54-fpcwb" Sep 04 22:28:33.749778 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:33.749742 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fecd36-795b599dfc-5nhnb"] Sep 04 22:28:33.750241 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:33.749971 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-fecd36-795b599dfc-5nhnb" podUID="2f64ef12-518c-4f18-adc4-99b085c05d37" containerName="payload-processing" containerID="cri-o://7e4f3f17c1515f3de81b371e5b87004fb9bb17d8495afbae876916cdaec98d69" gracePeriod=30 Sep 04 22:28:33.758154 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:33.758006 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fecd36-8557dfccfd-gr2gt"] Sep 04 22:28:33.758229 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:33.758207 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fecd36-8557dfccfd-gr2gt" podUID="e5ec485f-1099-45ab-8af3-c0878cac4233" containerName="payload-pre-processing" containerID="cri-o://8292474bf7752f32525e95f81978ebb0a6ca5c8f6fa236073ac94f1cf3125fac" gracePeriod=30 Sep 04 22:28:33.956925 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:33.956896 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fecd36-r5vvb"] Sep 04 22:28:33.960414 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:33.960382 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fecd36-r5vvb"] Sep 04 22:28:34.031075 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:34.031043 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fecd36-795b599dfc-5nhnb" Sep 04 22:28:34.033524 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:34.033468 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fecd36-8557dfccfd-gr2gt" Sep 04 22:28:34.132140 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:34.132107 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fbpq\" (UniqueName: \"kubernetes.io/projected/e5ec485f-1099-45ab-8af3-c0878cac4233-kube-api-access-4fbpq\") pod \"e5ec485f-1099-45ab-8af3-c0878cac4233\" (UID: \"e5ec485f-1099-45ab-8af3-c0878cac4233\") " Sep 04 22:28:34.132311 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:34.132156 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f64ef12-518c-4f18-adc4-99b085c05d37-plugins-config-volume\") pod \"2f64ef12-518c-4f18-adc4-99b085c05d37\" (UID: \"2f64ef12-518c-4f18-adc4-99b085c05d37\") " Sep 04 22:28:34.132311 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:34.132221 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5ec485f-1099-45ab-8af3-c0878cac4233-plugins-config-volume\") pod \"e5ec485f-1099-45ab-8af3-c0878cac4233\" (UID: \"e5ec485f-1099-45ab-8af3-c0878cac4233\") " Sep 04 22:28:34.132311 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:34.132268 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rd8hb\" (UniqueName: \"kubernetes.io/projected/2f64ef12-518c-4f18-adc4-99b085c05d37-kube-api-access-rd8hb\") pod \"2f64ef12-518c-4f18-adc4-99b085c05d37\" (UID: \"2f64ef12-518c-4f18-adc4-99b085c05d37\") " Sep 04 22:28:34.132655 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:34.132631 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f64ef12-518c-4f18-adc4-99b085c05d37-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2f64ef12-518c-4f18-adc4-99b085c05d37" (UID: "2f64ef12-518c-4f18-adc4-99b085c05d37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:28:34.132738 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:34.132654 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5ec485f-1099-45ab-8af3-c0878cac4233-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e5ec485f-1099-45ab-8af3-c0878cac4233" (UID: "e5ec485f-1099-45ab-8af3-c0878cac4233"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:28:34.134381 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:34.134353 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ec485f-1099-45ab-8af3-c0878cac4233-kube-api-access-4fbpq" (OuterVolumeSpecName: "kube-api-access-4fbpq") pod "e5ec485f-1099-45ab-8af3-c0878cac4233" (UID: "e5ec485f-1099-45ab-8af3-c0878cac4233"). InnerVolumeSpecName "kube-api-access-4fbpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:28:34.134381 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:34.134361 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f64ef12-518c-4f18-adc4-99b085c05d37-kube-api-access-rd8hb" (OuterVolumeSpecName: "kube-api-access-rd8hb") pod "2f64ef12-518c-4f18-adc4-99b085c05d37" (UID: "2f64ef12-518c-4f18-adc4-99b085c05d37"). InnerVolumeSpecName "kube-api-access-rd8hb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:28:34.233208 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:34.233174 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5ec485f-1099-45ab-8af3-c0878cac4233-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:28:34.233208 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:34.233202 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd8hb\" (UniqueName: \"kubernetes.io/projected/2f64ef12-518c-4f18-adc4-99b085c05d37-kube-api-access-rd8hb\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:28:34.233208 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:34.233211 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fbpq\" (UniqueName: \"kubernetes.io/projected/e5ec485f-1099-45ab-8af3-c0878cac4233-kube-api-access-4fbpq\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:28:34.233428 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:34.233219 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f64ef12-518c-4f18-adc4-99b085c05d37-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:28:34.414591 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:34.414473 2574 generic.go:356] "Generic (PLEG): container finished" podID="2f64ef12-518c-4f18-adc4-99b085c05d37" containerID="7e4f3f17c1515f3de81b371e5b87004fb9bb17d8495afbae876916cdaec98d69" exitCode=0 Sep 04 22:28:34.414591 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:34.414524 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fecd36-795b599dfc-5nhnb" Sep 04 22:28:34.414591 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:34.414582 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fecd36-795b599dfc-5nhnb" event={"ID":"2f64ef12-518c-4f18-adc4-99b085c05d37","Type":"ContainerDied","Data":"7e4f3f17c1515f3de81b371e5b87004fb9bb17d8495afbae876916cdaec98d69"} Sep 04 22:28:34.414890 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:34.414638 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fecd36-795b599dfc-5nhnb" event={"ID":"2f64ef12-518c-4f18-adc4-99b085c05d37","Type":"ContainerDied","Data":"eb52a229f11db329ef3989b1700581a0e5c4ef4de42e0b0d49cc8deebbde2f53"} Sep 04 22:28:34.414890 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:34.414661 2574 scope.go:117] "RemoveContainer" containerID="7e4f3f17c1515f3de81b371e5b87004fb9bb17d8495afbae876916cdaec98d69" Sep 04 22:28:34.416096 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:34.416077 2574 generic.go:356] "Generic (PLEG): container finished" podID="e5ec485f-1099-45ab-8af3-c0878cac4233" containerID="8292474bf7752f32525e95f81978ebb0a6ca5c8f6fa236073ac94f1cf3125fac" exitCode=0 Sep 04 22:28:34.416211 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:34.416107 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fecd36-8557dfccfd-gr2gt" event={"ID":"e5ec485f-1099-45ab-8af3-c0878cac4233","Type":"ContainerDied","Data":"8292474bf7752f32525e95f81978ebb0a6ca5c8f6fa236073ac94f1cf3125fac"} Sep 04 22:28:34.416211 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:34.416131 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fecd36-8557dfccfd-gr2gt" Sep 04 22:28:34.416211 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:34.416138 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fecd36-8557dfccfd-gr2gt" event={"ID":"e5ec485f-1099-45ab-8af3-c0878cac4233","Type":"ContainerDied","Data":"ee8562b3ea9cdf01a72603b635286491dcf80e9abec275ef7bb81c18f6410193"} Sep 04 22:28:34.425591 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:34.425567 2574 scope.go:117] "RemoveContainer" containerID="7e4f3f17c1515f3de81b371e5b87004fb9bb17d8495afbae876916cdaec98d69" Sep 04 22:28:34.425909 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:28:34.425825 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e4f3f17c1515f3de81b371e5b87004fb9bb17d8495afbae876916cdaec98d69\": container with ID starting with 7e4f3f17c1515f3de81b371e5b87004fb9bb17d8495afbae876916cdaec98d69 not found: ID does not exist" containerID="7e4f3f17c1515f3de81b371e5b87004fb9bb17d8495afbae876916cdaec98d69" Sep 04 22:28:34.425909 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:34.425849 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e4f3f17c1515f3de81b371e5b87004fb9bb17d8495afbae876916cdaec98d69"} err="failed to get container status \"7e4f3f17c1515f3de81b371e5b87004fb9bb17d8495afbae876916cdaec98d69\": rpc error: code = NotFound desc = could not find container \"7e4f3f17c1515f3de81b371e5b87004fb9bb17d8495afbae876916cdaec98d69\": container with ID starting with 7e4f3f17c1515f3de81b371e5b87004fb9bb17d8495afbae876916cdaec98d69 not found: ID does not exist" Sep 04 22:28:34.425909 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:34.425866 2574 scope.go:117] "RemoveContainer" containerID="8292474bf7752f32525e95f81978ebb0a6ca5c8f6fa236073ac94f1cf3125fac" Sep 04 22:28:34.433802 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:34.433784 2574 scope.go:117] "RemoveContainer" containerID="8292474bf7752f32525e95f81978ebb0a6ca5c8f6fa236073ac94f1cf3125fac" Sep 04 22:28:34.434072 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:28:34.434047 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8292474bf7752f32525e95f81978ebb0a6ca5c8f6fa236073ac94f1cf3125fac\": container with ID starting with 8292474bf7752f32525e95f81978ebb0a6ca5c8f6fa236073ac94f1cf3125fac not found: ID does not exist" containerID="8292474bf7752f32525e95f81978ebb0a6ca5c8f6fa236073ac94f1cf3125fac" Sep 04 22:28:34.434141 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:34.434085 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8292474bf7752f32525e95f81978ebb0a6ca5c8f6fa236073ac94f1cf3125fac"} err="failed to get container status \"8292474bf7752f32525e95f81978ebb0a6ca5c8f6fa236073ac94f1cf3125fac\": rpc error: code = NotFound desc = could not find container \"8292474bf7752f32525e95f81978ebb0a6ca5c8f6fa236073ac94f1cf3125fac\": container with ID starting with 8292474bf7752f32525e95f81978ebb0a6ca5c8f6fa236073ac94f1cf3125fac not found: ID does not exist" Sep 04 22:28:34.438192 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:34.438168 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fecd36-795b599dfc-5nhnb"] Sep 04 22:28:34.444884 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:34.444865 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fecd36-795b599dfc-5nhnb"] Sep 04 22:28:34.460567 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:34.458734 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fecd36-8557dfccfd-gr2gt"] Sep 04 22:28:34.463212 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:34.463179 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fecd36-8557dfccfd-gr2gt"] Sep 04 22:28:34.715508 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:34.715429 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f64ef12-518c-4f18-adc4-99b085c05d37" path="/var/lib/kubelet/pods/2f64ef12-518c-4f18-adc4-99b085c05d37/volumes" Sep 04 22:28:34.715781 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:34.715768 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a30eded0-076b-4bab-ba06-b584f0e86dbc" path="/var/lib/kubelet/pods/a30eded0-076b-4bab-ba06-b584f0e86dbc/volumes" Sep 04 22:28:34.716071 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:34.716060 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5ec485f-1099-45ab-8af3-c0878cac4233" path="/var/lib/kubelet/pods/e5ec485f-1099-45ab-8af3-c0878cac4233/volumes" Sep 04 22:28:46.823988 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:46.823951 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bc62f0-f7b44cbbb-c69qt"] Sep 04 22:28:46.824841 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:46.824768 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f64ef12-518c-4f18-adc4-99b085c05d37" containerName="payload-processing" Sep 04 22:28:46.824841 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:46.824794 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f64ef12-518c-4f18-adc4-99b085c05d37" containerName="payload-processing" Sep 04 22:28:46.824841 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:46.824819 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a30eded0-076b-4bab-ba06-b584f0e86dbc" containerName="revoke-keys" Sep 04 22:28:46.824841 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:46.824828 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a30eded0-076b-4bab-ba06-b584f0e86dbc" containerName="revoke-keys" Sep 04 22:28:46.825091 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:46.824863 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5ec485f-1099-45ab-8af3-c0878cac4233" containerName="payload-pre-processing" Sep 04 22:28:46.825091 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:46.824873 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5ec485f-1099-45ab-8af3-c0878cac4233" containerName="payload-pre-processing" Sep 04 22:28:46.825249 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:46.825230 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5ec485f-1099-45ab-8af3-c0878cac4233" containerName="payload-pre-processing" Sep 04 22:28:46.825416 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:46.825404 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a30eded0-076b-4bab-ba06-b584f0e86dbc" containerName="revoke-keys" Sep 04 22:28:46.825502 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:46.825492 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f64ef12-518c-4f18-adc4-99b085c05d37" containerName="payload-processing" Sep 04 22:28:46.829539 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:46.829514 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bc62f0-f7b44cbbb-c69qt" Sep 04 22:28:46.832188 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:46.832167 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-bc62f0\"" Sep 04 22:28:46.840724 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:46.840692 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bc62f0-f7b44cbbb-c69qt"] Sep 04 22:28:46.852642 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:46.852616 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-bc62f0-6d7d885fcc-vp2rs"] Sep 04 22:28:46.856041 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:46.856019 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bc62f0-6d7d885fcc-vp2rs" Sep 04 22:28:46.858434 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:46.858414 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-bc62f0-dockercfg-zcjcc\"" Sep 04 22:28:46.868649 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:46.868626 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-bc62f0-6d7d885fcc-vp2rs"] Sep 04 22:28:46.935210 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:46.935175 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b4b5a55-c0e2-46d7-9806-020b3aeffdbd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-bc62f0-6d7d885fcc-vp2rs\" (UID: \"6b4b5a55-c0e2-46d7-9806-020b3aeffdbd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bc62f0-6d7d885fcc-vp2rs" Sep 04 22:28:46.935392 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:46.935221 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25007bd5-fdb2-45aa-926f-bfaac7042319-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-bc62f0-f7b44cbbb-c69qt\" (UID: \"25007bd5-fdb2-45aa-926f-bfaac7042319\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bc62f0-f7b44cbbb-c69qt" Sep 04 22:28:46.935392 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:46.935249 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj2v5\" (UniqueName: \"kubernetes.io/projected/25007bd5-fdb2-45aa-926f-bfaac7042319-kube-api-access-pj2v5\") pod \"payload-pre-processing-e2e-shared-a-w3-bc62f0-f7b44cbbb-c69qt\" (UID: \"25007bd5-fdb2-45aa-926f-bfaac7042319\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bc62f0-f7b44cbbb-c69qt" Sep 04 22:28:46.935392 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:46.935275 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn2cd\" (UniqueName: \"kubernetes.io/projected/6b4b5a55-c0e2-46d7-9806-020b3aeffdbd-kube-api-access-bn2cd\") pod \"payload-processing-e2e-shared-a-w3-bc62f0-6d7d885fcc-vp2rs\" (UID: \"6b4b5a55-c0e2-46d7-9806-020b3aeffdbd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bc62f0-6d7d885fcc-vp2rs" Sep 04 22:28:47.035831 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:47.035801 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25007bd5-fdb2-45aa-926f-bfaac7042319-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-bc62f0-f7b44cbbb-c69qt\" (UID: \"25007bd5-fdb2-45aa-926f-bfaac7042319\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bc62f0-f7b44cbbb-c69qt" Sep 04 22:28:47.036022 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:47.035840 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj2v5\" (UniqueName: \"kubernetes.io/projected/25007bd5-fdb2-45aa-926f-bfaac7042319-kube-api-access-pj2v5\") pod \"payload-pre-processing-e2e-shared-a-w3-bc62f0-f7b44cbbb-c69qt\" (UID: \"25007bd5-fdb2-45aa-926f-bfaac7042319\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bc62f0-f7b44cbbb-c69qt" Sep 04 22:28:47.036022 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:47.035862 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn2cd\" (UniqueName: \"kubernetes.io/projected/6b4b5a55-c0e2-46d7-9806-020b3aeffdbd-kube-api-access-bn2cd\") pod \"payload-processing-e2e-shared-a-w3-bc62f0-6d7d885fcc-vp2rs\" (UID: \"6b4b5a55-c0e2-46d7-9806-020b3aeffdbd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bc62f0-6d7d885fcc-vp2rs" Sep 04 22:28:47.036022 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:47.035947 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b4b5a55-c0e2-46d7-9806-020b3aeffdbd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-bc62f0-6d7d885fcc-vp2rs\" (UID: \"6b4b5a55-c0e2-46d7-9806-020b3aeffdbd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bc62f0-6d7d885fcc-vp2rs" Sep 04 22:28:47.036587 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:47.036545 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25007bd5-fdb2-45aa-926f-bfaac7042319-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-bc62f0-f7b44cbbb-c69qt\" (UID: \"25007bd5-fdb2-45aa-926f-bfaac7042319\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bc62f0-f7b44cbbb-c69qt" Sep 04 22:28:47.036688 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:47.036648 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b4b5a55-c0e2-46d7-9806-020b3aeffdbd-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-bc62f0-6d7d885fcc-vp2rs\" (UID: \"6b4b5a55-c0e2-46d7-9806-020b3aeffdbd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bc62f0-6d7d885fcc-vp2rs" Sep 04 22:28:47.050024 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:47.049991 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj2v5\" (UniqueName: \"kubernetes.io/projected/25007bd5-fdb2-45aa-926f-bfaac7042319-kube-api-access-pj2v5\") pod \"payload-pre-processing-e2e-shared-a-w3-bc62f0-f7b44cbbb-c69qt\" (UID: \"25007bd5-fdb2-45aa-926f-bfaac7042319\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bc62f0-f7b44cbbb-c69qt" Sep 04 22:28:47.050395 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:47.050378 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn2cd\" (UniqueName: \"kubernetes.io/projected/6b4b5a55-c0e2-46d7-9806-020b3aeffdbd-kube-api-access-bn2cd\") pod \"payload-processing-e2e-shared-a-w3-bc62f0-6d7d885fcc-vp2rs\" (UID: \"6b4b5a55-c0e2-46d7-9806-020b3aeffdbd\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bc62f0-6d7d885fcc-vp2rs" Sep 04 22:28:47.141114 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:47.141024 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bc62f0-f7b44cbbb-c69qt" Sep 04 22:28:47.165477 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:47.165449 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bc62f0-6d7d885fcc-vp2rs" Sep 04 22:28:47.422790 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:47.422765 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bc62f0-f7b44cbbb-c69qt"] Sep 04 22:28:47.425382 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:28:47.425357 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25007bd5_fdb2_45aa_926f_bfaac7042319.slice/crio-057fd6304320448ec2a22778dee963ec1f1ccbef953f8c7b769937f93b1916e1 WatchSource:0}: Error finding container 057fd6304320448ec2a22778dee963ec1f1ccbef953f8c7b769937f93b1916e1: Status 404 returned error can't find the container with id 057fd6304320448ec2a22778dee963ec1f1ccbef953f8c7b769937f93b1916e1 Sep 04 22:28:47.461335 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:47.461312 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-bc62f0-6d7d885fcc-vp2rs"] Sep 04 22:28:47.462852 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:28:47.462824 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b4b5a55_c0e2_46d7_9806_020b3aeffdbd.slice/crio-e5aa2e459168bb3d8bbb58b67097bb4edd04f5adfced07d25b90b5ffd3f3676d WatchSource:0}: Error finding container e5aa2e459168bb3d8bbb58b67097bb4edd04f5adfced07d25b90b5ffd3f3676d: Status 404 returned error can't find the container with id e5aa2e459168bb3d8bbb58b67097bb4edd04f5adfced07d25b90b5ffd3f3676d Sep 04 22:28:47.494623 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:47.494592 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bc62f0-6d7d885fcc-vp2rs" event={"ID":"6b4b5a55-c0e2-46d7-9806-020b3aeffdbd","Type":"ContainerStarted","Data":"e5aa2e459168bb3d8bbb58b67097bb4edd04f5adfced07d25b90b5ffd3f3676d"} Sep 04 22:28:47.496610 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:47.496588 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bc62f0-f7b44cbbb-c69qt" event={"ID":"25007bd5-fdb2-45aa-926f-bfaac7042319","Type":"ContainerStarted","Data":"057fd6304320448ec2a22778dee963ec1f1ccbef953f8c7b769937f93b1916e1"} Sep 04 22:28:47.859690 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:47.859656 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a12dcf-fcb99bc45-56zrv"] Sep 04 22:28:47.862958 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:47.862940 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a12dcf-fcb99bc45-56zrv" Sep 04 22:28:47.865251 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:47.865226 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-a12dcf-dockercfg-5mkns\"" Sep 04 22:28:47.865361 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:47.865282 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-a12dcf\"" Sep 04 22:28:47.872976 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:47.872951 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a12dcf-fcb99bc45-56zrv"] Sep 04 22:28:47.950602 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:47.950566 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhjzx\" (UniqueName: \"kubernetes.io/projected/73657179-cc99-4ba4-acbf-7b9108cceaca-kube-api-access-qhjzx\") pod \"payload-processing-e2e-shared-b-w2-a12dcf-fcb99bc45-56zrv\" (UID: \"73657179-cc99-4ba4-acbf-7b9108cceaca\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a12dcf-fcb99bc45-56zrv" Sep 04 22:28:47.950787 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:47.950664 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73657179-cc99-4ba4-acbf-7b9108cceaca-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-a12dcf-fcb99bc45-56zrv\" (UID: \"73657179-cc99-4ba4-acbf-7b9108cceaca\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a12dcf-fcb99bc45-56zrv" Sep 04 22:28:48.051590 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:48.051514 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhjzx\" (UniqueName: \"kubernetes.io/projected/73657179-cc99-4ba4-acbf-7b9108cceaca-kube-api-access-qhjzx\") pod \"payload-processing-e2e-shared-b-w2-a12dcf-fcb99bc45-56zrv\" (UID: \"73657179-cc99-4ba4-acbf-7b9108cceaca\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a12dcf-fcb99bc45-56zrv" Sep 04 22:28:48.051773 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:48.051617 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73657179-cc99-4ba4-acbf-7b9108cceaca-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-a12dcf-fcb99bc45-56zrv\" (UID: \"73657179-cc99-4ba4-acbf-7b9108cceaca\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a12dcf-fcb99bc45-56zrv" Sep 04 22:28:48.052519 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:48.052496 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73657179-cc99-4ba4-acbf-7b9108cceaca-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-a12dcf-fcb99bc45-56zrv\" (UID: \"73657179-cc99-4ba4-acbf-7b9108cceaca\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a12dcf-fcb99bc45-56zrv" Sep 04 22:28:48.061043 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:48.061019 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhjzx\" (UniqueName: \"kubernetes.io/projected/73657179-cc99-4ba4-acbf-7b9108cceaca-kube-api-access-qhjzx\") pod \"payload-processing-e2e-shared-b-w2-a12dcf-fcb99bc45-56zrv\" (UID: \"73657179-cc99-4ba4-acbf-7b9108cceaca\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a12dcf-fcb99bc45-56zrv" Sep 04 22:28:48.174607 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:48.174501 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a12dcf-fcb99bc45-56zrv" Sep 04 22:28:48.451731 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:48.451653 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a12dcf-fcb99bc45-56zrv"] Sep 04 22:28:48.455249 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:28:48.455225 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73657179_cc99_4ba4_acbf_7b9108cceaca.slice/crio-fa87583869b15938dd89ee27f157d03c168da5c6f61823f3526185b150e332f7 WatchSource:0}: Error finding container fa87583869b15938dd89ee27f157d03c168da5c6f61823f3526185b150e332f7: Status 404 returned error can't find the container with id fa87583869b15938dd89ee27f157d03c168da5c6f61823f3526185b150e332f7 Sep 04 22:28:48.504249 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:48.504218 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bc62f0-f7b44cbbb-c69qt" event={"ID":"25007bd5-fdb2-45aa-926f-bfaac7042319","Type":"ContainerStarted","Data":"25c846ef843dd5499a06a8d7db000e0e7c310666aae0d21267b511f29bc4fda1"} Sep 04 22:28:48.504366 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:48.504300 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bc62f0-f7b44cbbb-c69qt" Sep 04 22:28:48.505941 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:48.505914 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bc62f0-6d7d885fcc-vp2rs" event={"ID":"6b4b5a55-c0e2-46d7-9806-020b3aeffdbd","Type":"ContainerStarted","Data":"891a42c58225b839b10dc1d7597fa13f95b9e702e52418e37aa06bf1f197d97f"} Sep 04 22:28:48.506353 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:48.506000 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bc62f0-6d7d885fcc-vp2rs" Sep 04 22:28:48.507530 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:48.507508 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a12dcf-fcb99bc45-56zrv" event={"ID":"73657179-cc99-4ba4-acbf-7b9108cceaca","Type":"ContainerStarted","Data":"fa87583869b15938dd89ee27f157d03c168da5c6f61823f3526185b150e332f7"} Sep 04 22:28:48.526467 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:48.526411 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bc62f0-f7b44cbbb-c69qt" podStartSLOduration=2.526388603 podStartE2EDuration="2.526388603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:28:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:28:48.52252989 +0000 UTC m=+1232.287005503" watchObservedRunningTime="2026-09-04 22:28:48.526388603 +0000 UTC m=+1232.290864218" Sep 04 22:28:48.541438 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:48.541378 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bc62f0-6d7d885fcc-vp2rs" podStartSLOduration=2.541357689 podStartE2EDuration="2.541357689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:28:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:28:48.539083844 +0000 UTC m=+1232.303559470" watchObservedRunningTime="2026-09-04 22:28:48.541357689 +0000 UTC m=+1232.305833304" Sep 04 22:28:49.513934 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:49.513899 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a12dcf-fcb99bc45-56zrv" event={"ID":"73657179-cc99-4ba4-acbf-7b9108cceaca","Type":"ContainerStarted","Data":"b3214360e16d0f2650f366bfb671f10256dc130a5e5400b2c24fd99a7baa1590"} Sep 04 22:28:49.514392 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:49.514285 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a12dcf-fcb99bc45-56zrv" Sep 04 22:28:49.539486 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:49.533311 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a12dcf-fcb99bc45-56zrv" podStartSLOduration=2.53329212 podStartE2EDuration="2.53329212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:28:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:28:49.531310008 +0000 UTC m=+1233.295785617" watchObservedRunningTime="2026-09-04 22:28:49.53329212 +0000 UTC m=+1233.297767734" Sep 04 22:28:59.516114 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:59.516080 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bc62f0-f7b44cbbb-c69qt" Sep 04 22:28:59.518379 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:28:59.516269 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bc62f0-6d7d885fcc-vp2rs" Sep 04 22:29:01.528179 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:01.528148 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a12dcf-fcb99bc45-56zrv" Sep 04 22:29:15.553940 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:15.553862 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf86db-699cd877cd-h9bv7"] Sep 04 22:29:15.557269 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:15.557248 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf86db-699cd877cd-h9bv7" Sep 04 22:29:15.559623 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:15.559602 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-bf86db-dockercfg-dgwpj\"" Sep 04 22:29:15.559702 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:15.559602 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-bf86db\"" Sep 04 22:29:15.566136 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:15.566113 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bf86db-845cf69cf7-gkcfw"] Sep 04 22:29:15.569867 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:15.569847 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf86db-699cd877cd-h9bv7"] Sep 04 22:29:15.569974 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:15.569946 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bf86db-845cf69cf7-gkcfw" Sep 04 22:29:15.582060 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:15.582032 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bf86db-845cf69cf7-gkcfw"] Sep 04 22:29:15.700763 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:15.700730 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xw65\" (UniqueName: \"kubernetes.io/projected/218c9dcd-c4f3-4ed6-b3db-f12019d81e8d-kube-api-access-8xw65\") pod \"payload-pre-processing-e2e-shared-b-w3-bf86db-699cd877cd-h9bv7\" (UID: \"218c9dcd-c4f3-4ed6-b3db-f12019d81e8d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf86db-699cd877cd-h9bv7" Sep 04 22:29:15.700925 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:15.700864 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/218c9dcd-c4f3-4ed6-b3db-f12019d81e8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-bf86db-699cd877cd-h9bv7\" (UID: \"218c9dcd-c4f3-4ed6-b3db-f12019d81e8d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf86db-699cd877cd-h9bv7" Sep 04 22:29:15.701010 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:15.700932 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd2fb\" (UniqueName: \"kubernetes.io/projected/a4e03bd9-1bbc-45b4-8482-73d487025325-kube-api-access-xd2fb\") pod \"payload-processing-e2e-shared-b-w3-bf86db-845cf69cf7-gkcfw\" (UID: \"a4e03bd9-1bbc-45b4-8482-73d487025325\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bf86db-845cf69cf7-gkcfw" Sep 04 22:29:15.701010 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:15.700979 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4e03bd9-1bbc-45b4-8482-73d487025325-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-bf86db-845cf69cf7-gkcfw\" (UID: \"a4e03bd9-1bbc-45b4-8482-73d487025325\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bf86db-845cf69cf7-gkcfw" Sep 04 22:29:15.802442 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:15.802406 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/218c9dcd-c4f3-4ed6-b3db-f12019d81e8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-bf86db-699cd877cd-h9bv7\" (UID: \"218c9dcd-c4f3-4ed6-b3db-f12019d81e8d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf86db-699cd877cd-h9bv7" Sep 04 22:29:15.802679 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:15.802455 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd2fb\" (UniqueName: \"kubernetes.io/projected/a4e03bd9-1bbc-45b4-8482-73d487025325-kube-api-access-xd2fb\") pod \"payload-processing-e2e-shared-b-w3-bf86db-845cf69cf7-gkcfw\" (UID: \"a4e03bd9-1bbc-45b4-8482-73d487025325\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bf86db-845cf69cf7-gkcfw" Sep 04 22:29:15.802679 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:15.802587 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4e03bd9-1bbc-45b4-8482-73d487025325-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-bf86db-845cf69cf7-gkcfw\" (UID: \"a4e03bd9-1bbc-45b4-8482-73d487025325\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bf86db-845cf69cf7-gkcfw" Sep 04 22:29:15.802825 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:15.802728 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xw65\" (UniqueName: \"kubernetes.io/projected/218c9dcd-c4f3-4ed6-b3db-f12019d81e8d-kube-api-access-8xw65\") pod \"payload-pre-processing-e2e-shared-b-w3-bf86db-699cd877cd-h9bv7\" (UID: \"218c9dcd-c4f3-4ed6-b3db-f12019d81e8d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf86db-699cd877cd-h9bv7" Sep 04 22:29:15.803264 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:15.803240 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/218c9dcd-c4f3-4ed6-b3db-f12019d81e8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-bf86db-699cd877cd-h9bv7\" (UID: \"218c9dcd-c4f3-4ed6-b3db-f12019d81e8d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf86db-699cd877cd-h9bv7" Sep 04 22:29:15.803356 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:15.803251 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4e03bd9-1bbc-45b4-8482-73d487025325-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-bf86db-845cf69cf7-gkcfw\" (UID: \"a4e03bd9-1bbc-45b4-8482-73d487025325\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bf86db-845cf69cf7-gkcfw" Sep 04 22:29:15.811182 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:15.811128 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd2fb\" (UniqueName: \"kubernetes.io/projected/a4e03bd9-1bbc-45b4-8482-73d487025325-kube-api-access-xd2fb\") pod \"payload-processing-e2e-shared-b-w3-bf86db-845cf69cf7-gkcfw\" (UID: \"a4e03bd9-1bbc-45b4-8482-73d487025325\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bf86db-845cf69cf7-gkcfw" Sep 04 22:29:15.811264 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:15.811243 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xw65\" (UniqueName: \"kubernetes.io/projected/218c9dcd-c4f3-4ed6-b3db-f12019d81e8d-kube-api-access-8xw65\") pod \"payload-pre-processing-e2e-shared-b-w3-bf86db-699cd877cd-h9bv7\" (UID: \"218c9dcd-c4f3-4ed6-b3db-f12019d81e8d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf86db-699cd877cd-h9bv7" Sep 04 22:29:15.869055 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:15.869027 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf86db-699cd877cd-h9bv7" Sep 04 22:29:15.880115 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:15.880092 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bf86db-845cf69cf7-gkcfw" Sep 04 22:29:16.144208 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:16.144176 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf86db-699cd877cd-h9bv7"] Sep 04 22:29:16.145848 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:29:16.145824 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod218c9dcd_c4f3_4ed6_b3db_f12019d81e8d.slice/crio-4a4f42741aef9c7ac7f83a24a1c01997cf624bfb9629b1546f3d37bf451875a1 WatchSource:0}: Error finding container 4a4f42741aef9c7ac7f83a24a1c01997cf624bfb9629b1546f3d37bf451875a1: Status 404 returned error can't find the container with id 4a4f42741aef9c7ac7f83a24a1c01997cf624bfb9629b1546f3d37bf451875a1 Sep 04 22:29:16.154854 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:16.154830 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bf86db-845cf69cf7-gkcfw"] Sep 04 22:29:16.159137 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:29:16.159117 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4e03bd9_1bbc_45b4_8482_73d487025325.slice/crio-942778bfbbe42fa1e75d69ffd3456d305fce27a74384032018de3fa20702009a WatchSource:0}: Error finding container 942778bfbbe42fa1e75d69ffd3456d305fce27a74384032018de3fa20702009a: Status 404 returned error can't find the container with id 942778bfbbe42fa1e75d69ffd3456d305fce27a74384032018de3fa20702009a Sep 04 22:29:16.672131 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:16.672091 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf86db-699cd877cd-h9bv7" event={"ID":"218c9dcd-c4f3-4ed6-b3db-f12019d81e8d","Type":"ContainerStarted","Data":"aaf622c50ac25253251ac84c9ab20aeb6432e6fdfee77669041be383f55cf1cc"} Sep 04 22:29:16.672606 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:16.672159 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf86db-699cd877cd-h9bv7" event={"ID":"218c9dcd-c4f3-4ed6-b3db-f12019d81e8d","Type":"ContainerStarted","Data":"4a4f42741aef9c7ac7f83a24a1c01997cf624bfb9629b1546f3d37bf451875a1"} Sep 04 22:29:16.672606 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:16.672192 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf86db-699cd877cd-h9bv7" Sep 04 22:29:16.673817 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:16.673785 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bf86db-845cf69cf7-gkcfw" event={"ID":"a4e03bd9-1bbc-45b4-8482-73d487025325","Type":"ContainerStarted","Data":"338c917bdeb095c898ffab3a2ac5f35b8a382599eaeeea7fad915a63dca1087e"} Sep 04 22:29:16.673912 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:16.673841 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bf86db-845cf69cf7-gkcfw" event={"ID":"a4e03bd9-1bbc-45b4-8482-73d487025325","Type":"ContainerStarted","Data":"942778bfbbe42fa1e75d69ffd3456d305fce27a74384032018de3fa20702009a"} Sep 04 22:29:16.689376 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:16.689335 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf86db-699cd877cd-h9bv7" podStartSLOduration=1.689317629 podStartE2EDuration="1.689317629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:29:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:29:16.687846258 +0000 UTC m=+1260.452321871" watchObservedRunningTime="2026-09-04 22:29:16.689317629 +0000 UTC m=+1260.453793242" Sep 04 22:29:16.705807 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:16.705762 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bf86db-845cf69cf7-gkcfw" podStartSLOduration=1.705748652 podStartE2EDuration="1.705748652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:29:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:29:16.704581344 +0000 UTC m=+1260.469056962" watchObservedRunningTime="2026-09-04 22:29:16.705748652 +0000 UTC m=+1260.470224255" Sep 04 22:29:17.678852 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:17.678821 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bf86db-845cf69cf7-gkcfw" Sep 04 22:29:25.143042 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:25.142998 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3949cd16-5877d64c7d-7ql27"] Sep 04 22:29:25.149695 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:25.149672 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3949cd16-5877d64c7d-7ql27" Sep 04 22:29:25.152231 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:25.152209 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-3949cd16\"" Sep 04 22:29:25.156683 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:25.156623 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3949cd16-5877d64c7d-7ql27"] Sep 04 22:29:25.159126 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:25.159105 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3949cd16-6f5fbfbd9-pqf89"] Sep 04 22:29:25.162892 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:25.162876 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3949cd16-6f5fbfbd9-pqf89" Sep 04 22:29:25.165380 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:25.165360 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-3949cd16-dockercfg-ddbp8\"" Sep 04 22:29:25.174366 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:25.174343 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3949cd16-6f5fbfbd9-pqf89"] Sep 04 22:29:25.177932 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:25.177911 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q86gn\" (UniqueName: \"kubernetes.io/projected/025bc6e3-b614-4125-b33a-3c0da2a8ae1f-kube-api-access-q86gn\") pod \"payload-processing-e2e-mt-3949cd16-6f5fbfbd9-pqf89\" (UID: \"025bc6e3-b614-4125-b33a-3c0da2a8ae1f\") " pod="openshift-ingress/payload-processing-e2e-mt-3949cd16-6f5fbfbd9-pqf89" Sep 04 22:29:25.178029 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:25.177941 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-899kd\" (UniqueName: \"kubernetes.io/projected/f9421f27-7fed-4bff-93ac-f1e55174fd4c-kube-api-access-899kd\") pod \"payload-pre-processing-e2e-mt-3949cd16-5877d64c7d-7ql27\" (UID: \"f9421f27-7fed-4bff-93ac-f1e55174fd4c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3949cd16-5877d64c7d-7ql27" Sep 04 22:29:25.178029 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:25.177971 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9421f27-7fed-4bff-93ac-f1e55174fd4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3949cd16-5877d64c7d-7ql27\" (UID: \"f9421f27-7fed-4bff-93ac-f1e55174fd4c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3949cd16-5877d64c7d-7ql27" Sep 04 22:29:25.178112 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:25.178048 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/025bc6e3-b614-4125-b33a-3c0da2a8ae1f-plugins-config-volume\") pod \"payload-processing-e2e-mt-3949cd16-6f5fbfbd9-pqf89\" (UID: \"025bc6e3-b614-4125-b33a-3c0da2a8ae1f\") " pod="openshift-ingress/payload-processing-e2e-mt-3949cd16-6f5fbfbd9-pqf89" Sep 04 22:29:25.278855 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:25.278816 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q86gn\" (UniqueName: \"kubernetes.io/projected/025bc6e3-b614-4125-b33a-3c0da2a8ae1f-kube-api-access-q86gn\") pod \"payload-processing-e2e-mt-3949cd16-6f5fbfbd9-pqf89\" (UID: \"025bc6e3-b614-4125-b33a-3c0da2a8ae1f\") " pod="openshift-ingress/payload-processing-e2e-mt-3949cd16-6f5fbfbd9-pqf89" Sep 04 22:29:25.279025 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:25.278867 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-899kd\" (UniqueName: \"kubernetes.io/projected/f9421f27-7fed-4bff-93ac-f1e55174fd4c-kube-api-access-899kd\") pod \"payload-pre-processing-e2e-mt-3949cd16-5877d64c7d-7ql27\" (UID: \"f9421f27-7fed-4bff-93ac-f1e55174fd4c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3949cd16-5877d64c7d-7ql27" Sep 04 22:29:25.279025 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:25.278907 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9421f27-7fed-4bff-93ac-f1e55174fd4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3949cd16-5877d64c7d-7ql27\" (UID: \"f9421f27-7fed-4bff-93ac-f1e55174fd4c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3949cd16-5877d64c7d-7ql27" Sep 04 22:29:25.279025 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:25.278940 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/025bc6e3-b614-4125-b33a-3c0da2a8ae1f-plugins-config-volume\") pod \"payload-processing-e2e-mt-3949cd16-6f5fbfbd9-pqf89\" (UID: \"025bc6e3-b614-4125-b33a-3c0da2a8ae1f\") " pod="openshift-ingress/payload-processing-e2e-mt-3949cd16-6f5fbfbd9-pqf89" Sep 04 22:29:25.279714 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:25.279687 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9421f27-7fed-4bff-93ac-f1e55174fd4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3949cd16-5877d64c7d-7ql27\" (UID: \"f9421f27-7fed-4bff-93ac-f1e55174fd4c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3949cd16-5877d64c7d-7ql27" Sep 04 22:29:25.279817 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:25.279693 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/025bc6e3-b614-4125-b33a-3c0da2a8ae1f-plugins-config-volume\") pod \"payload-processing-e2e-mt-3949cd16-6f5fbfbd9-pqf89\" (UID: \"025bc6e3-b614-4125-b33a-3c0da2a8ae1f\") " pod="openshift-ingress/payload-processing-e2e-mt-3949cd16-6f5fbfbd9-pqf89" Sep 04 22:29:25.288036 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:25.288007 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-899kd\" (UniqueName: \"kubernetes.io/projected/f9421f27-7fed-4bff-93ac-f1e55174fd4c-kube-api-access-899kd\") pod \"payload-pre-processing-e2e-mt-3949cd16-5877d64c7d-7ql27\" (UID: \"f9421f27-7fed-4bff-93ac-f1e55174fd4c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3949cd16-5877d64c7d-7ql27" Sep 04 22:29:25.288535 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:25.288514 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q86gn\" (UniqueName: \"kubernetes.io/projected/025bc6e3-b614-4125-b33a-3c0da2a8ae1f-kube-api-access-q86gn\") pod \"payload-processing-e2e-mt-3949cd16-6f5fbfbd9-pqf89\" (UID: \"025bc6e3-b614-4125-b33a-3c0da2a8ae1f\") " pod="openshift-ingress/payload-processing-e2e-mt-3949cd16-6f5fbfbd9-pqf89" Sep 04 22:29:25.462457 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:25.462368 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3949cd16-5877d64c7d-7ql27" Sep 04 22:29:25.474224 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:25.474203 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3949cd16-6f5fbfbd9-pqf89" Sep 04 22:29:25.743972 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:25.743943 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3949cd16-5877d64c7d-7ql27"] Sep 04 22:29:25.744881 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:29:25.744846 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9421f27_7fed_4bff_93ac_f1e55174fd4c.slice/crio-7bb973579cb1b042bd13b10a0a6bc59c4492d87ec9ac8bdd0c6d7b03bb99672e WatchSource:0}: Error finding container 7bb973579cb1b042bd13b10a0a6bc59c4492d87ec9ac8bdd0c6d7b03bb99672e: Status 404 returned error can't find the container with id 7bb973579cb1b042bd13b10a0a6bc59c4492d87ec9ac8bdd0c6d7b03bb99672e Sep 04 22:29:25.760293 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:25.760271 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3949cd16-6f5fbfbd9-pqf89"] Sep 04 22:29:25.761994 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:29:25.761971 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod025bc6e3_b614_4125_b33a_3c0da2a8ae1f.slice/crio-e94703d50a6c68c53afdf54e8dbc69de7fb6a12f431f3b8ee925893f48262877 WatchSource:0}: Error finding container e94703d50a6c68c53afdf54e8dbc69de7fb6a12f431f3b8ee925893f48262877: Status 404 returned error can't find the container with id e94703d50a6c68c53afdf54e8dbc69de7fb6a12f431f3b8ee925893f48262877 Sep 04 22:29:26.738016 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:26.737977 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3949cd16-5877d64c7d-7ql27" event={"ID":"f9421f27-7fed-4bff-93ac-f1e55174fd4c","Type":"ContainerStarted","Data":"0d3c221ad4878b13f2ea3a7f6159a1bfed1704c637440a4f8dcd28865cc031e5"} Sep 04 22:29:26.738513 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:26.738046 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3949cd16-5877d64c7d-7ql27" Sep 04 22:29:26.738513 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:26.738073 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3949cd16-5877d64c7d-7ql27" event={"ID":"f9421f27-7fed-4bff-93ac-f1e55174fd4c","Type":"ContainerStarted","Data":"7bb973579cb1b042bd13b10a0a6bc59c4492d87ec9ac8bdd0c6d7b03bb99672e"} Sep 04 22:29:26.739616 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:26.739592 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3949cd16-6f5fbfbd9-pqf89" event={"ID":"025bc6e3-b614-4125-b33a-3c0da2a8ae1f","Type":"ContainerStarted","Data":"41167f0736aab673d97d6e00bfc083e59a02a590049667335af94cc45834a932"} Sep 04 22:29:26.739731 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:26.739636 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3949cd16-6f5fbfbd9-pqf89" event={"ID":"025bc6e3-b614-4125-b33a-3c0da2a8ae1f","Type":"ContainerStarted","Data":"e94703d50a6c68c53afdf54e8dbc69de7fb6a12f431f3b8ee925893f48262877"} Sep 04 22:29:26.739731 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:26.739714 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-3949cd16-6f5fbfbd9-pqf89" Sep 04 22:29:26.754999 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:26.754957 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-3949cd16-5877d64c7d-7ql27" podStartSLOduration=1.754943439 podStartE2EDuration="1.754943439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:29:26.753792352 +0000 UTC m=+1270.518267967" watchObservedRunningTime="2026-09-04 22:29:26.754943439 +0000 UTC m=+1270.519419089" Sep 04 22:29:26.772012 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:26.771956 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-3949cd16-6f5fbfbd9-pqf89" podStartSLOduration=1.77193507 podStartE2EDuration="1.77193507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:29:26.76961893 +0000 UTC m=+1270.534094544" watchObservedRunningTime="2026-09-04 22:29:26.77193507 +0000 UTC m=+1270.536410685" Sep 04 22:29:27.680647 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:27.680612 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf86db-699cd877cd-h9bv7" Sep 04 22:29:28.690686 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:28.690652 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bf86db-845cf69cf7-gkcfw" Sep 04 22:29:37.753124 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:37.753089 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-3949cd16-6f5fbfbd9-pqf89" Sep 04 22:29:37.753497 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:29:37.753171 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3949cd16-5877d64c7d-7ql27" Sep 04 22:30:00.163476 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:00.163440 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bc62f0-29809350-7dfkv"] Sep 04 22:30:00.168691 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:00.168660 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3949cd16-29809350-2xxtc"] Sep 04 22:30:00.168850 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:00.168770 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bc62f0-29809350-7dfkv" Sep 04 22:30:00.170973 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:00.170951 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hc7x4\"" Sep 04 22:30:00.172160 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:00.172141 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf86db-29809350-brkhj"] Sep 04 22:30:00.172269 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:00.172256 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3949cd16-29809350-2xxtc" Sep 04 22:30:00.175959 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:00.175935 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf86db-29809350-brkhj" Sep 04 22:30:00.176392 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:00.176372 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bc62f0-29809350-7dfkv"] Sep 04 22:30:00.180910 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:00.180851 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3949cd16-29809350-2xxtc"] Sep 04 22:30:00.183044 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:00.183022 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bc62f0-29809350-7dfkv" Sep 04 22:30:00.185941 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:00.185917 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf86db-29809350-brkhj"] Sep 04 22:30:00.194320 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:00.194294 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3949cd16-29809350-2xxtc" Sep 04 22:30:00.195539 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:00.195516 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a12dcf-29809350-r8mh5"] Sep 04 22:30:00.198912 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:00.198892 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a12dcf-29809350-r8mh5" Sep 04 22:30:00.210623 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:00.210594 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a12dcf-29809350-r8mh5"] Sep 04 22:30:00.240936 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:00.240221 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf86db-29809350-brkhj" Sep 04 22:30:00.255356 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:00.253181 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a12dcf-29809350-r8mh5" Sep 04 22:30:00.468831 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:00.468803 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bc62f0-29809350-7dfkv"] Sep 04 22:30:00.470427 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:30:00.470407 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48ee60f3_45e8_4b65_bf0f_abf6a231bf27.slice/crio-c662c881e86d5c7372affaf6d648507ed51d17edd62c7e504f1a6825f4355da5 WatchSource:0}: Error finding container c662c881e86d5c7372affaf6d648507ed51d17edd62c7e504f1a6825f4355da5: Status 404 returned error can't find the container with id c662c881e86d5c7372affaf6d648507ed51d17edd62c7e504f1a6825f4355da5 Sep 04 22:30:00.496358 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:00.496333 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3949cd16-29809350-2xxtc"] Sep 04 22:30:00.506805 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:30:00.506780 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod540611f6_8a21_490c_aaab_71cd768443e8.slice/crio-860c8638c0cf5d0f0285a8352f8c1c7e7b036cd11f115ef8022b1447ca9a4574 WatchSource:0}: Error finding container 860c8638c0cf5d0f0285a8352f8c1c7e7b036cd11f115ef8022b1447ca9a4574: Status 404 returned error can't find the container with id 860c8638c0cf5d0f0285a8352f8c1c7e7b036cd11f115ef8022b1447ca9a4574 Sep 04 22:30:00.544780 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:00.544756 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a12dcf-29809350-r8mh5"] Sep 04 22:30:00.546491 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:30:00.546466 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcef4ead_0bb5_4476_9873_4ef8712f4821.slice/crio-434c882ae1e3de86af3894e473995bcec2409996244030ac1c002362eddaf605 WatchSource:0}: Error finding container 434c882ae1e3de86af3894e473995bcec2409996244030ac1c002362eddaf605: Status 404 returned error can't find the container with id 434c882ae1e3de86af3894e473995bcec2409996244030ac1c002362eddaf605 Sep 04 22:30:00.585597 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:00.585570 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf86db-29809350-brkhj"] Sep 04 22:30:00.586704 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:30:00.586674 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9b5b00b_13b6_4f14_a403_2c0478abe985.slice/crio-07a7fc4c08dc780e1bd0fa47531ef26c6ff4b6a40edd44491288fa6883ae6637 WatchSource:0}: Error finding container 07a7fc4c08dc780e1bd0fa47531ef26c6ff4b6a40edd44491288fa6883ae6637: Status 404 returned error can't find the container with id 07a7fc4c08dc780e1bd0fa47531ef26c6ff4b6a40edd44491288fa6883ae6637 Sep 04 22:30:00.943345 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:00.943307 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bc62f0-29809350-7dfkv" event={"ID":"48ee60f3-45e8-4b65-bf0f-abf6a231bf27","Type":"ContainerStarted","Data":"e2cf82d9f12eec04227301ecb119339243cce49933777b650f6bbc34b7e55d24"} Sep 04 22:30:00.943529 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:00.943375 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bc62f0-29809350-7dfkv" event={"ID":"48ee60f3-45e8-4b65-bf0f-abf6a231bf27","Type":"ContainerStarted","Data":"c662c881e86d5c7372affaf6d648507ed51d17edd62c7e504f1a6825f4355da5"} Sep 04 22:30:00.944943 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:00.944920 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3949cd16-29809350-2xxtc" event={"ID":"540611f6-8a21-490c-aaab-71cd768443e8","Type":"ContainerStarted","Data":"2a915238c7fde2b026655793d7eefb1d97deb30243a4ad57989b330169af2874"} Sep 04 22:30:00.945091 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:00.944962 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3949cd16-29809350-2xxtc" event={"ID":"540611f6-8a21-490c-aaab-71cd768443e8","Type":"ContainerStarted","Data":"860c8638c0cf5d0f0285a8352f8c1c7e7b036cd11f115ef8022b1447ca9a4574"} Sep 04 22:30:00.946449 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:00.946426 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a12dcf-29809350-r8mh5" event={"ID":"fcef4ead-0bb5-4476-9873-4ef8712f4821","Type":"ContainerStarted","Data":"c56533d481687ae6b52c0f39216c8a047e563c125281f639e309d009cc64f9a4"} Sep 04 22:30:00.946579 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:00.946478 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a12dcf-29809350-r8mh5" event={"ID":"fcef4ead-0bb5-4476-9873-4ef8712f4821","Type":"ContainerStarted","Data":"434c882ae1e3de86af3894e473995bcec2409996244030ac1c002362eddaf605"} Sep 04 22:30:00.948075 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:00.948052 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf86db-29809350-brkhj" event={"ID":"e9b5b00b-13b6-4f14-a403-2c0478abe985","Type":"ContainerStarted","Data":"8a0297cf1157a369bec37b1e2996c46101a513bdb5bc8752fce21a516a2b9f1a"} Sep 04 22:30:00.948185 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:00.948093 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf86db-29809350-brkhj" event={"ID":"e9b5b00b-13b6-4f14-a403-2c0478abe985","Type":"ContainerStarted","Data":"07a7fc4c08dc780e1bd0fa47531ef26c6ff4b6a40edd44491288fa6883ae6637"} Sep 04 22:30:00.960249 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:00.960203 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bc62f0-29809350-7dfkv" podStartSLOduration=0.960188505 podStartE2EDuration="960.188505ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:30:00.958269038 +0000 UTC m=+1304.722744651" watchObservedRunningTime="2026-09-04 22:30:00.960188505 +0000 UTC m=+1304.724664163" Sep 04 22:30:00.974066 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:00.973951 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf86db-29809350-brkhj" podStartSLOduration=0.973933256 podStartE2EDuration="973.933256ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:30:00.973304173 +0000 UTC m=+1304.737779785" watchObservedRunningTime="2026-09-04 22:30:00.973933256 +0000 UTC m=+1304.738408870" Sep 04 22:30:00.989903 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:00.989840 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3949cd16-29809350-2xxtc" podStartSLOduration=0.989819518 podStartE2EDuration="989.819518ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:30:00.987185261 +0000 UTC m=+1304.751660875" watchObservedRunningTime="2026-09-04 22:30:00.989819518 +0000 UTC m=+1304.754295130" Sep 04 22:30:01.002107 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:01.002031 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a12dcf-29809350-r8mh5" podStartSLOduration=1.002014222 podStartE2EDuration="1.002014222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:30:01.001474487 +0000 UTC m=+1304.765950116" watchObservedRunningTime="2026-09-04 22:30:01.002014222 +0000 UTC m=+1304.766489840" Sep 04 22:30:01.955461 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:01.955355 2574 generic.go:356] "Generic (PLEG): container finished" podID="fcef4ead-0bb5-4476-9873-4ef8712f4821" containerID="c56533d481687ae6b52c0f39216c8a047e563c125281f639e309d009cc64f9a4" exitCode=0 Sep 04 22:30:01.955461 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:01.955395 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a12dcf-29809350-r8mh5" event={"ID":"fcef4ead-0bb5-4476-9873-4ef8712f4821","Type":"ContainerDied","Data":"c56533d481687ae6b52c0f39216c8a047e563c125281f639e309d009cc64f9a4"} Sep 04 22:30:01.957104 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:01.957084 2574 generic.go:356] "Generic (PLEG): container finished" podID="e9b5b00b-13b6-4f14-a403-2c0478abe985" containerID="8a0297cf1157a369bec37b1e2996c46101a513bdb5bc8752fce21a516a2b9f1a" exitCode=0 Sep 04 22:30:01.957228 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:01.957114 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf86db-29809350-brkhj" event={"ID":"e9b5b00b-13b6-4f14-a403-2c0478abe985","Type":"ContainerDied","Data":"8a0297cf1157a369bec37b1e2996c46101a513bdb5bc8752fce21a516a2b9f1a"} Sep 04 22:30:01.958644 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:01.958620 2574 generic.go:356] "Generic (PLEG): container finished" podID="48ee60f3-45e8-4b65-bf0f-abf6a231bf27" containerID="e2cf82d9f12eec04227301ecb119339243cce49933777b650f6bbc34b7e55d24" exitCode=0 Sep 04 22:30:01.958735 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:01.958713 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bc62f0-29809350-7dfkv" event={"ID":"48ee60f3-45e8-4b65-bf0f-abf6a231bf27","Type":"ContainerDied","Data":"e2cf82d9f12eec04227301ecb119339243cce49933777b650f6bbc34b7e55d24"} Sep 04 22:30:01.960294 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:01.960275 2574 generic.go:356] "Generic (PLEG): container finished" podID="540611f6-8a21-490c-aaab-71cd768443e8" containerID="2a915238c7fde2b026655793d7eefb1d97deb30243a4ad57989b330169af2874" exitCode=0 Sep 04 22:30:01.960399 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:01.960318 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3949cd16-29809350-2xxtc" event={"ID":"540611f6-8a21-490c-aaab-71cd768443e8","Type":"ContainerDied","Data":"2a915238c7fde2b026655793d7eefb1d97deb30243a4ad57989b330169af2874"} Sep 04 22:30:03.136036 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:03.135998 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3949cd16-29809350-2xxtc" Sep 04 22:30:03.178892 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:03.178867 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf86db-29809350-brkhj" Sep 04 22:30:03.183119 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:03.183095 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a12dcf-29809350-r8mh5" Sep 04 22:30:03.186580 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:03.186562 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bc62f0-29809350-7dfkv" Sep 04 22:30:03.975083 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:03.975046 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bc62f0-29809350-7dfkv" Sep 04 22:30:03.975083 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:03.975061 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bc62f0-29809350-7dfkv" event={"ID":"48ee60f3-45e8-4b65-bf0f-abf6a231bf27","Type":"ContainerDied","Data":"c662c881e86d5c7372affaf6d648507ed51d17edd62c7e504f1a6825f4355da5"} Sep 04 22:30:03.975315 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:03.975109 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c662c881e86d5c7372affaf6d648507ed51d17edd62c7e504f1a6825f4355da5" Sep 04 22:30:03.976532 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:03.976505 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3949cd16-29809350-2xxtc" Sep 04 22:30:03.976705 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:03.976529 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3949cd16-29809350-2xxtc" event={"ID":"540611f6-8a21-490c-aaab-71cd768443e8","Type":"ContainerDied","Data":"860c8638c0cf5d0f0285a8352f8c1c7e7b036cd11f115ef8022b1447ca9a4574"} Sep 04 22:30:03.976705 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:03.976586 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="860c8638c0cf5d0f0285a8352f8c1c7e7b036cd11f115ef8022b1447ca9a4574" Sep 04 22:30:03.977962 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:03.977942 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a12dcf-29809350-r8mh5" Sep 04 22:30:03.978075 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:03.977939 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a12dcf-29809350-r8mh5" event={"ID":"fcef4ead-0bb5-4476-9873-4ef8712f4821","Type":"ContainerDied","Data":"434c882ae1e3de86af3894e473995bcec2409996244030ac1c002362eddaf605"} Sep 04 22:30:03.978075 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:03.978064 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="434c882ae1e3de86af3894e473995bcec2409996244030ac1c002362eddaf605" Sep 04 22:30:03.979394 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:03.979376 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf86db-29809350-brkhj" Sep 04 22:30:03.979498 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:03.979373 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf86db-29809350-brkhj" event={"ID":"e9b5b00b-13b6-4f14-a403-2c0478abe985","Type":"ContainerDied","Data":"07a7fc4c08dc780e1bd0fa47531ef26c6ff4b6a40edd44491288fa6883ae6637"} Sep 04 22:30:03.979576 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:03.979494 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07a7fc4c08dc780e1bd0fa47531ef26c6ff4b6a40edd44491288fa6883ae6637" Sep 04 22:30:16.867058 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:16.867025 2574 scope.go:117] "RemoveContainer" containerID="52928bf308cbabac4a0447a63209eac0ffa3f8279e632379024f2d428b074fe4" Sep 04 22:30:16.875962 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:16.875945 2574 scope.go:117] "RemoveContainer" containerID="cb398f6d928644fedbccad354b5c6b16a702d2c9e2cf132c8db965b6453f8ca4" Sep 04 22:30:38.381895 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:38.381859 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3949cd16-mxt9n"] Sep 04 22:30:38.383222 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:38.382574 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9b5b00b-13b6-4f14-a403-2c0478abe985" containerName="cleanup" Sep 04 22:30:38.383222 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:38.382600 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9b5b00b-13b6-4f14-a403-2c0478abe985" containerName="cleanup" Sep 04 22:30:38.383222 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:38.382619 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48ee60f3-45e8-4b65-bf0f-abf6a231bf27" containerName="cleanup" Sep 04 22:30:38.383222 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:38.382628 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="48ee60f3-45e8-4b65-bf0f-abf6a231bf27" containerName="cleanup" Sep 04 22:30:38.383222 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:38.382653 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcef4ead-0bb5-4476-9873-4ef8712f4821" containerName="cleanup" Sep 04 22:30:38.383222 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:38.382662 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcef4ead-0bb5-4476-9873-4ef8712f4821" containerName="cleanup" Sep 04 22:30:38.383222 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:38.382692 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="540611f6-8a21-490c-aaab-71cd768443e8" containerName="cleanup" Sep 04 22:30:38.383222 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:38.382707 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="540611f6-8a21-490c-aaab-71cd768443e8" containerName="cleanup" Sep 04 22:30:38.383222 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:38.382825 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcef4ead-0bb5-4476-9873-4ef8712f4821" containerName="cleanup" Sep 04 22:30:38.383222 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:38.382840 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="540611f6-8a21-490c-aaab-71cd768443e8" containerName="cleanup" Sep 04 22:30:38.383222 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:38.382852 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="48ee60f3-45e8-4b65-bf0f-abf6a231bf27" containerName="cleanup" Sep 04 22:30:38.383222 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:38.382860 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9b5b00b-13b6-4f14-a403-2c0478abe985" containerName="cleanup" Sep 04 22:30:38.387986 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:38.387966 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3949cd16-mxt9n" Sep 04 22:30:38.390822 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:38.390763 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hc7x4\"" Sep 04 22:30:38.391446 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:38.391422 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 04 22:30:38.392526 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:38.392502 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3949cd16-mxt9n"] Sep 04 22:30:38.517589 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:38.517527 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e5f379e-9ddd-49fa-8d74-928387170e43-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3949cd16-mxt9n\" (UID: \"7e5f379e-9ddd-49fa-8d74-928387170e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3949cd16-mxt9n" Sep 04 22:30:38.618071 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:38.618034 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e5f379e-9ddd-49fa-8d74-928387170e43-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3949cd16-mxt9n\" (UID: \"7e5f379e-9ddd-49fa-8d74-928387170e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3949cd16-mxt9n" Sep 04 22:30:38.618925 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:38.618901 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e5f379e-9ddd-49fa-8d74-928387170e43-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3949cd16-mxt9n\" (UID: \"7e5f379e-9ddd-49fa-8d74-928387170e43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3949cd16-mxt9n" Sep 04 22:30:38.701638 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:38.701534 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3949cd16-mxt9n" Sep 04 22:30:38.980493 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:38.975583 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3949cd16-mxt9n"] Sep 04 22:30:38.982749 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:30:38.982715 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e5f379e_9ddd_49fa_8d74_928387170e43.slice/crio-351f7aaf429f340f5466bff04059367c315ca7e3732f13c6ee6d89238824f6d8 WatchSource:0}: Error finding container 351f7aaf429f340f5466bff04059367c315ca7e3732f13c6ee6d89238824f6d8: Status 404 returned error can't find the container with id 351f7aaf429f340f5466bff04059367c315ca7e3732f13c6ee6d89238824f6d8 Sep 04 22:30:39.179775 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:39.179735 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3949cd16-mxt9n" event={"ID":"7e5f379e-9ddd-49fa-8d74-928387170e43","Type":"ContainerStarted","Data":"351f7aaf429f340f5466bff04059367c315ca7e3732f13c6ee6d89238824f6d8"} Sep 04 22:30:40.186807 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:40.186771 2574 generic.go:356] "Generic (PLEG): container finished" podID="7e5f379e-9ddd-49fa-8d74-928387170e43" containerID="aed11de8689b3a3028f74e9df2b65a1028530de08bbf6cb1e25d832b087a7944" exitCode=0 Sep 04 22:30:40.187176 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:40.186862 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3949cd16-mxt9n" event={"ID":"7e5f379e-9ddd-49fa-8d74-928387170e43","Type":"ContainerDied","Data":"aed11de8689b3a3028f74e9df2b65a1028530de08bbf6cb1e25d832b087a7944"} Sep 04 22:30:41.323165 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:41.323142 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3949cd16-mxt9n" Sep 04 22:30:41.442060 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:41.442023 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e5f379e-9ddd-49fa-8d74-928387170e43-maas-api-service-ca\") pod \"7e5f379e-9ddd-49fa-8d74-928387170e43\" (UID: \"7e5f379e-9ddd-49fa-8d74-928387170e43\") " Sep 04 22:30:41.442399 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:41.442373 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e5f379e-9ddd-49fa-8d74-928387170e43-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7e5f379e-9ddd-49fa-8d74-928387170e43" (UID: "7e5f379e-9ddd-49fa-8d74-928387170e43"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:30:41.543297 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:41.543263 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e5f379e-9ddd-49fa-8d74-928387170e43-maas-api-service-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:30:42.199561 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:42.199532 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3949cd16-mxt9n" Sep 04 22:30:42.199758 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:42.199570 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3949cd16-mxt9n" event={"ID":"7e5f379e-9ddd-49fa-8d74-928387170e43","Type":"ContainerDied","Data":"351f7aaf429f340f5466bff04059367c315ca7e3732f13c6ee6d89238824f6d8"} Sep 04 22:30:42.199758 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:42.199620 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="351f7aaf429f340f5466bff04059367c315ca7e3732f13c6ee6d89238824f6d8" Sep 04 22:30:54.001573 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:54.001511 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3949cd16-29809350-2xxtc"] Sep 04 22:30:54.005433 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:54.005404 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3949cd16-29809350-2xxtc"] Sep 04 22:30:54.031657 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:54.031619 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3949cd16-6f5fbfbd9-pqf89"] Sep 04 22:30:54.031928 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:54.031897 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-3949cd16-6f5fbfbd9-pqf89" podUID="025bc6e3-b614-4125-b33a-3c0da2a8ae1f" containerName="payload-processing" containerID="cri-o://41167f0736aab673d97d6e00bfc083e59a02a590049667335af94cc45834a932" gracePeriod=30 Sep 04 22:30:54.042675 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:54.042635 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3949cd16-5877d64c7d-7ql27"] Sep 04 22:30:54.043353 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:54.043053 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-3949cd16-5877d64c7d-7ql27" podUID="f9421f27-7fed-4bff-93ac-f1e55174fd4c" containerName="payload-pre-processing" containerID="cri-o://0d3c221ad4878b13f2ea3a7f6159a1bfed1704c637440a4f8dcd28865cc031e5" gracePeriod=30 Sep 04 22:30:54.245682 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:54.245648 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3949cd16-mxt9n"] Sep 04 22:30:54.249155 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:54.249123 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3949cd16-mxt9n"] Sep 04 22:30:54.270688 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:54.270661 2574 generic.go:356] "Generic (PLEG): container finished" podID="f9421f27-7fed-4bff-93ac-f1e55174fd4c" containerID="0d3c221ad4878b13f2ea3a7f6159a1bfed1704c637440a4f8dcd28865cc031e5" exitCode=0 Sep 04 22:30:54.270867 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:54.270740 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3949cd16-5877d64c7d-7ql27" event={"ID":"f9421f27-7fed-4bff-93ac-f1e55174fd4c","Type":"ContainerDied","Data":"0d3c221ad4878b13f2ea3a7f6159a1bfed1704c637440a4f8dcd28865cc031e5"} Sep 04 22:30:54.272305 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:54.272283 2574 generic.go:356] "Generic (PLEG): container finished" podID="025bc6e3-b614-4125-b33a-3c0da2a8ae1f" containerID="41167f0736aab673d97d6e00bfc083e59a02a590049667335af94cc45834a932" exitCode=0 Sep 04 22:30:54.272388 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:54.272347 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3949cd16-6f5fbfbd9-pqf89" event={"ID":"025bc6e3-b614-4125-b33a-3c0da2a8ae1f","Type":"ContainerDied","Data":"41167f0736aab673d97d6e00bfc083e59a02a590049667335af94cc45834a932"} Sep 04 22:30:54.291657 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:54.291632 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3949cd16-6f5fbfbd9-pqf89" Sep 04 22:30:54.328467 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:54.328446 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3949cd16-5877d64c7d-7ql27" Sep 04 22:30:54.452649 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:54.452613 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/025bc6e3-b614-4125-b33a-3c0da2a8ae1f-plugins-config-volume\") pod \"025bc6e3-b614-4125-b33a-3c0da2a8ae1f\" (UID: \"025bc6e3-b614-4125-b33a-3c0da2a8ae1f\") " Sep 04 22:30:54.452847 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:54.452670 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9421f27-7fed-4bff-93ac-f1e55174fd4c-plugins-config-volume\") pod \"f9421f27-7fed-4bff-93ac-f1e55174fd4c\" (UID: \"f9421f27-7fed-4bff-93ac-f1e55174fd4c\") " Sep 04 22:30:54.452847 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:54.452706 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q86gn\" (UniqueName: \"kubernetes.io/projected/025bc6e3-b614-4125-b33a-3c0da2a8ae1f-kube-api-access-q86gn\") pod \"025bc6e3-b614-4125-b33a-3c0da2a8ae1f\" (UID: \"025bc6e3-b614-4125-b33a-3c0da2a8ae1f\") " Sep 04 22:30:54.452847 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:54.452741 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-899kd\" (UniqueName: \"kubernetes.io/projected/f9421f27-7fed-4bff-93ac-f1e55174fd4c-kube-api-access-899kd\") pod \"f9421f27-7fed-4bff-93ac-f1e55174fd4c\" (UID: \"f9421f27-7fed-4bff-93ac-f1e55174fd4c\") " Sep 04 22:30:54.453085 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:54.453058 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9421f27-7fed-4bff-93ac-f1e55174fd4c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f9421f27-7fed-4bff-93ac-f1e55174fd4c" (UID: "f9421f27-7fed-4bff-93ac-f1e55174fd4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:30:54.453140 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:54.453085 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/025bc6e3-b614-4125-b33a-3c0da2a8ae1f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "025bc6e3-b614-4125-b33a-3c0da2a8ae1f" (UID: "025bc6e3-b614-4125-b33a-3c0da2a8ae1f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:30:54.455072 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:54.455026 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9421f27-7fed-4bff-93ac-f1e55174fd4c-kube-api-access-899kd" (OuterVolumeSpecName: "kube-api-access-899kd") pod "f9421f27-7fed-4bff-93ac-f1e55174fd4c" (UID: "f9421f27-7fed-4bff-93ac-f1e55174fd4c"). InnerVolumeSpecName "kube-api-access-899kd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:30:54.455187 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:54.455115 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/025bc6e3-b614-4125-b33a-3c0da2a8ae1f-kube-api-access-q86gn" (OuterVolumeSpecName: "kube-api-access-q86gn") pod "025bc6e3-b614-4125-b33a-3c0da2a8ae1f" (UID: "025bc6e3-b614-4125-b33a-3c0da2a8ae1f"). InnerVolumeSpecName "kube-api-access-q86gn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:30:54.554332 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:54.554239 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-899kd\" (UniqueName: \"kubernetes.io/projected/f9421f27-7fed-4bff-93ac-f1e55174fd4c-kube-api-access-899kd\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:30:54.554332 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:54.554271 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/025bc6e3-b614-4125-b33a-3c0da2a8ae1f-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:30:54.554332 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:54.554287 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9421f27-7fed-4bff-93ac-f1e55174fd4c-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:30:54.554332 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:54.554299 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q86gn\" (UniqueName: \"kubernetes.io/projected/025bc6e3-b614-4125-b33a-3c0da2a8ae1f-kube-api-access-q86gn\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:30:54.715473 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:54.715437 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="540611f6-8a21-490c-aaab-71cd768443e8" path="/var/lib/kubelet/pods/540611f6-8a21-490c-aaab-71cd768443e8/volumes" Sep 04 22:30:54.715856 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:54.715836 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e5f379e-9ddd-49fa-8d74-928387170e43" path="/var/lib/kubelet/pods/7e5f379e-9ddd-49fa-8d74-928387170e43/volumes" Sep 04 22:30:55.279822 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:55.279782 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3949cd16-5877d64c7d-7ql27" Sep 04 22:30:55.279822 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:55.279793 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3949cd16-5877d64c7d-7ql27" event={"ID":"f9421f27-7fed-4bff-93ac-f1e55174fd4c","Type":"ContainerDied","Data":"7bb973579cb1b042bd13b10a0a6bc59c4492d87ec9ac8bdd0c6d7b03bb99672e"} Sep 04 22:30:55.280378 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:55.279903 2574 scope.go:117] "RemoveContainer" containerID="0d3c221ad4878b13f2ea3a7f6159a1bfed1704c637440a4f8dcd28865cc031e5" Sep 04 22:30:55.281639 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:55.281614 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3949cd16-6f5fbfbd9-pqf89" Sep 04 22:30:55.281763 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:55.281611 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3949cd16-6f5fbfbd9-pqf89" event={"ID":"025bc6e3-b614-4125-b33a-3c0da2a8ae1f","Type":"ContainerDied","Data":"e94703d50a6c68c53afdf54e8dbc69de7fb6a12f431f3b8ee925893f48262877"} Sep 04 22:30:55.289797 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:55.289780 2574 scope.go:117] "RemoveContainer" containerID="41167f0736aab673d97d6e00bfc083e59a02a590049667335af94cc45834a932" Sep 04 22:30:55.298088 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:55.298034 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3949cd16-5877d64c7d-7ql27"] Sep 04 22:30:55.301943 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:55.301920 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3949cd16-5877d64c7d-7ql27"] Sep 04 22:30:55.315976 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:55.315950 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3949cd16-6f5fbfbd9-pqf89"] Sep 04 22:30:55.322611 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:55.322585 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3949cd16-6f5fbfbd9-pqf89"] Sep 04 22:30:56.715370 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:56.715335 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="025bc6e3-b614-4125-b33a-3c0da2a8ae1f" path="/var/lib/kubelet/pods/025bc6e3-b614-4125-b33a-3c0da2a8ae1f/volumes" Sep 04 22:30:56.715751 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:30:56.715698 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9421f27-7fed-4bff-93ac-f1e55174fd4c" path="/var/lib/kubelet/pods/f9421f27-7fed-4bff-93ac-f1e55174fd4c/volumes" Sep 04 22:31:16.898864 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:31:16.898835 2574 scope.go:117] "RemoveContainer" containerID="d6dc6560178fb697fab2ffac91e1c3e6ca9a5b7a91b84dc66047b6c01e6904c1" Sep 04 22:31:16.907117 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:31:16.907102 2574 scope.go:117] "RemoveContainer" containerID="b282440f064ebdfbb8ac4a9a12094869ebf73d5bd64fd53958f5526234f2dd30" Sep 04 22:32:16.939814 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:32:16.939737 2574 scope.go:117] "RemoveContainer" containerID="6ad373034975bf051aa09057cca01427c436cbc9722b65073417542189732bb4" Sep 04 22:33:04.331012 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:04.330971 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b852ba-6c77b577d-zzqpq"] Sep 04 22:33:04.331475 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:04.331350 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9421f27-7fed-4bff-93ac-f1e55174fd4c" containerName="payload-pre-processing" Sep 04 22:33:04.331475 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:04.331364 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9421f27-7fed-4bff-93ac-f1e55174fd4c" containerName="payload-pre-processing" Sep 04 22:33:04.331475 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:04.331376 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e5f379e-9ddd-49fa-8d74-928387170e43" containerName="revoke-keys" Sep 04 22:33:04.331475 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:04.331382 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e5f379e-9ddd-49fa-8d74-928387170e43" containerName="revoke-keys" Sep 04 22:33:04.331475 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:04.331397 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="025bc6e3-b614-4125-b33a-3c0da2a8ae1f" containerName="payload-processing" Sep 04 22:33:04.331475 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:04.331406 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="025bc6e3-b614-4125-b33a-3c0da2a8ae1f" containerName="payload-processing" Sep 04 22:33:04.331710 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:04.331484 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e5f379e-9ddd-49fa-8d74-928387170e43" containerName="revoke-keys" Sep 04 22:33:04.331710 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:04.331495 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9421f27-7fed-4bff-93ac-f1e55174fd4c" containerName="payload-pre-processing" Sep 04 22:33:04.331710 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:04.331504 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="025bc6e3-b614-4125-b33a-3c0da2a8ae1f" containerName="payload-processing" Sep 04 22:33:04.334559 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:04.334535 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b852ba-6c77b577d-zzqpq" Sep 04 22:33:04.337177 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:04.337155 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-b852ba\"" Sep 04 22:33:04.342415 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:04.342395 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-b852ba-dockercfg-66b2w\"" Sep 04 22:33:04.363812 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:04.363782 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b852ba-6c77b577d-zzqpq"] Sep 04 22:33:04.383058 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:04.383009 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrl88\" (UniqueName: \"kubernetes.io/projected/8e7c4bf4-ceaa-4e1c-b026-530f425bb523-kube-api-access-wrl88\") pod \"payload-pre-processing-e2e-api-a-b852ba-6c77b577d-zzqpq\" (UID: \"8e7c4bf4-ceaa-4e1c-b026-530f425bb523\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b852ba-6c77b577d-zzqpq" Sep 04 22:33:04.383254 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:04.383127 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e7c4bf4-ceaa-4e1c-b026-530f425bb523-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b852ba-6c77b577d-zzqpq\" (UID: \"8e7c4bf4-ceaa-4e1c-b026-530f425bb523\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b852ba-6c77b577d-zzqpq" Sep 04 22:33:04.408395 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:04.408361 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b852ba-848756dd6c-xd9jd"] Sep 04 22:33:04.411837 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:04.411817 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b852ba-848756dd6c-xd9jd" Sep 04 22:33:04.424313 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:04.424282 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b852ba-848756dd6c-xd9jd"] Sep 04 22:33:04.484444 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:04.484404 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbwtq\" (UniqueName: \"kubernetes.io/projected/8981136c-34d6-44bc-8c60-d79dbd4214d0-kube-api-access-gbwtq\") pod \"payload-processing-e2e-api-a-b852ba-848756dd6c-xd9jd\" (UID: \"8981136c-34d6-44bc-8c60-d79dbd4214d0\") " pod="openshift-ingress/payload-processing-e2e-api-a-b852ba-848756dd6c-xd9jd" Sep 04 22:33:04.484633 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:04.484468 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrl88\" (UniqueName: \"kubernetes.io/projected/8e7c4bf4-ceaa-4e1c-b026-530f425bb523-kube-api-access-wrl88\") pod \"payload-pre-processing-e2e-api-a-b852ba-6c77b577d-zzqpq\" (UID: \"8e7c4bf4-ceaa-4e1c-b026-530f425bb523\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b852ba-6c77b577d-zzqpq" Sep 04 22:33:04.484633 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:04.484487 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8981136c-34d6-44bc-8c60-d79dbd4214d0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b852ba-848756dd6c-xd9jd\" (UID: \"8981136c-34d6-44bc-8c60-d79dbd4214d0\") " pod="openshift-ingress/payload-processing-e2e-api-a-b852ba-848756dd6c-xd9jd" Sep 04 22:33:04.484633 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:04.484531 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e7c4bf4-ceaa-4e1c-b026-530f425bb523-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b852ba-6c77b577d-zzqpq\" (UID: \"8e7c4bf4-ceaa-4e1c-b026-530f425bb523\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b852ba-6c77b577d-zzqpq" Sep 04 22:33:04.485186 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:04.485166 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e7c4bf4-ceaa-4e1c-b026-530f425bb523-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b852ba-6c77b577d-zzqpq\" (UID: \"8e7c4bf4-ceaa-4e1c-b026-530f425bb523\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b852ba-6c77b577d-zzqpq" Sep 04 22:33:04.496990 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:04.496958 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrl88\" (UniqueName: \"kubernetes.io/projected/8e7c4bf4-ceaa-4e1c-b026-530f425bb523-kube-api-access-wrl88\") pod \"payload-pre-processing-e2e-api-a-b852ba-6c77b577d-zzqpq\" (UID: \"8e7c4bf4-ceaa-4e1c-b026-530f425bb523\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b852ba-6c77b577d-zzqpq" Sep 04 22:33:04.585865 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:04.585763 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8981136c-34d6-44bc-8c60-d79dbd4214d0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b852ba-848756dd6c-xd9jd\" (UID: \"8981136c-34d6-44bc-8c60-d79dbd4214d0\") " pod="openshift-ingress/payload-processing-e2e-api-a-b852ba-848756dd6c-xd9jd" Sep 04 22:33:04.586032 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:04.585888 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbwtq\" (UniqueName: \"kubernetes.io/projected/8981136c-34d6-44bc-8c60-d79dbd4214d0-kube-api-access-gbwtq\") pod \"payload-processing-e2e-api-a-b852ba-848756dd6c-xd9jd\" (UID: \"8981136c-34d6-44bc-8c60-d79dbd4214d0\") " pod="openshift-ingress/payload-processing-e2e-api-a-b852ba-848756dd6c-xd9jd" Sep 04 22:33:04.586487 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:04.586467 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8981136c-34d6-44bc-8c60-d79dbd4214d0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b852ba-848756dd6c-xd9jd\" (UID: \"8981136c-34d6-44bc-8c60-d79dbd4214d0\") " pod="openshift-ingress/payload-processing-e2e-api-a-b852ba-848756dd6c-xd9jd" Sep 04 22:33:04.595790 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:04.595763 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbwtq\" (UniqueName: \"kubernetes.io/projected/8981136c-34d6-44bc-8c60-d79dbd4214d0-kube-api-access-gbwtq\") pod \"payload-processing-e2e-api-a-b852ba-848756dd6c-xd9jd\" (UID: \"8981136c-34d6-44bc-8c60-d79dbd4214d0\") " pod="openshift-ingress/payload-processing-e2e-api-a-b852ba-848756dd6c-xd9jd" Sep 04 22:33:04.644880 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:04.644841 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b852ba-6c77b577d-zzqpq" Sep 04 22:33:04.723679 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:04.723641 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b852ba-848756dd6c-xd9jd" Sep 04 22:33:04.921950 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:04.921921 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b852ba-6c77b577d-zzqpq"] Sep 04 22:33:04.923512 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:33:04.923483 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e7c4bf4_ceaa_4e1c_b026_530f425bb523.slice/crio-61a71fea65cd5cda197e3572a1f06537eaeb93ddb7e23b5fd2fb3e83bced688a WatchSource:0}: Error finding container 61a71fea65cd5cda197e3572a1f06537eaeb93ddb7e23b5fd2fb3e83bced688a: Status 404 returned error can't find the container with id 61a71fea65cd5cda197e3572a1f06537eaeb93ddb7e23b5fd2fb3e83bced688a Sep 04 22:33:05.006801 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:05.006759 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b852ba-848756dd6c-xd9jd"] Sep 04 22:33:05.007956 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:33:05.007923 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8981136c_34d6_44bc_8c60_d79dbd4214d0.slice/crio-bcae762b0d3e209c4a5bad5e95c4e963ced8c5f305f5ab424cf1efde371ab9c5 WatchSource:0}: Error finding container bcae762b0d3e209c4a5bad5e95c4e963ced8c5f305f5ab424cf1efde371ab9c5: Status 404 returned error can't find the container with id bcae762b0d3e209c4a5bad5e95c4e963ced8c5f305f5ab424cf1efde371ab9c5 Sep 04 22:33:05.015039 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:05.015014 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b852ba-6c77b577d-zzqpq" event={"ID":"8e7c4bf4-ceaa-4e1c-b026-530f425bb523","Type":"ContainerStarted","Data":"9cedb6955a97b36d8a4ec06ccc53d4f0547cc9732e615f3de1cc4ec7f149454a"} Sep 04 22:33:05.015137 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:05.015069 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b852ba-6c77b577d-zzqpq" Sep 04 22:33:05.015137 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:05.015087 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b852ba-6c77b577d-zzqpq" event={"ID":"8e7c4bf4-ceaa-4e1c-b026-530f425bb523","Type":"ContainerStarted","Data":"61a71fea65cd5cda197e3572a1f06537eaeb93ddb7e23b5fd2fb3e83bced688a"} Sep 04 22:33:05.037154 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:05.037103 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b852ba-6c77b577d-zzqpq" podStartSLOduration=1.037084938 podStartE2EDuration="1.037084938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:33:05.03569031 +0000 UTC m=+1488.800165925" watchObservedRunningTime="2026-09-04 22:33:05.037084938 +0000 UTC m=+1488.801560551" Sep 04 22:33:06.025616 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:06.025575 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b852ba-848756dd6c-xd9jd" event={"ID":"8981136c-34d6-44bc-8c60-d79dbd4214d0","Type":"ContainerStarted","Data":"b980999909822102ae25c00f602fcc623c9dae890267188e199eb47f3afb535d"} Sep 04 22:33:06.026062 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:06.025638 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b852ba-848756dd6c-xd9jd" event={"ID":"8981136c-34d6-44bc-8c60-d79dbd4214d0","Type":"ContainerStarted","Data":"bcae762b0d3e209c4a5bad5e95c4e963ced8c5f305f5ab424cf1efde371ab9c5"} Sep 04 22:33:06.026062 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:06.025769 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-b852ba-848756dd6c-xd9jd" Sep 04 22:33:06.054122 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:06.054051 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-b852ba-848756dd6c-xd9jd" podStartSLOduration=2.054027267 podStartE2EDuration="2.054027267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:33:06.050457559 +0000 UTC m=+1489.814933186" watchObservedRunningTime="2026-09-04 22:33:06.054027267 +0000 UTC m=+1489.818502887" Sep 04 22:33:16.027735 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:16.027698 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b852ba-6c77b577d-zzqpq" Sep 04 22:33:17.034644 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:17.034615 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-b852ba-848756dd6c-xd9jd" Sep 04 22:33:33.491483 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:33.491445 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ef263a-788447bc8-6r2bm"] Sep 04 22:33:33.494929 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:33.494911 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ef263a-788447bc8-6r2bm" Sep 04 22:33:33.497522 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:33.497499 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-ef263a-dockercfg-xx7r9\"" Sep 04 22:33:33.497674 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:33.497524 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-ef263a\"" Sep 04 22:33:33.508274 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:33.508247 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ef263a-788447bc8-6r2bm"] Sep 04 22:33:33.556731 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:33.556696 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ef263a-5f7cfd74f9-z45xn"] Sep 04 22:33:33.560115 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:33.560094 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ef263a-5f7cfd74f9-z45xn" Sep 04 22:33:33.578522 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:33.578495 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ef263a-5f7cfd74f9-z45xn"] Sep 04 22:33:33.650289 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:33.650250 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/235d3947-3575-41a5-9adc-f6de9ecea65e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ef263a-5f7cfd74f9-z45xn\" (UID: \"235d3947-3575-41a5-9adc-f6de9ecea65e\") " pod="openshift-ingress/payload-processing-e2e-api-b-ef263a-5f7cfd74f9-z45xn" Sep 04 22:33:33.650469 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:33.650305 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8kns\" (UniqueName: \"kubernetes.io/projected/235d3947-3575-41a5-9adc-f6de9ecea65e-kube-api-access-d8kns\") pod \"payload-processing-e2e-api-b-ef263a-5f7cfd74f9-z45xn\" (UID: \"235d3947-3575-41a5-9adc-f6de9ecea65e\") " pod="openshift-ingress/payload-processing-e2e-api-b-ef263a-5f7cfd74f9-z45xn" Sep 04 22:33:33.650469 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:33.650396 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqmhj\" (UniqueName: \"kubernetes.io/projected/5cbbfe13-6f05-473e-87a1-08d845223559-kube-api-access-cqmhj\") pod \"payload-pre-processing-e2e-api-b-ef263a-788447bc8-6r2bm\" (UID: \"5cbbfe13-6f05-473e-87a1-08d845223559\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ef263a-788447bc8-6r2bm" Sep 04 22:33:33.650469 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:33.650437 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cbbfe13-6f05-473e-87a1-08d845223559-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ef263a-788447bc8-6r2bm\" (UID: \"5cbbfe13-6f05-473e-87a1-08d845223559\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ef263a-788447bc8-6r2bm" Sep 04 22:33:33.751913 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:33.751882 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/235d3947-3575-41a5-9adc-f6de9ecea65e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ef263a-5f7cfd74f9-z45xn\" (UID: \"235d3947-3575-41a5-9adc-f6de9ecea65e\") " pod="openshift-ingress/payload-processing-e2e-api-b-ef263a-5f7cfd74f9-z45xn" Sep 04 22:33:33.752072 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:33.751937 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8kns\" (UniqueName: \"kubernetes.io/projected/235d3947-3575-41a5-9adc-f6de9ecea65e-kube-api-access-d8kns\") pod \"payload-processing-e2e-api-b-ef263a-5f7cfd74f9-z45xn\" (UID: \"235d3947-3575-41a5-9adc-f6de9ecea65e\") " pod="openshift-ingress/payload-processing-e2e-api-b-ef263a-5f7cfd74f9-z45xn" Sep 04 22:33:33.752072 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:33.752001 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqmhj\" (UniqueName: \"kubernetes.io/projected/5cbbfe13-6f05-473e-87a1-08d845223559-kube-api-access-cqmhj\") pod \"payload-pre-processing-e2e-api-b-ef263a-788447bc8-6r2bm\" (UID: \"5cbbfe13-6f05-473e-87a1-08d845223559\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ef263a-788447bc8-6r2bm" Sep 04 22:33:33.752202 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:33.752179 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cbbfe13-6f05-473e-87a1-08d845223559-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ef263a-788447bc8-6r2bm\" (UID: \"5cbbfe13-6f05-473e-87a1-08d845223559\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ef263a-788447bc8-6r2bm" Sep 04 22:33:33.752899 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:33.752876 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/235d3947-3575-41a5-9adc-f6de9ecea65e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ef263a-5f7cfd74f9-z45xn\" (UID: \"235d3947-3575-41a5-9adc-f6de9ecea65e\") " pod="openshift-ingress/payload-processing-e2e-api-b-ef263a-5f7cfd74f9-z45xn" Sep 04 22:33:33.752945 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:33.752923 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cbbfe13-6f05-473e-87a1-08d845223559-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ef263a-788447bc8-6r2bm\" (UID: \"5cbbfe13-6f05-473e-87a1-08d845223559\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ef263a-788447bc8-6r2bm" Sep 04 22:33:33.765066 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:33.765039 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqmhj\" (UniqueName: \"kubernetes.io/projected/5cbbfe13-6f05-473e-87a1-08d845223559-kube-api-access-cqmhj\") pod \"payload-pre-processing-e2e-api-b-ef263a-788447bc8-6r2bm\" (UID: \"5cbbfe13-6f05-473e-87a1-08d845223559\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ef263a-788447bc8-6r2bm" Sep 04 22:33:33.765199 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:33.765183 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8kns\" (UniqueName: \"kubernetes.io/projected/235d3947-3575-41a5-9adc-f6de9ecea65e-kube-api-access-d8kns\") pod \"payload-processing-e2e-api-b-ef263a-5f7cfd74f9-z45xn\" (UID: \"235d3947-3575-41a5-9adc-f6de9ecea65e\") " pod="openshift-ingress/payload-processing-e2e-api-b-ef263a-5f7cfd74f9-z45xn" Sep 04 22:33:33.804621 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:33.804581 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ef263a-788447bc8-6r2bm" Sep 04 22:33:33.870279 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:33.870248 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ef263a-5f7cfd74f9-z45xn" Sep 04 22:33:34.084924 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:34.084899 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ef263a-788447bc8-6r2bm"] Sep 04 22:33:34.086497 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:33:34.086473 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cbbfe13_6f05_473e_87a1_08d845223559.slice/crio-256060aebd8ff497200c24dff7cb83e09fb8d0b9caaaa4346ff0a1f75b7cf957 WatchSource:0}: Error finding container 256060aebd8ff497200c24dff7cb83e09fb8d0b9caaaa4346ff0a1f75b7cf957: Status 404 returned error can't find the container with id 256060aebd8ff497200c24dff7cb83e09fb8d0b9caaaa4346ff0a1f75b7cf957 Sep 04 22:33:34.151149 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:34.151123 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ef263a-5f7cfd74f9-z45xn"] Sep 04 22:33:34.189644 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:34.189611 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ef263a-5f7cfd74f9-z45xn" event={"ID":"235d3947-3575-41a5-9adc-f6de9ecea65e","Type":"ContainerStarted","Data":"61cd2277240a8ef22a14c5c2c81d4465f7cc341e2e805569676688677315f5e1"} Sep 04 22:33:34.191453 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:34.191423 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ef263a-788447bc8-6r2bm" event={"ID":"5cbbfe13-6f05-473e-87a1-08d845223559","Type":"ContainerStarted","Data":"32796963c98079e614ca69fc2f3a988c5a892e2f78db0f94aeeb7db0fedcab44"} Sep 04 22:33:34.191587 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:34.191467 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ef263a-788447bc8-6r2bm" event={"ID":"5cbbfe13-6f05-473e-87a1-08d845223559","Type":"ContainerStarted","Data":"256060aebd8ff497200c24dff7cb83e09fb8d0b9caaaa4346ff0a1f75b7cf957"} Sep 04 22:33:34.191587 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:34.191539 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ef263a-788447bc8-6r2bm" Sep 04 22:33:34.217086 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:34.217034 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ef263a-788447bc8-6r2bm" podStartSLOduration=1.217013561 podStartE2EDuration="1.217013561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:33:34.217009173 +0000 UTC m=+1517.981484786" watchObservedRunningTime="2026-09-04 22:33:34.217013561 +0000 UTC m=+1517.981489175" Sep 04 22:33:35.199344 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:35.199300 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ef263a-5f7cfd74f9-z45xn" event={"ID":"235d3947-3575-41a5-9adc-f6de9ecea65e","Type":"ContainerStarted","Data":"d9f66b0a3b983e17fcf7e62cf66b75ccf7ede91326ab8ab07dcf52d013f86c70"} Sep 04 22:33:35.199784 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:35.199489 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-ef263a-5f7cfd74f9-z45xn" Sep 04 22:33:35.223391 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:35.223340 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-ef263a-5f7cfd74f9-z45xn" podStartSLOduration=2.223323522 podStartE2EDuration="2.223323522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:33:35.221038423 +0000 UTC m=+1518.985514048" watchObservedRunningTime="2026-09-04 22:33:35.223323522 +0000 UTC m=+1518.987799131" Sep 04 22:33:45.201370 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:45.201284 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ef263a-788447bc8-6r2bm" Sep 04 22:33:46.207152 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:33:46.207113 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-ef263a-5f7cfd74f9-z45xn" Sep 04 22:34:04.642070 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:04.642034 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b852ba-jxqnn"] Sep 04 22:34:04.645586 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:04.645539 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b852ba-jxqnn" Sep 04 22:34:04.648635 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:04.648606 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hc7x4\"" Sep 04 22:34:04.648635 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:04.648628 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 04 22:34:04.655107 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:04.655082 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b852ba-jxqnn"] Sep 04 22:34:04.726274 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:04.726244 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89c89889-7d51-4b9f-a88f-b68904274367-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b852ba-jxqnn\" (UID: \"89c89889-7d51-4b9f-a88f-b68904274367\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b852ba-jxqnn" Sep 04 22:34:04.827066 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:04.827019 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89c89889-7d51-4b9f-a88f-b68904274367-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b852ba-jxqnn\" (UID: \"89c89889-7d51-4b9f-a88f-b68904274367\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b852ba-jxqnn" Sep 04 22:34:04.827673 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:04.827654 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89c89889-7d51-4b9f-a88f-b68904274367-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-b852ba-jxqnn\" (UID: \"89c89889-7d51-4b9f-a88f-b68904274367\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b852ba-jxqnn" Sep 04 22:34:04.957057 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:04.956975 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b852ba-jxqnn" Sep 04 22:34:05.239725 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:05.239696 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b852ba-jxqnn"] Sep 04 22:34:05.241137 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:34:05.241112 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89c89889_7d51_4b9f_a88f_b68904274367.slice/crio-9cfd7cbf5202053cf8193eac22e245fac747dfa6e56b344d71016d056851d121 WatchSource:0}: Error finding container 9cfd7cbf5202053cf8193eac22e245fac747dfa6e56b344d71016d056851d121: Status 404 returned error can't find the container with id 9cfd7cbf5202053cf8193eac22e245fac747dfa6e56b344d71016d056851d121 Sep 04 22:34:05.371225 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:05.371189 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b852ba-jxqnn" event={"ID":"89c89889-7d51-4b9f-a88f-b68904274367","Type":"ContainerStarted","Data":"9cfd7cbf5202053cf8193eac22e245fac747dfa6e56b344d71016d056851d121"} Sep 04 22:34:06.378376 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:06.378335 2574 generic.go:356] "Generic (PLEG): container finished" podID="89c89889-7d51-4b9f-a88f-b68904274367" containerID="537d8d4cc7a9ae3334a795dfa6dd5421f2f679a473c883ca5777a052828f109c" exitCode=0 Sep 04 22:34:06.378769 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:06.378422 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b852ba-jxqnn" event={"ID":"89c89889-7d51-4b9f-a88f-b68904274367","Type":"ContainerDied","Data":"537d8d4cc7a9ae3334a795dfa6dd5421f2f679a473c883ca5777a052828f109c"} Sep 04 22:34:07.538486 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:07.538452 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b852ba-jxqnn" Sep 04 22:34:07.655172 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:07.655072 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89c89889-7d51-4b9f-a88f-b68904274367-maas-api-service-ca\") pod \"89c89889-7d51-4b9f-a88f-b68904274367\" (UID: \"89c89889-7d51-4b9f-a88f-b68904274367\") " Sep 04 22:34:07.655436 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:07.655411 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89c89889-7d51-4b9f-a88f-b68904274367-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "89c89889-7d51-4b9f-a88f-b68904274367" (UID: "89c89889-7d51-4b9f-a88f-b68904274367"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:34:07.756213 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:07.756174 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89c89889-7d51-4b9f-a88f-b68904274367-maas-api-service-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:34:08.391704 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:08.391673 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b852ba-jxqnn" Sep 04 22:34:08.391883 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:08.391738 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b852ba-jxqnn" event={"ID":"89c89889-7d51-4b9f-a88f-b68904274367","Type":"ContainerDied","Data":"9cfd7cbf5202053cf8193eac22e245fac747dfa6e56b344d71016d056851d121"} Sep 04 22:34:08.391883 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:08.391783 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9cfd7cbf5202053cf8193eac22e245fac747dfa6e56b344d71016d056851d121" Sep 04 22:34:16.968688 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:16.968656 2574 scope.go:117] "RemoveContainer" containerID="48fca6171c402cd0754a7807d822e2b2d81c3b65fcf53e1e02f307bd859503c3" Sep 04 22:34:16.979893 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:16.979870 2574 scope.go:117] "RemoveContainer" containerID="3f1ac0a7420107fcddfa8271bf5b0966179ea1736c25087178fd094416f64c11" Sep 04 22:34:16.988014 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:16.987994 2574 scope.go:117] "RemoveContainer" containerID="5ffd75a03b61d9bfa2827fc07c8e1e06e953f835cbad037788f10ad8458f232f" Sep 04 22:34:21.524260 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:21.524221 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b852ba-848756dd6c-xd9jd"] Sep 04 22:34:21.524634 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:21.524433 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-b852ba-848756dd6c-xd9jd" podUID="8981136c-34d6-44bc-8c60-d79dbd4214d0" containerName="payload-processing" containerID="cri-o://b980999909822102ae25c00f602fcc623c9dae890267188e199eb47f3afb535d" gracePeriod=30 Sep 04 22:34:21.532538 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:21.532501 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b852ba-6c77b577d-zzqpq"] Sep 04 22:34:21.532774 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:21.532727 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b852ba-6c77b577d-zzqpq" podUID="8e7c4bf4-ceaa-4e1c-b026-530f425bb523" containerName="payload-pre-processing" containerID="cri-o://9cedb6955a97b36d8a4ec06ccc53d4f0547cc9732e615f3de1cc4ec7f149454a" gracePeriod=30 Sep 04 22:34:21.800200 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:21.800175 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b852ba-848756dd6c-xd9jd" Sep 04 22:34:21.802988 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:21.802970 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b852ba-6c77b577d-zzqpq" Sep 04 22:34:21.982793 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:21.982758 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrl88\" (UniqueName: \"kubernetes.io/projected/8e7c4bf4-ceaa-4e1c-b026-530f425bb523-kube-api-access-wrl88\") pod \"8e7c4bf4-ceaa-4e1c-b026-530f425bb523\" (UID: \"8e7c4bf4-ceaa-4e1c-b026-530f425bb523\") " Sep 04 22:34:21.982793 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:21.982797 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbwtq\" (UniqueName: \"kubernetes.io/projected/8981136c-34d6-44bc-8c60-d79dbd4214d0-kube-api-access-gbwtq\") pod \"8981136c-34d6-44bc-8c60-d79dbd4214d0\" (UID: \"8981136c-34d6-44bc-8c60-d79dbd4214d0\") " Sep 04 22:34:21.983008 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:21.982837 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8981136c-34d6-44bc-8c60-d79dbd4214d0-plugins-config-volume\") pod \"8981136c-34d6-44bc-8c60-d79dbd4214d0\" (UID: \"8981136c-34d6-44bc-8c60-d79dbd4214d0\") " Sep 04 22:34:21.983008 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:21.982881 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e7c4bf4-ceaa-4e1c-b026-530f425bb523-plugins-config-volume\") pod \"8e7c4bf4-ceaa-4e1c-b026-530f425bb523\" (UID: \"8e7c4bf4-ceaa-4e1c-b026-530f425bb523\") " Sep 04 22:34:21.983267 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:21.983240 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8981136c-34d6-44bc-8c60-d79dbd4214d0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8981136c-34d6-44bc-8c60-d79dbd4214d0" (UID: "8981136c-34d6-44bc-8c60-d79dbd4214d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:34:21.983391 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:21.983338 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e7c4bf4-ceaa-4e1c-b026-530f425bb523-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8e7c4bf4-ceaa-4e1c-b026-530f425bb523" (UID: "8e7c4bf4-ceaa-4e1c-b026-530f425bb523"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:34:21.984894 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:21.984877 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e7c4bf4-ceaa-4e1c-b026-530f425bb523-kube-api-access-wrl88" (OuterVolumeSpecName: "kube-api-access-wrl88") pod "8e7c4bf4-ceaa-4e1c-b026-530f425bb523" (UID: "8e7c4bf4-ceaa-4e1c-b026-530f425bb523"). InnerVolumeSpecName "kube-api-access-wrl88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:34:21.985006 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:21.984991 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8981136c-34d6-44bc-8c60-d79dbd4214d0-kube-api-access-gbwtq" (OuterVolumeSpecName: "kube-api-access-gbwtq") pod "8981136c-34d6-44bc-8c60-d79dbd4214d0" (UID: "8981136c-34d6-44bc-8c60-d79dbd4214d0"). InnerVolumeSpecName "kube-api-access-gbwtq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:34:22.022851 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:22.022819 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b852ba-jxqnn"] Sep 04 22:34:22.032539 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:22.032467 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-b852ba-jxqnn"] Sep 04 22:34:22.084631 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:22.084590 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8981136c-34d6-44bc-8c60-d79dbd4214d0-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:34:22.084631 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:22.084634 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e7c4bf4-ceaa-4e1c-b026-530f425bb523-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:34:22.084885 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:22.084649 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrl88\" (UniqueName: \"kubernetes.io/projected/8e7c4bf4-ceaa-4e1c-b026-530f425bb523-kube-api-access-wrl88\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:34:22.084885 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:22.084664 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbwtq\" (UniqueName: \"kubernetes.io/projected/8981136c-34d6-44bc-8c60-d79dbd4214d0-kube-api-access-gbwtq\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:34:22.475109 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:22.475024 2574 generic.go:356] "Generic (PLEG): container finished" podID="8981136c-34d6-44bc-8c60-d79dbd4214d0" containerID="b980999909822102ae25c00f602fcc623c9dae890267188e199eb47f3afb535d" exitCode=0 Sep 04 22:34:22.475109 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:22.475073 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b852ba-848756dd6c-xd9jd" Sep 04 22:34:22.475338 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:22.475113 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b852ba-848756dd6c-xd9jd" event={"ID":"8981136c-34d6-44bc-8c60-d79dbd4214d0","Type":"ContainerDied","Data":"b980999909822102ae25c00f602fcc623c9dae890267188e199eb47f3afb535d"} Sep 04 22:34:22.475338 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:22.475163 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b852ba-848756dd6c-xd9jd" event={"ID":"8981136c-34d6-44bc-8c60-d79dbd4214d0","Type":"ContainerDied","Data":"bcae762b0d3e209c4a5bad5e95c4e963ced8c5f305f5ab424cf1efde371ab9c5"} Sep 04 22:34:22.475338 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:22.475188 2574 scope.go:117] "RemoveContainer" containerID="b980999909822102ae25c00f602fcc623c9dae890267188e199eb47f3afb535d" Sep 04 22:34:22.476413 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:22.476394 2574 generic.go:356] "Generic (PLEG): container finished" podID="8e7c4bf4-ceaa-4e1c-b026-530f425bb523" containerID="9cedb6955a97b36d8a4ec06ccc53d4f0547cc9732e615f3de1cc4ec7f149454a" exitCode=0 Sep 04 22:34:22.476485 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:22.476425 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b852ba-6c77b577d-zzqpq" event={"ID":"8e7c4bf4-ceaa-4e1c-b026-530f425bb523","Type":"ContainerDied","Data":"9cedb6955a97b36d8a4ec06ccc53d4f0547cc9732e615f3de1cc4ec7f149454a"} Sep 04 22:34:22.476485 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:22.476445 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b852ba-6c77b577d-zzqpq" Sep 04 22:34:22.476485 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:22.476471 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b852ba-6c77b577d-zzqpq" event={"ID":"8e7c4bf4-ceaa-4e1c-b026-530f425bb523","Type":"ContainerDied","Data":"61a71fea65cd5cda197e3572a1f06537eaeb93ddb7e23b5fd2fb3e83bced688a"} Sep 04 22:34:22.484906 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:22.484889 2574 scope.go:117] "RemoveContainer" containerID="b980999909822102ae25c00f602fcc623c9dae890267188e199eb47f3afb535d" Sep 04 22:34:22.485175 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:34:22.485153 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b980999909822102ae25c00f602fcc623c9dae890267188e199eb47f3afb535d\": container with ID starting with b980999909822102ae25c00f602fcc623c9dae890267188e199eb47f3afb535d not found: ID does not exist" containerID="b980999909822102ae25c00f602fcc623c9dae890267188e199eb47f3afb535d" Sep 04 22:34:22.485249 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:22.485185 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b980999909822102ae25c00f602fcc623c9dae890267188e199eb47f3afb535d"} err="failed to get container status \"b980999909822102ae25c00f602fcc623c9dae890267188e199eb47f3afb535d\": rpc error: code = NotFound desc = could not find container \"b980999909822102ae25c00f602fcc623c9dae890267188e199eb47f3afb535d\": container with ID starting with b980999909822102ae25c00f602fcc623c9dae890267188e199eb47f3afb535d not found: ID does not exist" Sep 04 22:34:22.485249 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:22.485203 2574 scope.go:117] "RemoveContainer" containerID="9cedb6955a97b36d8a4ec06ccc53d4f0547cc9732e615f3de1cc4ec7f149454a" Sep 04 22:34:22.493558 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:22.493528 2574 scope.go:117] "RemoveContainer" containerID="9cedb6955a97b36d8a4ec06ccc53d4f0547cc9732e615f3de1cc4ec7f149454a" Sep 04 22:34:22.493925 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:34:22.493903 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cedb6955a97b36d8a4ec06ccc53d4f0547cc9732e615f3de1cc4ec7f149454a\": container with ID starting with 9cedb6955a97b36d8a4ec06ccc53d4f0547cc9732e615f3de1cc4ec7f149454a not found: ID does not exist" containerID="9cedb6955a97b36d8a4ec06ccc53d4f0547cc9732e615f3de1cc4ec7f149454a" Sep 04 22:34:22.493997 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:22.493936 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cedb6955a97b36d8a4ec06ccc53d4f0547cc9732e615f3de1cc4ec7f149454a"} err="failed to get container status \"9cedb6955a97b36d8a4ec06ccc53d4f0547cc9732e615f3de1cc4ec7f149454a\": rpc error: code = NotFound desc = could not find container \"9cedb6955a97b36d8a4ec06ccc53d4f0547cc9732e615f3de1cc4ec7f149454a\": container with ID starting with 9cedb6955a97b36d8a4ec06ccc53d4f0547cc9732e615f3de1cc4ec7f149454a not found: ID does not exist" Sep 04 22:34:22.513107 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:22.513080 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b852ba-848756dd6c-xd9jd"] Sep 04 22:34:22.519902 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:22.519881 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b852ba-848756dd6c-xd9jd"] Sep 04 22:34:22.549604 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:22.549573 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b852ba-6c77b577d-zzqpq"] Sep 04 22:34:22.565211 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:22.565188 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b852ba-6c77b577d-zzqpq"] Sep 04 22:34:22.715030 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:22.714996 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8981136c-34d6-44bc-8c60-d79dbd4214d0" path="/var/lib/kubelet/pods/8981136c-34d6-44bc-8c60-d79dbd4214d0/volumes" Sep 04 22:34:22.715366 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:22.715352 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89c89889-7d51-4b9f-a88f-b68904274367" path="/var/lib/kubelet/pods/89c89889-7d51-4b9f-a88f-b68904274367/volumes" Sep 04 22:34:22.715722 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:22.715708 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e7c4bf4-ceaa-4e1c-b026-530f425bb523" path="/var/lib/kubelet/pods/8e7c4bf4-ceaa-4e1c-b026-530f425bb523/volumes" Sep 04 22:34:31.624694 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:31.624659 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ef263a-hk5xp"] Sep 04 22:34:31.625131 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:31.625048 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e7c4bf4-ceaa-4e1c-b026-530f425bb523" containerName="payload-pre-processing" Sep 04 22:34:31.625131 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:31.625060 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e7c4bf4-ceaa-4e1c-b026-530f425bb523" containerName="payload-pre-processing" Sep 04 22:34:31.625131 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:31.625074 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89c89889-7d51-4b9f-a88f-b68904274367" containerName="revoke-keys" Sep 04 22:34:31.625131 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:31.625080 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="89c89889-7d51-4b9f-a88f-b68904274367" containerName="revoke-keys" Sep 04 22:34:31.625131 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:31.625093 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8981136c-34d6-44bc-8c60-d79dbd4214d0" containerName="payload-processing" Sep 04 22:34:31.625131 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:31.625098 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8981136c-34d6-44bc-8c60-d79dbd4214d0" containerName="payload-processing" Sep 04 22:34:31.625327 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:31.625152 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="89c89889-7d51-4b9f-a88f-b68904274367" containerName="revoke-keys" Sep 04 22:34:31.625327 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:31.625160 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8981136c-34d6-44bc-8c60-d79dbd4214d0" containerName="payload-processing" Sep 04 22:34:31.625327 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:31.625168 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e7c4bf4-ceaa-4e1c-b026-530f425bb523" containerName="payload-pre-processing" Sep 04 22:34:31.629779 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:31.629762 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ef263a-hk5xp" Sep 04 22:34:31.632770 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:31.632747 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hc7x4\"" Sep 04 22:34:31.633135 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:31.633115 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 04 22:34:31.643422 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:31.643399 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ef263a-hk5xp"] Sep 04 22:34:31.668623 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:31.668586 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e95c6ff-bb24-4208-9be9-e04699b69f08-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ef263a-hk5xp\" (UID: \"9e95c6ff-bb24-4208-9be9-e04699b69f08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ef263a-hk5xp" Sep 04 22:34:31.769399 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:31.769361 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e95c6ff-bb24-4208-9be9-e04699b69f08-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ef263a-hk5xp\" (UID: \"9e95c6ff-bb24-4208-9be9-e04699b69f08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ef263a-hk5xp" Sep 04 22:34:31.770100 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:31.770079 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e95c6ff-bb24-4208-9be9-e04699b69f08-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ef263a-hk5xp\" (UID: \"9e95c6ff-bb24-4208-9be9-e04699b69f08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ef263a-hk5xp" Sep 04 22:34:31.939961 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:31.939867 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ef263a-hk5xp" Sep 04 22:34:32.236805 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:32.236708 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ef263a-hk5xp"] Sep 04 22:34:32.240705 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:34:32.240678 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e95c6ff_bb24_4208_9be9_e04699b69f08.slice/crio-c9a76919b803426ef64fdfedaadb853d01028cf21e1823f74059afd09a8b635f WatchSource:0}: Error finding container c9a76919b803426ef64fdfedaadb853d01028cf21e1823f74059afd09a8b635f: Status 404 returned error can't find the container with id c9a76919b803426ef64fdfedaadb853d01028cf21e1823f74059afd09a8b635f Sep 04 22:34:32.542023 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:32.541991 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ef263a-hk5xp" event={"ID":"9e95c6ff-bb24-4208-9be9-e04699b69f08","Type":"ContainerStarted","Data":"d92f1f9570c335d71fef58e662d8d260b72438b5782b2ed96c146f07b28091f5"} Sep 04 22:34:32.542156 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:32.542044 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ef263a-hk5xp" event={"ID":"9e95c6ff-bb24-4208-9be9-e04699b69f08","Type":"ContainerStarted","Data":"c9a76919b803426ef64fdfedaadb853d01028cf21e1823f74059afd09a8b635f"} Sep 04 22:34:32.590349 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:32.590297 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ef263a-hk5xp" podStartSLOduration=1.590280038 podStartE2EDuration="1.590280038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:34:32.589473201 +0000 UTC m=+1576.353948807" watchObservedRunningTime="2026-09-04 22:34:32.590280038 +0000 UTC m=+1576.354755652" Sep 04 22:34:33.549738 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:33.549704 2574 generic.go:356] "Generic (PLEG): container finished" podID="9e95c6ff-bb24-4208-9be9-e04699b69f08" containerID="d92f1f9570c335d71fef58e662d8d260b72438b5782b2ed96c146f07b28091f5" exitCode=0 Sep 04 22:34:33.550123 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:33.549788 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ef263a-hk5xp" event={"ID":"9e95c6ff-bb24-4208-9be9-e04699b69f08","Type":"ContainerDied","Data":"d92f1f9570c335d71fef58e662d8d260b72438b5782b2ed96c146f07b28091f5"} Sep 04 22:34:34.685849 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:34.685825 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ef263a-hk5xp" Sep 04 22:34:34.691125 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:34.691108 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e95c6ff-bb24-4208-9be9-e04699b69f08-maas-api-service-ca\") pod \"9e95c6ff-bb24-4208-9be9-e04699b69f08\" (UID: \"9e95c6ff-bb24-4208-9be9-e04699b69f08\") " Sep 04 22:34:34.691420 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:34.691396 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e95c6ff-bb24-4208-9be9-e04699b69f08-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9e95c6ff-bb24-4208-9be9-e04699b69f08" (UID: "9e95c6ff-bb24-4208-9be9-e04699b69f08"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:34:34.791967 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:34.791924 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e95c6ff-bb24-4208-9be9-e04699b69f08-maas-api-service-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:34:35.564240 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:35.564211 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ef263a-hk5xp" Sep 04 22:34:35.564422 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:35.564248 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ef263a-hk5xp" event={"ID":"9e95c6ff-bb24-4208-9be9-e04699b69f08","Type":"ContainerDied","Data":"c9a76919b803426ef64fdfedaadb853d01028cf21e1823f74059afd09a8b635f"} Sep 04 22:34:35.564422 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:35.564316 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9a76919b803426ef64fdfedaadb853d01028cf21e1823f74059afd09a8b635f" Sep 04 22:34:48.375729 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:48.375695 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ef263a-5f7cfd74f9-z45xn"] Sep 04 22:34:48.376151 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:48.375907 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-ef263a-5f7cfd74f9-z45xn" podUID="235d3947-3575-41a5-9adc-f6de9ecea65e" containerName="payload-processing" containerID="cri-o://d9f66b0a3b983e17fcf7e62cf66b75ccf7ede91326ab8ab07dcf52d013f86c70" gracePeriod=30 Sep 04 22:34:48.409898 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:48.409867 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ef263a-788447bc8-6r2bm"] Sep 04 22:34:48.410154 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:48.410125 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ef263a-788447bc8-6r2bm" podUID="5cbbfe13-6f05-473e-87a1-08d845223559" containerName="payload-pre-processing" containerID="cri-o://32796963c98079e614ca69fc2f3a988c5a892e2f78db0f94aeeb7db0fedcab44" gracePeriod=30 Sep 04 22:34:48.647186 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:48.647087 2574 generic.go:356] "Generic (PLEG): container finished" podID="235d3947-3575-41a5-9adc-f6de9ecea65e" containerID="d9f66b0a3b983e17fcf7e62cf66b75ccf7ede91326ab8ab07dcf52d013f86c70" exitCode=0 Sep 04 22:34:48.647186 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:48.647159 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ef263a-5f7cfd74f9-z45xn" event={"ID":"235d3947-3575-41a5-9adc-f6de9ecea65e","Type":"ContainerDied","Data":"d9f66b0a3b983e17fcf7e62cf66b75ccf7ede91326ab8ab07dcf52d013f86c70"} Sep 04 22:34:48.648878 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:48.648853 2574 generic.go:356] "Generic (PLEG): container finished" podID="5cbbfe13-6f05-473e-87a1-08d845223559" containerID="32796963c98079e614ca69fc2f3a988c5a892e2f78db0f94aeeb7db0fedcab44" exitCode=0 Sep 04 22:34:48.649000 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:48.648918 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ef263a-788447bc8-6r2bm" event={"ID":"5cbbfe13-6f05-473e-87a1-08d845223559","Type":"ContainerDied","Data":"32796963c98079e614ca69fc2f3a988c5a892e2f78db0f94aeeb7db0fedcab44"} Sep 04 22:34:48.702412 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:48.702382 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ef263a-5f7cfd74f9-z45xn" Sep 04 22:34:48.705563 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:48.705526 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ef263a-788447bc8-6r2bm" Sep 04 22:34:48.778512 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:48.778472 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ef263a-hk5xp"] Sep 04 22:34:48.786013 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:48.785985 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ef263a-hk5xp"] Sep 04 22:34:48.817715 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:48.817685 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cbbfe13-6f05-473e-87a1-08d845223559-plugins-config-volume\") pod \"5cbbfe13-6f05-473e-87a1-08d845223559\" (UID: \"5cbbfe13-6f05-473e-87a1-08d845223559\") " Sep 04 22:34:48.817854 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:48.817823 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/235d3947-3575-41a5-9adc-f6de9ecea65e-plugins-config-volume\") pod \"235d3947-3575-41a5-9adc-f6de9ecea65e\" (UID: \"235d3947-3575-41a5-9adc-f6de9ecea65e\") " Sep 04 22:34:48.817904 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:48.817854 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8kns\" (UniqueName: \"kubernetes.io/projected/235d3947-3575-41a5-9adc-f6de9ecea65e-kube-api-access-d8kns\") pod \"235d3947-3575-41a5-9adc-f6de9ecea65e\" (UID: \"235d3947-3575-41a5-9adc-f6de9ecea65e\") " Sep 04 22:34:48.817904 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:48.817890 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqmhj\" (UniqueName: \"kubernetes.io/projected/5cbbfe13-6f05-473e-87a1-08d845223559-kube-api-access-cqmhj\") pod \"5cbbfe13-6f05-473e-87a1-08d845223559\" (UID: \"5cbbfe13-6f05-473e-87a1-08d845223559\") " Sep 04 22:34:48.818217 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:48.818196 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cbbfe13-6f05-473e-87a1-08d845223559-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5cbbfe13-6f05-473e-87a1-08d845223559" (UID: "5cbbfe13-6f05-473e-87a1-08d845223559"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:34:48.818300 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:48.818246 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/235d3947-3575-41a5-9adc-f6de9ecea65e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "235d3947-3575-41a5-9adc-f6de9ecea65e" (UID: "235d3947-3575-41a5-9adc-f6de9ecea65e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:34:48.819908 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:48.819878 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/235d3947-3575-41a5-9adc-f6de9ecea65e-kube-api-access-d8kns" (OuterVolumeSpecName: "kube-api-access-d8kns") pod "235d3947-3575-41a5-9adc-f6de9ecea65e" (UID: "235d3947-3575-41a5-9adc-f6de9ecea65e"). InnerVolumeSpecName "kube-api-access-d8kns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:34:48.820021 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:48.819998 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cbbfe13-6f05-473e-87a1-08d845223559-kube-api-access-cqmhj" (OuterVolumeSpecName: "kube-api-access-cqmhj") pod "5cbbfe13-6f05-473e-87a1-08d845223559" (UID: "5cbbfe13-6f05-473e-87a1-08d845223559"). InnerVolumeSpecName "kube-api-access-cqmhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:34:48.918803 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:48.918734 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/235d3947-3575-41a5-9adc-f6de9ecea65e-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:34:48.918803 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:48.918758 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8kns\" (UniqueName: \"kubernetes.io/projected/235d3947-3575-41a5-9adc-f6de9ecea65e-kube-api-access-d8kns\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:34:48.918803 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:48.918772 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqmhj\" (UniqueName: \"kubernetes.io/projected/5cbbfe13-6f05-473e-87a1-08d845223559-kube-api-access-cqmhj\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:34:48.918803 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:48.918781 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cbbfe13-6f05-473e-87a1-08d845223559-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:34:49.656811 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:49.656779 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ef263a-788447bc8-6r2bm" Sep 04 22:34:49.657240 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:49.656823 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ef263a-788447bc8-6r2bm" event={"ID":"5cbbfe13-6f05-473e-87a1-08d845223559","Type":"ContainerDied","Data":"256060aebd8ff497200c24dff7cb83e09fb8d0b9caaaa4346ff0a1f75b7cf957"} Sep 04 22:34:49.657240 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:49.656881 2574 scope.go:117] "RemoveContainer" containerID="32796963c98079e614ca69fc2f3a988c5a892e2f78db0f94aeeb7db0fedcab44" Sep 04 22:34:49.658339 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:49.658318 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ef263a-5f7cfd74f9-z45xn" event={"ID":"235d3947-3575-41a5-9adc-f6de9ecea65e","Type":"ContainerDied","Data":"61cd2277240a8ef22a14c5c2c81d4465f7cc341e2e805569676688677315f5e1"} Sep 04 22:34:49.658425 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:49.658339 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ef263a-5f7cfd74f9-z45xn" Sep 04 22:34:49.666196 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:49.666178 2574 scope.go:117] "RemoveContainer" containerID="d9f66b0a3b983e17fcf7e62cf66b75ccf7ede91326ab8ab07dcf52d013f86c70" Sep 04 22:34:49.696498 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:49.696474 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ef263a-5f7cfd74f9-z45xn"] Sep 04 22:34:49.704380 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:49.704358 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ef263a-5f7cfd74f9-z45xn"] Sep 04 22:34:49.720869 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:49.720843 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ef263a-788447bc8-6r2bm"] Sep 04 22:34:49.727891 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:49.727872 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ef263a-788447bc8-6r2bm"] Sep 04 22:34:50.715527 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:50.715482 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="235d3947-3575-41a5-9adc-f6de9ecea65e" path="/var/lib/kubelet/pods/235d3947-3575-41a5-9adc-f6de9ecea65e/volumes" Sep 04 22:34:50.716072 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:50.716048 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cbbfe13-6f05-473e-87a1-08d845223559" path="/var/lib/kubelet/pods/5cbbfe13-6f05-473e-87a1-08d845223559/volumes" Sep 04 22:34:50.716412 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:50.716392 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e95c6ff-bb24-4208-9be9-e04699b69f08" path="/var/lib/kubelet/pods/9e95c6ff-bb24-4208-9be9-e04699b69f08/volumes" Sep 04 22:34:56.919115 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:56.919074 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f64b20-swm7n"] Sep 04 22:34:56.919490 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:56.919470 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cbbfe13-6f05-473e-87a1-08d845223559" containerName="payload-pre-processing" Sep 04 22:34:56.919490 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:56.919486 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cbbfe13-6f05-473e-87a1-08d845223559" containerName="payload-pre-processing" Sep 04 22:34:56.919620 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:56.919494 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="235d3947-3575-41a5-9adc-f6de9ecea65e" containerName="payload-processing" Sep 04 22:34:56.919620 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:56.919500 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="235d3947-3575-41a5-9adc-f6de9ecea65e" containerName="payload-processing" Sep 04 22:34:56.919620 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:56.919509 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e95c6ff-bb24-4208-9be9-e04699b69f08" containerName="revoke-keys" Sep 04 22:34:56.919620 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:56.919516 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e95c6ff-bb24-4208-9be9-e04699b69f08" containerName="revoke-keys" Sep 04 22:34:56.919620 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:56.919596 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e95c6ff-bb24-4208-9be9-e04699b69f08" containerName="revoke-keys" Sep 04 22:34:56.919620 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:56.919605 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cbbfe13-6f05-473e-87a1-08d845223559" containerName="payload-pre-processing" Sep 04 22:34:56.919620 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:56.919613 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="235d3947-3575-41a5-9adc-f6de9ecea65e" containerName="payload-processing" Sep 04 22:34:56.923897 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:56.923877 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f64b20-swm7n" Sep 04 22:34:56.926471 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:56.926449 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 04 22:34:56.926596 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:56.926465 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hc7x4\"" Sep 04 22:34:56.934636 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:56.934611 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f64b20-swm7n"] Sep 04 22:34:56.988374 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:56.988335 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5bfd5bf-6cac-44aa-a9ff-3ed3bce1a473-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f64b20-swm7n\" (UID: \"e5bfd5bf-6cac-44aa-a9ff-3ed3bce1a473\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f64b20-swm7n" Sep 04 22:34:57.089463 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:57.089425 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5bfd5bf-6cac-44aa-a9ff-3ed3bce1a473-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f64b20-swm7n\" (UID: \"e5bfd5bf-6cac-44aa-a9ff-3ed3bce1a473\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f64b20-swm7n" Sep 04 22:34:57.090066 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:57.090047 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5bfd5bf-6cac-44aa-a9ff-3ed3bce1a473-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-f64b20-swm7n\" (UID: \"e5bfd5bf-6cac-44aa-a9ff-3ed3bce1a473\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f64b20-swm7n" Sep 04 22:34:57.235225 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:57.235118 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f64b20-swm7n" Sep 04 22:34:57.519341 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:57.519315 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f64b20-swm7n"] Sep 04 22:34:57.520925 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:34:57.520897 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5bfd5bf_6cac_44aa_a9ff_3ed3bce1a473.slice/crio-59426e0e8c3308a67180a521abf4c2ec181c1b159921fb97719e837e30d8de9e WatchSource:0}: Error finding container 59426e0e8c3308a67180a521abf4c2ec181c1b159921fb97719e837e30d8de9e: Status 404 returned error can't find the container with id 59426e0e8c3308a67180a521abf4c2ec181c1b159921fb97719e837e30d8de9e Sep 04 22:34:57.709390 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:57.709356 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f64b20-swm7n" event={"ID":"e5bfd5bf-6cac-44aa-a9ff-3ed3bce1a473","Type":"ContainerStarted","Data":"59426e0e8c3308a67180a521abf4c2ec181c1b159921fb97719e837e30d8de9e"} Sep 04 22:34:58.716923 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:58.716883 2574 generic.go:356] "Generic (PLEG): container finished" podID="e5bfd5bf-6cac-44aa-a9ff-3ed3bce1a473" containerID="00c0d472f61e9e2f1b2e27c006f0ea4acefd1bd85391923a8bebd13cfa0b98f5" exitCode=0 Sep 04 22:34:58.717293 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:58.716965 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f64b20-swm7n" event={"ID":"e5bfd5bf-6cac-44aa-a9ff-3ed3bce1a473","Type":"ContainerDied","Data":"00c0d472f61e9e2f1b2e27c006f0ea4acefd1bd85391923a8bebd13cfa0b98f5"} Sep 04 22:34:59.851661 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:59.851635 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f64b20-swm7n" Sep 04 22:34:59.916975 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:59.916945 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5bfd5bf-6cac-44aa-a9ff-3ed3bce1a473-maas-api-service-ca\") pod \"e5bfd5bf-6cac-44aa-a9ff-3ed3bce1a473\" (UID: \"e5bfd5bf-6cac-44aa-a9ff-3ed3bce1a473\") " Sep 04 22:34:59.917178 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:34:59.917155 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5bfd5bf-6cac-44aa-a9ff-3ed3bce1a473-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e5bfd5bf-6cac-44aa-a9ff-3ed3bce1a473" (UID: "e5bfd5bf-6cac-44aa-a9ff-3ed3bce1a473"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:35:00.017730 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:00.017700 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5bfd5bf-6cac-44aa-a9ff-3ed3bce1a473-maas-api-service-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:35:00.728779 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:00.728751 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f64b20-swm7n" event={"ID":"e5bfd5bf-6cac-44aa-a9ff-3ed3bce1a473","Type":"ContainerDied","Data":"59426e0e8c3308a67180a521abf4c2ec181c1b159921fb97719e837e30d8de9e"} Sep 04 22:35:00.728908 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:00.728805 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59426e0e8c3308a67180a521abf4c2ec181c1b159921fb97719e837e30d8de9e" Sep 04 22:35:00.728908 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:00.728758 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f64b20-swm7n" Sep 04 22:35:07.691123 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:07.691093 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f64b20-768cdb7d5-f5vz6"] Sep 04 22:35:07.691651 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:07.691302 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f64b20-768cdb7d5-f5vz6" podUID="fd097038-b010-4ea0-91c8-8ab9fe80a57e" containerName="payload-processing" containerID="cri-o://b96b51b92bc911b066dcdce076a97344ebdb6dd1fbc9336c462de5d800eadc86" gracePeriod=30 Sep 04 22:35:07.716268 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:07.716228 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64b20-fb5bd6b54-fpcwb"] Sep 04 22:35:07.716525 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:07.716475 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64b20-fb5bd6b54-fpcwb" podUID="9abe2458-1046-4ebd-8c55-108ec7c14c6d" containerName="payload-pre-processing" containerID="cri-o://cf6f78d15180594b7d850f2f5cef00dff5f3e4dc0f6be7c62ad5657e9e263201" gracePeriod=30 Sep 04 22:35:07.996024 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:07.995990 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f64b20-768cdb7d5-f5vz6" Sep 04 22:35:07.998707 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:07.998687 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64b20-fb5bd6b54-fpcwb" Sep 04 22:35:08.074243 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:08.074210 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f64b20-swm7n"] Sep 04 22:35:08.079208 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:08.079174 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-f64b20-swm7n"] Sep 04 22:35:08.088181 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:08.088154 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd097038-b010-4ea0-91c8-8ab9fe80a57e-plugins-config-volume\") pod \"fd097038-b010-4ea0-91c8-8ab9fe80a57e\" (UID: \"fd097038-b010-4ea0-91c8-8ab9fe80a57e\") " Sep 04 22:35:08.088336 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:08.088196 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4llf\" (UniqueName: \"kubernetes.io/projected/9abe2458-1046-4ebd-8c55-108ec7c14c6d-kube-api-access-w4llf\") pod \"9abe2458-1046-4ebd-8c55-108ec7c14c6d\" (UID: \"9abe2458-1046-4ebd-8c55-108ec7c14c6d\") " Sep 04 22:35:08.088336 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:08.088267 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9abe2458-1046-4ebd-8c55-108ec7c14c6d-plugins-config-volume\") pod \"9abe2458-1046-4ebd-8c55-108ec7c14c6d\" (UID: \"9abe2458-1046-4ebd-8c55-108ec7c14c6d\") " Sep 04 22:35:08.088336 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:08.088300 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktvwp\" (UniqueName: \"kubernetes.io/projected/fd097038-b010-4ea0-91c8-8ab9fe80a57e-kube-api-access-ktvwp\") pod \"fd097038-b010-4ea0-91c8-8ab9fe80a57e\" (UID: \"fd097038-b010-4ea0-91c8-8ab9fe80a57e\") " Sep 04 22:35:08.088689 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:08.088661 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9abe2458-1046-4ebd-8c55-108ec7c14c6d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9abe2458-1046-4ebd-8c55-108ec7c14c6d" (UID: "9abe2458-1046-4ebd-8c55-108ec7c14c6d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:35:08.088689 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:08.088656 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd097038-b010-4ea0-91c8-8ab9fe80a57e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fd097038-b010-4ea0-91c8-8ab9fe80a57e" (UID: "fd097038-b010-4ea0-91c8-8ab9fe80a57e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:35:08.090458 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:08.090428 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9abe2458-1046-4ebd-8c55-108ec7c14c6d-kube-api-access-w4llf" (OuterVolumeSpecName: "kube-api-access-w4llf") pod "9abe2458-1046-4ebd-8c55-108ec7c14c6d" (UID: "9abe2458-1046-4ebd-8c55-108ec7c14c6d"). InnerVolumeSpecName "kube-api-access-w4llf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:35:08.090544 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:08.090484 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd097038-b010-4ea0-91c8-8ab9fe80a57e-kube-api-access-ktvwp" (OuterVolumeSpecName: "kube-api-access-ktvwp") pod "fd097038-b010-4ea0-91c8-8ab9fe80a57e" (UID: "fd097038-b010-4ea0-91c8-8ab9fe80a57e"). InnerVolumeSpecName "kube-api-access-ktvwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:35:08.189151 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:08.189118 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktvwp\" (UniqueName: \"kubernetes.io/projected/fd097038-b010-4ea0-91c8-8ab9fe80a57e-kube-api-access-ktvwp\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:35:08.189151 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:08.189144 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd097038-b010-4ea0-91c8-8ab9fe80a57e-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:35:08.189151 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:08.189153 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4llf\" (UniqueName: \"kubernetes.io/projected/9abe2458-1046-4ebd-8c55-108ec7c14c6d-kube-api-access-w4llf\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:35:08.189376 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:08.189164 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9abe2458-1046-4ebd-8c55-108ec7c14c6d-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:35:08.714871 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:08.714571 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5bfd5bf-6cac-44aa-a9ff-3ed3bce1a473" path="/var/lib/kubelet/pods/e5bfd5bf-6cac-44aa-a9ff-3ed3bce1a473/volumes" Sep 04 22:35:08.771897 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:08.771864 2574 generic.go:356] "Generic (PLEG): container finished" podID="9abe2458-1046-4ebd-8c55-108ec7c14c6d" containerID="cf6f78d15180594b7d850f2f5cef00dff5f3e4dc0f6be7c62ad5657e9e263201" exitCode=0 Sep 04 22:35:08.772069 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:08.771913 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64b20-fb5bd6b54-fpcwb" Sep 04 22:35:08.772069 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:08.771955 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64b20-fb5bd6b54-fpcwb" event={"ID":"9abe2458-1046-4ebd-8c55-108ec7c14c6d","Type":"ContainerDied","Data":"cf6f78d15180594b7d850f2f5cef00dff5f3e4dc0f6be7c62ad5657e9e263201"} Sep 04 22:35:08.772069 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:08.772005 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64b20-fb5bd6b54-fpcwb" event={"ID":"9abe2458-1046-4ebd-8c55-108ec7c14c6d","Type":"ContainerDied","Data":"ceb37ef365b4ac845631b40b2b051b0be3f712cc456b1fa69ee5b80b83c63423"} Sep 04 22:35:08.772069 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:08.772031 2574 scope.go:117] "RemoveContainer" containerID="cf6f78d15180594b7d850f2f5cef00dff5f3e4dc0f6be7c62ad5657e9e263201" Sep 04 22:35:08.773463 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:08.773443 2574 generic.go:356] "Generic (PLEG): container finished" podID="fd097038-b010-4ea0-91c8-8ab9fe80a57e" containerID="b96b51b92bc911b066dcdce076a97344ebdb6dd1fbc9336c462de5d800eadc86" exitCode=0 Sep 04 22:35:08.773541 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:08.773482 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f64b20-768cdb7d5-f5vz6" Sep 04 22:35:08.773541 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:08.773496 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f64b20-768cdb7d5-f5vz6" event={"ID":"fd097038-b010-4ea0-91c8-8ab9fe80a57e","Type":"ContainerDied","Data":"b96b51b92bc911b066dcdce076a97344ebdb6dd1fbc9336c462de5d800eadc86"} Sep 04 22:35:08.773694 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:08.773585 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-f64b20-768cdb7d5-f5vz6" event={"ID":"fd097038-b010-4ea0-91c8-8ab9fe80a57e","Type":"ContainerDied","Data":"f2411bcd616f1b584ccfae12b734f8c1d74379db539d534e9a4b5abd1758d5dd"} Sep 04 22:35:08.781218 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:08.781199 2574 scope.go:117] "RemoveContainer" containerID="cf6f78d15180594b7d850f2f5cef00dff5f3e4dc0f6be7c62ad5657e9e263201" Sep 04 22:35:08.781492 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:35:08.781475 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf6f78d15180594b7d850f2f5cef00dff5f3e4dc0f6be7c62ad5657e9e263201\": container with ID starting with cf6f78d15180594b7d850f2f5cef00dff5f3e4dc0f6be7c62ad5657e9e263201 not found: ID does not exist" containerID="cf6f78d15180594b7d850f2f5cef00dff5f3e4dc0f6be7c62ad5657e9e263201" Sep 04 22:35:08.781540 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:08.781500 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf6f78d15180594b7d850f2f5cef00dff5f3e4dc0f6be7c62ad5657e9e263201"} err="failed to get container status \"cf6f78d15180594b7d850f2f5cef00dff5f3e4dc0f6be7c62ad5657e9e263201\": rpc error: code = NotFound desc = could not find container \"cf6f78d15180594b7d850f2f5cef00dff5f3e4dc0f6be7c62ad5657e9e263201\": container with ID starting with cf6f78d15180594b7d850f2f5cef00dff5f3e4dc0f6be7c62ad5657e9e263201 not found: ID does not exist" Sep 04 22:35:08.781540 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:08.781516 2574 scope.go:117] "RemoveContainer" containerID="b96b51b92bc911b066dcdce076a97344ebdb6dd1fbc9336c462de5d800eadc86" Sep 04 22:35:08.789333 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:08.789318 2574 scope.go:117] "RemoveContainer" containerID="b96b51b92bc911b066dcdce076a97344ebdb6dd1fbc9336c462de5d800eadc86" Sep 04 22:35:08.789622 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:35:08.789606 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b96b51b92bc911b066dcdce076a97344ebdb6dd1fbc9336c462de5d800eadc86\": container with ID starting with b96b51b92bc911b066dcdce076a97344ebdb6dd1fbc9336c462de5d800eadc86 not found: ID does not exist" containerID="b96b51b92bc911b066dcdce076a97344ebdb6dd1fbc9336c462de5d800eadc86" Sep 04 22:35:08.789671 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:08.789628 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b96b51b92bc911b066dcdce076a97344ebdb6dd1fbc9336c462de5d800eadc86"} err="failed to get container status \"b96b51b92bc911b066dcdce076a97344ebdb6dd1fbc9336c462de5d800eadc86\": rpc error: code = NotFound desc = could not find container \"b96b51b92bc911b066dcdce076a97344ebdb6dd1fbc9336c462de5d800eadc86\": container with ID starting with b96b51b92bc911b066dcdce076a97344ebdb6dd1fbc9336c462de5d800eadc86 not found: ID does not exist" Sep 04 22:35:08.805986 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:08.805966 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f64b20-768cdb7d5-f5vz6"] Sep 04 22:35:08.810645 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:08.810623 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-f64b20-768cdb7d5-f5vz6"] Sep 04 22:35:08.841929 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:08.841906 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64b20-fb5bd6b54-fpcwb"] Sep 04 22:35:08.845257 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:08.845236 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-f64b20-fb5bd6b54-fpcwb"] Sep 04 22:35:10.715164 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:10.715060 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9abe2458-1046-4ebd-8c55-108ec7c14c6d" path="/var/lib/kubelet/pods/9abe2458-1046-4ebd-8c55-108ec7c14c6d/volumes" Sep 04 22:35:10.715642 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:10.715602 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd097038-b010-4ea0-91c8-8ab9fe80a57e" path="/var/lib/kubelet/pods/fd097038-b010-4ea0-91c8-8ab9fe80a57e/volumes" Sep 04 22:35:17.016918 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:17.016888 2574 scope.go:117] "RemoveContainer" containerID="bfe71659afa5745b9b29863341babf9a59fe7a5b39edac749d2c97265391f053" Sep 04 22:35:18.106280 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:18.106249 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a12dcf-l27xh"] Sep 04 22:35:18.106693 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:18.106588 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5bfd5bf-6cac-44aa-a9ff-3ed3bce1a473" containerName="revoke-keys" Sep 04 22:35:18.106693 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:18.106599 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5bfd5bf-6cac-44aa-a9ff-3ed3bce1a473" containerName="revoke-keys" Sep 04 22:35:18.106693 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:18.106616 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9abe2458-1046-4ebd-8c55-108ec7c14c6d" containerName="payload-pre-processing" Sep 04 22:35:18.106693 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:18.106621 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9abe2458-1046-4ebd-8c55-108ec7c14c6d" containerName="payload-pre-processing" Sep 04 22:35:18.106693 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:18.106630 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd097038-b010-4ea0-91c8-8ab9fe80a57e" containerName="payload-processing" Sep 04 22:35:18.106693 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:18.106636 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd097038-b010-4ea0-91c8-8ab9fe80a57e" containerName="payload-processing" Sep 04 22:35:18.106693 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:18.106693 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd097038-b010-4ea0-91c8-8ab9fe80a57e" containerName="payload-processing" Sep 04 22:35:18.106952 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:18.106700 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9abe2458-1046-4ebd-8c55-108ec7c14c6d" containerName="payload-pre-processing" Sep 04 22:35:18.106952 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:18.106708 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5bfd5bf-6cac-44aa-a9ff-3ed3bce1a473" containerName="revoke-keys" Sep 04 22:35:18.109680 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:18.109663 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a12dcf-l27xh" Sep 04 22:35:18.112716 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:18.112695 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hc7x4\"" Sep 04 22:35:18.113172 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:18.113151 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 04 22:35:18.132507 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:18.132482 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a12dcf-l27xh"] Sep 04 22:35:18.275930 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:18.275890 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/961158a0-51d2-42d2-a63f-f79283ce58ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a12dcf-l27xh\" (UID: \"961158a0-51d2-42d2-a63f-f79283ce58ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a12dcf-l27xh" Sep 04 22:35:18.377132 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:18.377004 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/961158a0-51d2-42d2-a63f-f79283ce58ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a12dcf-l27xh\" (UID: \"961158a0-51d2-42d2-a63f-f79283ce58ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a12dcf-l27xh" Sep 04 22:35:18.377835 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:18.377803 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/961158a0-51d2-42d2-a63f-f79283ce58ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-a12dcf-l27xh\" (UID: \"961158a0-51d2-42d2-a63f-f79283ce58ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a12dcf-l27xh" Sep 04 22:35:18.419716 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:18.419647 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a12dcf-l27xh" Sep 04 22:35:18.705663 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:18.705637 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a12dcf-l27xh"] Sep 04 22:35:18.707245 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:35:18.707218 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod961158a0_51d2_42d2_a63f_f79283ce58ea.slice/crio-ec375736f2896a0b73b13dda3694e8bae3e55bee81d0a250ea5e731d74cef6ec WatchSource:0}: Error finding container ec375736f2896a0b73b13dda3694e8bae3e55bee81d0a250ea5e731d74cef6ec: Status 404 returned error can't find the container with id ec375736f2896a0b73b13dda3694e8bae3e55bee81d0a250ea5e731d74cef6ec Sep 04 22:35:18.832316 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:18.832281 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a12dcf-l27xh" event={"ID":"961158a0-51d2-42d2-a63f-f79283ce58ea","Type":"ContainerStarted","Data":"ec375736f2896a0b73b13dda3694e8bae3e55bee81d0a250ea5e731d74cef6ec"} Sep 04 22:35:19.841020 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:19.840979 2574 generic.go:356] "Generic (PLEG): container finished" podID="961158a0-51d2-42d2-a63f-f79283ce58ea" containerID="47fc8b7c9852614ef923d603595ad29c45277e66b15a5bc2d0a38e775ad3f8fb" exitCode=0 Sep 04 22:35:19.841417 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:19.841068 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a12dcf-l27xh" event={"ID":"961158a0-51d2-42d2-a63f-f79283ce58ea","Type":"ContainerDied","Data":"47fc8b7c9852614ef923d603595ad29c45277e66b15a5bc2d0a38e775ad3f8fb"} Sep 04 22:35:20.973944 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:20.973920 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a12dcf-l27xh" Sep 04 22:35:21.102722 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:21.102634 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/961158a0-51d2-42d2-a63f-f79283ce58ea-maas-api-service-ca\") pod \"961158a0-51d2-42d2-a63f-f79283ce58ea\" (UID: \"961158a0-51d2-42d2-a63f-f79283ce58ea\") " Sep 04 22:35:21.103022 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:21.102997 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/961158a0-51d2-42d2-a63f-f79283ce58ea-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "961158a0-51d2-42d2-a63f-f79283ce58ea" (UID: "961158a0-51d2-42d2-a63f-f79283ce58ea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:35:21.203806 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:21.203777 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/961158a0-51d2-42d2-a63f-f79283ce58ea-maas-api-service-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:35:21.854074 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:21.854045 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a12dcf-l27xh" Sep 04 22:35:21.854259 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:21.854083 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a12dcf-l27xh" event={"ID":"961158a0-51d2-42d2-a63f-f79283ce58ea","Type":"ContainerDied","Data":"ec375736f2896a0b73b13dda3694e8bae3e55bee81d0a250ea5e731d74cef6ec"} Sep 04 22:35:21.854259 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:21.854130 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec375736f2896a0b73b13dda3694e8bae3e55bee81d0a250ea5e731d74cef6ec" Sep 04 22:35:29.104716 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:29.104682 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a12dcf-29809350-r8mh5"] Sep 04 22:35:29.110908 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:29.110878 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a12dcf-29809350-r8mh5"] Sep 04 22:35:29.180517 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:29.180485 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a12dcf-fcb99bc45-56zrv"] Sep 04 22:35:29.180776 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:29.180751 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a12dcf-fcb99bc45-56zrv" podUID="73657179-cc99-4ba4-acbf-7b9108cceaca" containerName="payload-processing" containerID="cri-o://b3214360e16d0f2650f366bfb671f10256dc130a5e5400b2c24fd99a7baa1590" gracePeriod=30 Sep 04 22:35:29.429487 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:29.429459 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a12dcf-fcb99bc45-56zrv" Sep 04 22:35:29.470887 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:29.470853 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhjzx\" (UniqueName: \"kubernetes.io/projected/73657179-cc99-4ba4-acbf-7b9108cceaca-kube-api-access-qhjzx\") pod \"73657179-cc99-4ba4-acbf-7b9108cceaca\" (UID: \"73657179-cc99-4ba4-acbf-7b9108cceaca\") " Sep 04 22:35:29.471092 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:29.470903 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73657179-cc99-4ba4-acbf-7b9108cceaca-plugins-config-volume\") pod \"73657179-cc99-4ba4-acbf-7b9108cceaca\" (UID: \"73657179-cc99-4ba4-acbf-7b9108cceaca\") " Sep 04 22:35:29.471325 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:29.471298 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73657179-cc99-4ba4-acbf-7b9108cceaca-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "73657179-cc99-4ba4-acbf-7b9108cceaca" (UID: "73657179-cc99-4ba4-acbf-7b9108cceaca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:35:29.474124 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:29.473599 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73657179-cc99-4ba4-acbf-7b9108cceaca-kube-api-access-qhjzx" (OuterVolumeSpecName: "kube-api-access-qhjzx") pod "73657179-cc99-4ba4-acbf-7b9108cceaca" (UID: "73657179-cc99-4ba4-acbf-7b9108cceaca"). InnerVolumeSpecName "kube-api-access-qhjzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:35:29.572030 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:29.572002 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhjzx\" (UniqueName: \"kubernetes.io/projected/73657179-cc99-4ba4-acbf-7b9108cceaca-kube-api-access-qhjzx\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:35:29.572030 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:29.572028 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73657179-cc99-4ba4-acbf-7b9108cceaca-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:35:29.689216 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:29.689130 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a12dcf-l27xh"] Sep 04 22:35:29.696924 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:29.696894 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-a12dcf-l27xh"] Sep 04 22:35:29.899983 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:29.899941 2574 generic.go:356] "Generic (PLEG): container finished" podID="73657179-cc99-4ba4-acbf-7b9108cceaca" containerID="b3214360e16d0f2650f366bfb671f10256dc130a5e5400b2c24fd99a7baa1590" exitCode=0 Sep 04 22:35:29.899983 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:29.899984 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a12dcf-fcb99bc45-56zrv" Sep 04 22:35:29.900181 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:29.899985 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a12dcf-fcb99bc45-56zrv" event={"ID":"73657179-cc99-4ba4-acbf-7b9108cceaca","Type":"ContainerDied","Data":"b3214360e16d0f2650f366bfb671f10256dc130a5e5400b2c24fd99a7baa1590"} Sep 04 22:35:29.900181 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:29.900113 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-a12dcf-fcb99bc45-56zrv" event={"ID":"73657179-cc99-4ba4-acbf-7b9108cceaca","Type":"ContainerDied","Data":"fa87583869b15938dd89ee27f157d03c168da5c6f61823f3526185b150e332f7"} Sep 04 22:35:29.900181 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:29.900144 2574 scope.go:117] "RemoveContainer" containerID="b3214360e16d0f2650f366bfb671f10256dc130a5e5400b2c24fd99a7baa1590" Sep 04 22:35:29.909805 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:29.909787 2574 scope.go:117] "RemoveContainer" containerID="b3214360e16d0f2650f366bfb671f10256dc130a5e5400b2c24fd99a7baa1590" Sep 04 22:35:29.910089 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:35:29.910071 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3214360e16d0f2650f366bfb671f10256dc130a5e5400b2c24fd99a7baa1590\": container with ID starting with b3214360e16d0f2650f366bfb671f10256dc130a5e5400b2c24fd99a7baa1590 not found: ID does not exist" containerID="b3214360e16d0f2650f366bfb671f10256dc130a5e5400b2c24fd99a7baa1590" Sep 04 22:35:29.910160 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:29.910099 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3214360e16d0f2650f366bfb671f10256dc130a5e5400b2c24fd99a7baa1590"} err="failed to get container status \"b3214360e16d0f2650f366bfb671f10256dc130a5e5400b2c24fd99a7baa1590\": rpc error: code = NotFound desc = could not find container \"b3214360e16d0f2650f366bfb671f10256dc130a5e5400b2c24fd99a7baa1590\": container with ID starting with b3214360e16d0f2650f366bfb671f10256dc130a5e5400b2c24fd99a7baa1590 not found: ID does not exist" Sep 04 22:35:29.932888 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:29.932859 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a12dcf-fcb99bc45-56zrv"] Sep 04 22:35:29.938124 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:29.938101 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-a12dcf-fcb99bc45-56zrv"] Sep 04 22:35:30.714977 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:30.714940 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73657179-cc99-4ba4-acbf-7b9108cceaca" path="/var/lib/kubelet/pods/73657179-cc99-4ba4-acbf-7b9108cceaca/volumes" Sep 04 22:35:30.715463 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:30.715404 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="961158a0-51d2-42d2-a63f-f79283ce58ea" path="/var/lib/kubelet/pods/961158a0-51d2-42d2-a63f-f79283ce58ea/volumes" Sep 04 22:35:30.715724 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:30.715705 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcef4ead-0bb5-4476-9873-4ef8712f4821" path="/var/lib/kubelet/pods/fcef4ead-0bb5-4476-9873-4ef8712f4821/volumes" Sep 04 22:35:51.129271 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:51.129233 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2df128-d65c6b75f-nn7pw"] Sep 04 22:35:51.129813 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:51.129569 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73657179-cc99-4ba4-acbf-7b9108cceaca" containerName="payload-processing" Sep 04 22:35:51.129813 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:51.129581 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="73657179-cc99-4ba4-acbf-7b9108cceaca" containerName="payload-processing" Sep 04 22:35:51.129813 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:51.129598 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="961158a0-51d2-42d2-a63f-f79283ce58ea" containerName="revoke-keys" Sep 04 22:35:51.129813 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:51.129603 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="961158a0-51d2-42d2-a63f-f79283ce58ea" containerName="revoke-keys" Sep 04 22:35:51.129813 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:51.129654 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="961158a0-51d2-42d2-a63f-f79283ce58ea" containerName="revoke-keys" Sep 04 22:35:51.129813 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:51.129664 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="73657179-cc99-4ba4-acbf-7b9108cceaca" containerName="payload-processing" Sep 04 22:35:51.132723 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:51.132704 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2df128-d65c6b75f-nn7pw" Sep 04 22:35:51.133752 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:51.133734 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2df128-64b8579d9d-ljk6b"] Sep 04 22:35:51.136907 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:51.136893 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2df128-64b8579d9d-ljk6b" Sep 04 22:35:51.146356 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:51.146339 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2df128\"" Sep 04 22:35:51.153537 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:51.153519 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2df128-dockercfg-2zzdh\"" Sep 04 22:35:51.193957 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:51.192823 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2df128-d65c6b75f-nn7pw"] Sep 04 22:35:51.212376 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:51.212354 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2df128-64b8579d9d-ljk6b"] Sep 04 22:35:51.262402 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:51.262367 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbzdn\" (UniqueName: \"kubernetes.io/projected/acbd20b5-5426-4b59-bd4d-027faa54adb0-kube-api-access-jbzdn\") pod \"payload-pre-processing-e2e-ipp-a-2df128-64b8579d9d-ljk6b\" (UID: \"acbd20b5-5426-4b59-bd4d-027faa54adb0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2df128-64b8579d9d-ljk6b" Sep 04 22:35:51.262402 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:51.262407 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1594f79a-a774-4a97-9aba-0ccb03c381bc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2df128-d65c6b75f-nn7pw\" (UID: \"1594f79a-a774-4a97-9aba-0ccb03c381bc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2df128-d65c6b75f-nn7pw" Sep 04 22:35:51.262629 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:51.262439 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjtld\" (UniqueName: \"kubernetes.io/projected/1594f79a-a774-4a97-9aba-0ccb03c381bc-kube-api-access-bjtld\") pod \"payload-processing-e2e-ipp-a-2df128-d65c6b75f-nn7pw\" (UID: \"1594f79a-a774-4a97-9aba-0ccb03c381bc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2df128-d65c6b75f-nn7pw" Sep 04 22:35:51.262629 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:51.262494 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acbd20b5-5426-4b59-bd4d-027faa54adb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2df128-64b8579d9d-ljk6b\" (UID: \"acbd20b5-5426-4b59-bd4d-027faa54adb0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2df128-64b8579d9d-ljk6b" Sep 04 22:35:51.363654 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:51.363621 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acbd20b5-5426-4b59-bd4d-027faa54adb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2df128-64b8579d9d-ljk6b\" (UID: \"acbd20b5-5426-4b59-bd4d-027faa54adb0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2df128-64b8579d9d-ljk6b" Sep 04 22:35:51.363814 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:51.363718 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbzdn\" (UniqueName: \"kubernetes.io/projected/acbd20b5-5426-4b59-bd4d-027faa54adb0-kube-api-access-jbzdn\") pod \"payload-pre-processing-e2e-ipp-a-2df128-64b8579d9d-ljk6b\" (UID: \"acbd20b5-5426-4b59-bd4d-027faa54adb0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2df128-64b8579d9d-ljk6b" Sep 04 22:35:51.363814 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:51.363772 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1594f79a-a774-4a97-9aba-0ccb03c381bc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2df128-d65c6b75f-nn7pw\" (UID: \"1594f79a-a774-4a97-9aba-0ccb03c381bc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2df128-d65c6b75f-nn7pw" Sep 04 22:35:51.363896 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:51.363816 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjtld\" (UniqueName: \"kubernetes.io/projected/1594f79a-a774-4a97-9aba-0ccb03c381bc-kube-api-access-bjtld\") pod \"payload-processing-e2e-ipp-a-2df128-d65c6b75f-nn7pw\" (UID: \"1594f79a-a774-4a97-9aba-0ccb03c381bc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2df128-d65c6b75f-nn7pw" Sep 04 22:35:51.364417 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:51.364397 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acbd20b5-5426-4b59-bd4d-027faa54adb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2df128-64b8579d9d-ljk6b\" (UID: \"acbd20b5-5426-4b59-bd4d-027faa54adb0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2df128-64b8579d9d-ljk6b" Sep 04 22:35:51.364484 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:51.364467 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1594f79a-a774-4a97-9aba-0ccb03c381bc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2df128-d65c6b75f-nn7pw\" (UID: \"1594f79a-a774-4a97-9aba-0ccb03c381bc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2df128-d65c6b75f-nn7pw" Sep 04 22:35:51.392628 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:51.392567 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjtld\" (UniqueName: \"kubernetes.io/projected/1594f79a-a774-4a97-9aba-0ccb03c381bc-kube-api-access-bjtld\") pod \"payload-processing-e2e-ipp-a-2df128-d65c6b75f-nn7pw\" (UID: \"1594f79a-a774-4a97-9aba-0ccb03c381bc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2df128-d65c6b75f-nn7pw" Sep 04 22:35:51.404633 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:51.404609 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbzdn\" (UniqueName: \"kubernetes.io/projected/acbd20b5-5426-4b59-bd4d-027faa54adb0-kube-api-access-jbzdn\") pod \"payload-pre-processing-e2e-ipp-a-2df128-64b8579d9d-ljk6b\" (UID: \"acbd20b5-5426-4b59-bd4d-027faa54adb0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2df128-64b8579d9d-ljk6b" Sep 04 22:35:51.443877 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:51.443855 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2df128-d65c6b75f-nn7pw" Sep 04 22:35:51.449956 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:51.449940 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2df128-64b8579d9d-ljk6b" Sep 04 22:35:51.733591 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:51.733562 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2df128-d65c6b75f-nn7pw"] Sep 04 22:35:51.735251 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:35:51.735218 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1594f79a_a774_4a97_9aba_0ccb03c381bc.slice/crio-10530d33e1b78eda26dc1aed6e55fa919e37018a36eaf3f073b9e16019fb58a5 WatchSource:0}: Error finding container 10530d33e1b78eda26dc1aed6e55fa919e37018a36eaf3f073b9e16019fb58a5: Status 404 returned error can't find the container with id 10530d33e1b78eda26dc1aed6e55fa919e37018a36eaf3f073b9e16019fb58a5 Sep 04 22:35:51.769514 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:51.769483 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2df128-64b8579d9d-ljk6b"] Sep 04 22:35:51.772712 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:35:51.772680 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacbd20b5_5426_4b59_bd4d_027faa54adb0.slice/crio-a6224874750fe14cbd053771af204ca2103ca735dc4cfbb3883d1d88402c95a5 WatchSource:0}: Error finding container a6224874750fe14cbd053771af204ca2103ca735dc4cfbb3883d1d88402c95a5: Status 404 returned error can't find the container with id a6224874750fe14cbd053771af204ca2103ca735dc4cfbb3883d1d88402c95a5 Sep 04 22:35:52.026120 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:52.026071 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2df128-d65c6b75f-nn7pw" event={"ID":"1594f79a-a774-4a97-9aba-0ccb03c381bc","Type":"ContainerStarted","Data":"67654b607575eb78fead4ee6a3c468bf9eee65098a4ae75c2da565c3938f2563"} Sep 04 22:35:52.026317 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:52.026138 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2df128-d65c6b75f-nn7pw" event={"ID":"1594f79a-a774-4a97-9aba-0ccb03c381bc","Type":"ContainerStarted","Data":"10530d33e1b78eda26dc1aed6e55fa919e37018a36eaf3f073b9e16019fb58a5"} Sep 04 22:35:52.026317 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:52.026168 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2df128-d65c6b75f-nn7pw" Sep 04 22:35:52.028043 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:52.028005 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2df128-64b8579d9d-ljk6b" event={"ID":"acbd20b5-5426-4b59-bd4d-027faa54adb0","Type":"ContainerStarted","Data":"ca799df6ba6919b0bf255d71c1b152253ac1eed51f9f75f99292337802eaacbc"} Sep 04 22:35:52.028177 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:52.028060 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2df128-64b8579d9d-ljk6b" event={"ID":"acbd20b5-5426-4b59-bd4d-027faa54adb0","Type":"ContainerStarted","Data":"a6224874750fe14cbd053771af204ca2103ca735dc4cfbb3883d1d88402c95a5"} Sep 04 22:35:52.028177 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:52.028092 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2df128-64b8579d9d-ljk6b" Sep 04 22:35:52.076707 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:52.076661 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-2df128-d65c6b75f-nn7pw" podStartSLOduration=1.076644945 podStartE2EDuration="1.076644945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:35:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:35:52.069363444 +0000 UTC m=+1655.833839058" watchObservedRunningTime="2026-09-04 22:35:52.076644945 +0000 UTC m=+1655.841120624" Sep 04 22:35:52.120654 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:35:52.120604 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2df128-64b8579d9d-ljk6b" podStartSLOduration=1.12059039 podStartE2EDuration="1.12059039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:35:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:35:52.118300393 +0000 UTC m=+1655.882776005" watchObservedRunningTime="2026-09-04 22:35:52.12059039 +0000 UTC m=+1655.885066006" Sep 04 22:36:03.036085 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:36:03.036047 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2df128-64b8579d9d-ljk6b" Sep 04 22:36:03.036665 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:36:03.036213 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2df128-d65c6b75f-nn7pw" Sep 04 22:36:17.062404 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:36:17.062372 2574 scope.go:117] "RemoveContainer" containerID="2a915238c7fde2b026655793d7eefb1d97deb30243a4ad57989b330169af2874" Sep 04 22:36:17.070306 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:36:17.070287 2574 scope.go:117] "RemoveContainer" containerID="c56533d481687ae6b52c0f39216c8a047e563c125281f639e309d009cc64f9a4" Sep 04 22:36:19.580021 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:36:19.579994 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ecfcad-7d79987f5f-czgts"] Sep 04 22:36:19.583343 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:36:19.583327 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ecfcad-7d79987f5f-czgts" Sep 04 22:36:19.586036 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:36:19.586007 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ecfcad\"" Sep 04 22:36:19.607132 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:36:19.607107 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kwl9\" (UniqueName: \"kubernetes.io/projected/6c2c6b3a-2589-45fa-9e93-9483f742ede4-kube-api-access-6kwl9\") pod \"payload-processing-e2e-ipp-b-ecfcad-7d79987f5f-czgts\" (UID: \"6c2c6b3a-2589-45fa-9e93-9483f742ede4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ecfcad-7d79987f5f-czgts" Sep 04 22:36:19.607242 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:36:19.607155 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c2c6b3a-2589-45fa-9e93-9483f742ede4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ecfcad-7d79987f5f-czgts\" (UID: \"6c2c6b3a-2589-45fa-9e93-9483f742ede4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ecfcad-7d79987f5f-czgts" Sep 04 22:36:19.610862 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:36:19.610842 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ecfcad-7d79987f5f-czgts"] Sep 04 22:36:19.708650 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:36:19.708623 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kwl9\" (UniqueName: \"kubernetes.io/projected/6c2c6b3a-2589-45fa-9e93-9483f742ede4-kube-api-access-6kwl9\") pod \"payload-processing-e2e-ipp-b-ecfcad-7d79987f5f-czgts\" (UID: \"6c2c6b3a-2589-45fa-9e93-9483f742ede4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ecfcad-7d79987f5f-czgts" Sep 04 22:36:19.708810 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:36:19.708659 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c2c6b3a-2589-45fa-9e93-9483f742ede4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ecfcad-7d79987f5f-czgts\" (UID: \"6c2c6b3a-2589-45fa-9e93-9483f742ede4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ecfcad-7d79987f5f-czgts" Sep 04 22:36:19.709297 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:36:19.709279 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c2c6b3a-2589-45fa-9e93-9483f742ede4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ecfcad-7d79987f5f-czgts\" (UID: \"6c2c6b3a-2589-45fa-9e93-9483f742ede4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ecfcad-7d79987f5f-czgts" Sep 04 22:36:19.719429 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:36:19.719403 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kwl9\" (UniqueName: \"kubernetes.io/projected/6c2c6b3a-2589-45fa-9e93-9483f742ede4-kube-api-access-6kwl9\") pod \"payload-processing-e2e-ipp-b-ecfcad-7d79987f5f-czgts\" (UID: \"6c2c6b3a-2589-45fa-9e93-9483f742ede4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ecfcad-7d79987f5f-czgts" Sep 04 22:36:19.893933 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:36:19.893851 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ecfcad-7d79987f5f-czgts" Sep 04 22:36:20.174718 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:36:20.174686 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c2c6b3a_2589_45fa_9e93_9483f742ede4.slice/crio-ee01dcbdab774d37bb7df58ab5d6b28d911e70d48d6c272df3c601443631a20a WatchSource:0}: Error finding container ee01dcbdab774d37bb7df58ab5d6b28d911e70d48d6c272df3c601443631a20a: Status 404 returned error can't find the container with id ee01dcbdab774d37bb7df58ab5d6b28d911e70d48d6c272df3c601443631a20a Sep 04 22:36:20.180679 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:36:20.180652 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ecfcad-7d79987f5f-czgts"] Sep 04 22:36:20.186502 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:36:20.186472 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ecfcad-7d79987f5f-czgts" event={"ID":"6c2c6b3a-2589-45fa-9e93-9483f742ede4","Type":"ContainerStarted","Data":"ee01dcbdab774d37bb7df58ab5d6b28d911e70d48d6c272df3c601443631a20a"} Sep 04 22:36:21.193865 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:36:21.193804 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ecfcad-7d79987f5f-czgts" event={"ID":"6c2c6b3a-2589-45fa-9e93-9483f742ede4","Type":"ContainerStarted","Data":"6aa16c64c9f5789a15f53ce556ea7e2f2773674ec6a5c5f78ee06ea55ec2f081"} Sep 04 22:36:21.194286 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:36:21.193922 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ecfcad-7d79987f5f-czgts" Sep 04 22:36:21.227539 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:36:21.227488 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-ecfcad-7d79987f5f-czgts" podStartSLOduration=2.227473363 podStartE2EDuration="2.227473363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:36:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:36:21.225646369 +0000 UTC m=+1684.990121987" watchObservedRunningTime="2026-09-04 22:36:21.227473363 +0000 UTC m=+1684.991948975" Sep 04 22:36:32.202106 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:36:32.202075 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ecfcad-7d79987f5f-czgts" Sep 04 22:36:47.664320 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:36:47.664233 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-xj6cg_8e12172f-2c9e-4eaa-b780-076e4c686406/payload-processing/0.log" Sep 04 22:36:47.792665 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:36:47.792635 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-ecfcad-7d79987f5f-czgts_6c2c6b3a-2589-45fa-9e93-9483f742ede4/payload-processing/0.log" Sep 04 22:37:17.096689 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:17.096659 2574 scope.go:117] "RemoveContainer" containerID="aed11de8689b3a3028f74e9df2b65a1028530de08bbf6cb1e25d832b087a7944" Sep 04 22:37:20.465249 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:20.465215 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-2df128-d65c6b75f-nn7pw_1594f79a-a774-4a97-9aba-0ccb03c381bc/payload-processing/0.log" Sep 04 22:37:20.584196 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:20.584152 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-xj6cg_8e12172f-2c9e-4eaa-b780-076e4c686406/payload-processing/0.log" Sep 04 22:37:20.703093 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:20.703066 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-ecfcad-7d79987f5f-czgts_6c2c6b3a-2589-45fa-9e93-9483f742ede4/payload-processing/0.log" Sep 04 22:37:26.566996 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:26.566960 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-919fd8-7549d7b85c-6c7b6"] Sep 04 22:37:26.570440 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:26.570424 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-919fd8-7549d7b85c-6c7b6" Sep 04 22:37:26.573359 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:26.573334 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-919fd8\"" Sep 04 22:37:26.585238 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:26.585209 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-919fd8-7549d7b85c-6c7b6"] Sep 04 22:37:26.644216 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:26.644185 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ng54\" (UniqueName: \"kubernetes.io/projected/3a71fcc3-c1cc-472a-bb9c-816eb1fbb4e9-kube-api-access-8ng54\") pod \"payload-processing-e2e-ipp-cleanup-919fd8-7549d7b85c-6c7b6\" (UID: \"3a71fcc3-c1cc-472a-bb9c-816eb1fbb4e9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-919fd8-7549d7b85c-6c7b6" Sep 04 22:37:26.644352 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:26.644241 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a71fcc3-c1cc-472a-bb9c-816eb1fbb4e9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-919fd8-7549d7b85c-6c7b6\" (UID: \"3a71fcc3-c1cc-472a-bb9c-816eb1fbb4e9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-919fd8-7549d7b85c-6c7b6" Sep 04 22:37:26.647987 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:26.647951 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-919fd8-6dff475786-gdj9p"] Sep 04 22:37:26.651182 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:26.651166 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-919fd8-6dff475786-gdj9p" Sep 04 22:37:26.653885 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:26.653861 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-919fd8-dockercfg-t2f9g\"" Sep 04 22:37:26.667740 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:26.667695 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-919fd8-6dff475786-gdj9p"] Sep 04 22:37:26.745325 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:26.745298 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfwsc\" (UniqueName: \"kubernetes.io/projected/62ff63ca-b056-402c-ad51-a666191d40e1-kube-api-access-jfwsc\") pod \"payload-pre-processing-e2e-ipp-cleanup-919fd8-6dff475786-gdj9p\" (UID: \"62ff63ca-b056-402c-ad51-a666191d40e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-919fd8-6dff475786-gdj9p" Sep 04 22:37:26.745509 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:26.745337 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ng54\" (UniqueName: \"kubernetes.io/projected/3a71fcc3-c1cc-472a-bb9c-816eb1fbb4e9-kube-api-access-8ng54\") pod \"payload-processing-e2e-ipp-cleanup-919fd8-7549d7b85c-6c7b6\" (UID: \"3a71fcc3-c1cc-472a-bb9c-816eb1fbb4e9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-919fd8-7549d7b85c-6c7b6" Sep 04 22:37:26.745509 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:26.745382 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a71fcc3-c1cc-472a-bb9c-816eb1fbb4e9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-919fd8-7549d7b85c-6c7b6\" (UID: \"3a71fcc3-c1cc-472a-bb9c-816eb1fbb4e9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-919fd8-7549d7b85c-6c7b6" Sep 04 22:37:26.745509 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:26.745398 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62ff63ca-b056-402c-ad51-a666191d40e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-919fd8-6dff475786-gdj9p\" (UID: \"62ff63ca-b056-402c-ad51-a666191d40e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-919fd8-6dff475786-gdj9p" Sep 04 22:37:26.746031 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:26.746013 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a71fcc3-c1cc-472a-bb9c-816eb1fbb4e9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-919fd8-7549d7b85c-6c7b6\" (UID: \"3a71fcc3-c1cc-472a-bb9c-816eb1fbb4e9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-919fd8-7549d7b85c-6c7b6" Sep 04 22:37:26.755738 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:26.755718 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ng54\" (UniqueName: \"kubernetes.io/projected/3a71fcc3-c1cc-472a-bb9c-816eb1fbb4e9-kube-api-access-8ng54\") pod \"payload-processing-e2e-ipp-cleanup-919fd8-7549d7b85c-6c7b6\" (UID: \"3a71fcc3-c1cc-472a-bb9c-816eb1fbb4e9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-919fd8-7549d7b85c-6c7b6" Sep 04 22:37:26.846663 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:26.846593 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62ff63ca-b056-402c-ad51-a666191d40e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-919fd8-6dff475786-gdj9p\" (UID: \"62ff63ca-b056-402c-ad51-a666191d40e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-919fd8-6dff475786-gdj9p" Sep 04 22:37:26.846663 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:26.846656 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfwsc\" (UniqueName: \"kubernetes.io/projected/62ff63ca-b056-402c-ad51-a666191d40e1-kube-api-access-jfwsc\") pod \"payload-pre-processing-e2e-ipp-cleanup-919fd8-6dff475786-gdj9p\" (UID: \"62ff63ca-b056-402c-ad51-a666191d40e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-919fd8-6dff475786-gdj9p" Sep 04 22:37:26.847301 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:26.847277 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62ff63ca-b056-402c-ad51-a666191d40e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-919fd8-6dff475786-gdj9p\" (UID: \"62ff63ca-b056-402c-ad51-a666191d40e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-919fd8-6dff475786-gdj9p" Sep 04 22:37:26.856776 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:26.856749 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfwsc\" (UniqueName: \"kubernetes.io/projected/62ff63ca-b056-402c-ad51-a666191d40e1-kube-api-access-jfwsc\") pod \"payload-pre-processing-e2e-ipp-cleanup-919fd8-6dff475786-gdj9p\" (UID: \"62ff63ca-b056-402c-ad51-a666191d40e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-919fd8-6dff475786-gdj9p" Sep 04 22:37:26.886613 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:26.886588 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-919fd8-7549d7b85c-6c7b6" Sep 04 22:37:26.961349 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:26.961312 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-919fd8-6dff475786-gdj9p" Sep 04 22:37:27.165973 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:27.165942 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-919fd8-7549d7b85c-6c7b6"] Sep 04 22:37:27.166903 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:37:27.166873 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a71fcc3_c1cc_472a_bb9c_816eb1fbb4e9.slice/crio-ffe0e59674da1516147413edbef7a50637a3ab3486d7185ec919b02b4ab9f6e3 WatchSource:0}: Error finding container ffe0e59674da1516147413edbef7a50637a3ab3486d7185ec919b02b4ab9f6e3: Status 404 returned error can't find the container with id ffe0e59674da1516147413edbef7a50637a3ab3486d7185ec919b02b4ab9f6e3 Sep 04 22:37:27.236405 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:27.236383 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-919fd8-6dff475786-gdj9p"] Sep 04 22:37:27.236700 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:37:27.236652 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62ff63ca_b056_402c_ad51_a666191d40e1.slice/crio-360ec0d63e6ebc96b26a88da35cb913c2c44f9a2d8e4311fb5474641693a3917 WatchSource:0}: Error finding container 360ec0d63e6ebc96b26a88da35cb913c2c44f9a2d8e4311fb5474641693a3917: Status 404 returned error can't find the container with id 360ec0d63e6ebc96b26a88da35cb913c2c44f9a2d8e4311fb5474641693a3917 Sep 04 22:37:27.563368 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:27.563327 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-919fd8-7549d7b85c-6c7b6" event={"ID":"3a71fcc3-c1cc-472a-bb9c-816eb1fbb4e9","Type":"ContainerStarted","Data":"6406d41630ac4ec956feedf5236e094a5bcb540d4f401ff8167dce80da86e64b"} Sep 04 22:37:27.563592 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:27.563395 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-919fd8-7549d7b85c-6c7b6" event={"ID":"3a71fcc3-c1cc-472a-bb9c-816eb1fbb4e9","Type":"ContainerStarted","Data":"ffe0e59674da1516147413edbef7a50637a3ab3486d7185ec919b02b4ab9f6e3"} Sep 04 22:37:27.563592 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:27.563527 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-919fd8-7549d7b85c-6c7b6" Sep 04 22:37:27.565015 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:27.564991 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-919fd8-6dff475786-gdj9p" event={"ID":"62ff63ca-b056-402c-ad51-a666191d40e1","Type":"ContainerStarted","Data":"299b73b37664610bd90a5eda3ccfaeb89ea32f6a90c45caa0c41c5c1427cf451"} Sep 04 22:37:27.565126 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:27.565031 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-919fd8-6dff475786-gdj9p" event={"ID":"62ff63ca-b056-402c-ad51-a666191d40e1","Type":"ContainerStarted","Data":"360ec0d63e6ebc96b26a88da35cb913c2c44f9a2d8e4311fb5474641693a3917"} Sep 04 22:37:27.565126 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:27.565093 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-919fd8-6dff475786-gdj9p" Sep 04 22:37:27.583087 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:27.583036 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-919fd8-7549d7b85c-6c7b6" podStartSLOduration=1.583019559 podStartE2EDuration="1.583019559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:37:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:37:27.580905322 +0000 UTC m=+1751.345380937" watchObservedRunningTime="2026-09-04 22:37:27.583019559 +0000 UTC m=+1751.347495174" Sep 04 22:37:27.599069 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:27.599026 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-919fd8-6dff475786-gdj9p" podStartSLOduration=1.599012484 podStartE2EDuration="1.599012484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:37:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:37:27.598073587 +0000 UTC m=+1751.362549199" watchObservedRunningTime="2026-09-04 22:37:27.599012484 +0000 UTC m=+1751.363488096" Sep 04 22:37:38.573291 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:38.573255 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-919fd8-6dff475786-gdj9p" Sep 04 22:37:38.573755 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:38.573427 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-919fd8-7549d7b85c-6c7b6" Sep 04 22:37:50.244524 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:50.244490 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-919fd8-8f84q"] Sep 04 22:37:50.247938 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:50.247920 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-919fd8-8f84q" Sep 04 22:37:50.251101 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:50.251084 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 04 22:37:50.251253 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:50.251233 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hc7x4\"" Sep 04 22:37:50.260871 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:50.260848 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-919fd8-8f84q"] Sep 04 22:37:50.347687 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:50.347645 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3ef7200-5f94-451e-a7b2-2db2342722d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-919fd8-8f84q\" (UID: \"b3ef7200-5f94-451e-a7b2-2db2342722d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-919fd8-8f84q" Sep 04 22:37:50.448709 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:50.448673 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3ef7200-5f94-451e-a7b2-2db2342722d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-919fd8-8f84q\" (UID: \"b3ef7200-5f94-451e-a7b2-2db2342722d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-919fd8-8f84q" Sep 04 22:37:50.449303 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:50.449283 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3ef7200-5f94-451e-a7b2-2db2342722d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-919fd8-8f84q\" (UID: \"b3ef7200-5f94-451e-a7b2-2db2342722d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-919fd8-8f84q" Sep 04 22:37:50.558458 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:50.558426 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-919fd8-8f84q" Sep 04 22:37:50.831735 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:50.831704 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-919fd8-8f84q"] Sep 04 22:37:50.833218 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:37:50.833181 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3ef7200_5f94_451e_a7b2_2db2342722d9.slice/crio-42efb2ff266c20c0d66fffe4908d4cf30628229b7e398d21b3d55474797f0be5 WatchSource:0}: Error finding container 42efb2ff266c20c0d66fffe4908d4cf30628229b7e398d21b3d55474797f0be5: Status 404 returned error can't find the container with id 42efb2ff266c20c0d66fffe4908d4cf30628229b7e398d21b3d55474797f0be5 Sep 04 22:37:51.705657 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:51.705618 2574 generic.go:356] "Generic (PLEG): container finished" podID="b3ef7200-5f94-451e-a7b2-2db2342722d9" containerID="3ade6ac4dac46580055beb5af784e8e4d410a85ef3d6402cf9ddd28ae80713c9" exitCode=0 Sep 04 22:37:51.706125 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:51.705708 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-919fd8-8f84q" event={"ID":"b3ef7200-5f94-451e-a7b2-2db2342722d9","Type":"ContainerDied","Data":"3ade6ac4dac46580055beb5af784e8e4d410a85ef3d6402cf9ddd28ae80713c9"} Sep 04 22:37:51.706125 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:51.705759 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-919fd8-8f84q" event={"ID":"b3ef7200-5f94-451e-a7b2-2db2342722d9","Type":"ContainerStarted","Data":"42efb2ff266c20c0d66fffe4908d4cf30628229b7e398d21b3d55474797f0be5"} Sep 04 22:37:52.845060 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:52.845034 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-919fd8-8f84q" Sep 04 22:37:52.970901 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:52.970809 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3ef7200-5f94-451e-a7b2-2db2342722d9-maas-api-service-ca\") pod \"b3ef7200-5f94-451e-a7b2-2db2342722d9\" (UID: \"b3ef7200-5f94-451e-a7b2-2db2342722d9\") " Sep 04 22:37:52.971152 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:52.971128 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3ef7200-5f94-451e-a7b2-2db2342722d9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b3ef7200-5f94-451e-a7b2-2db2342722d9" (UID: "b3ef7200-5f94-451e-a7b2-2db2342722d9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:37:53.072396 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:53.072352 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3ef7200-5f94-451e-a7b2-2db2342722d9-maas-api-service-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:37:53.718385 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:53.718353 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-919fd8-8f84q" Sep 04 22:37:53.718574 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:53.718399 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-919fd8-8f84q" event={"ID":"b3ef7200-5f94-451e-a7b2-2db2342722d9","Type":"ContainerDied","Data":"42efb2ff266c20c0d66fffe4908d4cf30628229b7e398d21b3d55474797f0be5"} Sep 04 22:37:53.718574 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:53.718439 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42efb2ff266c20c0d66fffe4908d4cf30628229b7e398d21b3d55474797f0be5" Sep 04 22:37:56.447192 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:56.447163 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-919fd8-7549d7b85c-6c7b6"] Sep 04 22:37:56.447587 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:56.447407 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-919fd8-7549d7b85c-6c7b6" podUID="3a71fcc3-c1cc-472a-bb9c-816eb1fbb4e9" containerName="payload-processing" containerID="cri-o://6406d41630ac4ec956feedf5236e094a5bcb540d4f401ff8167dce80da86e64b" gracePeriod=30 Sep 04 22:37:56.451698 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:56.451621 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-919fd8-6dff475786-gdj9p"] Sep 04 22:37:56.451904 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:56.451873 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-919fd8-6dff475786-gdj9p" podUID="62ff63ca-b056-402c-ad51-a666191d40e1" containerName="payload-pre-processing" containerID="cri-o://299b73b37664610bd90a5eda3ccfaeb89ea32f6a90c45caa0c41c5c1427cf451" gracePeriod=30 Sep 04 22:37:56.683695 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:56.683637 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-919fd8-8f84q"] Sep 04 22:37:56.687414 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:56.687387 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-919fd8-8f84q"] Sep 04 22:37:56.717345 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:56.717314 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3ef7200-5f94-451e-a7b2-2db2342722d9" path="/var/lib/kubelet/pods/b3ef7200-5f94-451e-a7b2-2db2342722d9/volumes" Sep 04 22:37:56.741164 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:56.741139 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-919fd8-7549d7b85c-6c7b6" Sep 04 22:37:56.745255 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:56.745234 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-919fd8-6dff475786-gdj9p" Sep 04 22:37:56.745845 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:56.745822 2574 generic.go:356] "Generic (PLEG): container finished" podID="3a71fcc3-c1cc-472a-bb9c-816eb1fbb4e9" containerID="6406d41630ac4ec956feedf5236e094a5bcb540d4f401ff8167dce80da86e64b" exitCode=0 Sep 04 22:37:56.745977 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:56.745892 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-919fd8-7549d7b85c-6c7b6" event={"ID":"3a71fcc3-c1cc-472a-bb9c-816eb1fbb4e9","Type":"ContainerDied","Data":"6406d41630ac4ec956feedf5236e094a5bcb540d4f401ff8167dce80da86e64b"} Sep 04 22:37:56.745977 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:56.745968 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-919fd8-7549d7b85c-6c7b6" event={"ID":"3a71fcc3-c1cc-472a-bb9c-816eb1fbb4e9","Type":"ContainerDied","Data":"ffe0e59674da1516147413edbef7a50637a3ab3486d7185ec919b02b4ab9f6e3"} Sep 04 22:37:56.746055 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:56.745997 2574 scope.go:117] "RemoveContainer" containerID="6406d41630ac4ec956feedf5236e094a5bcb540d4f401ff8167dce80da86e64b" Sep 04 22:37:56.747270 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:56.747251 2574 generic.go:356] "Generic (PLEG): container finished" podID="62ff63ca-b056-402c-ad51-a666191d40e1" containerID="299b73b37664610bd90a5eda3ccfaeb89ea32f6a90c45caa0c41c5c1427cf451" exitCode=0 Sep 04 22:37:56.747348 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:56.747286 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-919fd8-6dff475786-gdj9p" Sep 04 22:37:56.747348 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:56.747327 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-919fd8-6dff475786-gdj9p" event={"ID":"62ff63ca-b056-402c-ad51-a666191d40e1","Type":"ContainerDied","Data":"299b73b37664610bd90a5eda3ccfaeb89ea32f6a90c45caa0c41c5c1427cf451"} Sep 04 22:37:56.747442 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:56.747364 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-919fd8-6dff475786-gdj9p" event={"ID":"62ff63ca-b056-402c-ad51-a666191d40e1","Type":"ContainerDied","Data":"360ec0d63e6ebc96b26a88da35cb913c2c44f9a2d8e4311fb5474641693a3917"} Sep 04 22:37:56.754672 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:56.754656 2574 scope.go:117] "RemoveContainer" containerID="6406d41630ac4ec956feedf5236e094a5bcb540d4f401ff8167dce80da86e64b" Sep 04 22:37:56.754936 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:37:56.754916 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6406d41630ac4ec956feedf5236e094a5bcb540d4f401ff8167dce80da86e64b\": container with ID starting with 6406d41630ac4ec956feedf5236e094a5bcb540d4f401ff8167dce80da86e64b not found: ID does not exist" containerID="6406d41630ac4ec956feedf5236e094a5bcb540d4f401ff8167dce80da86e64b" Sep 04 22:37:56.754990 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:56.754944 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6406d41630ac4ec956feedf5236e094a5bcb540d4f401ff8167dce80da86e64b"} err="failed to get container status \"6406d41630ac4ec956feedf5236e094a5bcb540d4f401ff8167dce80da86e64b\": rpc error: code = NotFound desc = could not find container \"6406d41630ac4ec956feedf5236e094a5bcb540d4f401ff8167dce80da86e64b\": container with ID starting with 6406d41630ac4ec956feedf5236e094a5bcb540d4f401ff8167dce80da86e64b not found: ID does not exist" Sep 04 22:37:56.754990 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:56.754961 2574 scope.go:117] "RemoveContainer" containerID="299b73b37664610bd90a5eda3ccfaeb89ea32f6a90c45caa0c41c5c1427cf451" Sep 04 22:37:56.764625 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:56.763627 2574 scope.go:117] "RemoveContainer" containerID="299b73b37664610bd90a5eda3ccfaeb89ea32f6a90c45caa0c41c5c1427cf451" Sep 04 22:37:56.764625 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:37:56.764022 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"299b73b37664610bd90a5eda3ccfaeb89ea32f6a90c45caa0c41c5c1427cf451\": container with ID starting with 299b73b37664610bd90a5eda3ccfaeb89ea32f6a90c45caa0c41c5c1427cf451 not found: ID does not exist" containerID="299b73b37664610bd90a5eda3ccfaeb89ea32f6a90c45caa0c41c5c1427cf451" Sep 04 22:37:56.764625 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:56.764115 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"299b73b37664610bd90a5eda3ccfaeb89ea32f6a90c45caa0c41c5c1427cf451"} err="failed to get container status \"299b73b37664610bd90a5eda3ccfaeb89ea32f6a90c45caa0c41c5c1427cf451\": rpc error: code = NotFound desc = could not find container \"299b73b37664610bd90a5eda3ccfaeb89ea32f6a90c45caa0c41c5c1427cf451\": container with ID starting with 299b73b37664610bd90a5eda3ccfaeb89ea32f6a90c45caa0c41c5c1427cf451 not found: ID does not exist" Sep 04 22:37:56.802849 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:56.802807 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a71fcc3-c1cc-472a-bb9c-816eb1fbb4e9-plugins-config-volume\") pod \"3a71fcc3-c1cc-472a-bb9c-816eb1fbb4e9\" (UID: \"3a71fcc3-c1cc-472a-bb9c-816eb1fbb4e9\") " Sep 04 22:37:56.802849 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:56.802854 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ng54\" (UniqueName: \"kubernetes.io/projected/3a71fcc3-c1cc-472a-bb9c-816eb1fbb4e9-kube-api-access-8ng54\") pod \"3a71fcc3-c1cc-472a-bb9c-816eb1fbb4e9\" (UID: \"3a71fcc3-c1cc-472a-bb9c-816eb1fbb4e9\") " Sep 04 22:37:56.803253 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:56.803226 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a71fcc3-c1cc-472a-bb9c-816eb1fbb4e9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3a71fcc3-c1cc-472a-bb9c-816eb1fbb4e9" (UID: "3a71fcc3-c1cc-472a-bb9c-816eb1fbb4e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:37:56.805499 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:56.805476 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a71fcc3-c1cc-472a-bb9c-816eb1fbb4e9-kube-api-access-8ng54" (OuterVolumeSpecName: "kube-api-access-8ng54") pod "3a71fcc3-c1cc-472a-bb9c-816eb1fbb4e9" (UID: "3a71fcc3-c1cc-472a-bb9c-816eb1fbb4e9"). InnerVolumeSpecName "kube-api-access-8ng54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:37:56.904324 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:56.904287 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62ff63ca-b056-402c-ad51-a666191d40e1-plugins-config-volume\") pod \"62ff63ca-b056-402c-ad51-a666191d40e1\" (UID: \"62ff63ca-b056-402c-ad51-a666191d40e1\") " Sep 04 22:37:56.904513 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:56.904372 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfwsc\" (UniqueName: \"kubernetes.io/projected/62ff63ca-b056-402c-ad51-a666191d40e1-kube-api-access-jfwsc\") pod \"62ff63ca-b056-402c-ad51-a666191d40e1\" (UID: \"62ff63ca-b056-402c-ad51-a666191d40e1\") " Sep 04 22:37:56.904623 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:56.904534 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a71fcc3-c1cc-472a-bb9c-816eb1fbb4e9-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:37:56.904623 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:56.904570 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ng54\" (UniqueName: \"kubernetes.io/projected/3a71fcc3-c1cc-472a-bb9c-816eb1fbb4e9-kube-api-access-8ng54\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:37:56.904738 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:56.904678 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62ff63ca-b056-402c-ad51-a666191d40e1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "62ff63ca-b056-402c-ad51-a666191d40e1" (UID: "62ff63ca-b056-402c-ad51-a666191d40e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:37:56.906419 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:56.906396 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62ff63ca-b056-402c-ad51-a666191d40e1-kube-api-access-jfwsc" (OuterVolumeSpecName: "kube-api-access-jfwsc") pod "62ff63ca-b056-402c-ad51-a666191d40e1" (UID: "62ff63ca-b056-402c-ad51-a666191d40e1"). InnerVolumeSpecName "kube-api-access-jfwsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:37:57.005458 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:57.005431 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62ff63ca-b056-402c-ad51-a666191d40e1-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:37:57.005458 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:57.005458 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfwsc\" (UniqueName: \"kubernetes.io/projected/62ff63ca-b056-402c-ad51-a666191d40e1-kube-api-access-jfwsc\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:37:57.075083 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:57.075052 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-919fd8-6dff475786-gdj9p"] Sep 04 22:37:57.079339 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:57.079317 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-919fd8-6dff475786-gdj9p"] Sep 04 22:37:57.754290 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:57.754263 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-919fd8-7549d7b85c-6c7b6" Sep 04 22:37:57.783869 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:57.783833 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-919fd8-7549d7b85c-6c7b6"] Sep 04 22:37:57.791045 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:57.791019 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-919fd8-7549d7b85c-6c7b6"] Sep 04 22:37:58.714908 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:58.714876 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a71fcc3-c1cc-472a-bb9c-816eb1fbb4e9" path="/var/lib/kubelet/pods/3a71fcc3-c1cc-472a-bb9c-816eb1fbb4e9/volumes" Sep 04 22:37:58.715250 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:37:58.715237 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62ff63ca-b056-402c-ad51-a666191d40e1" path="/var/lib/kubelet/pods/62ff63ca-b056-402c-ad51-a666191d40e1/volumes" Sep 04 22:38:07.381851 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:07.381809 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2df128-qd2kz"] Sep 04 22:38:07.382313 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:07.382185 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a71fcc3-c1cc-472a-bb9c-816eb1fbb4e9" containerName="payload-processing" Sep 04 22:38:07.382313 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:07.382197 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a71fcc3-c1cc-472a-bb9c-816eb1fbb4e9" containerName="payload-processing" Sep 04 22:38:07.382313 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:07.382208 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3ef7200-5f94-451e-a7b2-2db2342722d9" containerName="revoke-keys" Sep 04 22:38:07.382313 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:07.382217 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3ef7200-5f94-451e-a7b2-2db2342722d9" containerName="revoke-keys" Sep 04 22:38:07.382313 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:07.382230 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62ff63ca-b056-402c-ad51-a666191d40e1" containerName="payload-pre-processing" Sep 04 22:38:07.382313 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:07.382236 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="62ff63ca-b056-402c-ad51-a666191d40e1" containerName="payload-pre-processing" Sep 04 22:38:07.382313 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:07.382308 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="62ff63ca-b056-402c-ad51-a666191d40e1" containerName="payload-pre-processing" Sep 04 22:38:07.382313 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:07.382317 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a71fcc3-c1cc-472a-bb9c-816eb1fbb4e9" containerName="payload-processing" Sep 04 22:38:07.382620 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:07.382322 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3ef7200-5f94-451e-a7b2-2db2342722d9" containerName="revoke-keys" Sep 04 22:38:07.386948 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:07.386930 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2df128-qd2kz" Sep 04 22:38:07.389916 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:07.389893 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 04 22:38:07.390113 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:07.390099 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hc7x4\"" Sep 04 22:38:07.398962 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:07.398937 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2df128-qd2kz"] Sep 04 22:38:07.504218 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:07.504178 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f8ec30-b15a-4664-83c8-cda279dbb367-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2df128-qd2kz\" (UID: \"94f8ec30-b15a-4664-83c8-cda279dbb367\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2df128-qd2kz" Sep 04 22:38:07.604865 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:07.604832 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f8ec30-b15a-4664-83c8-cda279dbb367-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2df128-qd2kz\" (UID: \"94f8ec30-b15a-4664-83c8-cda279dbb367\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2df128-qd2kz" Sep 04 22:38:07.605419 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:07.605400 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f8ec30-b15a-4664-83c8-cda279dbb367-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2df128-qd2kz\" (UID: \"94f8ec30-b15a-4664-83c8-cda279dbb367\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2df128-qd2kz" Sep 04 22:38:07.700523 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:07.700444 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2df128-qd2kz" Sep 04 22:38:07.999383 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:07.999348 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2df128-qd2kz"] Sep 04 22:38:08.007858 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:38:08.007826 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94f8ec30_b15a_4664_83c8_cda279dbb367.slice/crio-af91734828c3642403d5a4bb8aade38c753607e765679b6ce88f4d4ecf36256a WatchSource:0}: Error finding container af91734828c3642403d5a4bb8aade38c753607e765679b6ce88f4d4ecf36256a: Status 404 returned error can't find the container with id af91734828c3642403d5a4bb8aade38c753607e765679b6ce88f4d4ecf36256a Sep 04 22:38:08.819010 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:08.818974 2574 generic.go:356] "Generic (PLEG): container finished" podID="94f8ec30-b15a-4664-83c8-cda279dbb367" containerID="18da19ae4a9365697599a9de12e07e189e5f60df69dc883b916126c1f11ec463" exitCode=0 Sep 04 22:38:08.819375 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:08.819013 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2df128-qd2kz" event={"ID":"94f8ec30-b15a-4664-83c8-cda279dbb367","Type":"ContainerDied","Data":"18da19ae4a9365697599a9de12e07e189e5f60df69dc883b916126c1f11ec463"} Sep 04 22:38:08.819375 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:08.819067 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2df128-qd2kz" event={"ID":"94f8ec30-b15a-4664-83c8-cda279dbb367","Type":"ContainerStarted","Data":"af91734828c3642403d5a4bb8aade38c753607e765679b6ce88f4d4ecf36256a"} Sep 04 22:38:09.958703 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:09.958678 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2df128-qd2kz" Sep 04 22:38:10.125187 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:10.125106 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f8ec30-b15a-4664-83c8-cda279dbb367-maas-api-service-ca\") pod \"94f8ec30-b15a-4664-83c8-cda279dbb367\" (UID: \"94f8ec30-b15a-4664-83c8-cda279dbb367\") " Sep 04 22:38:10.125448 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:10.125424 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94f8ec30-b15a-4664-83c8-cda279dbb367-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "94f8ec30-b15a-4664-83c8-cda279dbb367" (UID: "94f8ec30-b15a-4664-83c8-cda279dbb367"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:38:10.226081 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:10.226054 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94f8ec30-b15a-4664-83c8-cda279dbb367-maas-api-service-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:38:10.830790 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:10.830768 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2df128-qd2kz" Sep 04 22:38:10.830960 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:10.830820 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2df128-qd2kz" event={"ID":"94f8ec30-b15a-4664-83c8-cda279dbb367","Type":"ContainerDied","Data":"af91734828c3642403d5a4bb8aade38c753607e765679b6ce88f4d4ecf36256a"} Sep 04 22:38:10.830960 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:10.830878 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af91734828c3642403d5a4bb8aade38c753607e765679b6ce88f4d4ecf36256a" Sep 04 22:38:23.078378 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:23.078329 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2df128-d65c6b75f-nn7pw"] Sep 04 22:38:23.079326 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:23.078623 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-2df128-d65c6b75f-nn7pw" podUID="1594f79a-a774-4a97-9aba-0ccb03c381bc" containerName="payload-processing" containerID="cri-o://67654b607575eb78fead4ee6a3c468bf9eee65098a4ae75c2da565c3938f2563" gracePeriod=30 Sep 04 22:38:23.086988 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:23.086959 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2df128-64b8579d9d-ljk6b"] Sep 04 22:38:23.087227 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:23.087199 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2df128-64b8579d9d-ljk6b" podUID="acbd20b5-5426-4b59-bd4d-027faa54adb0" containerName="payload-pre-processing" containerID="cri-o://ca799df6ba6919b0bf255d71c1b152253ac1eed51f9f75f99292337802eaacbc" gracePeriod=30 Sep 04 22:38:23.358501 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:23.358477 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2df128-d65c6b75f-nn7pw" Sep 04 22:38:23.365333 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:23.365307 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2df128-64b8579d9d-ljk6b" Sep 04 22:38:23.368998 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:23.368970 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2df128-qd2kz"] Sep 04 22:38:23.372397 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:23.372369 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2df128-qd2kz"] Sep 04 22:38:23.439626 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:23.439587 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1594f79a-a774-4a97-9aba-0ccb03c381bc-plugins-config-volume\") pod \"1594f79a-a774-4a97-9aba-0ccb03c381bc\" (UID: \"1594f79a-a774-4a97-9aba-0ccb03c381bc\") " Sep 04 22:38:23.439822 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:23.439654 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbzdn\" (UniqueName: \"kubernetes.io/projected/acbd20b5-5426-4b59-bd4d-027faa54adb0-kube-api-access-jbzdn\") pod \"acbd20b5-5426-4b59-bd4d-027faa54adb0\" (UID: \"acbd20b5-5426-4b59-bd4d-027faa54adb0\") " Sep 04 22:38:23.439822 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:23.439674 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjtld\" (UniqueName: \"kubernetes.io/projected/1594f79a-a774-4a97-9aba-0ccb03c381bc-kube-api-access-bjtld\") pod \"1594f79a-a774-4a97-9aba-0ccb03c381bc\" (UID: \"1594f79a-a774-4a97-9aba-0ccb03c381bc\") " Sep 04 22:38:23.439822 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:23.439720 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acbd20b5-5426-4b59-bd4d-027faa54adb0-plugins-config-volume\") pod \"acbd20b5-5426-4b59-bd4d-027faa54adb0\" (UID: \"acbd20b5-5426-4b59-bd4d-027faa54adb0\") " Sep 04 22:38:23.440075 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:23.440036 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1594f79a-a774-4a97-9aba-0ccb03c381bc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1594f79a-a774-4a97-9aba-0ccb03c381bc" (UID: "1594f79a-a774-4a97-9aba-0ccb03c381bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:38:23.440149 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:23.440130 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acbd20b5-5426-4b59-bd4d-027faa54adb0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "acbd20b5-5426-4b59-bd4d-027faa54adb0" (UID: "acbd20b5-5426-4b59-bd4d-027faa54adb0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:38:23.441913 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:23.441884 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acbd20b5-5426-4b59-bd4d-027faa54adb0-kube-api-access-jbzdn" (OuterVolumeSpecName: "kube-api-access-jbzdn") pod "acbd20b5-5426-4b59-bd4d-027faa54adb0" (UID: "acbd20b5-5426-4b59-bd4d-027faa54adb0"). InnerVolumeSpecName "kube-api-access-jbzdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:38:23.441913 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:23.441901 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1594f79a-a774-4a97-9aba-0ccb03c381bc-kube-api-access-bjtld" (OuterVolumeSpecName: "kube-api-access-bjtld") pod "1594f79a-a774-4a97-9aba-0ccb03c381bc" (UID: "1594f79a-a774-4a97-9aba-0ccb03c381bc"). InnerVolumeSpecName "kube-api-access-bjtld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:38:23.540799 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:23.540746 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acbd20b5-5426-4b59-bd4d-027faa54adb0-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:38:23.540799 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:23.540791 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1594f79a-a774-4a97-9aba-0ccb03c381bc-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:38:23.540799 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:23.540801 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbzdn\" (UniqueName: \"kubernetes.io/projected/acbd20b5-5426-4b59-bd4d-027faa54adb0-kube-api-access-jbzdn\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:38:23.540799 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:23.540811 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjtld\" (UniqueName: \"kubernetes.io/projected/1594f79a-a774-4a97-9aba-0ccb03c381bc-kube-api-access-bjtld\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:38:23.912569 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:23.912517 2574 generic.go:356] "Generic (PLEG): container finished" podID="acbd20b5-5426-4b59-bd4d-027faa54adb0" containerID="ca799df6ba6919b0bf255d71c1b152253ac1eed51f9f75f99292337802eaacbc" exitCode=0 Sep 04 22:38:23.912755 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:23.912586 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2df128-64b8579d9d-ljk6b" Sep 04 22:38:23.912755 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:23.912603 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2df128-64b8579d9d-ljk6b" event={"ID":"acbd20b5-5426-4b59-bd4d-027faa54adb0","Type":"ContainerDied","Data":"ca799df6ba6919b0bf255d71c1b152253ac1eed51f9f75f99292337802eaacbc"} Sep 04 22:38:23.912755 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:23.912658 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2df128-64b8579d9d-ljk6b" event={"ID":"acbd20b5-5426-4b59-bd4d-027faa54adb0","Type":"ContainerDied","Data":"a6224874750fe14cbd053771af204ca2103ca735dc4cfbb3883d1d88402c95a5"} Sep 04 22:38:23.912755 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:23.912683 2574 scope.go:117] "RemoveContainer" containerID="ca799df6ba6919b0bf255d71c1b152253ac1eed51f9f75f99292337802eaacbc" Sep 04 22:38:23.913984 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:23.913963 2574 generic.go:356] "Generic (PLEG): container finished" podID="1594f79a-a774-4a97-9aba-0ccb03c381bc" containerID="67654b607575eb78fead4ee6a3c468bf9eee65098a4ae75c2da565c3938f2563" exitCode=0 Sep 04 22:38:23.914080 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:23.914007 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2df128-d65c6b75f-nn7pw" Sep 04 22:38:23.914080 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:23.914021 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2df128-d65c6b75f-nn7pw" event={"ID":"1594f79a-a774-4a97-9aba-0ccb03c381bc","Type":"ContainerDied","Data":"67654b607575eb78fead4ee6a3c468bf9eee65098a4ae75c2da565c3938f2563"} Sep 04 22:38:23.914080 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:23.914054 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2df128-d65c6b75f-nn7pw" event={"ID":"1594f79a-a774-4a97-9aba-0ccb03c381bc","Type":"ContainerDied","Data":"10530d33e1b78eda26dc1aed6e55fa919e37018a36eaf3f073b9e16019fb58a5"} Sep 04 22:38:23.928454 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:23.924939 2574 scope.go:117] "RemoveContainer" containerID="ca799df6ba6919b0bf255d71c1b152253ac1eed51f9f75f99292337802eaacbc" Sep 04 22:38:23.928454 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:38:23.925499 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca799df6ba6919b0bf255d71c1b152253ac1eed51f9f75f99292337802eaacbc\": container with ID starting with ca799df6ba6919b0bf255d71c1b152253ac1eed51f9f75f99292337802eaacbc not found: ID does not exist" containerID="ca799df6ba6919b0bf255d71c1b152253ac1eed51f9f75f99292337802eaacbc" Sep 04 22:38:23.928454 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:23.925540 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca799df6ba6919b0bf255d71c1b152253ac1eed51f9f75f99292337802eaacbc"} err="failed to get container status \"ca799df6ba6919b0bf255d71c1b152253ac1eed51f9f75f99292337802eaacbc\": rpc error: code = NotFound desc = could not find container \"ca799df6ba6919b0bf255d71c1b152253ac1eed51f9f75f99292337802eaacbc\": container with ID starting with ca799df6ba6919b0bf255d71c1b152253ac1eed51f9f75f99292337802eaacbc not found: ID does not exist" Sep 04 22:38:23.928454 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:23.925791 2574 scope.go:117] "RemoveContainer" containerID="67654b607575eb78fead4ee6a3c468bf9eee65098a4ae75c2da565c3938f2563" Sep 04 22:38:23.937701 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:23.937683 2574 scope.go:117] "RemoveContainer" containerID="67654b607575eb78fead4ee6a3c468bf9eee65098a4ae75c2da565c3938f2563" Sep 04 22:38:23.938004 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:38:23.937986 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67654b607575eb78fead4ee6a3c468bf9eee65098a4ae75c2da565c3938f2563\": container with ID starting with 67654b607575eb78fead4ee6a3c468bf9eee65098a4ae75c2da565c3938f2563 not found: ID does not exist" containerID="67654b607575eb78fead4ee6a3c468bf9eee65098a4ae75c2da565c3938f2563" Sep 04 22:38:23.938067 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:23.938013 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67654b607575eb78fead4ee6a3c468bf9eee65098a4ae75c2da565c3938f2563"} err="failed to get container status \"67654b607575eb78fead4ee6a3c468bf9eee65098a4ae75c2da565c3938f2563\": rpc error: code = NotFound desc = could not find container \"67654b607575eb78fead4ee6a3c468bf9eee65098a4ae75c2da565c3938f2563\": container with ID starting with 67654b607575eb78fead4ee6a3c468bf9eee65098a4ae75c2da565c3938f2563 not found: ID does not exist" Sep 04 22:38:23.949488 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:23.949464 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2df128-64b8579d9d-ljk6b"] Sep 04 22:38:23.956316 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:23.956295 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2df128-64b8579d9d-ljk6b"] Sep 04 22:38:23.969634 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:23.969607 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2df128-d65c6b75f-nn7pw"] Sep 04 22:38:23.975696 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:23.975670 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2df128-d65c6b75f-nn7pw"] Sep 04 22:38:24.715246 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:24.715209 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1594f79a-a774-4a97-9aba-0ccb03c381bc" path="/var/lib/kubelet/pods/1594f79a-a774-4a97-9aba-0ccb03c381bc/volumes" Sep 04 22:38:24.715680 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:24.715536 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94f8ec30-b15a-4664-83c8-cda279dbb367" path="/var/lib/kubelet/pods/94f8ec30-b15a-4664-83c8-cda279dbb367/volumes" Sep 04 22:38:24.715903 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:38:24.715888 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acbd20b5-5426-4b59-bd4d-027faa54adb0" path="/var/lib/kubelet/pods/acbd20b5-5426-4b59-bd4d-027faa54adb0/volumes" Sep 04 22:39:13.728436 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:13.728386 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ecfcad-srgcf"] Sep 04 22:39:13.730939 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:13.728908 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94f8ec30-b15a-4664-83c8-cda279dbb367" containerName="revoke-keys" Sep 04 22:39:13.730939 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:13.728926 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="94f8ec30-b15a-4664-83c8-cda279dbb367" containerName="revoke-keys" Sep 04 22:39:13.730939 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:13.728935 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1594f79a-a774-4a97-9aba-0ccb03c381bc" containerName="payload-processing" Sep 04 22:39:13.730939 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:13.728941 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1594f79a-a774-4a97-9aba-0ccb03c381bc" containerName="payload-processing" Sep 04 22:39:13.730939 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:13.728954 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acbd20b5-5426-4b59-bd4d-027faa54adb0" containerName="payload-pre-processing" Sep 04 22:39:13.730939 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:13.728959 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="acbd20b5-5426-4b59-bd4d-027faa54adb0" containerName="payload-pre-processing" Sep 04 22:39:13.730939 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:13.729025 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="94f8ec30-b15a-4664-83c8-cda279dbb367" containerName="revoke-keys" Sep 04 22:39:13.730939 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:13.729036 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="acbd20b5-5426-4b59-bd4d-027faa54adb0" containerName="payload-pre-processing" Sep 04 22:39:13.730939 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:13.729042 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1594f79a-a774-4a97-9aba-0ccb03c381bc" containerName="payload-processing" Sep 04 22:39:13.731963 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:13.731944 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ecfcad-srgcf" Sep 04 22:39:13.735452 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:13.735431 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 04 22:39:13.735636 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:13.735618 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hc7x4\"" Sep 04 22:39:13.741493 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:13.741471 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ecfcad-srgcf"] Sep 04 22:39:13.758352 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:13.758327 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dfa1ba9-a51d-400b-a09f-6243680cd4a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ecfcad-srgcf\" (UID: \"7dfa1ba9-a51d-400b-a09f-6243680cd4a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ecfcad-srgcf" Sep 04 22:39:13.858988 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:13.858958 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dfa1ba9-a51d-400b-a09f-6243680cd4a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ecfcad-srgcf\" (UID: \"7dfa1ba9-a51d-400b-a09f-6243680cd4a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ecfcad-srgcf" Sep 04 22:39:13.859529 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:13.859510 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dfa1ba9-a51d-400b-a09f-6243680cd4a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ecfcad-srgcf\" (UID: \"7dfa1ba9-a51d-400b-a09f-6243680cd4a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ecfcad-srgcf" Sep 04 22:39:14.042538 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:14.042499 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ecfcad-srgcf" Sep 04 22:39:14.317225 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:14.317199 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ecfcad-srgcf"] Sep 04 22:39:14.318424 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:39:14.318396 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dfa1ba9_a51d_400b_a09f_6243680cd4a7.slice/crio-5df4cb4e242b1ed6d1b0cb70d84ddc900173c2cc0621684342c1e90f0518e829 WatchSource:0}: Error finding container 5df4cb4e242b1ed6d1b0cb70d84ddc900173c2cc0621684342c1e90f0518e829: Status 404 returned error can't find the container with id 5df4cb4e242b1ed6d1b0cb70d84ddc900173c2cc0621684342c1e90f0518e829 Sep 04 22:39:15.197729 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:15.197692 2574 generic.go:356] "Generic (PLEG): container finished" podID="7dfa1ba9-a51d-400b-a09f-6243680cd4a7" containerID="49fcd0ba6f146f25c7a685930b659ea69b62ba5b1b0b14b2fb43d52074188646" exitCode=0 Sep 04 22:39:15.198152 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:15.197774 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ecfcad-srgcf" event={"ID":"7dfa1ba9-a51d-400b-a09f-6243680cd4a7","Type":"ContainerDied","Data":"49fcd0ba6f146f25c7a685930b659ea69b62ba5b1b0b14b2fb43d52074188646"} Sep 04 22:39:15.198152 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:15.197823 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ecfcad-srgcf" event={"ID":"7dfa1ba9-a51d-400b-a09f-6243680cd4a7","Type":"ContainerStarted","Data":"5df4cb4e242b1ed6d1b0cb70d84ddc900173c2cc0621684342c1e90f0518e829"} Sep 04 22:39:16.329078 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:16.329058 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ecfcad-srgcf" Sep 04 22:39:16.377921 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:16.377893 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dfa1ba9-a51d-400b-a09f-6243680cd4a7-maas-api-service-ca\") pod \"7dfa1ba9-a51d-400b-a09f-6243680cd4a7\" (UID: \"7dfa1ba9-a51d-400b-a09f-6243680cd4a7\") " Sep 04 22:39:16.378208 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:16.378188 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dfa1ba9-a51d-400b-a09f-6243680cd4a7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7dfa1ba9-a51d-400b-a09f-6243680cd4a7" (UID: "7dfa1ba9-a51d-400b-a09f-6243680cd4a7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:39:16.479084 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:16.479010 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dfa1ba9-a51d-400b-a09f-6243680cd4a7-maas-api-service-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:39:17.208698 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:17.208677 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ecfcad-srgcf" Sep 04 22:39:17.208698 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:17.208676 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ecfcad-srgcf" event={"ID":"7dfa1ba9-a51d-400b-a09f-6243680cd4a7","Type":"ContainerDied","Data":"5df4cb4e242b1ed6d1b0cb70d84ddc900173c2cc0621684342c1e90f0518e829"} Sep 04 22:39:17.208841 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:17.208727 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5df4cb4e242b1ed6d1b0cb70d84ddc900173c2cc0621684342c1e90f0518e829" Sep 04 22:39:24.442942 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:24.442907 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ecfcad-7d79987f5f-czgts"] Sep 04 22:39:24.443710 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:24.443675 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-ecfcad-7d79987f5f-czgts" podUID="6c2c6b3a-2589-45fa-9e93-9483f742ede4" containerName="payload-processing" containerID="cri-o://6aa16c64c9f5789a15f53ce556ea7e2f2773674ec6a5c5f78ee06ea55ec2f081" gracePeriod=30 Sep 04 22:39:24.659767 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:24.659724 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ecfcad-srgcf"] Sep 04 22:39:24.663001 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:24.662974 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ecfcad-srgcf"] Sep 04 22:39:24.700734 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:24.700683 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ecfcad-7d79987f5f-czgts" Sep 04 22:39:24.714660 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:24.714637 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7dfa1ba9-a51d-400b-a09f-6243680cd4a7" path="/var/lib/kubelet/pods/7dfa1ba9-a51d-400b-a09f-6243680cd4a7/volumes" Sep 04 22:39:24.744837 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:24.744807 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c2c6b3a-2589-45fa-9e93-9483f742ede4-plugins-config-volume\") pod \"6c2c6b3a-2589-45fa-9e93-9483f742ede4\" (UID: \"6c2c6b3a-2589-45fa-9e93-9483f742ede4\") " Sep 04 22:39:24.744967 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:24.744870 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6kwl9\" (UniqueName: \"kubernetes.io/projected/6c2c6b3a-2589-45fa-9e93-9483f742ede4-kube-api-access-6kwl9\") pod \"6c2c6b3a-2589-45fa-9e93-9483f742ede4\" (UID: \"6c2c6b3a-2589-45fa-9e93-9483f742ede4\") " Sep 04 22:39:24.745210 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:24.745179 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c2c6b3a-2589-45fa-9e93-9483f742ede4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6c2c6b3a-2589-45fa-9e93-9483f742ede4" (UID: "6c2c6b3a-2589-45fa-9e93-9483f742ede4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:39:24.746865 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:24.746842 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c2c6b3a-2589-45fa-9e93-9483f742ede4-kube-api-access-6kwl9" (OuterVolumeSpecName: "kube-api-access-6kwl9") pod "6c2c6b3a-2589-45fa-9e93-9483f742ede4" (UID: "6c2c6b3a-2589-45fa-9e93-9483f742ede4"). InnerVolumeSpecName "kube-api-access-6kwl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:39:24.845542 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:24.845511 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kwl9\" (UniqueName: \"kubernetes.io/projected/6c2c6b3a-2589-45fa-9e93-9483f742ede4-kube-api-access-6kwl9\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:39:24.845703 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:24.845542 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c2c6b3a-2589-45fa-9e93-9483f742ede4-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:39:25.255017 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:25.254980 2574 generic.go:356] "Generic (PLEG): container finished" podID="6c2c6b3a-2589-45fa-9e93-9483f742ede4" containerID="6aa16c64c9f5789a15f53ce556ea7e2f2773674ec6a5c5f78ee06ea55ec2f081" exitCode=0 Sep 04 22:39:25.255215 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:25.255037 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ecfcad-7d79987f5f-czgts" Sep 04 22:39:25.255215 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:25.255062 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ecfcad-7d79987f5f-czgts" event={"ID":"6c2c6b3a-2589-45fa-9e93-9483f742ede4","Type":"ContainerDied","Data":"6aa16c64c9f5789a15f53ce556ea7e2f2773674ec6a5c5f78ee06ea55ec2f081"} Sep 04 22:39:25.255215 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:25.255113 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ecfcad-7d79987f5f-czgts" event={"ID":"6c2c6b3a-2589-45fa-9e93-9483f742ede4","Type":"ContainerDied","Data":"ee01dcbdab774d37bb7df58ab5d6b28d911e70d48d6c272df3c601443631a20a"} Sep 04 22:39:25.255215 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:25.255139 2574 scope.go:117] "RemoveContainer" containerID="6aa16c64c9f5789a15f53ce556ea7e2f2773674ec6a5c5f78ee06ea55ec2f081" Sep 04 22:39:25.264000 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:25.263980 2574 scope.go:117] "RemoveContainer" containerID="6aa16c64c9f5789a15f53ce556ea7e2f2773674ec6a5c5f78ee06ea55ec2f081" Sep 04 22:39:25.264302 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:39:25.264282 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aa16c64c9f5789a15f53ce556ea7e2f2773674ec6a5c5f78ee06ea55ec2f081\": container with ID starting with 6aa16c64c9f5789a15f53ce556ea7e2f2773674ec6a5c5f78ee06ea55ec2f081 not found: ID does not exist" containerID="6aa16c64c9f5789a15f53ce556ea7e2f2773674ec6a5c5f78ee06ea55ec2f081" Sep 04 22:39:25.264411 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:25.264308 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aa16c64c9f5789a15f53ce556ea7e2f2773674ec6a5c5f78ee06ea55ec2f081"} err="failed to get container status \"6aa16c64c9f5789a15f53ce556ea7e2f2773674ec6a5c5f78ee06ea55ec2f081\": rpc error: code = NotFound desc = could not find container \"6aa16c64c9f5789a15f53ce556ea7e2f2773674ec6a5c5f78ee06ea55ec2f081\": container with ID starting with 6aa16c64c9f5789a15f53ce556ea7e2f2773674ec6a5c5f78ee06ea55ec2f081 not found: ID does not exist" Sep 04 22:39:25.279884 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:25.279853 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ecfcad-7d79987f5f-czgts"] Sep 04 22:39:25.284165 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:25.284144 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ecfcad-7d79987f5f-czgts"] Sep 04 22:39:26.714560 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:26.714523 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c2c6b3a-2589-45fa-9e93-9483f742ede4" path="/var/lib/kubelet/pods/6c2c6b3a-2589-45fa-9e93-9483f742ede4/volumes" Sep 04 22:39:32.149411 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:32.149380 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bc62f0-4wxvd"] Sep 04 22:39:32.149889 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:32.149729 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dfa1ba9-a51d-400b-a09f-6243680cd4a7" containerName="revoke-keys" Sep 04 22:39:32.149889 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:32.149742 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dfa1ba9-a51d-400b-a09f-6243680cd4a7" containerName="revoke-keys" Sep 04 22:39:32.149889 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:32.149752 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c2c6b3a-2589-45fa-9e93-9483f742ede4" containerName="payload-processing" Sep 04 22:39:32.149889 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:32.149758 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c2c6b3a-2589-45fa-9e93-9483f742ede4" containerName="payload-processing" Sep 04 22:39:32.149889 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:32.149821 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c2c6b3a-2589-45fa-9e93-9483f742ede4" containerName="payload-processing" Sep 04 22:39:32.149889 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:32.149829 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dfa1ba9-a51d-400b-a09f-6243680cd4a7" containerName="revoke-keys" Sep 04 22:39:32.154033 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:32.154016 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bc62f0-4wxvd" Sep 04 22:39:32.156534 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:32.156512 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 04 22:39:32.156735 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:32.156715 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hc7x4\"" Sep 04 22:39:32.162287 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:32.162268 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bc62f0-4wxvd"] Sep 04 22:39:32.206660 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:32.206632 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/387c661e-dcc3-4aa8-bf76-36e0b94c94e6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-bc62f0-4wxvd\" (UID: \"387c661e-dcc3-4aa8-bf76-36e0b94c94e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bc62f0-4wxvd" Sep 04 22:39:32.307828 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:32.307797 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/387c661e-dcc3-4aa8-bf76-36e0b94c94e6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-bc62f0-4wxvd\" (UID: \"387c661e-dcc3-4aa8-bf76-36e0b94c94e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bc62f0-4wxvd" Sep 04 22:39:32.308418 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:32.308397 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/387c661e-dcc3-4aa8-bf76-36e0b94c94e6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-bc62f0-4wxvd\" (UID: \"387c661e-dcc3-4aa8-bf76-36e0b94c94e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bc62f0-4wxvd" Sep 04 22:39:32.465784 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:32.465708 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bc62f0-4wxvd" Sep 04 22:39:32.743865 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:32.743837 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bc62f0-4wxvd"] Sep 04 22:39:32.745797 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:39:32.745767 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod387c661e_dcc3_4aa8_bf76_36e0b94c94e6.slice/crio-0db6dde48ffc9efc1ea823b366a5d7eccc7ade03c03e0db7eb365ebab2b142be WatchSource:0}: Error finding container 0db6dde48ffc9efc1ea823b366a5d7eccc7ade03c03e0db7eb365ebab2b142be: Status 404 returned error can't find the container with id 0db6dde48ffc9efc1ea823b366a5d7eccc7ade03c03e0db7eb365ebab2b142be Sep 04 22:39:33.303927 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:33.303894 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bc62f0-4wxvd" event={"ID":"387c661e-dcc3-4aa8-bf76-36e0b94c94e6","Type":"ContainerStarted","Data":"8deb8045c6ed69210758e8ef82c3a871c889d502898f70fd40dbbc6cd7e7515a"} Sep 04 22:39:33.304334 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:33.303954 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bc62f0-4wxvd" event={"ID":"387c661e-dcc3-4aa8-bf76-36e0b94c94e6","Type":"ContainerStarted","Data":"0db6dde48ffc9efc1ea823b366a5d7eccc7ade03c03e0db7eb365ebab2b142be"} Sep 04 22:39:33.322655 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:33.322604 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bc62f0-4wxvd" podStartSLOduration=1.3225908610000001 podStartE2EDuration="1.322590861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:39:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:39:33.319749905 +0000 UTC m=+1877.084225530" watchObservedRunningTime="2026-09-04 22:39:33.322590861 +0000 UTC m=+1877.087066468" Sep 04 22:39:34.316937 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:34.316893 2574 generic.go:356] "Generic (PLEG): container finished" podID="387c661e-dcc3-4aa8-bf76-36e0b94c94e6" containerID="8deb8045c6ed69210758e8ef82c3a871c889d502898f70fd40dbbc6cd7e7515a" exitCode=0 Sep 04 22:39:34.317309 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:34.316947 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bc62f0-4wxvd" event={"ID":"387c661e-dcc3-4aa8-bf76-36e0b94c94e6","Type":"ContainerDied","Data":"8deb8045c6ed69210758e8ef82c3a871c889d502898f70fd40dbbc6cd7e7515a"} Sep 04 22:39:35.449877 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:35.449854 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bc62f0-4wxvd" Sep 04 22:39:35.535069 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:35.535039 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/387c661e-dcc3-4aa8-bf76-36e0b94c94e6-maas-api-service-ca\") pod \"387c661e-dcc3-4aa8-bf76-36e0b94c94e6\" (UID: \"387c661e-dcc3-4aa8-bf76-36e0b94c94e6\") " Sep 04 22:39:35.535424 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:35.535403 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/387c661e-dcc3-4aa8-bf76-36e0b94c94e6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "387c661e-dcc3-4aa8-bf76-36e0b94c94e6" (UID: "387c661e-dcc3-4aa8-bf76-36e0b94c94e6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:39:35.636074 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:35.636015 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/387c661e-dcc3-4aa8-bf76-36e0b94c94e6-maas-api-service-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:39:36.329461 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:36.329421 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bc62f0-4wxvd" event={"ID":"387c661e-dcc3-4aa8-bf76-36e0b94c94e6","Type":"ContainerDied","Data":"0db6dde48ffc9efc1ea823b366a5d7eccc7ade03c03e0db7eb365ebab2b142be"} Sep 04 22:39:36.329461 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:36.329452 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bc62f0-4wxvd" Sep 04 22:39:36.329685 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:36.329481 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0db6dde48ffc9efc1ea823b366a5d7eccc7ade03c03e0db7eb365ebab2b142be" Sep 04 22:39:43.146004 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:43.145933 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bc62f0-29809350-7dfkv"] Sep 04 22:39:43.150111 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:43.150083 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bc62f0-29809350-7dfkv"] Sep 04 22:39:43.185421 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:43.185393 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-bc62f0-6d7d885fcc-vp2rs"] Sep 04 22:39:43.185762 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:43.185703 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bc62f0-6d7d885fcc-vp2rs" podUID="6b4b5a55-c0e2-46d7-9806-020b3aeffdbd" containerName="payload-processing" containerID="cri-o://891a42c58225b839b10dc1d7597fa13f95b9e702e52418e37aa06bf1f197d97f" gracePeriod=30 Sep 04 22:39:43.189067 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:43.189029 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bc62f0-f7b44cbbb-c69qt"] Sep 04 22:39:43.189304 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:43.189256 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bc62f0-f7b44cbbb-c69qt" podUID="25007bd5-fdb2-45aa-926f-bfaac7042319" containerName="payload-pre-processing" containerID="cri-o://25c846ef843dd5499a06a8d7db000e0e7c310666aae0d21267b511f29bc4fda1" gracePeriod=30 Sep 04 22:39:43.371384 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:43.371349 2574 generic.go:356] "Generic (PLEG): container finished" podID="25007bd5-fdb2-45aa-926f-bfaac7042319" containerID="25c846ef843dd5499a06a8d7db000e0e7c310666aae0d21267b511f29bc4fda1" exitCode=0 Sep 04 22:39:43.371582 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:43.371542 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bc62f0-f7b44cbbb-c69qt" event={"ID":"25007bd5-fdb2-45aa-926f-bfaac7042319","Type":"ContainerDied","Data":"25c846ef843dd5499a06a8d7db000e0e7c310666aae0d21267b511f29bc4fda1"} Sep 04 22:39:43.373242 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:43.373215 2574 generic.go:356] "Generic (PLEG): container finished" podID="6b4b5a55-c0e2-46d7-9806-020b3aeffdbd" containerID="891a42c58225b839b10dc1d7597fa13f95b9e702e52418e37aa06bf1f197d97f" exitCode=0 Sep 04 22:39:43.373358 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:43.373270 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bc62f0-6d7d885fcc-vp2rs" event={"ID":"6b4b5a55-c0e2-46d7-9806-020b3aeffdbd","Type":"ContainerDied","Data":"891a42c58225b839b10dc1d7597fa13f95b9e702e52418e37aa06bf1f197d97f"} Sep 04 22:39:43.409895 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:43.409825 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bc62f0-4wxvd"] Sep 04 22:39:43.414941 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:43.414914 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bc62f0-4wxvd"] Sep 04 22:39:43.459429 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:43.459407 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bc62f0-6d7d885fcc-vp2rs" Sep 04 22:39:43.461908 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:43.461891 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bc62f0-f7b44cbbb-c69qt" Sep 04 22:39:43.598288 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:43.598262 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25007bd5-fdb2-45aa-926f-bfaac7042319-plugins-config-volume\") pod \"25007bd5-fdb2-45aa-926f-bfaac7042319\" (UID: \"25007bd5-fdb2-45aa-926f-bfaac7042319\") " Sep 04 22:39:43.598465 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:43.598298 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bn2cd\" (UniqueName: \"kubernetes.io/projected/6b4b5a55-c0e2-46d7-9806-020b3aeffdbd-kube-api-access-bn2cd\") pod \"6b4b5a55-c0e2-46d7-9806-020b3aeffdbd\" (UID: \"6b4b5a55-c0e2-46d7-9806-020b3aeffdbd\") " Sep 04 22:39:43.598465 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:43.598366 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pj2v5\" (UniqueName: \"kubernetes.io/projected/25007bd5-fdb2-45aa-926f-bfaac7042319-kube-api-access-pj2v5\") pod \"25007bd5-fdb2-45aa-926f-bfaac7042319\" (UID: \"25007bd5-fdb2-45aa-926f-bfaac7042319\") " Sep 04 22:39:43.598465 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:43.598406 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b4b5a55-c0e2-46d7-9806-020b3aeffdbd-plugins-config-volume\") pod \"6b4b5a55-c0e2-46d7-9806-020b3aeffdbd\" (UID: \"6b4b5a55-c0e2-46d7-9806-020b3aeffdbd\") " Sep 04 22:39:43.598757 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:43.598647 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25007bd5-fdb2-45aa-926f-bfaac7042319-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "25007bd5-fdb2-45aa-926f-bfaac7042319" (UID: "25007bd5-fdb2-45aa-926f-bfaac7042319"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:39:43.598847 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:43.598809 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b4b5a55-c0e2-46d7-9806-020b3aeffdbd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6b4b5a55-c0e2-46d7-9806-020b3aeffdbd" (UID: "6b4b5a55-c0e2-46d7-9806-020b3aeffdbd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:39:43.600581 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:43.600518 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25007bd5-fdb2-45aa-926f-bfaac7042319-kube-api-access-pj2v5" (OuterVolumeSpecName: "kube-api-access-pj2v5") pod "25007bd5-fdb2-45aa-926f-bfaac7042319" (UID: "25007bd5-fdb2-45aa-926f-bfaac7042319"). InnerVolumeSpecName "kube-api-access-pj2v5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:39:43.600581 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:43.600575 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b4b5a55-c0e2-46d7-9806-020b3aeffdbd-kube-api-access-bn2cd" (OuterVolumeSpecName: "kube-api-access-bn2cd") pod "6b4b5a55-c0e2-46d7-9806-020b3aeffdbd" (UID: "6b4b5a55-c0e2-46d7-9806-020b3aeffdbd"). InnerVolumeSpecName "kube-api-access-bn2cd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:39:43.699293 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:43.699233 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25007bd5-fdb2-45aa-926f-bfaac7042319-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:39:43.699293 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:43.699257 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn2cd\" (UniqueName: \"kubernetes.io/projected/6b4b5a55-c0e2-46d7-9806-020b3aeffdbd-kube-api-access-bn2cd\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:39:43.699293 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:43.699266 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj2v5\" (UniqueName: \"kubernetes.io/projected/25007bd5-fdb2-45aa-926f-bfaac7042319-kube-api-access-pj2v5\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:39:43.699293 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:43.699274 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b4b5a55-c0e2-46d7-9806-020b3aeffdbd-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:39:44.380881 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:44.380850 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bc62f0-6d7d885fcc-vp2rs" Sep 04 22:39:44.381319 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:44.380891 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bc62f0-6d7d885fcc-vp2rs" event={"ID":"6b4b5a55-c0e2-46d7-9806-020b3aeffdbd","Type":"ContainerDied","Data":"e5aa2e459168bb3d8bbb58b67097bb4edd04f5adfced07d25b90b5ffd3f3676d"} Sep 04 22:39:44.381319 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:44.380952 2574 scope.go:117] "RemoveContainer" containerID="891a42c58225b839b10dc1d7597fa13f95b9e702e52418e37aa06bf1f197d97f" Sep 04 22:39:44.382341 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:44.382314 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bc62f0-f7b44cbbb-c69qt" event={"ID":"25007bd5-fdb2-45aa-926f-bfaac7042319","Type":"ContainerDied","Data":"057fd6304320448ec2a22778dee963ec1f1ccbef953f8c7b769937f93b1916e1"} Sep 04 22:39:44.382424 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:44.382327 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bc62f0-f7b44cbbb-c69qt" Sep 04 22:39:44.391275 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:44.391252 2574 scope.go:117] "RemoveContainer" containerID="25c846ef843dd5499a06a8d7db000e0e7c310666aae0d21267b511f29bc4fda1" Sep 04 22:39:44.412871 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:44.411520 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-bc62f0-6d7d885fcc-vp2rs"] Sep 04 22:39:44.416173 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:44.416147 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-bc62f0-6d7d885fcc-vp2rs"] Sep 04 22:39:44.427612 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:44.427588 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bc62f0-f7b44cbbb-c69qt"] Sep 04 22:39:44.432961 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:44.432943 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bc62f0-f7b44cbbb-c69qt"] Sep 04 22:39:44.715376 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:44.715304 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25007bd5-fdb2-45aa-926f-bfaac7042319" path="/var/lib/kubelet/pods/25007bd5-fdb2-45aa-926f-bfaac7042319/volumes" Sep 04 22:39:44.715640 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:44.715626 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="387c661e-dcc3-4aa8-bf76-36e0b94c94e6" path="/var/lib/kubelet/pods/387c661e-dcc3-4aa8-bf76-36e0b94c94e6/volumes" Sep 04 22:39:44.715862 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:44.715851 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48ee60f3-45e8-4b65-bf0f-abf6a231bf27" path="/var/lib/kubelet/pods/48ee60f3-45e8-4b65-bf0f-abf6a231bf27/volumes" Sep 04 22:39:44.716118 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:44.716107 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b4b5a55-c0e2-46d7-9806-020b3aeffdbd" path="/var/lib/kubelet/pods/6b4b5a55-c0e2-46d7-9806-020b3aeffdbd/volumes" Sep 04 22:39:51.158222 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:51.158185 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bf86db-pbd6w"] Sep 04 22:39:51.158703 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:51.158491 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="387c661e-dcc3-4aa8-bf76-36e0b94c94e6" containerName="revoke-keys" Sep 04 22:39:51.158703 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:51.158502 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="387c661e-dcc3-4aa8-bf76-36e0b94c94e6" containerName="revoke-keys" Sep 04 22:39:51.158703 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:51.158509 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25007bd5-fdb2-45aa-926f-bfaac7042319" containerName="payload-pre-processing" Sep 04 22:39:51.158703 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:51.158515 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="25007bd5-fdb2-45aa-926f-bfaac7042319" containerName="payload-pre-processing" Sep 04 22:39:51.158703 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:51.158528 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b4b5a55-c0e2-46d7-9806-020b3aeffdbd" containerName="payload-processing" Sep 04 22:39:51.158703 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:51.158533 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b4b5a55-c0e2-46d7-9806-020b3aeffdbd" containerName="payload-processing" Sep 04 22:39:51.158703 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:51.158604 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b4b5a55-c0e2-46d7-9806-020b3aeffdbd" containerName="payload-processing" Sep 04 22:39:51.158703 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:51.158615 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="25007bd5-fdb2-45aa-926f-bfaac7042319" containerName="payload-pre-processing" Sep 04 22:39:51.158703 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:51.158622 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="387c661e-dcc3-4aa8-bf76-36e0b94c94e6" containerName="revoke-keys" Sep 04 22:39:51.161578 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:51.161537 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bf86db-pbd6w" Sep 04 22:39:51.164335 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:51.164311 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 04 22:39:51.164485 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:51.164471 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hc7x4\"" Sep 04 22:39:51.171670 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:51.171646 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bf86db-pbd6w"] Sep 04 22:39:51.262167 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:51.262136 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7abe3c73-8f3c-4535-b1ee-74239a307cdc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bf86db-pbd6w\" (UID: \"7abe3c73-8f3c-4535-b1ee-74239a307cdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bf86db-pbd6w" Sep 04 22:39:51.362917 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:51.362886 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7abe3c73-8f3c-4535-b1ee-74239a307cdc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bf86db-pbd6w\" (UID: \"7abe3c73-8f3c-4535-b1ee-74239a307cdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bf86db-pbd6w" Sep 04 22:39:51.363597 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:51.363577 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7abe3c73-8f3c-4535-b1ee-74239a307cdc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-bf86db-pbd6w\" (UID: \"7abe3c73-8f3c-4535-b1ee-74239a307cdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bf86db-pbd6w" Sep 04 22:39:51.472968 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:51.472909 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bf86db-pbd6w" Sep 04 22:39:51.750102 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:51.750077 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bf86db-pbd6w"] Sep 04 22:39:51.752024 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:39:51.751997 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7abe3c73_8f3c_4535_b1ee_74239a307cdc.slice/crio-e5dee8ba3f8ed7ac01993a1d68ce0f407925b11a1c820d201b9d737893ec9762 WatchSource:0}: Error finding container e5dee8ba3f8ed7ac01993a1d68ce0f407925b11a1c820d201b9d737893ec9762: Status 404 returned error can't find the container with id e5dee8ba3f8ed7ac01993a1d68ce0f407925b11a1c820d201b9d737893ec9762 Sep 04 22:39:52.428713 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:52.428677 2574 generic.go:356] "Generic (PLEG): container finished" podID="7abe3c73-8f3c-4535-b1ee-74239a307cdc" containerID="7cc5f6d24ed221ad67e7a3aeffaa01376d53e527f41d05c7f92bec2d04ef14ef" exitCode=0 Sep 04 22:39:52.429074 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:52.428763 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bf86db-pbd6w" event={"ID":"7abe3c73-8f3c-4535-b1ee-74239a307cdc","Type":"ContainerDied","Data":"7cc5f6d24ed221ad67e7a3aeffaa01376d53e527f41d05c7f92bec2d04ef14ef"} Sep 04 22:39:52.429074 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:52.428829 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bf86db-pbd6w" event={"ID":"7abe3c73-8f3c-4535-b1ee-74239a307cdc","Type":"ContainerStarted","Data":"e5dee8ba3f8ed7ac01993a1d68ce0f407925b11a1c820d201b9d737893ec9762"} Sep 04 22:39:53.559363 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:53.559340 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bf86db-pbd6w" Sep 04 22:39:53.683517 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:53.683485 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7abe3c73-8f3c-4535-b1ee-74239a307cdc-maas-api-service-ca\") pod \"7abe3c73-8f3c-4535-b1ee-74239a307cdc\" (UID: \"7abe3c73-8f3c-4535-b1ee-74239a307cdc\") " Sep 04 22:39:53.683832 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:53.683808 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7abe3c73-8f3c-4535-b1ee-74239a307cdc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7abe3c73-8f3c-4535-b1ee-74239a307cdc" (UID: "7abe3c73-8f3c-4535-b1ee-74239a307cdc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:39:53.784046 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:53.784018 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7abe3c73-8f3c-4535-b1ee-74239a307cdc-maas-api-service-ca\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:39:54.445796 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:54.445750 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bf86db-pbd6w" Sep 04 22:39:54.445796 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:54.445785 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bf86db-pbd6w" event={"ID":"7abe3c73-8f3c-4535-b1ee-74239a307cdc","Type":"ContainerDied","Data":"e5dee8ba3f8ed7ac01993a1d68ce0f407925b11a1c820d201b9d737893ec9762"} Sep 04 22:39:54.446000 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:39:54.445854 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5dee8ba3f8ed7ac01993a1d68ce0f407925b11a1c820d201b9d737893ec9762" Sep 04 22:40:01.820927 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:01.820895 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf86db-29809350-brkhj"] Sep 04 22:40:01.826677 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:01.826648 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bf86db-29809350-brkhj"] Sep 04 22:40:01.872879 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:01.872850 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bf86db-845cf69cf7-gkcfw"] Sep 04 22:40:01.873083 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:01.873058 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bf86db-845cf69cf7-gkcfw" podUID="a4e03bd9-1bbc-45b4-8482-73d487025325" containerName="payload-processing" containerID="cri-o://338c917bdeb095c898ffab3a2ac5f35b8a382599eaeeea7fad915a63dca1087e" gracePeriod=30 Sep 04 22:40:01.886260 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:01.886231 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf86db-699cd877cd-h9bv7"] Sep 04 22:40:01.886504 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:01.886462 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf86db-699cd877cd-h9bv7" podUID="218c9dcd-c4f3-4ed6-b3db-f12019d81e8d" containerName="payload-pre-processing" containerID="cri-o://aaf622c50ac25253251ac84c9ab20aeb6432e6fdfee77669041be383f55cf1cc" gracePeriod=30 Sep 04 22:40:02.108019 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:02.107991 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bf86db-pbd6w"] Sep 04 22:40:02.113514 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:02.113489 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-bf86db-pbd6w"] Sep 04 22:40:02.151346 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:02.151321 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bf86db-845cf69cf7-gkcfw" Sep 04 22:40:02.154023 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:02.154009 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf86db-699cd877cd-h9bv7" Sep 04 22:40:02.249190 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:02.249163 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/218c9dcd-c4f3-4ed6-b3db-f12019d81e8d-plugins-config-volume\") pod \"218c9dcd-c4f3-4ed6-b3db-f12019d81e8d\" (UID: \"218c9dcd-c4f3-4ed6-b3db-f12019d81e8d\") " Sep 04 22:40:02.249344 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:02.249216 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4e03bd9-1bbc-45b4-8482-73d487025325-plugins-config-volume\") pod \"a4e03bd9-1bbc-45b4-8482-73d487025325\" (UID: \"a4e03bd9-1bbc-45b4-8482-73d487025325\") " Sep 04 22:40:02.249344 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:02.249268 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xd2fb\" (UniqueName: \"kubernetes.io/projected/a4e03bd9-1bbc-45b4-8482-73d487025325-kube-api-access-xd2fb\") pod \"a4e03bd9-1bbc-45b4-8482-73d487025325\" (UID: \"a4e03bd9-1bbc-45b4-8482-73d487025325\") " Sep 04 22:40:02.249473 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:02.249343 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xw65\" (UniqueName: \"kubernetes.io/projected/218c9dcd-c4f3-4ed6-b3db-f12019d81e8d-kube-api-access-8xw65\") pod \"218c9dcd-c4f3-4ed6-b3db-f12019d81e8d\" (UID: \"218c9dcd-c4f3-4ed6-b3db-f12019d81e8d\") " Sep 04 22:40:02.249634 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:02.249613 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/218c9dcd-c4f3-4ed6-b3db-f12019d81e8d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "218c9dcd-c4f3-4ed6-b3db-f12019d81e8d" (UID: "218c9dcd-c4f3-4ed6-b3db-f12019d81e8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:40:02.249705 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:02.249681 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4e03bd9-1bbc-45b4-8482-73d487025325-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a4e03bd9-1bbc-45b4-8482-73d487025325" (UID: "a4e03bd9-1bbc-45b4-8482-73d487025325"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 04 22:40:02.251432 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:02.251410 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4e03bd9-1bbc-45b4-8482-73d487025325-kube-api-access-xd2fb" (OuterVolumeSpecName: "kube-api-access-xd2fb") pod "a4e03bd9-1bbc-45b4-8482-73d487025325" (UID: "a4e03bd9-1bbc-45b4-8482-73d487025325"). InnerVolumeSpecName "kube-api-access-xd2fb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:40:02.251515 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:02.251431 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/218c9dcd-c4f3-4ed6-b3db-f12019d81e8d-kube-api-access-8xw65" (OuterVolumeSpecName: "kube-api-access-8xw65") pod "218c9dcd-c4f3-4ed6-b3db-f12019d81e8d" (UID: "218c9dcd-c4f3-4ed6-b3db-f12019d81e8d"). InnerVolumeSpecName "kube-api-access-8xw65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 04 22:40:02.350840 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:02.350785 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4e03bd9-1bbc-45b4-8482-73d487025325-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:40:02.350840 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:02.350808 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd2fb\" (UniqueName: \"kubernetes.io/projected/a4e03bd9-1bbc-45b4-8482-73d487025325-kube-api-access-xd2fb\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:40:02.350840 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:02.350817 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xw65\" (UniqueName: \"kubernetes.io/projected/218c9dcd-c4f3-4ed6-b3db-f12019d81e8d-kube-api-access-8xw65\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:40:02.350840 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:02.350826 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/218c9dcd-c4f3-4ed6-b3db-f12019d81e8d-plugins-config-volume\") on node \"ip-10-0-137-219.ec2.internal\" DevicePath \"\"" Sep 04 22:40:02.494184 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:02.494152 2574 generic.go:356] "Generic (PLEG): container finished" podID="218c9dcd-c4f3-4ed6-b3db-f12019d81e8d" containerID="aaf622c50ac25253251ac84c9ab20aeb6432e6fdfee77669041be383f55cf1cc" exitCode=0 Sep 04 22:40:02.494295 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:02.494204 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf86db-699cd877cd-h9bv7" Sep 04 22:40:02.494295 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:02.494233 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf86db-699cd877cd-h9bv7" event={"ID":"218c9dcd-c4f3-4ed6-b3db-f12019d81e8d","Type":"ContainerDied","Data":"aaf622c50ac25253251ac84c9ab20aeb6432e6fdfee77669041be383f55cf1cc"} Sep 04 22:40:02.494295 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:02.494285 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf86db-699cd877cd-h9bv7" event={"ID":"218c9dcd-c4f3-4ed6-b3db-f12019d81e8d","Type":"ContainerDied","Data":"4a4f42741aef9c7ac7f83a24a1c01997cf624bfb9629b1546f3d37bf451875a1"} Sep 04 22:40:02.494460 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:02.494311 2574 scope.go:117] "RemoveContainer" containerID="aaf622c50ac25253251ac84c9ab20aeb6432e6fdfee77669041be383f55cf1cc" Sep 04 22:40:02.495739 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:02.495716 2574 generic.go:356] "Generic (PLEG): container finished" podID="a4e03bd9-1bbc-45b4-8482-73d487025325" containerID="338c917bdeb095c898ffab3a2ac5f35b8a382599eaeeea7fad915a63dca1087e" exitCode=0 Sep 04 22:40:02.495838 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:02.495760 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bf86db-845cf69cf7-gkcfw" Sep 04 22:40:02.495838 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:02.495804 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bf86db-845cf69cf7-gkcfw" event={"ID":"a4e03bd9-1bbc-45b4-8482-73d487025325","Type":"ContainerDied","Data":"338c917bdeb095c898ffab3a2ac5f35b8a382599eaeeea7fad915a63dca1087e"} Sep 04 22:40:02.495971 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:02.495843 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bf86db-845cf69cf7-gkcfw" event={"ID":"a4e03bd9-1bbc-45b4-8482-73d487025325","Type":"ContainerDied","Data":"942778bfbbe42fa1e75d69ffd3456d305fce27a74384032018de3fa20702009a"} Sep 04 22:40:02.504125 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:02.504111 2574 scope.go:117] "RemoveContainer" containerID="aaf622c50ac25253251ac84c9ab20aeb6432e6fdfee77669041be383f55cf1cc" Sep 04 22:40:02.504384 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:40:02.504366 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaf622c50ac25253251ac84c9ab20aeb6432e6fdfee77669041be383f55cf1cc\": container with ID starting with aaf622c50ac25253251ac84c9ab20aeb6432e6fdfee77669041be383f55cf1cc not found: ID does not exist" containerID="aaf622c50ac25253251ac84c9ab20aeb6432e6fdfee77669041be383f55cf1cc" Sep 04 22:40:02.504470 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:02.504390 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaf622c50ac25253251ac84c9ab20aeb6432e6fdfee77669041be383f55cf1cc"} err="failed to get container status \"aaf622c50ac25253251ac84c9ab20aeb6432e6fdfee77669041be383f55cf1cc\": rpc error: code = NotFound desc = could not find container \"aaf622c50ac25253251ac84c9ab20aeb6432e6fdfee77669041be383f55cf1cc\": container with ID starting with aaf622c50ac25253251ac84c9ab20aeb6432e6fdfee77669041be383f55cf1cc not found: ID does not exist" Sep 04 22:40:02.504470 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:02.504404 2574 scope.go:117] "RemoveContainer" containerID="338c917bdeb095c898ffab3a2ac5f35b8a382599eaeeea7fad915a63dca1087e" Sep 04 22:40:02.512463 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:02.512446 2574 scope.go:117] "RemoveContainer" containerID="338c917bdeb095c898ffab3a2ac5f35b8a382599eaeeea7fad915a63dca1087e" Sep 04 22:40:02.512722 ip-10-0-137-219 kubenswrapper[2574]: E0904 22:40:02.512707 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"338c917bdeb095c898ffab3a2ac5f35b8a382599eaeeea7fad915a63dca1087e\": container with ID starting with 338c917bdeb095c898ffab3a2ac5f35b8a382599eaeeea7fad915a63dca1087e not found: ID does not exist" containerID="338c917bdeb095c898ffab3a2ac5f35b8a382599eaeeea7fad915a63dca1087e" Sep 04 22:40:02.512768 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:02.512727 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"338c917bdeb095c898ffab3a2ac5f35b8a382599eaeeea7fad915a63dca1087e"} err="failed to get container status \"338c917bdeb095c898ffab3a2ac5f35b8a382599eaeeea7fad915a63dca1087e\": rpc error: code = NotFound desc = could not find container \"338c917bdeb095c898ffab3a2ac5f35b8a382599eaeeea7fad915a63dca1087e\": container with ID starting with 338c917bdeb095c898ffab3a2ac5f35b8a382599eaeeea7fad915a63dca1087e not found: ID does not exist" Sep 04 22:40:02.521121 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:02.521100 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf86db-699cd877cd-h9bv7"] Sep 04 22:40:02.525412 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:02.525393 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bf86db-699cd877cd-h9bv7"] Sep 04 22:40:02.540684 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:02.540665 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bf86db-845cf69cf7-gkcfw"] Sep 04 22:40:02.547464 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:02.547444 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bf86db-845cf69cf7-gkcfw"] Sep 04 22:40:02.715575 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:02.715486 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="218c9dcd-c4f3-4ed6-b3db-f12019d81e8d" path="/var/lib/kubelet/pods/218c9dcd-c4f3-4ed6-b3db-f12019d81e8d/volumes" Sep 04 22:40:02.715843 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:02.715828 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7abe3c73-8f3c-4535-b1ee-74239a307cdc" path="/var/lib/kubelet/pods/7abe3c73-8f3c-4535-b1ee-74239a307cdc/volumes" Sep 04 22:40:02.716148 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:02.716136 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4e03bd9-1bbc-45b4-8482-73d487025325" path="/var/lib/kubelet/pods/a4e03bd9-1bbc-45b4-8482-73d487025325/volumes" Sep 04 22:40:02.716371 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:02.716359 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9b5b00b-13b6-4f14-a403-2c0478abe985" path="/var/lib/kubelet/pods/e9b5b00b-13b6-4f14-a403-2c0478abe985/volumes" Sep 04 22:40:17.152201 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:17.152166 2574 scope.go:117] "RemoveContainer" containerID="e2cf82d9f12eec04227301ecb119339243cce49933777b650f6bbc34b7e55d24" Sep 04 22:40:17.160582 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:17.160557 2574 scope.go:117] "RemoveContainer" containerID="8a0297cf1157a369bec37b1e2996c46101a513bdb5bc8752fce21a516a2b9f1a" Sep 04 22:40:17.167874 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:40:17.167854 2574 scope.go:117] "RemoveContainer" containerID="537d8d4cc7a9ae3334a795dfa6dd5421f2f679a473c883ca5777a052828f109c" Sep 04 22:41:17.215688 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:41:17.215604 2574 scope.go:117] "RemoveContainer" containerID="00c0d472f61e9e2f1b2e27c006f0ea4acefd1bd85391923a8bebd13cfa0b98f5" Sep 04 22:41:17.225099 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:41:17.225082 2574 scope.go:117] "RemoveContainer" containerID="d92f1f9570c335d71fef58e662d8d260b72438b5782b2ed96c146f07b28091f5" Sep 04 22:42:17.247269 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:42:17.247238 2574 scope.go:117] "RemoveContainer" containerID="47fc8b7c9852614ef923d603595ad29c45277e66b15a5bc2d0a38e775ad3f8fb" Sep 04 22:44:17.272634 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:44:17.272605 2574 scope.go:117] "RemoveContainer" containerID="18da19ae4a9365697599a9de12e07e189e5f60df69dc883b916126c1f11ec463" Sep 04 22:44:17.282007 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:44:17.281983 2574 scope.go:117] "RemoveContainer" containerID="3ade6ac4dac46580055beb5af784e8e4d410a85ef3d6402cf9ddd28ae80713c9" Sep 04 22:45:00.133970 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:45:00.133938 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29809365-kfm9w"] Sep 04 22:45:00.134464 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:45:00.134233 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7abe3c73-8f3c-4535-b1ee-74239a307cdc" containerName="revoke-keys" Sep 04 22:45:00.134464 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:45:00.134245 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7abe3c73-8f3c-4535-b1ee-74239a307cdc" containerName="revoke-keys" Sep 04 22:45:00.134464 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:45:00.134254 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="218c9dcd-c4f3-4ed6-b3db-f12019d81e8d" containerName="payload-pre-processing" Sep 04 22:45:00.134464 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:45:00.134261 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="218c9dcd-c4f3-4ed6-b3db-f12019d81e8d" containerName="payload-pre-processing" Sep 04 22:45:00.134464 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:45:00.134276 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4e03bd9-1bbc-45b4-8482-73d487025325" containerName="payload-processing" Sep 04 22:45:00.134464 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:45:00.134282 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4e03bd9-1bbc-45b4-8482-73d487025325" containerName="payload-processing" Sep 04 22:45:00.134464 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:45:00.134339 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7abe3c73-8f3c-4535-b1ee-74239a307cdc" containerName="revoke-keys" Sep 04 22:45:00.134464 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:45:00.134348 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4e03bd9-1bbc-45b4-8482-73d487025325" containerName="payload-processing" Sep 04 22:45:00.134464 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:45:00.134353 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="218c9dcd-c4f3-4ed6-b3db-f12019d81e8d" containerName="payload-pre-processing" Sep 04 22:45:00.137221 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:45:00.137206 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29809365-kfm9w" Sep 04 22:45:00.139420 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:45:00.139396 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hc7x4\"" Sep 04 22:45:00.144732 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:45:00.144710 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29809365-kfm9w"] Sep 04 22:45:00.148056 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:45:00.148037 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29809365-kfm9w" Sep 04 22:45:00.419738 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:45:00.419671 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29809365-kfm9w"] Sep 04 22:45:00.422005 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:45:00.421979 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c80dda0_213d_431b_bbcd_fb2d0af2d2ba.slice/crio-a7dc0a08570f5b90069b2ad38bc7c6431e449e7642c6a7f5c4e25e25b057f1b6 WatchSource:0}: Error finding container a7dc0a08570f5b90069b2ad38bc7c6431e449e7642c6a7f5c4e25e25b057f1b6: Status 404 returned error can't find the container with id a7dc0a08570f5b90069b2ad38bc7c6431e449e7642c6a7f5c4e25e25b057f1b6 Sep 04 22:45:01.029370 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:45:01.029337 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29809365-kfm9w" event={"ID":"9c80dda0-213d-431b-bbcd-fb2d0af2d2ba","Type":"ContainerStarted","Data":"c0058f2d6de7d41e0ab8765b911d89cead00cdff66c6605a5d2c3287ae3a5d11"} Sep 04 22:45:01.029532 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:45:01.029388 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29809365-kfm9w" event={"ID":"9c80dda0-213d-431b-bbcd-fb2d0af2d2ba","Type":"ContainerStarted","Data":"a7dc0a08570f5b90069b2ad38bc7c6431e449e7642c6a7f5c4e25e25b057f1b6"} Sep 04 22:45:01.046810 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:45:01.046762 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29809365-kfm9w" podStartSLOduration=1.046742457 podStartE2EDuration="1.046742457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:45:01.044255748 +0000 UTC m=+2204.808731364" watchObservedRunningTime="2026-09-04 22:45:01.046742457 +0000 UTC m=+2204.811218070" Sep 04 22:45:02.035315 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:45:02.035281 2574 generic.go:356] "Generic (PLEG): container finished" podID="9c80dda0-213d-431b-bbcd-fb2d0af2d2ba" containerID="c0058f2d6de7d41e0ab8765b911d89cead00cdff66c6605a5d2c3287ae3a5d11" exitCode=0 Sep 04 22:45:02.035787 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:45:02.035359 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29809365-kfm9w" event={"ID":"9c80dda0-213d-431b-bbcd-fb2d0af2d2ba","Type":"ContainerDied","Data":"c0058f2d6de7d41e0ab8765b911d89cead00cdff66c6605a5d2c3287ae3a5d11"} Sep 04 22:45:03.162908 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:45:03.162887 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29809365-kfm9w" Sep 04 22:45:04.047125 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:45:04.047099 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29809365-kfm9w" Sep 04 22:45:04.047288 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:45:04.047144 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29809365-kfm9w" event={"ID":"9c80dda0-213d-431b-bbcd-fb2d0af2d2ba","Type":"ContainerDied","Data":"a7dc0a08570f5b90069b2ad38bc7c6431e449e7642c6a7f5c4e25e25b057f1b6"} Sep 04 22:45:04.047288 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:45:04.047189 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7dc0a08570f5b90069b2ad38bc7c6431e449e7642c6a7f5c4e25e25b057f1b6" Sep 04 22:45:17.302235 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:45:17.302205 2574 scope.go:117] "RemoveContainer" containerID="49fcd0ba6f146f25c7a685930b659ea69b62ba5b1b0b14b2fb43d52074188646" Sep 04 22:46:14.480699 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:14.480667 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-glhzp_65d33876-fc29-4b81-883c-9d982a830ef8/manager/0.log" Sep 04 22:46:14.839344 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:14.839316 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cd686f5d7-hxv2l_67333fdd-f1ff-41b9-bb3a-504f1d3fb547/copy-manifests/0.log" Sep 04 22:46:14.852866 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:14.852836 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cd686f5d7-hxv2l_67333fdd-f1ff-41b9-bb3a-504f1d3fb547/manager/0.log" Sep 04 22:46:15.522246 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:15.522220 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29809365-kfm9w_9c80dda0-213d-431b-bbcd-fb2d0af2d2ba/cleanup/0.log" Sep 04 22:46:17.319959 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:17.319924 2574 scope.go:117] "RemoveContainer" containerID="7cc5f6d24ed221ad67e7a3aeffaa01376d53e527f41d05c7f92bec2d04ef14ef" Sep 04 22:46:17.327904 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:17.327888 2574 scope.go:117] "RemoveContainer" containerID="8deb8045c6ed69210758e8ef82c3a871c889d502898f70fd40dbbc6cd7e7515a" Sep 04 22:46:17.580907 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:17.580817 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-b885p_25883a31-d829-4bb6-b434-8d525c5f7368/discovery/0.log" Sep 04 22:46:18.168240 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:18.168211 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-xj6cg_8e12172f-2c9e-4eaa-b780-076e4c686406/payload-processing/0.log" Sep 04 22:46:18.279712 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:18.279668 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-599484b46b-xdvdt_de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94/router/0.log" Sep 04 22:46:18.728799 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:18.728769 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-gdkld_1ac44222-4d72-4c15-b2a5-e6db9e022242/main/0.log" Sep 04 22:46:18.843864 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:18.843830 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-lp9j6_7ded97bf-a59e-4609-bed2-616814538e8c/main/0.log" Sep 04 22:46:19.064066 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:19.064035 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7ctb6_7caf7d48-54df-4902-861d-3bd25adc8de0/main/0.log" Sep 04 22:46:27.477296 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:27.477242 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ndzbl_62c2d50e-f517-47e6-8042-31b2455981fb/global-pull-secret-syncer/0.log" Sep 04 22:46:27.622278 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:27.622238 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9gcxl_1373f05f-60fa-4414-a0da-93106632c578/konnectivity-agent/0.log" Sep 04 22:46:27.703912 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:27.703889 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-219.ec2.internal_5c4c794f844e4275466dd646fd4a6840/haproxy/0.log" Sep 04 22:46:34.645326 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:34.645297 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k7vtb_e8026ebe-4c89-4066-9ef4-39769575e3c3/node-exporter/0.log" Sep 04 22:46:34.668691 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:34.668662 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k7vtb_e8026ebe-4c89-4066-9ef4-39769575e3c3/kube-rbac-proxy/0.log" Sep 04 22:46:34.699047 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:34.699031 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k7vtb_e8026ebe-4c89-4066-9ef4-39769575e3c3/init-textfile/0.log" Sep 04 22:46:37.653852 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:37.653815 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2c9z6/perf-node-gather-daemonset-ckxlj"] Sep 04 22:46:37.654228 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:37.654125 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c80dda0-213d-431b-bbcd-fb2d0af2d2ba" containerName="cleanup" Sep 04 22:46:37.654228 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:37.654136 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c80dda0-213d-431b-bbcd-fb2d0af2d2ba" containerName="cleanup" Sep 04 22:46:37.654228 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:37.654188 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c80dda0-213d-431b-bbcd-fb2d0af2d2ba" containerName="cleanup" Sep 04 22:46:37.656781 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:37.656764 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2c9z6/perf-node-gather-daemonset-ckxlj" Sep 04 22:46:37.659038 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:37.659017 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2c9z6\"/\"openshift-service-ca.crt\"" Sep 04 22:46:37.659898 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:37.659878 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2c9z6\"/\"default-dockercfg-6c2zj\"" Sep 04 22:46:37.659989 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:37.659883 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2c9z6\"/\"kube-root-ca.crt\"" Sep 04 22:46:37.665776 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:37.665758 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2c9z6/perf-node-gather-daemonset-ckxlj"] Sep 04 22:46:37.743984 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:37.743950 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25jbd\" (UniqueName: \"kubernetes.io/projected/cee10974-1255-49bc-913e-d6cecc494343-kube-api-access-25jbd\") pod \"perf-node-gather-daemonset-ckxlj\" (UID: \"cee10974-1255-49bc-913e-d6cecc494343\") " pod="openshift-must-gather-2c9z6/perf-node-gather-daemonset-ckxlj" Sep 04 22:46:37.744150 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:37.743993 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cee10974-1255-49bc-913e-d6cecc494343-sys\") pod \"perf-node-gather-daemonset-ckxlj\" (UID: \"cee10974-1255-49bc-913e-d6cecc494343\") " pod="openshift-must-gather-2c9z6/perf-node-gather-daemonset-ckxlj" Sep 04 22:46:37.744150 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:37.744038 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cee10974-1255-49bc-913e-d6cecc494343-lib-modules\") pod \"perf-node-gather-daemonset-ckxlj\" (UID: \"cee10974-1255-49bc-913e-d6cecc494343\") " pod="openshift-must-gather-2c9z6/perf-node-gather-daemonset-ckxlj" Sep 04 22:46:37.744150 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:37.744067 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cee10974-1255-49bc-913e-d6cecc494343-proc\") pod \"perf-node-gather-daemonset-ckxlj\" (UID: \"cee10974-1255-49bc-913e-d6cecc494343\") " pod="openshift-must-gather-2c9z6/perf-node-gather-daemonset-ckxlj" Sep 04 22:46:37.744150 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:37.744088 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cee10974-1255-49bc-913e-d6cecc494343-podres\") pod \"perf-node-gather-daemonset-ckxlj\" (UID: \"cee10974-1255-49bc-913e-d6cecc494343\") " pod="openshift-must-gather-2c9z6/perf-node-gather-daemonset-ckxlj" Sep 04 22:46:37.845034 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:37.845005 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25jbd\" (UniqueName: \"kubernetes.io/projected/cee10974-1255-49bc-913e-d6cecc494343-kube-api-access-25jbd\") pod \"perf-node-gather-daemonset-ckxlj\" (UID: \"cee10974-1255-49bc-913e-d6cecc494343\") " pod="openshift-must-gather-2c9z6/perf-node-gather-daemonset-ckxlj" Sep 04 22:46:37.845205 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:37.845038 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cee10974-1255-49bc-913e-d6cecc494343-sys\") pod \"perf-node-gather-daemonset-ckxlj\" (UID: \"cee10974-1255-49bc-913e-d6cecc494343\") " pod="openshift-must-gather-2c9z6/perf-node-gather-daemonset-ckxlj" Sep 04 22:46:37.845205 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:37.845071 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cee10974-1255-49bc-913e-d6cecc494343-lib-modules\") pod \"perf-node-gather-daemonset-ckxlj\" (UID: \"cee10974-1255-49bc-913e-d6cecc494343\") " pod="openshift-must-gather-2c9z6/perf-node-gather-daemonset-ckxlj" Sep 04 22:46:37.845205 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:37.845097 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cee10974-1255-49bc-913e-d6cecc494343-proc\") pod \"perf-node-gather-daemonset-ckxlj\" (UID: \"cee10974-1255-49bc-913e-d6cecc494343\") " pod="openshift-must-gather-2c9z6/perf-node-gather-daemonset-ckxlj" Sep 04 22:46:37.845205 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:37.845111 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cee10974-1255-49bc-913e-d6cecc494343-podres\") pod \"perf-node-gather-daemonset-ckxlj\" (UID: \"cee10974-1255-49bc-913e-d6cecc494343\") " pod="openshift-must-gather-2c9z6/perf-node-gather-daemonset-ckxlj" Sep 04 22:46:37.845205 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:37.845140 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cee10974-1255-49bc-913e-d6cecc494343-sys\") pod \"perf-node-gather-daemonset-ckxlj\" (UID: \"cee10974-1255-49bc-913e-d6cecc494343\") " pod="openshift-must-gather-2c9z6/perf-node-gather-daemonset-ckxlj" Sep 04 22:46:37.845205 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:37.845191 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cee10974-1255-49bc-913e-d6cecc494343-proc\") pod \"perf-node-gather-daemonset-ckxlj\" (UID: \"cee10974-1255-49bc-913e-d6cecc494343\") " pod="openshift-must-gather-2c9z6/perf-node-gather-daemonset-ckxlj" Sep 04 22:46:37.845528 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:37.845221 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cee10974-1255-49bc-913e-d6cecc494343-podres\") pod \"perf-node-gather-daemonset-ckxlj\" (UID: \"cee10974-1255-49bc-913e-d6cecc494343\") " pod="openshift-must-gather-2c9z6/perf-node-gather-daemonset-ckxlj" Sep 04 22:46:37.845528 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:37.845250 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cee10974-1255-49bc-913e-d6cecc494343-lib-modules\") pod \"perf-node-gather-daemonset-ckxlj\" (UID: \"cee10974-1255-49bc-913e-d6cecc494343\") " pod="openshift-must-gather-2c9z6/perf-node-gather-daemonset-ckxlj" Sep 04 22:46:37.854604 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:37.854572 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25jbd\" (UniqueName: \"kubernetes.io/projected/cee10974-1255-49bc-913e-d6cecc494343-kube-api-access-25jbd\") pod \"perf-node-gather-daemonset-ckxlj\" (UID: \"cee10974-1255-49bc-913e-d6cecc494343\") " pod="openshift-must-gather-2c9z6/perf-node-gather-daemonset-ckxlj" Sep 04 22:46:37.968596 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:37.968511 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2c9z6/perf-node-gather-daemonset-ckxlj" Sep 04 22:46:38.187775 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:38.187748 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-867cfc6f4d-2kp9k_514c0fa6-6c00-4791-9c9f-05f743605730/console/0.log" Sep 04 22:46:38.238117 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:38.238092 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2c9z6/perf-node-gather-daemonset-ckxlj"] Sep 04 22:46:38.239850 ip-10-0-137-219 kubenswrapper[2574]: W0904 22:46:38.239828 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podcee10974_1255_49bc_913e_d6cecc494343.slice/crio-62789d8af3488579c592dab18ae915c4a7e118db862acbd1c885a1005fa27007 WatchSource:0}: Error finding container 62789d8af3488579c592dab18ae915c4a7e118db862acbd1c885a1005fa27007: Status 404 returned error can't find the container with id 62789d8af3488579c592dab18ae915c4a7e118db862acbd1c885a1005fa27007 Sep 04 22:46:38.524538 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:38.524506 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2c9z6/perf-node-gather-daemonset-ckxlj" event={"ID":"cee10974-1255-49bc-913e-d6cecc494343","Type":"ContainerStarted","Data":"e9348b72d9f7d3917d9e218307f6559409bd6b46e96e934470388169fa679345"} Sep 04 22:46:38.524732 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:38.524585 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2c9z6/perf-node-gather-daemonset-ckxlj" event={"ID":"cee10974-1255-49bc-913e-d6cecc494343","Type":"ContainerStarted","Data":"62789d8af3488579c592dab18ae915c4a7e118db862acbd1c885a1005fa27007"} Sep 04 22:46:38.524732 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:38.524710 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2c9z6/perf-node-gather-daemonset-ckxlj" Sep 04 22:46:38.542582 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:38.542520 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2c9z6/perf-node-gather-daemonset-ckxlj" podStartSLOduration=1.542505649 podStartE2EDuration="1.542505649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 22:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:46:38.540310506 +0000 UTC m=+2302.304786118" watchObservedRunningTime="2026-09-04 22:46:38.542505649 +0000 UTC m=+2302.306981262" Sep 04 22:46:40.120878 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:40.120851 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9jzzc_059b83a0-e9ee-4f9f-a3ae-250d85b3387d/dns/0.log" Sep 04 22:46:40.145529 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:40.145501 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9jzzc_059b83a0-e9ee-4f9f-a3ae-250d85b3387d/kube-rbac-proxy/0.log" Sep 04 22:46:40.280598 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:40.280574 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-p9mdt_c8319608-4bcc-405e-b607-e3fea2c4e9a8/dns-node-resolver/0.log" Sep 04 22:46:40.864498 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:40.864467 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d5j8h_54f49670-6d36-4464-b9c4-87718e384fb2/node-ca/0.log" Sep 04 22:46:42.174273 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:42.174243 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-b885p_25883a31-d829-4bb6-b434-8d525c5f7368/discovery/0.log" Sep 04 22:46:42.338244 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:42.338220 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-xj6cg_8e12172f-2c9e-4eaa-b780-076e4c686406/payload-processing/0.log" Sep 04 22:46:42.363993 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:42.363971 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-599484b46b-xdvdt_de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94/router/0.log" Sep 04 22:46:43.346916 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:43.346880 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-c2jbn_6d30375e-bff3-4f2c-9322-6e97a3c0e109/serve-healthcheck-canary/0.log" Sep 04 22:46:43.989631 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:43.989606 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-84kv4_c932b83b-a9b3-46dc-963d-af429480dc45/kube-rbac-proxy/0.log" Sep 04 22:46:44.014181 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:44.014148 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-84kv4_c932b83b-a9b3-46dc-963d-af429480dc45/exporter/0.log" Sep 04 22:46:44.039236 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:44.039208 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-84kv4_c932b83b-a9b3-46dc-963d-af429480dc45/extractor/0.log" Sep 04 22:46:44.539686 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:44.539663 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2c9z6/perf-node-gather-daemonset-ckxlj" Sep 04 22:46:46.736218 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:46.736191 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-glhzp_65d33876-fc29-4b81-883c-9d982a830ef8/manager/0.log" Sep 04 22:46:46.942203 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:46.942132 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cd686f5d7-hxv2l_67333fdd-f1ff-41b9-bb3a-504f1d3fb547/manager/0.log" Sep 04 22:46:46.963753 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:46.963730 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cd686f5d7-hxv2l_67333fdd-f1ff-41b9-bb3a-504f1d3fb547/copy-manifests/0.log" Sep 04 22:46:48.392231 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:48.392196 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-868df85875-2qvbh_5a157212-e6ef-47fa-9c00-f522c947d76d/manager/0.log" Sep 04 22:46:48.456122 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:48.456102 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-hh5sw_b1f71386-c859-4117-80ce-9817285e5888/openshift-lws-operator/0.log" Sep 04 22:46:48.821754 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:48.821726 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-b885p_25883a31-d829-4bb6-b434-8d525c5f7368/discovery/0.log" Sep 04 22:46:48.935828 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:48.935801 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-xj6cg_8e12172f-2c9e-4eaa-b780-076e4c686406/payload-processing/0.log" Sep 04 22:46:48.953338 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:48.953314 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-599484b46b-xdvdt_de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94/router/0.log" Sep 04 22:46:50.706924 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:50.706893 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-b885p_25883a31-d829-4bb6-b434-8d525c5f7368/discovery/0.log" Sep 04 22:46:50.837736 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:50.837707 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-xj6cg_8e12172f-2c9e-4eaa-b780-076e4c686406/payload-processing/0.log" Sep 04 22:46:50.855627 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:50.855599 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-599484b46b-xdvdt_de0ec74b-29ce-42dc-bb2d-be0dcfdc0a94/router/0.log" Sep 04 22:46:55.302331 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:55.302302 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b49j5_95ade170-0175-41e0-b78b-ede111ef37af/kube-multus-additional-cni-plugins/0.log" Sep 04 22:46:55.329465 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:55.329444 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b49j5_95ade170-0175-41e0-b78b-ede111ef37af/egress-router-binary-copy/0.log" Sep 04 22:46:55.356342 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:55.356319 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b49j5_95ade170-0175-41e0-b78b-ede111ef37af/cni-plugins/0.log" Sep 04 22:46:55.385343 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:55.385322 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b49j5_95ade170-0175-41e0-b78b-ede111ef37af/bond-cni-plugin/0.log" Sep 04 22:46:55.412516 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:55.412499 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b49j5_95ade170-0175-41e0-b78b-ede111ef37af/routeoverride-cni/0.log" Sep 04 22:46:55.444014 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:55.443996 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b49j5_95ade170-0175-41e0-b78b-ede111ef37af/whereabouts-cni-bincopy/0.log" Sep 04 22:46:55.470405 ip-10-0-137-219 kubenswrapper[2574]: I0904 22:46:55.470388 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-b49j5_95ade170-0175-41e0-b78b-ede111ef37af/whereabouts-cni/0.log"