Sep 14 11:00:54.093972 ip-10-0-137-230 systemd[1]: Starting Kubernetes Kubelet... Sep 14 11:00:54.516175 ip-10-0-137-230 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 14 11:00:54.516175 ip-10-0-137-230 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 14 11:00:54.516175 ip-10-0-137-230 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 14 11:00:54.516175 ip-10-0-137-230 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 14 11:00:54.516175 ip-10-0-137-230 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 14 11:00:54.517296 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.517213 2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 14 11:00:54.521315 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521300 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 11:00:54.521315 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521314 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 11:00:54.521315 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521317 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 11:00:54.521418 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521321 2574 feature_gate.go:328] unrecognized feature gate: Example Sep 14 11:00:54.521418 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521324 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 11:00:54.521418 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521327 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 11:00:54.521418 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521329 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 11:00:54.521418 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521332 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 11:00:54.521418 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521335 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 11:00:54.521418 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521338 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 11:00:54.521418 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521342 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 11:00:54.521418 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521344 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 11:00:54.521418 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521347 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 11:00:54.521418 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521349 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 11:00:54.521418 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521352 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 11:00:54.521418 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521355 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 11:00:54.521418 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521358 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 11:00:54.521418 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521366 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 11:00:54.521418 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521369 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 11:00:54.521418 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521371 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 11:00:54.521418 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521374 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 11:00:54.521418 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521376 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 11:00:54.521418 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521379 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 11:00:54.521936 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521382 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 11:00:54.521936 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521384 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 11:00:54.521936 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521387 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 11:00:54.521936 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521389 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 11:00:54.521936 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521392 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 11:00:54.521936 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521394 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 11:00:54.521936 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521396 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 11:00:54.521936 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521399 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 11:00:54.521936 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521401 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 11:00:54.521936 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521404 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 11:00:54.521936 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521406 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 11:00:54.521936 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521410 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 11:00:54.521936 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521413 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 11:00:54.521936 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521415 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 11:00:54.521936 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521418 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 11:00:54.521936 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521421 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 11:00:54.521936 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521424 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 11:00:54.521936 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521426 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 11:00:54.521936 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521429 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 11:00:54.521936 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521431 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 11:00:54.522465 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521434 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 11:00:54.522465 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521436 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 11:00:54.522465 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521439 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 11:00:54.522465 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521441 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 11:00:54.522465 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521444 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 11:00:54.522465 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521446 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 11:00:54.522465 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521448 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 11:00:54.522465 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521452 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 11:00:54.522465 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521454 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 11:00:54.522465 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521457 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 11:00:54.522465 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521459 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 11:00:54.522465 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521461 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 11:00:54.522465 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521464 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 11:00:54.522465 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521466 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 11:00:54.522465 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521469 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 11:00:54.522465 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521471 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 11:00:54.522465 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521474 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 11:00:54.522465 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521476 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 11:00:54.522465 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521478 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 11:00:54.522465 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521481 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 11:00:54.523000 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521485 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 11:00:54.523000 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521488 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 11:00:54.523000 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521491 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 11:00:54.523000 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521495 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 11:00:54.523000 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521499 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 11:00:54.523000 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521503 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 11:00:54.523000 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521506 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 11:00:54.523000 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521508 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 11:00:54.523000 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521511 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 11:00:54.523000 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521514 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 11:00:54.523000 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521516 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 11:00:54.523000 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521519 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 11:00:54.523000 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521521 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 11:00:54.523000 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521524 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 11:00:54.523000 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521527 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 11:00:54.523000 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521529 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 11:00:54.523000 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521532 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 11:00:54.523000 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521534 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 11:00:54.523469 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521537 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 11:00:54.523469 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521539 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 11:00:54.523469 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521542 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 11:00:54.523469 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521544 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 11:00:54.523469 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521548 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 11:00:54.523469 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521551 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 11:00:54.523469 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521553 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 11:00:54.523469 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521555 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 11:00:54.523469 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521558 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 11:00:54.523469 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521560 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 11:00:54.523469 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521563 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 11:00:54.523469 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521565 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 11:00:54.523469 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521568 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 11:00:54.523469 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521571 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 11:00:54.523469 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521575 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 11:00:54.523469 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521578 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 11:00:54.523469 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521580 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 11:00:54.523469 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.521583 2574 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 11:00:54.525633 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525616 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 11:00:54.525633 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525631 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 11:00:54.525633 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525636 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 11:00:54.525779 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525641 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 11:00:54.525779 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525645 2574 feature_gate.go:328] unrecognized feature gate: Example Sep 14 11:00:54.525779 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525648 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 11:00:54.525779 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525651 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 11:00:54.525779 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525654 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 11:00:54.525779 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525657 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 11:00:54.525779 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525660 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 11:00:54.525779 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525663 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 11:00:54.525779 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525683 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 11:00:54.525779 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525697 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 11:00:54.525779 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525700 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 11:00:54.525779 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525703 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 11:00:54.525779 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525706 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 11:00:54.525779 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525709 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 11:00:54.525779 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525711 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 11:00:54.525779 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525714 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 11:00:54.525779 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525717 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 11:00:54.525779 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525719 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 11:00:54.525779 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525722 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 11:00:54.525779 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525724 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 11:00:54.526377 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525727 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 11:00:54.526377 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525729 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 11:00:54.526377 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525731 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 11:00:54.526377 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525735 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 11:00:54.526377 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525739 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 11:00:54.526377 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525741 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 11:00:54.526377 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525744 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 11:00:54.526377 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525746 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 11:00:54.526377 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525750 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 11:00:54.526377 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525752 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 11:00:54.526377 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525755 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 11:00:54.526377 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525758 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 11:00:54.526377 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525761 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 11:00:54.526377 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525763 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 11:00:54.526377 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525766 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 11:00:54.526377 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525769 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 11:00:54.526377 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525772 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 11:00:54.526377 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525775 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 11:00:54.526377 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525777 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 11:00:54.526898 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525780 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 11:00:54.526898 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525782 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 11:00:54.526898 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525785 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 11:00:54.526898 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525788 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 11:00:54.526898 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525791 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 11:00:54.526898 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525793 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 11:00:54.526898 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525795 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 11:00:54.526898 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525798 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 11:00:54.526898 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525801 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 11:00:54.526898 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525803 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 11:00:54.526898 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525805 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 11:00:54.526898 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525808 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 11:00:54.526898 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525811 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 11:00:54.526898 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525813 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 11:00:54.526898 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525816 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 11:00:54.526898 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525818 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 11:00:54.526898 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525820 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 11:00:54.526898 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525823 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 11:00:54.526898 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525825 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 11:00:54.526898 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525828 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 11:00:54.527420 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525830 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 11:00:54.527420 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525833 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 11:00:54.527420 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525835 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 11:00:54.527420 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525838 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 11:00:54.527420 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525840 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 11:00:54.527420 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525843 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 11:00:54.527420 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525846 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 11:00:54.527420 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525848 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 11:00:54.527420 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525851 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 11:00:54.527420 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525855 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 11:00:54.527420 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525857 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 11:00:54.527420 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525859 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 11:00:54.527420 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525862 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 11:00:54.527420 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525865 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 11:00:54.527420 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525867 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 11:00:54.527420 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525881 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 11:00:54.527420 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525884 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 11:00:54.527420 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525886 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 11:00:54.527420 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525889 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 11:00:54.527420 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525891 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 11:00:54.527959 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525895 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 11:00:54.527959 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525899 2574 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 11:00:54.527959 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525902 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 11:00:54.527959 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525905 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 11:00:54.527959 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525907 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 11:00:54.527959 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525910 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 11:00:54.527959 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525913 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 11:00:54.527959 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525916 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 11:00:54.527959 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525919 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 11:00:54.527959 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525921 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 11:00:54.527959 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525924 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 11:00:54.527959 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525927 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 11:00:54.527959 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525930 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 11:00:54.527959 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525933 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 11:00:54.527959 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525936 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 11:00:54.527959 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525939 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 11:00:54.527959 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.525942 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 11:00:54.527959 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526313 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 11:00:54.527959 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526318 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 11:00:54.527959 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526322 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 11:00:54.528478 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526325 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 11:00:54.528478 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526327 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 11:00:54.528478 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526330 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 11:00:54.528478 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526333 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 11:00:54.528478 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526336 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 11:00:54.528478 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526339 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 11:00:54.528478 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526343 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 11:00:54.528478 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526346 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 11:00:54.528478 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526349 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 11:00:54.528478 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526351 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 11:00:54.528478 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526353 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 11:00:54.528478 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526356 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 11:00:54.528478 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526358 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 11:00:54.528478 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526361 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 11:00:54.528478 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526364 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 11:00:54.528478 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526367 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 11:00:54.528478 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526369 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 11:00:54.528478 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526372 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 11:00:54.528478 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526374 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 11:00:54.528991 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526377 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 11:00:54.528991 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526380 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 11:00:54.528991 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526382 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 11:00:54.528991 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526385 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 11:00:54.528991 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526388 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 11:00:54.528991 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526390 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 11:00:54.528991 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526393 2574 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 11:00:54.528991 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526396 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 11:00:54.528991 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526398 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 11:00:54.528991 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526400 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 11:00:54.528991 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526403 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 11:00:54.528991 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526406 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 11:00:54.528991 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526410 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 11:00:54.528991 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526413 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 11:00:54.528991 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526415 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 11:00:54.528991 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526418 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 11:00:54.528991 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526420 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 11:00:54.528991 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526423 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 11:00:54.528991 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526425 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 11:00:54.529494 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526428 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 11:00:54.529494 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526430 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 11:00:54.529494 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526433 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 11:00:54.529494 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526436 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 11:00:54.529494 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526438 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 11:00:54.529494 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526441 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 11:00:54.529494 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526443 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 11:00:54.529494 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526446 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 11:00:54.529494 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526449 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 11:00:54.529494 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526451 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 11:00:54.529494 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526454 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 11:00:54.529494 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526456 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 11:00:54.529494 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526459 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 11:00:54.529494 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526461 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 11:00:54.529494 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526463 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 11:00:54.529494 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526466 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 11:00:54.529494 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526469 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 11:00:54.529494 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526473 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 11:00:54.529494 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526477 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 11:00:54.530001 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526480 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 11:00:54.530001 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526483 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 11:00:54.530001 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526485 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 11:00:54.530001 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526488 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 11:00:54.530001 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526491 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 11:00:54.530001 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526494 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 11:00:54.530001 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526497 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 11:00:54.530001 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526500 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 11:00:54.530001 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526502 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 11:00:54.530001 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526505 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 11:00:54.530001 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526507 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 11:00:54.530001 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526510 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 11:00:54.530001 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526512 2574 feature_gate.go:328] unrecognized feature gate: Example Sep 14 11:00:54.530001 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526515 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 11:00:54.530001 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526517 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 11:00:54.530001 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526520 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 11:00:54.530001 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526522 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 11:00:54.530001 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526525 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 11:00:54.530001 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526527 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 11:00:54.530001 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526530 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 11:00:54.530523 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526532 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 11:00:54.530523 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526535 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 11:00:54.530523 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526538 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 11:00:54.530523 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526540 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 11:00:54.530523 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526543 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 11:00:54.530523 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526545 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 11:00:54.530523 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526548 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 11:00:54.530523 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526550 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 11:00:54.530523 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526552 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 11:00:54.530523 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526555 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 11:00:54.530523 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526557 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 11:00:54.530523 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526562 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 11:00:54.530523 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526565 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 11:00:54.530523 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526568 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 11:00:54.530523 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526571 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 11:00:54.530523 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526574 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 11:00:54.530523 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526576 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 11:00:54.530523 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526579 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 11:00:54.530523 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.526581 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 11:00:54.530523 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526652 2574 flags.go:64] FLAG: --address="0.0.0.0" Sep 14 11:00:54.531054 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526710 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 14 11:00:54.531054 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526718 2574 flags.go:64] FLAG: --anonymous-auth="true" Sep 14 11:00:54.531054 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526723 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 14 11:00:54.531054 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526731 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 14 11:00:54.531054 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526735 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 14 11:00:54.531054 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526739 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 14 11:00:54.531054 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526743 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 14 11:00:54.531054 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526746 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 14 11:00:54.531054 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526749 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 14 11:00:54.531054 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526752 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 14 11:00:54.531054 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526755 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 14 11:00:54.531054 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526758 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 14 11:00:54.531054 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526761 2574 flags.go:64] FLAG: --cgroup-root="" Sep 14 11:00:54.531054 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526763 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 14 11:00:54.531054 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526766 2574 flags.go:64] FLAG: --client-ca-file="" Sep 14 11:00:54.531054 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526769 2574 flags.go:64] FLAG: --cloud-provider="external" Sep 14 11:00:54.531054 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526772 2574 flags.go:64] FLAG: --cluster-dns="[]" Sep 14 11:00:54.531054 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526776 2574 flags.go:64] FLAG: --cluster-domain="" Sep 14 11:00:54.531054 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526778 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 14 11:00:54.531054 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526781 2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 14 11:00:54.531054 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526784 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 14 11:00:54.531054 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526788 2574 flags.go:64] FLAG: --container-log-max-files="5" Sep 14 11:00:54.531054 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526791 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 14 11:00:54.531054 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526794 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 14 11:00:54.531814 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526797 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 14 11:00:54.531814 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526800 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 14 11:00:54.531814 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526803 2574 flags.go:64] FLAG: --contention-profiling="false" Sep 14 11:00:54.531814 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526806 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 14 11:00:54.531814 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526809 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 14 11:00:54.531814 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526811 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 14 11:00:54.531814 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526814 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 14 11:00:54.531814 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526818 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 14 11:00:54.531814 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526821 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 14 11:00:54.531814 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526824 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 14 11:00:54.531814 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526827 2574 flags.go:64] FLAG: --enable-load-reader="false" Sep 14 11:00:54.531814 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526830 2574 flags.go:64] FLAG: --enable-server="true" Sep 14 11:00:54.531814 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526833 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 14 11:00:54.531814 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526837 2574 flags.go:64] FLAG: --event-burst="100" Sep 14 11:00:54.531814 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526840 2574 flags.go:64] FLAG: --event-qps="50" Sep 14 11:00:54.531814 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526843 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 14 11:00:54.531814 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526846 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 14 11:00:54.531814 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526848 2574 flags.go:64] FLAG: --eviction-hard="" Sep 14 11:00:54.531814 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526854 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 14 11:00:54.531814 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526857 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 14 11:00:54.531814 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526860 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 14 11:00:54.531814 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526863 2574 flags.go:64] FLAG: --eviction-soft="" Sep 14 11:00:54.531814 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526865 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 14 11:00:54.531814 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526868 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 14 11:00:54.531814 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526871 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 14 11:00:54.532461 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526874 2574 flags.go:64] FLAG: --experimental-mounter-path="" Sep 14 11:00:54.532461 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526877 2574 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 14 11:00:54.532461 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526880 2574 flags.go:64] FLAG: --fail-swap-on="true" Sep 14 11:00:54.532461 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526883 2574 flags.go:64] FLAG: --feature-gates="" Sep 14 11:00:54.532461 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526887 2574 flags.go:64] FLAG: --file-check-frequency="20s" Sep 14 11:00:54.532461 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526890 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 14 11:00:54.532461 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526893 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 14 11:00:54.532461 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526896 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 14 11:00:54.532461 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526899 2574 flags.go:64] FLAG: --healthz-port="10248" Sep 14 11:00:54.532461 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526904 2574 flags.go:64] FLAG: --help="false" Sep 14 11:00:54.532461 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526908 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-137-230.ec2.internal" Sep 14 11:00:54.532461 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526911 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 14 11:00:54.532461 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526914 2574 flags.go:64] FLAG: --http-check-frequency="20s" Sep 14 11:00:54.532461 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526917 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 14 11:00:54.532461 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526920 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 14 11:00:54.532461 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526923 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 14 11:00:54.532461 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526926 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 14 11:00:54.532461 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526929 2574 flags.go:64] FLAG: --image-service-endpoint="" Sep 14 11:00:54.532461 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526931 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 14 11:00:54.532461 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526934 2574 flags.go:64] FLAG: --kube-api-burst="100" Sep 14 11:00:54.532461 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526937 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 14 11:00:54.532461 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526940 2574 flags.go:64] FLAG: --kube-api-qps="50" Sep 14 11:00:54.532461 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526943 2574 flags.go:64] FLAG: --kube-reserved="" Sep 14 11:00:54.532461 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526945 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 14 11:00:54.533105 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526948 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 14 11:00:54.533105 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526951 2574 flags.go:64] FLAG: --kubelet-cgroups="" Sep 14 11:00:54.533105 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526954 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 14 11:00:54.533105 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526956 2574 flags.go:64] FLAG: --lock-file="" Sep 14 11:00:54.533105 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526959 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 14 11:00:54.533105 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526962 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 14 11:00:54.533105 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526965 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 14 11:00:54.533105 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526970 2574 flags.go:64] FLAG: --log-json-split-stream="false" Sep 14 11:00:54.533105 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526974 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 14 11:00:54.533105 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526977 2574 flags.go:64] FLAG: --log-text-split-stream="false" Sep 14 11:00:54.533105 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526980 2574 flags.go:64] FLAG: --logging-format="text" Sep 14 11:00:54.533105 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526983 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 14 11:00:54.533105 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526986 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 14 11:00:54.533105 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526989 2574 flags.go:64] FLAG: --manifest-url="" Sep 14 11:00:54.533105 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526992 2574 flags.go:64] FLAG: --manifest-url-header="" Sep 14 11:00:54.533105 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526996 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 14 11:00:54.533105 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.526999 2574 flags.go:64] FLAG: --max-open-files="1000000" Sep 14 11:00:54.533105 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527004 2574 flags.go:64] FLAG: --max-pods="110" Sep 14 11:00:54.533105 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527007 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 14 11:00:54.533105 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527010 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 14 11:00:54.533105 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527013 2574 flags.go:64] FLAG: --memory-manager-policy="None" Sep 14 11:00:54.533105 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527016 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 14 11:00:54.533105 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527019 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 14 11:00:54.533105 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527022 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 14 11:00:54.533105 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527024 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 14 11:00:54.533763 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527032 2574 flags.go:64] FLAG: --node-status-max-images="50" Sep 14 11:00:54.533763 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527035 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 14 11:00:54.533763 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527038 2574 flags.go:64] FLAG: --oom-score-adj="-999" Sep 14 11:00:54.533763 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527040 2574 flags.go:64] FLAG: --pod-cidr="" Sep 14 11:00:54.533763 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527044 2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 14 11:00:54.533763 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527049 2574 flags.go:64] FLAG: --pod-manifest-path="" Sep 14 11:00:54.533763 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527052 2574 flags.go:64] FLAG: --pod-max-pids="-1" Sep 14 11:00:54.533763 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527055 2574 flags.go:64] FLAG: --pods-per-core="0" Sep 14 11:00:54.533763 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527057 2574 flags.go:64] FLAG: --port="10250" Sep 14 11:00:54.533763 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527060 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 14 11:00:54.533763 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527063 2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c1cf0c8a4e0c51bc" Sep 14 11:00:54.533763 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527066 2574 flags.go:64] FLAG: --qos-reserved="" Sep 14 11:00:54.533763 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527069 2574 flags.go:64] FLAG: --read-only-port="10255" Sep 14 11:00:54.533763 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527072 2574 flags.go:64] FLAG: --register-node="true" Sep 14 11:00:54.533763 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527074 2574 flags.go:64] FLAG: --register-with-taints="" Sep 14 11:00:54.533763 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527078 2574 flags.go:64] FLAG: --registry-burst="10" Sep 14 11:00:54.533763 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527081 2574 flags.go:64] FLAG: --registry-qps="5" Sep 14 11:00:54.533763 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527084 2574 flags.go:64] FLAG: --reserved-cpus="" Sep 14 11:00:54.533763 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527088 2574 flags.go:64] FLAG: --reserved-memory="" Sep 14 11:00:54.533763 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527092 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 14 11:00:54.533763 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527094 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 14 11:00:54.533763 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527097 2574 flags.go:64] FLAG: --rotate-certificates="false" Sep 14 11:00:54.533763 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527100 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 14 11:00:54.533763 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527103 2574 flags.go:64] FLAG: --runonce="false" Sep 14 11:00:54.533763 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527106 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 14 11:00:54.534441 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527110 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 14 11:00:54.534441 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527113 2574 flags.go:64] FLAG: --seccomp-default="false" Sep 14 11:00:54.534441 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527115 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 14 11:00:54.534441 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527118 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 14 11:00:54.534441 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527121 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 14 11:00:54.534441 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527124 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 14 11:00:54.534441 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527127 2574 flags.go:64] FLAG: --storage-driver-password="root" Sep 14 11:00:54.534441 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527130 2574 flags.go:64] FLAG: --storage-driver-secure="false" Sep 14 11:00:54.534441 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527133 2574 flags.go:64] FLAG: --storage-driver-table="stats" Sep 14 11:00:54.534441 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527136 2574 flags.go:64] FLAG: --storage-driver-user="root" Sep 14 11:00:54.534441 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527138 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 14 11:00:54.534441 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527141 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 14 11:00:54.534441 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527144 2574 flags.go:64] FLAG: --system-cgroups="" Sep 14 11:00:54.534441 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527147 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 14 11:00:54.534441 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527152 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 14 11:00:54.534441 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527154 2574 flags.go:64] FLAG: --tls-cert-file="" Sep 14 11:00:54.534441 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527157 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 14 11:00:54.534441 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527161 2574 flags.go:64] FLAG: --tls-min-version="" Sep 14 11:00:54.534441 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527163 2574 flags.go:64] FLAG: --tls-private-key-file="" Sep 14 11:00:54.534441 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527166 2574 flags.go:64] FLAG: --topology-manager-policy="none" Sep 14 11:00:54.534441 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527169 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 14 11:00:54.534441 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527172 2574 flags.go:64] FLAG: --topology-manager-scope="container" Sep 14 11:00:54.534441 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527174 2574 flags.go:64] FLAG: --v="2" Sep 14 11:00:54.534441 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527178 2574 flags.go:64] FLAG: --version="false" Sep 14 11:00:54.534441 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527182 2574 flags.go:64] FLAG: --vmodule="" Sep 14 11:00:54.535166 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527187 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 14 11:00:54.535166 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.527190 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 14 11:00:54.535166 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527320 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 11:00:54.535166 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527324 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 11:00:54.535166 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527328 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 11:00:54.535166 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527331 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 11:00:54.535166 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527333 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 11:00:54.535166 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527337 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 11:00:54.535166 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527340 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 11:00:54.535166 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527342 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 11:00:54.535166 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527346 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 11:00:54.535166 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527349 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 11:00:54.535166 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527352 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 11:00:54.535166 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527355 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 11:00:54.535166 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527357 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 11:00:54.535166 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527360 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 11:00:54.535166 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527362 2574 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 11:00:54.535166 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527365 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 11:00:54.535166 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527367 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 11:00:54.535166 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527370 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 11:00:54.535734 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527372 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 11:00:54.535734 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527375 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 11:00:54.535734 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527377 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 11:00:54.535734 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527379 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 11:00:54.535734 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527382 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 11:00:54.535734 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527384 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 11:00:54.535734 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527387 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 11:00:54.535734 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527389 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 11:00:54.535734 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527392 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 11:00:54.535734 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527394 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 11:00:54.535734 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527397 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 11:00:54.535734 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527399 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 11:00:54.535734 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527404 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 11:00:54.535734 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527406 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 11:00:54.535734 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527408 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 11:00:54.535734 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527411 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 11:00:54.535734 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527413 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 11:00:54.535734 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527416 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 11:00:54.535734 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527419 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 11:00:54.535734 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527422 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 11:00:54.536249 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527425 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 11:00:54.536249 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527428 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 11:00:54.536249 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527431 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 11:00:54.536249 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527433 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 11:00:54.536249 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527438 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 11:00:54.536249 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527441 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 11:00:54.536249 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527444 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 11:00:54.536249 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527447 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 11:00:54.536249 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527449 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 11:00:54.536249 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527452 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 11:00:54.536249 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527455 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 11:00:54.536249 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527457 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 11:00:54.536249 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527460 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 11:00:54.536249 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527462 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 11:00:54.536249 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527465 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 11:00:54.536249 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527467 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 11:00:54.536249 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527470 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 11:00:54.536249 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527472 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 11:00:54.536249 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527475 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 11:00:54.536249 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527477 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 11:00:54.536798 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527480 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 11:00:54.536798 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527482 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 11:00:54.536798 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527485 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 11:00:54.536798 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527488 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 11:00:54.536798 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527492 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 11:00:54.536798 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527495 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 11:00:54.536798 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527498 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 11:00:54.536798 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527500 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 11:00:54.536798 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527503 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 11:00:54.536798 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527505 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 11:00:54.536798 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527508 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 11:00:54.536798 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527511 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 11:00:54.536798 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527514 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 11:00:54.536798 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527517 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 11:00:54.536798 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527519 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 11:00:54.536798 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527521 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 11:00:54.536798 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527524 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 11:00:54.536798 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527527 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 11:00:54.536798 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527529 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 11:00:54.537290 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527531 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 11:00:54.537290 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527534 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 11:00:54.537290 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527537 2574 feature_gate.go:328] unrecognized feature gate: Example Sep 14 11:00:54.537290 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527539 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 11:00:54.537290 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527541 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 11:00:54.537290 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527544 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 11:00:54.537290 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527548 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 11:00:54.537290 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527551 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 11:00:54.537290 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527554 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 11:00:54.537290 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527557 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 11:00:54.537290 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527560 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 11:00:54.537290 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527562 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 11:00:54.537290 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527565 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 11:00:54.537290 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527568 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 11:00:54.537290 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527570 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 11:00:54.537290 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527573 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 11:00:54.537290 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527575 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 11:00:54.537290 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527579 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 11:00:54.537290 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527582 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 11:00:54.537290 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527584 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 11:00:54.537833 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527587 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 11:00:54.537833 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.527589 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 11:00:54.537833 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.528324 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 14 11:00:54.537833 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.535855 2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 14 11:00:54.537833 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.535871 2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 14 11:00:54.537833 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.536880 2574 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 14 11:00:54.537833 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.536887 2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 14 11:00:54.537833 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.536941 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 11:00:54.537833 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.536946 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 11:00:54.537833 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.536951 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 11:00:54.537833 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.536955 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 11:00:54.537833 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.536958 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 11:00:54.537833 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.536961 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 11:00:54.537833 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.536964 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 11:00:54.538200 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.536967 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 11:00:54.538200 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.536969 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 11:00:54.538200 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.536972 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 11:00:54.538200 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.536975 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 11:00:54.538200 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.536977 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 11:00:54.538200 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.536980 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 11:00:54.538200 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.536983 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 11:00:54.538200 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.536985 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 11:00:54.538200 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.536988 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 11:00:54.538200 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.536990 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 11:00:54.538200 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.536993 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 11:00:54.538200 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.536995 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 11:00:54.538200 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.536998 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 11:00:54.538200 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537000 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 11:00:54.538200 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537003 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 11:00:54.538200 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537005 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 11:00:54.538200 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537008 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 11:00:54.538200 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537010 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 11:00:54.538200 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537013 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 11:00:54.538200 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537016 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 11:00:54.538757 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537019 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 11:00:54.538757 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537021 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 11:00:54.538757 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537025 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 11:00:54.538757 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537028 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 11:00:54.538757 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537031 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 11:00:54.538757 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537033 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 11:00:54.538757 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537036 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 11:00:54.538757 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537038 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 11:00:54.538757 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537041 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 11:00:54.538757 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537043 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 11:00:54.538757 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537046 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 11:00:54.538757 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537048 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 11:00:54.538757 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537051 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 11:00:54.538757 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537053 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 11:00:54.538757 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537056 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 11:00:54.538757 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537058 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 11:00:54.538757 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537061 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 11:00:54.538757 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537063 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 11:00:54.538757 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537066 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 11:00:54.538757 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537068 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 11:00:54.539307 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537071 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 11:00:54.539307 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537073 2574 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 11:00:54.539307 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537076 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 11:00:54.539307 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537078 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 11:00:54.539307 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537081 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 11:00:54.539307 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537084 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 11:00:54.539307 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537086 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 11:00:54.539307 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537089 2574 feature_gate.go:328] unrecognized feature gate: Example Sep 14 11:00:54.539307 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537091 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 11:00:54.539307 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537094 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 11:00:54.539307 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537097 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 11:00:54.539307 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537099 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 11:00:54.539307 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537102 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 11:00:54.539307 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537104 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 11:00:54.539307 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537109 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 11:00:54.539307 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537113 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 11:00:54.539307 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537116 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 11:00:54.539307 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537118 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 11:00:54.539307 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537121 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 11:00:54.539307 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537124 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 11:00:54.539838 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537127 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 11:00:54.539838 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537129 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 11:00:54.539838 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537133 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 11:00:54.539838 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537137 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 11:00:54.539838 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537140 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 11:00:54.539838 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537143 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 11:00:54.539838 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537146 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 11:00:54.539838 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537149 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 11:00:54.539838 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537151 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 11:00:54.539838 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537154 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 11:00:54.539838 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537157 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 11:00:54.539838 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537159 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 11:00:54.539838 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537162 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 11:00:54.539838 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537165 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 11:00:54.539838 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537167 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 11:00:54.539838 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537170 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 11:00:54.539838 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537173 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 11:00:54.539838 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537175 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 11:00:54.539838 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537177 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 11:00:54.540367 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537180 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 11:00:54.540367 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537182 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 11:00:54.540367 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537185 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 11:00:54.540367 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537188 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 11:00:54.540367 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537191 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 11:00:54.540367 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537193 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 11:00:54.540367 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537196 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 11:00:54.540367 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537198 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 11:00:54.540367 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537201 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 11:00:54.540367 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537204 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 11:00:54.540367 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537206 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 11:00:54.540367 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537209 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 11:00:54.540367 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537211 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 11:00:54.540367 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.537216 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 14 11:00:54.540367 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537322 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 11:00:54.540779 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537327 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 11:00:54.540779 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537330 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 11:00:54.540779 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537333 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 11:00:54.540779 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537336 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 11:00:54.540779 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537338 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 11:00:54.540779 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537341 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 11:00:54.540779 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537344 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 11:00:54.540779 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537346 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 11:00:54.540779 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537349 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 11:00:54.540779 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537352 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 11:00:54.540779 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537354 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 11:00:54.540779 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537357 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 11:00:54.540779 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537359 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 11:00:54.540779 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537362 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 11:00:54.540779 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537364 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 11:00:54.540779 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537367 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 11:00:54.540779 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537369 2574 feature_gate.go:328] unrecognized feature gate: Example Sep 14 11:00:54.540779 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537373 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 11:00:54.540779 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537377 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 11:00:54.541283 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537380 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 11:00:54.541283 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537383 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 11:00:54.541283 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537386 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 11:00:54.541283 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537389 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 11:00:54.541283 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537391 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 11:00:54.541283 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537394 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 11:00:54.541283 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537397 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 11:00:54.541283 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537399 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 11:00:54.541283 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537402 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 11:00:54.541283 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537405 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 11:00:54.541283 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537407 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 11:00:54.541283 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537410 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 11:00:54.541283 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537412 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 11:00:54.541283 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537415 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 11:00:54.541283 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537417 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 11:00:54.541283 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537420 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 11:00:54.541283 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537422 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 11:00:54.541283 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537424 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 11:00:54.541283 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537427 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 11:00:54.541283 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537429 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 11:00:54.541925 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537432 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 11:00:54.541925 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537434 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 11:00:54.541925 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537436 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 11:00:54.541925 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537439 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 11:00:54.541925 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537441 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 11:00:54.541925 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537444 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 11:00:54.541925 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537446 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 11:00:54.541925 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537449 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 11:00:54.541925 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537452 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 11:00:54.541925 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537454 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 11:00:54.541925 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537457 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 11:00:54.541925 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537459 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 11:00:54.541925 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537461 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 11:00:54.541925 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537464 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 11:00:54.541925 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537467 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 11:00:54.541925 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537470 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 11:00:54.541925 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537474 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 11:00:54.541925 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537477 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 11:00:54.541925 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537480 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 11:00:54.542421 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537482 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 11:00:54.542421 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537485 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 11:00:54.542421 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537488 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 11:00:54.542421 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537491 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 11:00:54.542421 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537493 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 11:00:54.542421 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537495 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 11:00:54.542421 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537498 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 11:00:54.542421 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537500 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 11:00:54.542421 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537503 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 11:00:54.542421 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537505 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 11:00:54.542421 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537508 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 11:00:54.542421 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537510 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 11:00:54.542421 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537512 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 11:00:54.542421 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537515 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 11:00:54.542421 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537517 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 11:00:54.542421 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537520 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 11:00:54.542421 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537522 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 11:00:54.542421 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537525 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 11:00:54.542421 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537527 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 11:00:54.542421 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537529 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 11:00:54.542944 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537532 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 11:00:54.542944 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537534 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 11:00:54.542944 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537537 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 11:00:54.542944 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537540 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 11:00:54.542944 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537542 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 11:00:54.542944 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537544 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 11:00:54.542944 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537547 2574 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 11:00:54.542944 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537550 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 11:00:54.542944 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537552 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 11:00:54.542944 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537555 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 11:00:54.542944 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537558 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 11:00:54.542944 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537560 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 11:00:54.542944 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537563 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 11:00:54.542944 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537566 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 11:00:54.542944 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537568 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 11:00:54.542944 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537571 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 11:00:54.542944 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537574 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 11:00:54.542944 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537577 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 11:00:54.542944 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537581 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 11:00:54.542944 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:54.537584 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 11:00:54.543467 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.537589 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 14 11:00:54.543467 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.537720 2574 server.go:962] "Client rotation is on, will bootstrap in background" Sep 14 11:00:54.543467 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.540116 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 14 11:00:54.543467 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.541176 2574 server.go:1019] "Starting client certificate rotation" Sep 14 11:00:54.543467 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.541271 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 14 11:00:54.543467 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.541321 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 14 11:00:54.544149 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.544136 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 14 11:00:54.549819 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.549796 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 14 11:00:54.564465 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.564449 2574 log.go:25] "Validated CRI v1 runtime API" Sep 14 11:00:54.567525 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.567509 2574 log.go:25] "Validated CRI v1 image API" Sep 14 11:00:54.568403 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.568385 2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 14 11:00:54.571099 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.571077 2574 fs.go:135] Filesystem UUIDs: map[08951213-0158-45eb-ae48-b5c3f8861368:/dev/nvme0n1p3 1392c5a9-4a46-4294-bcc5-b643527df40f:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Sep 14 11:00:54.571157 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.571101 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 14 11:00:54.577220 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.577114 2574 manager.go:216] Machine: {Timestamp:2026-09-14 11:00:54.575740674 +0000 UTC m=+0.374551297 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3106677 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27c0a64f2da80381309a6449874ec8 SystemUUID:ec27c0a6-4f2d-a803-8130-9a6449874ec8 BootID:2f73fd4a-f191-4192-b6c7-0e7db306c3b7 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:fa:9b:7d:11 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:fa:9b:7d:11 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:76:a8:b2:47:d1:de Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 14 11:00:54.577220 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.577215 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 14 11:00:54.577334 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.577292 2574 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 14 11:00:54.580926 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.580899 2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 14 11:00:54.581055 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.580929 2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-230.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 14 11:00:54.581439 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.581423 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 14 11:00:54.581764 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.581753 2574 topology_manager.go:138] "Creating topology manager with none policy" Sep 14 11:00:54.581808 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.581766 2574 container_manager_linux.go:309] "Creating device plugin manager" Sep 14 11:00:54.581808 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.581773 2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 14 11:00:54.582597 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.582586 2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 14 11:00:54.583826 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.583815 2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 14 11:00:54.588395 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.588383 2574 state_mem.go:36] "Initialized new in-memory state store" Sep 14 11:00:54.588496 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.588487 2574 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 14 11:00:54.588533 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.588515 2574 kubelet.go:486] "Attempting to sync node with API server" Sep 14 11:00:54.588563 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.588542 2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 14 11:00:54.588563 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.588555 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 14 11:00:54.588616 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.588567 2574 kubelet.go:398] "Adding apiserver pod source" Sep 14 11:00:54.588616 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.588578 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 14 11:00:54.588924 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.588913 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 14 11:00:54.588970 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.588931 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 14 11:00:54.591325 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.591306 2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 14 11:00:54.594075 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.594059 2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 14 11:00:54.594146 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.594091 2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 14 11:00:54.596244 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.596233 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 14 11:00:54.596288 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.596249 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 14 11:00:54.596288 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.596257 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 14 11:00:54.596288 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.596263 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 14 11:00:54.596288 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.596274 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 14 11:00:54.596288 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.596280 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 14 11:00:54.596478 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.596295 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 14 11:00:54.596478 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.596301 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 14 11:00:54.596478 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.596309 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 14 11:00:54.596478 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.596315 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 14 11:00:54.596478 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.596325 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 14 11:00:54.596478 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.596331 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 14 11:00:54.597253 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.597240 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 14 11:00:54.597253 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.597251 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 14 11:00:54.599482 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:54.599456 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-230.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 14 11:00:54.599552 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:54.599511 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 14 11:00:54.601171 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.601159 2574 server.go:1268] "Started kubelet" Sep 14 11:00:54.601248 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.601228 2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 14 11:00:54.602018 ip-10-0-137-230 systemd[1]: Started Kubernetes Kubelet. Sep 14 11:00:54.602345 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.601795 2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 14 11:00:54.602470 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.602448 2574 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 14 11:00:54.607631 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.607581 2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 14 11:00:54.608179 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.608073 2574 server.go:310] "Adding debug handlers to kubelet server" Sep 14 11:00:54.608617 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:54.608595 2574 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 14 11:00:54.608876 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.608858 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-khgvw" Sep 14 11:00:54.613747 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.613720 2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-230.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 14 11:00:54.614591 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:54.613410 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-230.ec2.internal.18d52af40b6403a9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-230.ec2.internal,UID:ip-10-0-137-230.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-230.ec2.internal,},FirstTimestamp:2026-09-14 11:00:54.601130921 +0000 UTC m=+0.399941529,LastTimestamp:2026-09-14 11:00:54.601130921 +0000 UTC m=+0.399941529,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-230.ec2.internal,}" Sep 14 11:00:54.614804 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.614785 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 14 11:00:54.615488 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.615474 2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 14 11:00:54.615581 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:54.615569 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-230.ec2.internal\" not found" Sep 14 11:00:54.616136 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.616120 2574 factory.go:55] Registering systemd factory Sep 14 11:00:54.616136 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.616135 2574 factory.go:223] Registration of the systemd container factory successfully Sep 14 11:00:54.616440 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.616187 2574 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 14 11:00:54.616440 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.616201 2574 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 14 11:00:54.616440 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.616261 2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 14 11:00:54.616440 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.616330 2574 reconstruct.go:97] "Volume reconstruction finished" Sep 14 11:00:54.616440 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.616337 2574 reconciler.go:29] "Reconciler: start to sync state" Sep 14 11:00:54.616440 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.616396 2574 factory.go:153] Registering CRI-O factory Sep 14 11:00:54.616440 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.616407 2574 factory.go:223] Registration of the crio container factory successfully Sep 14 11:00:54.616795 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.616468 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 14 11:00:54.616795 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.616483 2574 factory.go:103] Registering Raw factory Sep 14 11:00:54.616795 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.616494 2574 manager.go:1196] Started watching for new ooms in manager Sep 14 11:00:54.616795 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.616789 2574 manager.go:349] Starting recovery of all containers Sep 14 11:00:54.617080 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.617060 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-khgvw" Sep 14 11:00:54.619446 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:54.619362 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 14 11:00:54.619933 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:54.619908 2574 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-230.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 14 11:00:54.625091 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.625074 2574 manager.go:354] Recovery completed Sep 14 11:00:54.628980 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.628969 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 11:00:54.631098 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.631086 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-230.ec2.internal" event="NodeHasSufficientMemory" Sep 14 11:00:54.631153 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.631113 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-230.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 11:00:54.631153 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.631128 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-230.ec2.internal" event="NodeHasSufficientPID" Sep 14 11:00:54.631723 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.631706 2574 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 14 11:00:54.631723 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.631723 2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 14 11:00:54.631840 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.631742 2574 state_mem.go:36] "Initialized new in-memory state store" Sep 14 11:00:54.631840 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.631724 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 11:00:54.632494 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.632478 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-230.ec2.internal" event="NodeHasSufficientMemory" Sep 14 11:00:54.632523 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.632511 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-230.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 11:00:54.632554 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.632525 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-230.ec2.internal" event="NodeHasSufficientPID" Sep 14 11:00:54.633850 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.633834 2574 policy_none.go:49] "None policy: Start" Sep 14 11:00:54.633850 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.633850 2574 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 14 11:00:54.633975 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.633859 2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 14 11:00:54.635853 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.635840 2574 policy_none.go:47] "Start" Sep 14 11:00:54.671280 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.671266 2574 manager.go:337] "Starting Device Plugin manager" Sep 14 11:00:54.693238 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:54.671309 2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 14 11:00:54.693238 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.671320 2574 server.go:87] "Starting device plugin registration server" Sep 14 11:00:54.693238 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.671581 2574 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 14 11:00:54.693238 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.671590 2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 14 11:00:54.693238 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.671692 2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 14 11:00:54.693238 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.671766 2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 14 11:00:54.693238 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.671773 2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 14 11:00:54.693238 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:54.672328 2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 14 11:00:54.693238 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:54.672369 2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-230.ec2.internal\" not found" Sep 14 11:00:54.712140 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.712115 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 14 11:00:54.713431 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.713410 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 14 11:00:54.713505 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.713464 2574 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 14 11:00:54.713505 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.713489 2574 kubelet.go:2448] "Starting kubelet main sync loop" Sep 14 11:00:54.713597 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:54.713532 2574 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 14 11:00:54.715865 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.715845 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 11:00:54.771842 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.771782 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 11:00:54.772538 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.772524 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-230.ec2.internal" event="NodeHasSufficientMemory" Sep 14 11:00:54.772599 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.772551 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-230.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 11:00:54.772599 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.772562 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-230.ec2.internal" event="NodeHasSufficientPID" Sep 14 11:00:54.772599 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.772580 2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-230.ec2.internal" Sep 14 11:00:54.779511 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.779497 2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-230.ec2.internal" Sep 14 11:00:54.779594 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:54.779520 2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-230.ec2.internal\": node \"ip-10-0-137-230.ec2.internal\" not found" Sep 14 11:00:54.809094 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:54.809070 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-230.ec2.internal\" not found" Sep 14 11:00:54.814706 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.814683 2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-230.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-230.ec2.internal"] Sep 14 11:00:54.814753 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.814744 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 11:00:54.815519 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.815505 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-230.ec2.internal" event="NodeHasSufficientMemory" Sep 14 11:00:54.815578 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.815536 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-230.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 11:00:54.815578 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.815552 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-230.ec2.internal" event="NodeHasSufficientPID" Sep 14 11:00:54.817957 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.817945 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 11:00:54.818083 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.818069 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-230.ec2.internal" Sep 14 11:00:54.818140 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.818104 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 11:00:54.818615 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.818600 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-230.ec2.internal" event="NodeHasSufficientMemory" Sep 14 11:00:54.818692 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.818622 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-230.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 11:00:54.818692 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.818624 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-230.ec2.internal" event="NodeHasSufficientMemory" Sep 14 11:00:54.818692 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.818635 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-230.ec2.internal" event="NodeHasSufficientPID" Sep 14 11:00:54.818692 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.818646 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-230.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 11:00:54.818692 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.818657 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-230.ec2.internal" event="NodeHasSufficientPID" Sep 14 11:00:54.820858 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.820842 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-230.ec2.internal" Sep 14 11:00:54.820913 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.820868 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 11:00:54.822153 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.822133 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-230.ec2.internal" event="NodeHasSufficientMemory" Sep 14 11:00:54.822232 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.822160 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-230.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 11:00:54.822232 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.822171 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-230.ec2.internal" event="NodeHasSufficientPID" Sep 14 11:00:54.843232 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:54.843211 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-230.ec2.internal\" not found" node="ip-10-0-137-230.ec2.internal" Sep 14 11:00:54.847543 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:54.847528 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-230.ec2.internal\" not found" node="ip-10-0-137-230.ec2.internal" Sep 14 11:00:54.909350 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:54.909330 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-230.ec2.internal\" not found" Sep 14 11:00:54.917617 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.917600 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6be7bad72619c3572acbefbc29cd3390-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-230.ec2.internal\" (UID: \"6be7bad72619c3572acbefbc29cd3390\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-230.ec2.internal" Sep 14 11:00:54.917687 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.917622 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6be7bad72619c3572acbefbc29cd3390-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-230.ec2.internal\" (UID: \"6be7bad72619c3572acbefbc29cd3390\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-230.ec2.internal" Sep 14 11:00:54.917687 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:54.917639 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc8ab4f3c567cd8f16d2622c33e7fe29-config\") pod \"kube-apiserver-proxy-ip-10-0-137-230.ec2.internal\" (UID: \"fc8ab4f3c567cd8f16d2622c33e7fe29\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-230.ec2.internal" Sep 14 11:00:55.009401 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:55.009381 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-230.ec2.internal\" not found" Sep 14 11:00:55.018087 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.018069 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6be7bad72619c3572acbefbc29cd3390-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-230.ec2.internal\" (UID: \"6be7bad72619c3572acbefbc29cd3390\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-230.ec2.internal" Sep 14 11:00:55.018160 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.018092 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6be7bad72619c3572acbefbc29cd3390-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-230.ec2.internal\" (UID: \"6be7bad72619c3572acbefbc29cd3390\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-230.ec2.internal" Sep 14 11:00:55.018197 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.018156 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6be7bad72619c3572acbefbc29cd3390-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-230.ec2.internal\" (UID: \"6be7bad72619c3572acbefbc29cd3390\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-230.ec2.internal" Sep 14 11:00:55.018197 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.018163 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6be7bad72619c3572acbefbc29cd3390-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-230.ec2.internal\" (UID: \"6be7bad72619c3572acbefbc29cd3390\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-230.ec2.internal" Sep 14 11:00:55.018197 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.018186 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc8ab4f3c567cd8f16d2622c33e7fe29-config\") pod \"kube-apiserver-proxy-ip-10-0-137-230.ec2.internal\" (UID: \"fc8ab4f3c567cd8f16d2622c33e7fe29\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-230.ec2.internal" Sep 14 11:00:55.018306 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.018220 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc8ab4f3c567cd8f16d2622c33e7fe29-config\") pod \"kube-apiserver-proxy-ip-10-0-137-230.ec2.internal\" (UID: \"fc8ab4f3c567cd8f16d2622c33e7fe29\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-230.ec2.internal" Sep 14 11:00:55.110432 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:55.110382 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-230.ec2.internal\" not found" Sep 14 11:00:55.145844 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.145830 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-230.ec2.internal" Sep 14 11:00:55.150638 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.150614 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-230.ec2.internal" Sep 14 11:00:55.211300 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:55.211272 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-230.ec2.internal\" not found" Sep 14 11:00:55.311760 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:55.311734 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-230.ec2.internal\" not found" Sep 14 11:00:55.412319 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:55.412268 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-230.ec2.internal\" not found" Sep 14 11:00:55.512793 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:55.512763 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-230.ec2.internal\" not found" Sep 14 11:00:55.541291 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.541273 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 14 11:00:55.541661 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.541412 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 14 11:00:55.579939 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.579918 2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 11:00:55.585206 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.585192 2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 11:00:55.589371 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.589358 2574 apiserver.go:52] "Watching apiserver" Sep 14 11:00:55.597129 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.597112 2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 14 11:00:55.598016 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.597997 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-89vlx","openshift-image-registry/node-ca-sfnlb","openshift-multus/multus-additional-cni-plugins-xpv2v","openshift-multus/network-metrics-daemon-jkg6q","openshift-network-operator/iptables-alerter-b5r9g","openshift-ovn-kubernetes/ovnkube-node-xpwhs","kube-system/konnectivity-agent-jfdk9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mwsc","openshift-dns/node-resolver-lcx2b","openshift-multus/multus-hkdj4","openshift-network-diagnostics/network-check-target-t6jcs","kube-system/global-pull-secret-syncer-2sk6p"] Sep 14 11:00:55.600600 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.600585 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xpv2v" Sep 14 11:00:55.602755 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.602739 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lcx2b" Sep 14 11:00:55.605939 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.605917 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-89vlx" Sep 14 11:00:55.608092 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.608077 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sfnlb" Sep 14 11:00:55.609545 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.609526 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 14 11:00:55.609934 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.609912 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 14 11:00:55.610060 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.609941 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 14 11:00:55.610060 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.609968 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 14 11:00:55.610060 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.609997 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 14 11:00:55.610060 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.610034 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 14 11:00:55.610060 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.610050 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8hb8d\"" Sep 14 11:00:55.610318 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.610174 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 14 11:00:55.610318 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.610236 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 14 11:00:55.610408 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.610333 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jkg6q" Sep 14 11:00:55.610408 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:55.610390 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jkg6q" podUID="dfd621b6-51b0-4f2e-b1aa-ea198b732626" Sep 14 11:00:55.610503 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.610426 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5c74t\"" Sep 14 11:00:55.610503 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.610432 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zmppr\"" Sep 14 11:00:55.610503 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.610429 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dl6mn\"" Sep 14 11:00:55.611073 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.611060 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 14 11:00:55.611293 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.611280 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 14 11:00:55.612586 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.612568 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-b5r9g" Sep 14 11:00:55.613019 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.613004 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 14 11:00:55.613556 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.613540 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 14 11:00:55.614928 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.614895 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 14 11:00:55.615026 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.614940 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.616288 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.616268 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-230.ec2.internal" Sep 14 11:00:55.617196 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.617179 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jfdk9" Sep 14 11:00:55.619434 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.619418 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mwsc" Sep 14 11:00:55.621296 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.621255 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-13 10:55:55 +0000 UTC" deadline="2028-02-25 18:29:50.728915857 +0000 UTC" Sep 14 11:00:55.621296 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.621292 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12703h28m55.107625633s" Sep 14 11:00:55.621436 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.621316 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c961b4ad-d3a5-4032-aa9a-a96c167e3df8-hosts-file\") pod \"node-resolver-lcx2b\" (UID: \"c961b4ad-d3a5-4032-aa9a-a96c167e3df8\") " pod="openshift-dns/node-resolver-lcx2b" Sep 14 11:00:55.621436 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.621341 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/807c9dd2-6b4e-4994-b996-6b86893e87c2-etc-kubernetes\") pod \"tuned-89vlx\" (UID: \"807c9dd2-6b4e-4994-b996-6b86893e87c2\") " pod="openshift-cluster-node-tuning-operator/tuned-89vlx" Sep 14 11:00:55.621436 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.621357 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/807c9dd2-6b4e-4994-b996-6b86893e87c2-etc-sysctl-conf\") pod \"tuned-89vlx\" (UID: \"807c9dd2-6b4e-4994-b996-6b86893e87c2\") " pod="openshift-cluster-node-tuning-operator/tuned-89vlx" Sep 14 11:00:55.621436 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.621371 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-var-lib-openvswitch\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.621436 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.621386 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-node-log\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.621436 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.621400 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/807c9dd2-6b4e-4994-b996-6b86893e87c2-sys\") pod \"tuned-89vlx\" (UID: \"807c9dd2-6b4e-4994-b996-6b86893e87c2\") " pod="openshift-cluster-node-tuning-operator/tuned-89vlx" Sep 14 11:00:55.621436 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.621432 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5ce405d8-5bb3-453d-8c40-cf94ed8ac8a3-iptables-alerter-script\") pod \"iptables-alerter-b5r9g\" (UID: \"5ce405d8-5bb3-453d-8c40-cf94ed8ac8a3\") " pod="openshift-network-operator/iptables-alerter-b5r9g" Sep 14 11:00:55.621815 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.621470 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.621815 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.621523 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7px44\" (UniqueName: \"kubernetes.io/projected/7ffda2ea-6f19-4043-a160-d116acf60812-kube-api-access-7px44\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.621815 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.621549 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5ce405d8-5bb3-453d-8c40-cf94ed8ac8a3-host-slash\") pod \"iptables-alerter-b5r9g\" (UID: \"5ce405d8-5bb3-453d-8c40-cf94ed8ac8a3\") " pod="openshift-network-operator/iptables-alerter-b5r9g" Sep 14 11:00:55.621815 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.621565 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xpv2v\" (UID: \"3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f\") " pod="openshift-multus/multus-additional-cni-plugins-xpv2v" Sep 14 11:00:55.621815 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.621583 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4px7\" (UniqueName: \"kubernetes.io/projected/3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f-kube-api-access-k4px7\") pod \"multus-additional-cni-plugins-xpv2v\" (UID: \"3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f\") " pod="openshift-multus/multus-additional-cni-plugins-xpv2v" Sep 14 11:00:55.621815 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.621600 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-host-kubelet\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.621815 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.621616 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/978d6d43-2346-476b-9bff-274d816f1175-serviceca\") pod \"node-ca-sfnlb\" (UID: \"978d6d43-2346-476b-9bff-274d816f1175\") " pod="openshift-image-registry/node-ca-sfnlb" Sep 14 11:00:55.621815 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.621641 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/caa56301-04e7-4389-b0e8-b39bcc58b6ad-konnectivity-ca\") pod \"konnectivity-agent-jfdk9\" (UID: \"caa56301-04e7-4389-b0e8-b39bcc58b6ad\") " pod="kube-system/konnectivity-agent-jfdk9" Sep 14 11:00:55.621815 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.621686 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-host-slash\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.621815 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.621689 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.621815 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.621719 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-host-cni-netd\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.621815 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.621746 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/caa56301-04e7-4389-b0e8-b39bcc58b6ad-agent-certs\") pod \"konnectivity-agent-jfdk9\" (UID: \"caa56301-04e7-4389-b0e8-b39bcc58b6ad\") " pod="kube-system/konnectivity-agent-jfdk9" Sep 14 11:00:55.621815 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.621769 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7ffda2ea-6f19-4043-a160-d116acf60812-ovn-node-metrics-cert\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.621815 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.621796 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/807c9dd2-6b4e-4994-b996-6b86893e87c2-lib-modules\") pod \"tuned-89vlx\" (UID: \"807c9dd2-6b4e-4994-b996-6b86893e87c2\") " pod="openshift-cluster-node-tuning-operator/tuned-89vlx" Sep 14 11:00:55.621815 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.621820 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/807c9dd2-6b4e-4994-b996-6b86893e87c2-var-lib-kubelet\") pod \"tuned-89vlx\" (UID: \"807c9dd2-6b4e-4994-b996-6b86893e87c2\") " pod="openshift-cluster-node-tuning-operator/tuned-89vlx" Sep 14 11:00:55.622548 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.621842 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c961b4ad-d3a5-4032-aa9a-a96c167e3df8-tmp-dir\") pod \"node-resolver-lcx2b\" (UID: \"c961b4ad-d3a5-4032-aa9a-a96c167e3df8\") " pod="openshift-dns/node-resolver-lcx2b" Sep 14 11:00:55.622548 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.621866 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xpv2v\" (UID: \"3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f\") " pod="openshift-multus/multus-additional-cni-plugins-xpv2v" Sep 14 11:00:55.622548 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.621892 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-systemd-units\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.622548 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.621928 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-host-run-netns\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.622548 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.621947 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-run-ovn\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.622548 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.621968 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-host-cni-bin\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.622548 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.621989 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/807c9dd2-6b4e-4994-b996-6b86893e87c2-tmp\") pod \"tuned-89vlx\" (UID: \"807c9dd2-6b4e-4994-b996-6b86893e87c2\") " pod="openshift-cluster-node-tuning-operator/tuned-89vlx" Sep 14 11:00:55.622548 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.622013 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd8tz\" (UniqueName: \"kubernetes.io/projected/978d6d43-2346-476b-9bff-274d816f1175-kube-api-access-bd8tz\") pod \"node-ca-sfnlb\" (UID: \"978d6d43-2346-476b-9bff-274d816f1175\") " pod="openshift-image-registry/node-ca-sfnlb" Sep 14 11:00:55.622548 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.622036 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-run-openvswitch\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.622548 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.622065 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/807c9dd2-6b4e-4994-b996-6b86893e87c2-etc-sysconfig\") pod \"tuned-89vlx\" (UID: \"807c9dd2-6b4e-4994-b996-6b86893e87c2\") " pod="openshift-cluster-node-tuning-operator/tuned-89vlx" Sep 14 11:00:55.622548 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.622083 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/807c9dd2-6b4e-4994-b996-6b86893e87c2-etc-systemd\") pod \"tuned-89vlx\" (UID: \"807c9dd2-6b4e-4994-b996-6b86893e87c2\") " pod="openshift-cluster-node-tuning-operator/tuned-89vlx" Sep 14 11:00:55.622548 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.622098 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfd621b6-51b0-4f2e-b1aa-ea198b732626-metrics-certs\") pod \"network-metrics-daemon-jkg6q\" (UID: \"dfd621b6-51b0-4f2e-b1aa-ea198b732626\") " pod="openshift-multus/network-metrics-daemon-jkg6q" Sep 14 11:00:55.622548 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.622112 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xpv2v\" (UID: \"3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f\") " pod="openshift-multus/multus-additional-cni-plugins-xpv2v" Sep 14 11:00:55.622548 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.622131 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/978d6d43-2346-476b-9bff-274d816f1175-host\") pod \"node-ca-sfnlb\" (UID: \"978d6d43-2346-476b-9bff-274d816f1175\") " pod="openshift-image-registry/node-ca-sfnlb" Sep 14 11:00:55.622548 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.622153 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7ffda2ea-6f19-4043-a160-d116acf60812-ovnkube-script-lib\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.622548 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.622170 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tds9w\" (UniqueName: \"kubernetes.io/projected/c961b4ad-d3a5-4032-aa9a-a96c167e3df8-kube-api-access-tds9w\") pod \"node-resolver-lcx2b\" (UID: \"c961b4ad-d3a5-4032-aa9a-a96c167e3df8\") " pod="openshift-dns/node-resolver-lcx2b" Sep 14 11:00:55.623218 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.622193 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnbzr\" (UniqueName: \"kubernetes.io/projected/dfd621b6-51b0-4f2e-b1aa-ea198b732626-kube-api-access-lnbzr\") pod \"network-metrics-daemon-jkg6q\" (UID: \"dfd621b6-51b0-4f2e-b1aa-ea198b732626\") " pod="openshift-multus/network-metrics-daemon-jkg6q" Sep 14 11:00:55.623218 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.622214 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-run-systemd\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.623218 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.622228 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/807c9dd2-6b4e-4994-b996-6b86893e87c2-etc-modprobe-d\") pod \"tuned-89vlx\" (UID: \"807c9dd2-6b4e-4994-b996-6b86893e87c2\") " pod="openshift-cluster-node-tuning-operator/tuned-89vlx" Sep 14 11:00:55.623218 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.622242 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-host-run-ovn-kubernetes\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.623218 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.622255 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7ffda2ea-6f19-4043-a160-d116acf60812-env-overrides\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.623218 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.622278 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-etc-openvswitch\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.623218 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.622293 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7ffda2ea-6f19-4043-a160-d116acf60812-ovnkube-config\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.623218 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.622325 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/807c9dd2-6b4e-4994-b996-6b86893e87c2-etc-tuned\") pod \"tuned-89vlx\" (UID: \"807c9dd2-6b4e-4994-b996-6b86893e87c2\") " pod="openshift-cluster-node-tuning-operator/tuned-89vlx" Sep 14 11:00:55.623218 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.622352 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/807c9dd2-6b4e-4994-b996-6b86893e87c2-etc-sysctl-d\") pod \"tuned-89vlx\" (UID: \"807c9dd2-6b4e-4994-b996-6b86893e87c2\") " pod="openshift-cluster-node-tuning-operator/tuned-89vlx" Sep 14 11:00:55.623218 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.622373 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/807c9dd2-6b4e-4994-b996-6b86893e87c2-host\") pod \"tuned-89vlx\" (UID: \"807c9dd2-6b4e-4994-b996-6b86893e87c2\") " pod="openshift-cluster-node-tuning-operator/tuned-89vlx" Sep 14 11:00:55.623218 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.622399 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4f26\" (UniqueName: \"kubernetes.io/projected/807c9dd2-6b4e-4994-b996-6b86893e87c2-kube-api-access-q4f26\") pod \"tuned-89vlx\" (UID: \"807c9dd2-6b4e-4994-b996-6b86893e87c2\") " pod="openshift-cluster-node-tuning-operator/tuned-89vlx" Sep 14 11:00:55.623218 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.622418 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4szfj\" (UniqueName: \"kubernetes.io/projected/5ce405d8-5bb3-453d-8c40-cf94ed8ac8a3-kube-api-access-4szfj\") pod \"iptables-alerter-b5r9g\" (UID: \"5ce405d8-5bb3-453d-8c40-cf94ed8ac8a3\") " pod="openshift-network-operator/iptables-alerter-b5r9g" Sep 14 11:00:55.623218 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.622433 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f-cnibin\") pod \"multus-additional-cni-plugins-xpv2v\" (UID: \"3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f\") " pod="openshift-multus/multus-additional-cni-plugins-xpv2v" Sep 14 11:00:55.623218 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.622448 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-log-socket\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.623218 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.622462 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/807c9dd2-6b4e-4994-b996-6b86893e87c2-run\") pod \"tuned-89vlx\" (UID: \"807c9dd2-6b4e-4994-b996-6b86893e87c2\") " pod="openshift-cluster-node-tuning-operator/tuned-89vlx" Sep 14 11:00:55.623218 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.622489 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f-system-cni-dir\") pod \"multus-additional-cni-plugins-xpv2v\" (UID: \"3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f\") " pod="openshift-multus/multus-additional-cni-plugins-xpv2v" Sep 14 11:00:55.623983 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.622517 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f-cni-binary-copy\") pod \"multus-additional-cni-plugins-xpv2v\" (UID: \"3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f\") " pod="openshift-multus/multus-additional-cni-plugins-xpv2v" Sep 14 11:00:55.623983 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.623648 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 14 11:00:55.623983 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.623749 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 14 11:00:55.623983 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.623777 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 14 11:00:55.624169 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.624010 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 14 11:00:55.624169 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.624116 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6jcs" Sep 14 11:00:55.624270 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:55.624163 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6jcs" podUID="1bb6499d-6175-4cc7-83de-a6dd85437535" Sep 14 11:00:55.624328 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.624277 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 14 11:00:55.624944 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.624922 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 14 11:00:55.625042 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.624945 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 14 11:00:55.625042 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.625015 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-g9rkp\"" Sep 14 11:00:55.625198 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.625123 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-svjvk\"" Sep 14 11:00:55.625271 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.625212 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-t76lc\"" Sep 14 11:00:55.625460 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.625443 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gcxrq\"" Sep 14 11:00:55.625530 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.625500 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 14 11:00:55.625530 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.625516 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9g28p\"" Sep 14 11:00:55.625640 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.625450 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 14 11:00:55.625727 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.625630 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 14 11:00:55.625792 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.625754 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 14 11:00:55.625792 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.625763 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 14 11:00:55.625899 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.625816 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 14 11:00:55.625899 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.625874 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 14 11:00:55.626880 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.626861 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2sk6p" Sep 14 11:00:55.626965 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:55.626923 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2sk6p" podUID="07adbd4b-794d-4f1e-b566-02280ae8390a" Sep 14 11:00:55.629109 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.629095 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 14 11:00:55.639816 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.639800 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-137-230.ec2.internal"] Sep 14 11:00:55.640367 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.640354 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 14 11:00:55.640438 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.640414 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-230.ec2.internal" Sep 14 11:00:55.652514 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.652498 2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 11:00:55.655614 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.655599 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 14 11:00:55.657681 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.657657 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-230.ec2.internal"] Sep 14 11:00:55.658489 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.658478 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 14 11:00:55.669578 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.669561 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zrpwf" Sep 14 11:00:55.680107 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.680090 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zrpwf" Sep 14 11:00:55.717794 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.717778 2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 14 11:00:55.722995 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.722973 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c961b4ad-d3a5-4032-aa9a-a96c167e3df8-hosts-file\") pod \"node-resolver-lcx2b\" (UID: \"c961b4ad-d3a5-4032-aa9a-a96c167e3df8\") " pod="openshift-dns/node-resolver-lcx2b" Sep 14 11:00:55.723082 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.722998 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/807c9dd2-6b4e-4994-b996-6b86893e87c2-etc-kubernetes\") pod \"tuned-89vlx\" (UID: \"807c9dd2-6b4e-4994-b996-6b86893e87c2\") " pod="openshift-cluster-node-tuning-operator/tuned-89vlx" Sep 14 11:00:55.723082 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.723014 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/807c9dd2-6b4e-4994-b996-6b86893e87c2-etc-sysctl-conf\") pod \"tuned-89vlx\" (UID: \"807c9dd2-6b4e-4994-b996-6b86893e87c2\") " pod="openshift-cluster-node-tuning-operator/tuned-89vlx" Sep 14 11:00:55.723082 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.723031 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24a7349e-c9f6-4234-8e4f-effab7979852-cni-binary-copy\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.723082 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.723053 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24a7349e-c9f6-4234-8e4f-effab7979852-etc-kubernetes\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.723082 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.723068 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6qnc\" (UniqueName: \"kubernetes.io/projected/99ab5d24-81bd-4f38-8812-bc1794de173a-kube-api-access-x6qnc\") pod \"aws-ebs-csi-driver-node-7mwsc\" (UID: \"99ab5d24-81bd-4f38-8812-bc1794de173a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mwsc" Sep 14 11:00:55.723082 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.723070 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c961b4ad-d3a5-4032-aa9a-a96c167e3df8-hosts-file\") pod \"node-resolver-lcx2b\" (UID: \"c961b4ad-d3a5-4032-aa9a-a96c167e3df8\") " pod="openshift-dns/node-resolver-lcx2b" Sep 14 11:00:55.723345 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.723110 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-var-lib-openvswitch\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.723345 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.723135 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/807c9dd2-6b4e-4994-b996-6b86893e87c2-etc-kubernetes\") pod \"tuned-89vlx\" (UID: \"807c9dd2-6b4e-4994-b996-6b86893e87c2\") " pod="openshift-cluster-node-tuning-operator/tuned-89vlx" Sep 14 11:00:55.723345 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.723142 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-node-log\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.723345 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.723152 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/807c9dd2-6b4e-4994-b996-6b86893e87c2-etc-sysctl-conf\") pod \"tuned-89vlx\" (UID: \"807c9dd2-6b4e-4994-b996-6b86893e87c2\") " pod="openshift-cluster-node-tuning-operator/tuned-89vlx" Sep 14 11:00:55.723345 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.723163 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-var-lib-openvswitch\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.723345 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.723177 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/807c9dd2-6b4e-4994-b996-6b86893e87c2-sys\") pod \"tuned-89vlx\" (UID: \"807c9dd2-6b4e-4994-b996-6b86893e87c2\") " pod="openshift-cluster-node-tuning-operator/tuned-89vlx" Sep 14 11:00:55.723345 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.723186 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-node-log\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.723345 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.723196 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5ce405d8-5bb3-453d-8c40-cf94ed8ac8a3-iptables-alerter-script\") pod \"iptables-alerter-b5r9g\" (UID: \"5ce405d8-5bb3-453d-8c40-cf94ed8ac8a3\") " pod="openshift-network-operator/iptables-alerter-b5r9g" Sep 14 11:00:55.723345 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.723216 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24a7349e-c9f6-4234-8e4f-effab7979852-system-cni-dir\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.723345 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.723217 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/807c9dd2-6b4e-4994-b996-6b86893e87c2-sys\") pod \"tuned-89vlx\" (UID: \"807c9dd2-6b4e-4994-b996-6b86893e87c2\") " pod="openshift-cluster-node-tuning-operator/tuned-89vlx" Sep 14 11:00:55.723345 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.723238 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24a7349e-c9f6-4234-8e4f-effab7979852-host-var-lib-kubelet\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.723345 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.723259 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/24a7349e-c9f6-4234-8e4f-effab7979852-hostroot\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.723345 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.723276 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.723345 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.723293 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7px44\" (UniqueName: \"kubernetes.io/projected/7ffda2ea-6f19-4043-a160-d116acf60812-kube-api-access-7px44\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.723345 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.723307 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5ce405d8-5bb3-453d-8c40-cf94ed8ac8a3-host-slash\") pod \"iptables-alerter-b5r9g\" (UID: \"5ce405d8-5bb3-453d-8c40-cf94ed8ac8a3\") " pod="openshift-network-operator/iptables-alerter-b5r9g" Sep 14 11:00:55.723345 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.723323 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xpv2v\" (UID: \"3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f\") " pod="openshift-multus/multus-additional-cni-plugins-xpv2v" Sep 14 11:00:55.724773 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.723375 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4px7\" (UniqueName: \"kubernetes.io/projected/3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f-kube-api-access-k4px7\") pod \"multus-additional-cni-plugins-xpv2v\" (UID: \"3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f\") " pod="openshift-multus/multus-additional-cni-plugins-xpv2v" Sep 14 11:00:55.724773 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.723392 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/24a7349e-c9f6-4234-8e4f-effab7979852-multus-socket-dir-parent\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.724773 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.723410 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.724773 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.723414 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-host-kubelet\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.724773 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.723454 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/978d6d43-2346-476b-9bff-274d816f1175-serviceca\") pod \"node-ca-sfnlb\" (UID: \"978d6d43-2346-476b-9bff-274d816f1175\") " pod="openshift-image-registry/node-ca-sfnlb" Sep 14 11:00:55.724773 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.723473 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-host-kubelet\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.724773 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.723475 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/caa56301-04e7-4389-b0e8-b39bcc58b6ad-konnectivity-ca\") pod \"konnectivity-agent-jfdk9\" (UID: \"caa56301-04e7-4389-b0e8-b39bcc58b6ad\") " pod="kube-system/konnectivity-agent-jfdk9" Sep 14 11:00:55.724773 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.723516 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24a7349e-c9f6-4234-8e4f-effab7979852-cnibin\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.724773 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.723542 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/24a7349e-c9f6-4234-8e4f-effab7979852-host-run-k8s-cni-cncf-io\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.724773 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.723569 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-host-slash\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.724773 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.723594 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-host-cni-netd\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.724773 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.723686 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/caa56301-04e7-4389-b0e8-b39bcc58b6ad-agent-certs\") pod \"konnectivity-agent-jfdk9\" (UID: \"caa56301-04e7-4389-b0e8-b39bcc58b6ad\") " pod="kube-system/konnectivity-agent-jfdk9" Sep 14 11:00:55.724773 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.723726 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-host-slash\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.724773 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.723761 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/24a7349e-c9f6-4234-8e4f-effab7979852-host-var-lib-cni-multus\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.724773 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.723800 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-host-cni-netd\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.724773 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.723785 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/99ab5d24-81bd-4f38-8812-bc1794de173a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7mwsc\" (UID: \"99ab5d24-81bd-4f38-8812-bc1794de173a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mwsc" Sep 14 11:00:55.724773 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.723842 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/99ab5d24-81bd-4f38-8812-bc1794de173a-registration-dir\") pod \"aws-ebs-csi-driver-node-7mwsc\" (UID: \"99ab5d24-81bd-4f38-8812-bc1794de173a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mwsc" Sep 14 11:00:55.726101 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.723865 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5ce405d8-5bb3-453d-8c40-cf94ed8ac8a3-host-slash\") pod \"iptables-alerter-b5r9g\" (UID: \"5ce405d8-5bb3-453d-8c40-cf94ed8ac8a3\") " pod="openshift-network-operator/iptables-alerter-b5r9g" Sep 14 11:00:55.726101 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.723873 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7ffda2ea-6f19-4043-a160-d116acf60812-ovn-node-metrics-cert\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.726101 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.723922 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/807c9dd2-6b4e-4994-b996-6b86893e87c2-lib-modules\") pod \"tuned-89vlx\" (UID: \"807c9dd2-6b4e-4994-b996-6b86893e87c2\") " pod="openshift-cluster-node-tuning-operator/tuned-89vlx" Sep 14 11:00:55.726101 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.723948 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/807c9dd2-6b4e-4994-b996-6b86893e87c2-var-lib-kubelet\") pod \"tuned-89vlx\" (UID: \"807c9dd2-6b4e-4994-b996-6b86893e87c2\") " pod="openshift-cluster-node-tuning-operator/tuned-89vlx" Sep 14 11:00:55.726101 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.723973 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/99ab5d24-81bd-4f38-8812-bc1794de173a-etc-selinux\") pod \"aws-ebs-csi-driver-node-7mwsc\" (UID: \"99ab5d24-81bd-4f38-8812-bc1794de173a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mwsc" Sep 14 11:00:55.726101 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.723998 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c961b4ad-d3a5-4032-aa9a-a96c167e3df8-tmp-dir\") pod \"node-resolver-lcx2b\" (UID: \"c961b4ad-d3a5-4032-aa9a-a96c167e3df8\") " pod="openshift-dns/node-resolver-lcx2b" Sep 14 11:00:55.726101 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.724029 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/caa56301-04e7-4389-b0e8-b39bcc58b6ad-konnectivity-ca\") pod \"konnectivity-agent-jfdk9\" (UID: \"caa56301-04e7-4389-b0e8-b39bcc58b6ad\") " pod="kube-system/konnectivity-agent-jfdk9" Sep 14 11:00:55.726101 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.724022 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xpv2v\" (UID: \"3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f\") " pod="openshift-multus/multus-additional-cni-plugins-xpv2v" Sep 14 11:00:55.726101 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.724063 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24a7349e-c9f6-4234-8e4f-effab7979852-multus-conf-dir\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.726101 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.724106 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/978d6d43-2346-476b-9bff-274d816f1175-serviceca\") pod \"node-ca-sfnlb\" (UID: \"978d6d43-2346-476b-9bff-274d816f1175\") " pod="openshift-image-registry/node-ca-sfnlb" Sep 14 11:00:55.726101 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.724111 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/807c9dd2-6b4e-4994-b996-6b86893e87c2-lib-modules\") pod \"tuned-89vlx\" (UID: \"807c9dd2-6b4e-4994-b996-6b86893e87c2\") " pod="openshift-cluster-node-tuning-operator/tuned-89vlx" Sep 14 11:00:55.726101 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.724185 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/807c9dd2-6b4e-4994-b996-6b86893e87c2-var-lib-kubelet\") pod \"tuned-89vlx\" (UID: \"807c9dd2-6b4e-4994-b996-6b86893e87c2\") " pod="openshift-cluster-node-tuning-operator/tuned-89vlx" Sep 14 11:00:55.726101 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.724202 2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 14 11:00:55.726101 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.724213 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/24a7349e-c9f6-4234-8e4f-effab7979852-multus-daemon-config\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.726101 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.724244 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-systemd-units\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.726101 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.724279 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-host-run-netns\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.726101 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.724291 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-systemd-units\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.726101 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.724327 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-run-ovn\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.726756 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.724350 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-host-cni-bin\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.726756 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.724388 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5ce405d8-5bb3-453d-8c40-cf94ed8ac8a3-iptables-alerter-script\") pod \"iptables-alerter-b5r9g\" (UID: \"5ce405d8-5bb3-453d-8c40-cf94ed8ac8a3\") " pod="openshift-network-operator/iptables-alerter-b5r9g" Sep 14 11:00:55.726756 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.724399 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-run-ovn\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.726756 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.724391 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c961b4ad-d3a5-4032-aa9a-a96c167e3df8-tmp-dir\") pod \"node-resolver-lcx2b\" (UID: \"c961b4ad-d3a5-4032-aa9a-a96c167e3df8\") " pod="openshift-dns/node-resolver-lcx2b" Sep 14 11:00:55.726756 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.724425 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/807c9dd2-6b4e-4994-b996-6b86893e87c2-tmp\") pod \"tuned-89vlx\" (UID: \"807c9dd2-6b4e-4994-b996-6b86893e87c2\") " pod="openshift-cluster-node-tuning-operator/tuned-89vlx" Sep 14 11:00:55.726756 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.724450 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/24a7349e-c9f6-4234-8e4f-effab7979852-host-run-netns\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.726756 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.724458 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-host-cni-bin\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.726756 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.724510 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-host-run-netns\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.726756 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.724533 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/24a7349e-c9f6-4234-8e4f-effab7979852-host-run-multus-certs\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.726756 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.724565 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdc5t\" (UniqueName: \"kubernetes.io/projected/1bb6499d-6175-4cc7-83de-a6dd85437535-kube-api-access-hdc5t\") pod \"network-check-target-t6jcs\" (UID: \"1bb6499d-6175-4cc7-83de-a6dd85437535\") " pod="openshift-network-diagnostics/network-check-target-t6jcs" Sep 14 11:00:55.726756 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.724594 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd8tz\" (UniqueName: \"kubernetes.io/projected/978d6d43-2346-476b-9bff-274d816f1175-kube-api-access-bd8tz\") pod \"node-ca-sfnlb\" (UID: \"978d6d43-2346-476b-9bff-274d816f1175\") " pod="openshift-image-registry/node-ca-sfnlb" Sep 14 11:00:55.726756 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.724617 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-run-openvswitch\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.726756 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.724639 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/807c9dd2-6b4e-4994-b996-6b86893e87c2-etc-sysconfig\") pod \"tuned-89vlx\" (UID: \"807c9dd2-6b4e-4994-b996-6b86893e87c2\") " pod="openshift-cluster-node-tuning-operator/tuned-89vlx" Sep 14 11:00:55.726756 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.724659 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/807c9dd2-6b4e-4994-b996-6b86893e87c2-etc-systemd\") pod \"tuned-89vlx\" (UID: \"807c9dd2-6b4e-4994-b996-6b86893e87c2\") " pod="openshift-cluster-node-tuning-operator/tuned-89vlx" Sep 14 11:00:55.726756 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.724695 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfd621b6-51b0-4f2e-b1aa-ea198b732626-metrics-certs\") pod \"network-metrics-daemon-jkg6q\" (UID: \"dfd621b6-51b0-4f2e-b1aa-ea198b732626\") " pod="openshift-multus/network-metrics-daemon-jkg6q" Sep 14 11:00:55.726756 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.724715 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xpv2v\" (UID: \"3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f\") " pod="openshift-multus/multus-additional-cni-plugins-xpv2v" Sep 14 11:00:55.726756 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.724733 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/99ab5d24-81bd-4f38-8812-bc1794de173a-sys-fs\") pod \"aws-ebs-csi-driver-node-7mwsc\" (UID: \"99ab5d24-81bd-4f38-8812-bc1794de173a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mwsc" Sep 14 11:00:55.727504 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.724733 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xpv2v\" (UID: \"3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f\") " pod="openshift-multus/multus-additional-cni-plugins-xpv2v" Sep 14 11:00:55.727504 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.724753 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/978d6d43-2346-476b-9bff-274d816f1175-host\") pod \"node-ca-sfnlb\" (UID: \"978d6d43-2346-476b-9bff-274d816f1175\") " pod="openshift-image-registry/node-ca-sfnlb" Sep 14 11:00:55.727504 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.724779 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/978d6d43-2346-476b-9bff-274d816f1175-host\") pod \"node-ca-sfnlb\" (UID: \"978d6d43-2346-476b-9bff-274d816f1175\") " pod="openshift-image-registry/node-ca-sfnlb" Sep 14 11:00:55.727504 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.724772 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xpv2v\" (UID: \"3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f\") " pod="openshift-multus/multus-additional-cni-plugins-xpv2v" Sep 14 11:00:55.727504 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.724806 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7ffda2ea-6f19-4043-a160-d116acf60812-ovnkube-script-lib\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.727504 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.724820 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/807c9dd2-6b4e-4994-b996-6b86893e87c2-etc-systemd\") pod \"tuned-89vlx\" (UID: \"807c9dd2-6b4e-4994-b996-6b86893e87c2\") " pod="openshift-cluster-node-tuning-operator/tuned-89vlx" Sep 14 11:00:55.727504 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.724855 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-run-openvswitch\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.727504 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.724866 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/807c9dd2-6b4e-4994-b996-6b86893e87c2-etc-sysconfig\") pod \"tuned-89vlx\" (UID: \"807c9dd2-6b4e-4994-b996-6b86893e87c2\") " pod="openshift-cluster-node-tuning-operator/tuned-89vlx" Sep 14 11:00:55.727504 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:55.724879 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 11:00:55.727504 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.724890 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tds9w\" (UniqueName: \"kubernetes.io/projected/c961b4ad-d3a5-4032-aa9a-a96c167e3df8-kube-api-access-tds9w\") pod \"node-resolver-lcx2b\" (UID: \"c961b4ad-d3a5-4032-aa9a-a96c167e3df8\") " pod="openshift-dns/node-resolver-lcx2b" Sep 14 11:00:55.727504 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.724968 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnbzr\" (UniqueName: \"kubernetes.io/projected/dfd621b6-51b0-4f2e-b1aa-ea198b732626-kube-api-access-lnbzr\") pod \"network-metrics-daemon-jkg6q\" (UID: \"dfd621b6-51b0-4f2e-b1aa-ea198b732626\") " pod="openshift-multus/network-metrics-daemon-jkg6q" Sep 14 11:00:55.727504 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.724987 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xpv2v\" (UID: \"3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f\") " pod="openshift-multus/multus-additional-cni-plugins-xpv2v" Sep 14 11:00:55.727504 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:55.724993 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfd621b6-51b0-4f2e-b1aa-ea198b732626-metrics-certs podName:dfd621b6-51b0-4f2e-b1aa-ea198b732626 nodeName:}" failed. No retries permitted until 2026-09-14 11:00:56.224955892 +0000 UTC m=+2.023766487 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfd621b6-51b0-4f2e-b1aa-ea198b732626-metrics-certs") pod "network-metrics-daemon-jkg6q" (UID: "dfd621b6-51b0-4f2e-b1aa-ea198b732626") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 11:00:55.727504 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.725130 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58v9z\" (UniqueName: \"kubernetes.io/projected/24a7349e-c9f6-4234-8e4f-effab7979852-kube-api-access-58v9z\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.727504 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.725204 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-run-systemd\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.727504 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.725226 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/807c9dd2-6b4e-4994-b996-6b86893e87c2-etc-modprobe-d\") pod \"tuned-89vlx\" (UID: \"807c9dd2-6b4e-4994-b996-6b86893e87c2\") " pod="openshift-cluster-node-tuning-operator/tuned-89vlx" Sep 14 11:00:55.727504 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.725274 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/07adbd4b-794d-4f1e-b566-02280ae8390a-dbus\") pod \"global-pull-secret-syncer-2sk6p\" (UID: \"07adbd4b-794d-4f1e-b566-02280ae8390a\") " pod="kube-system/global-pull-secret-syncer-2sk6p" Sep 14 11:00:55.728242 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.725276 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-run-systemd\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.728242 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.725298 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-host-run-ovn-kubernetes\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.728242 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.725338 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7ffda2ea-6f19-4043-a160-d116acf60812-env-overrides\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.728242 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.725365 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/99ab5d24-81bd-4f38-8812-bc1794de173a-socket-dir\") pod \"aws-ebs-csi-driver-node-7mwsc\" (UID: \"99ab5d24-81bd-4f38-8812-bc1794de173a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mwsc" Sep 14 11:00:55.728242 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.725394 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-etc-openvswitch\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.728242 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.725414 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7ffda2ea-6f19-4043-a160-d116acf60812-ovnkube-config\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.728242 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.725437 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/807c9dd2-6b4e-4994-b996-6b86893e87c2-etc-tuned\") pod \"tuned-89vlx\" (UID: \"807c9dd2-6b4e-4994-b996-6b86893e87c2\") " pod="openshift-cluster-node-tuning-operator/tuned-89vlx" Sep 14 11:00:55.728242 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.725445 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/807c9dd2-6b4e-4994-b996-6b86893e87c2-etc-modprobe-d\") pod \"tuned-89vlx\" (UID: \"807c9dd2-6b4e-4994-b996-6b86893e87c2\") " pod="openshift-cluster-node-tuning-operator/tuned-89vlx" Sep 14 11:00:55.728242 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.725446 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-host-run-ovn-kubernetes\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.728242 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.725499 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/807c9dd2-6b4e-4994-b996-6b86893e87c2-etc-sysctl-d\") pod \"tuned-89vlx\" (UID: \"807c9dd2-6b4e-4994-b996-6b86893e87c2\") " pod="openshift-cluster-node-tuning-operator/tuned-89vlx" Sep 14 11:00:55.728242 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.725526 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/807c9dd2-6b4e-4994-b996-6b86893e87c2-host\") pod \"tuned-89vlx\" (UID: \"807c9dd2-6b4e-4994-b996-6b86893e87c2\") " pod="openshift-cluster-node-tuning-operator/tuned-89vlx" Sep 14 11:00:55.728242 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.725550 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4f26\" (UniqueName: \"kubernetes.io/projected/807c9dd2-6b4e-4994-b996-6b86893e87c2-kube-api-access-q4f26\") pod \"tuned-89vlx\" (UID: \"807c9dd2-6b4e-4994-b996-6b86893e87c2\") " pod="openshift-cluster-node-tuning-operator/tuned-89vlx" Sep 14 11:00:55.728242 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.725696 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/807c9dd2-6b4e-4994-b996-6b86893e87c2-etc-sysctl-d\") pod \"tuned-89vlx\" (UID: \"807c9dd2-6b4e-4994-b996-6b86893e87c2\") " pod="openshift-cluster-node-tuning-operator/tuned-89vlx" Sep 14 11:00:55.728242 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.725751 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-etc-openvswitch\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.728242 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.725826 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7ffda2ea-6f19-4043-a160-d116acf60812-env-overrides\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.728242 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.726059 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/807c9dd2-6b4e-4994-b996-6b86893e87c2-host\") pod \"tuned-89vlx\" (UID: \"807c9dd2-6b4e-4994-b996-6b86893e87c2\") " pod="openshift-cluster-node-tuning-operator/tuned-89vlx" Sep 14 11:00:55.728242 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.726320 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7ffda2ea-6f19-4043-a160-d116acf60812-ovnkube-config\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.728750 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.725606 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4szfj\" (UniqueName: \"kubernetes.io/projected/5ce405d8-5bb3-453d-8c40-cf94ed8ac8a3-kube-api-access-4szfj\") pod \"iptables-alerter-b5r9g\" (UID: \"5ce405d8-5bb3-453d-8c40-cf94ed8ac8a3\") " pod="openshift-network-operator/iptables-alerter-b5r9g" Sep 14 11:00:55.728750 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.726383 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f-cnibin\") pod \"multus-additional-cni-plugins-xpv2v\" (UID: \"3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f\") " pod="openshift-multus/multus-additional-cni-plugins-xpv2v" Sep 14 11:00:55.728750 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.726420 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24a7349e-c9f6-4234-8e4f-effab7979852-multus-cni-dir\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.728750 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.726458 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/24a7349e-c9f6-4234-8e4f-effab7979852-host-var-lib-cni-bin\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.728750 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.726482 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07adbd4b-794d-4f1e-b566-02280ae8390a-original-pull-secret\") pod \"global-pull-secret-syncer-2sk6p\" (UID: \"07adbd4b-794d-4f1e-b566-02280ae8390a\") " pod="kube-system/global-pull-secret-syncer-2sk6p" Sep 14 11:00:55.728750 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.726511 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-log-socket\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.728750 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.726533 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/807c9dd2-6b4e-4994-b996-6b86893e87c2-run\") pod \"tuned-89vlx\" (UID: \"807c9dd2-6b4e-4994-b996-6b86893e87c2\") " pod="openshift-cluster-node-tuning-operator/tuned-89vlx" Sep 14 11:00:55.728750 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.726558 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f-system-cni-dir\") pod \"multus-additional-cni-plugins-xpv2v\" (UID: \"3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f\") " pod="openshift-multus/multus-additional-cni-plugins-xpv2v" Sep 14 11:00:55.728750 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.726585 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f-cni-binary-copy\") pod \"multus-additional-cni-plugins-xpv2v\" (UID: \"3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f\") " pod="openshift-multus/multus-additional-cni-plugins-xpv2v" Sep 14 11:00:55.728750 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.726609 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/99ab5d24-81bd-4f38-8812-bc1794de173a-device-dir\") pod \"aws-ebs-csi-driver-node-7mwsc\" (UID: \"99ab5d24-81bd-4f38-8812-bc1794de173a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mwsc" Sep 14 11:00:55.728750 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.726631 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/07adbd4b-794d-4f1e-b566-02280ae8390a-kubelet-config\") pod \"global-pull-secret-syncer-2sk6p\" (UID: \"07adbd4b-794d-4f1e-b566-02280ae8390a\") " pod="kube-system/global-pull-secret-syncer-2sk6p" Sep 14 11:00:55.728750 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.726741 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f-cnibin\") pod \"multus-additional-cni-plugins-xpv2v\" (UID: \"3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f\") " pod="openshift-multus/multus-additional-cni-plugins-xpv2v" Sep 14 11:00:55.728750 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.726814 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7ffda2ea-6f19-4043-a160-d116acf60812-log-socket\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.728750 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.726862 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/807c9dd2-6b4e-4994-b996-6b86893e87c2-run\") pod \"tuned-89vlx\" (UID: \"807c9dd2-6b4e-4994-b996-6b86893e87c2\") " pod="openshift-cluster-node-tuning-operator/tuned-89vlx" Sep 14 11:00:55.728750 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.726901 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f-system-cni-dir\") pod \"multus-additional-cni-plugins-xpv2v\" (UID: \"3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f\") " pod="openshift-multus/multus-additional-cni-plugins-xpv2v" Sep 14 11:00:55.728750 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.727347 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f-cni-binary-copy\") pod \"multus-additional-cni-plugins-xpv2v\" (UID: \"3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f\") " pod="openshift-multus/multus-additional-cni-plugins-xpv2v" Sep 14 11:00:55.728750 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.727479 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7ffda2ea-6f19-4043-a160-d116acf60812-ovnkube-script-lib\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.729203 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.728068 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/807c9dd2-6b4e-4994-b996-6b86893e87c2-tmp\") pod \"tuned-89vlx\" (UID: \"807c9dd2-6b4e-4994-b996-6b86893e87c2\") " pod="openshift-cluster-node-tuning-operator/tuned-89vlx" Sep 14 11:00:55.729203 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.728188 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/807c9dd2-6b4e-4994-b996-6b86893e87c2-etc-tuned\") pod \"tuned-89vlx\" (UID: \"807c9dd2-6b4e-4994-b996-6b86893e87c2\") " pod="openshift-cluster-node-tuning-operator/tuned-89vlx" Sep 14 11:00:55.729203 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.728267 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7ffda2ea-6f19-4043-a160-d116acf60812-ovn-node-metrics-cert\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.729203 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.728328 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/caa56301-04e7-4389-b0e8-b39bcc58b6ad-agent-certs\") pod \"konnectivity-agent-jfdk9\" (UID: \"caa56301-04e7-4389-b0e8-b39bcc58b6ad\") " pod="kube-system/konnectivity-agent-jfdk9" Sep 14 11:00:55.744533 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.744508 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7px44\" (UniqueName: \"kubernetes.io/projected/7ffda2ea-6f19-4043-a160-d116acf60812-kube-api-access-7px44\") pod \"ovnkube-node-xpwhs\" (UID: \"7ffda2ea-6f19-4043-a160-d116acf60812\") " pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:55.754461 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.754442 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tds9w\" (UniqueName: \"kubernetes.io/projected/c961b4ad-d3a5-4032-aa9a-a96c167e3df8-kube-api-access-tds9w\") pod \"node-resolver-lcx2b\" (UID: \"c961b4ad-d3a5-4032-aa9a-a96c167e3df8\") " pod="openshift-dns/node-resolver-lcx2b" Sep 14 11:00:55.757198 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.757180 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4px7\" (UniqueName: \"kubernetes.io/projected/3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f-kube-api-access-k4px7\") pod \"multus-additional-cni-plugins-xpv2v\" (UID: \"3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f\") " pod="openshift-multus/multus-additional-cni-plugins-xpv2v" Sep 14 11:00:55.758073 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.758022 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4f26\" (UniqueName: \"kubernetes.io/projected/807c9dd2-6b4e-4994-b996-6b86893e87c2-kube-api-access-q4f26\") pod \"tuned-89vlx\" (UID: \"807c9dd2-6b4e-4994-b996-6b86893e87c2\") " pod="openshift-cluster-node-tuning-operator/tuned-89vlx" Sep 14 11:00:55.759487 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.759469 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd8tz\" (UniqueName: \"kubernetes.io/projected/978d6d43-2346-476b-9bff-274d816f1175-kube-api-access-bd8tz\") pod \"node-ca-sfnlb\" (UID: \"978d6d43-2346-476b-9bff-274d816f1175\") " pod="openshift-image-registry/node-ca-sfnlb" Sep 14 11:00:55.759831 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.759815 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnbzr\" (UniqueName: \"kubernetes.io/projected/dfd621b6-51b0-4f2e-b1aa-ea198b732626-kube-api-access-lnbzr\") pod \"network-metrics-daemon-jkg6q\" (UID: \"dfd621b6-51b0-4f2e-b1aa-ea198b732626\") " pod="openshift-multus/network-metrics-daemon-jkg6q" Sep 14 11:00:55.764324 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.764307 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4szfj\" (UniqueName: \"kubernetes.io/projected/5ce405d8-5bb3-453d-8c40-cf94ed8ac8a3-kube-api-access-4szfj\") pod \"iptables-alerter-b5r9g\" (UID: \"5ce405d8-5bb3-453d-8c40-cf94ed8ac8a3\") " pod="openshift-network-operator/iptables-alerter-b5r9g" Sep 14 11:00:55.827255 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.827235 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/24a7349e-c9f6-4234-8e4f-effab7979852-host-run-netns\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.827370 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.827259 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/24a7349e-c9f6-4234-8e4f-effab7979852-host-run-multus-certs\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.827370 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.827274 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdc5t\" (UniqueName: \"kubernetes.io/projected/1bb6499d-6175-4cc7-83de-a6dd85437535-kube-api-access-hdc5t\") pod \"network-check-target-t6jcs\" (UID: \"1bb6499d-6175-4cc7-83de-a6dd85437535\") " pod="openshift-network-diagnostics/network-check-target-t6jcs" Sep 14 11:00:55.827370 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.827303 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/99ab5d24-81bd-4f38-8812-bc1794de173a-sys-fs\") pod \"aws-ebs-csi-driver-node-7mwsc\" (UID: \"99ab5d24-81bd-4f38-8812-bc1794de173a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mwsc" Sep 14 11:00:55.827370 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.827324 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58v9z\" (UniqueName: \"kubernetes.io/projected/24a7349e-c9f6-4234-8e4f-effab7979852-kube-api-access-58v9z\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.827370 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.827345 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/24a7349e-c9f6-4234-8e4f-effab7979852-host-run-netns\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.827611 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.827365 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/24a7349e-c9f6-4234-8e4f-effab7979852-host-run-multus-certs\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.827611 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.827408 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/99ab5d24-81bd-4f38-8812-bc1794de173a-sys-fs\") pod \"aws-ebs-csi-driver-node-7mwsc\" (UID: \"99ab5d24-81bd-4f38-8812-bc1794de173a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mwsc" Sep 14 11:00:55.827611 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.827451 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/07adbd4b-794d-4f1e-b566-02280ae8390a-dbus\") pod \"global-pull-secret-syncer-2sk6p\" (UID: \"07adbd4b-794d-4f1e-b566-02280ae8390a\") " pod="kube-system/global-pull-secret-syncer-2sk6p" Sep 14 11:00:55.827611 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.827476 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/99ab5d24-81bd-4f38-8812-bc1794de173a-socket-dir\") pod \"aws-ebs-csi-driver-node-7mwsc\" (UID: \"99ab5d24-81bd-4f38-8812-bc1794de173a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mwsc" Sep 14 11:00:55.827611 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.827496 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24a7349e-c9f6-4234-8e4f-effab7979852-multus-cni-dir\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.827611 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.827513 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/24a7349e-c9f6-4234-8e4f-effab7979852-host-var-lib-cni-bin\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.827611 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.827528 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07adbd4b-794d-4f1e-b566-02280ae8390a-original-pull-secret\") pod \"global-pull-secret-syncer-2sk6p\" (UID: \"07adbd4b-794d-4f1e-b566-02280ae8390a\") " pod="kube-system/global-pull-secret-syncer-2sk6p" Sep 14 11:00:55.827611 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.827530 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/07adbd4b-794d-4f1e-b566-02280ae8390a-dbus\") pod \"global-pull-secret-syncer-2sk6p\" (UID: \"07adbd4b-794d-4f1e-b566-02280ae8390a\") " pod="kube-system/global-pull-secret-syncer-2sk6p" Sep 14 11:00:55.827611 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.827547 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/99ab5d24-81bd-4f38-8812-bc1794de173a-device-dir\") pod \"aws-ebs-csi-driver-node-7mwsc\" (UID: \"99ab5d24-81bd-4f38-8812-bc1794de173a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mwsc" Sep 14 11:00:55.827611 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.827571 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/24a7349e-c9f6-4234-8e4f-effab7979852-host-var-lib-cni-bin\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.827611 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.827577 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/07adbd4b-794d-4f1e-b566-02280ae8390a-kubelet-config\") pod \"global-pull-secret-syncer-2sk6p\" (UID: \"07adbd4b-794d-4f1e-b566-02280ae8390a\") " pod="kube-system/global-pull-secret-syncer-2sk6p" Sep 14 11:00:55.827611 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.827572 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24a7349e-c9f6-4234-8e4f-effab7979852-multus-cni-dir\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.827611 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.827596 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/99ab5d24-81bd-4f38-8812-bc1794de173a-device-dir\") pod \"aws-ebs-csi-driver-node-7mwsc\" (UID: \"99ab5d24-81bd-4f38-8812-bc1794de173a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mwsc" Sep 14 11:00:55.827611 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.827599 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24a7349e-c9f6-4234-8e4f-effab7979852-cni-binary-copy\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.827611 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.827614 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24a7349e-c9f6-4234-8e4f-effab7979852-etc-kubernetes\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.828290 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.827634 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6qnc\" (UniqueName: \"kubernetes.io/projected/99ab5d24-81bd-4f38-8812-bc1794de173a-kube-api-access-x6qnc\") pod \"aws-ebs-csi-driver-node-7mwsc\" (UID: \"99ab5d24-81bd-4f38-8812-bc1794de173a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mwsc" Sep 14 11:00:55.828290 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:55.827647 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 11:00:55.828290 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.827659 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24a7349e-c9f6-4234-8e4f-effab7979852-system-cni-dir\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.828290 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.827701 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24a7349e-c9f6-4234-8e4f-effab7979852-host-var-lib-kubelet\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.828290 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.827728 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/24a7349e-c9f6-4234-8e4f-effab7979852-hostroot\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.828290 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:55.827749 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07adbd4b-794d-4f1e-b566-02280ae8390a-original-pull-secret podName:07adbd4b-794d-4f1e-b566-02280ae8390a nodeName:}" failed. No retries permitted until 2026-09-14 11:00:56.327731827 +0000 UTC m=+2.126542422 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/07adbd4b-794d-4f1e-b566-02280ae8390a-original-pull-secret") pod "global-pull-secret-syncer-2sk6p" (UID: "07adbd4b-794d-4f1e-b566-02280ae8390a") : object "kube-system"/"original-pull-secret" not registered Sep 14 11:00:55.828290 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.827775 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/07adbd4b-794d-4f1e-b566-02280ae8390a-kubelet-config\") pod \"global-pull-secret-syncer-2sk6p\" (UID: \"07adbd4b-794d-4f1e-b566-02280ae8390a\") " pod="kube-system/global-pull-secret-syncer-2sk6p" Sep 14 11:00:55.828290 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.827810 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/99ab5d24-81bd-4f38-8812-bc1794de173a-socket-dir\") pod \"aws-ebs-csi-driver-node-7mwsc\" (UID: \"99ab5d24-81bd-4f38-8812-bc1794de173a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mwsc" Sep 14 11:00:55.828290 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.827813 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24a7349e-c9f6-4234-8e4f-effab7979852-system-cni-dir\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.828290 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.827813 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24a7349e-c9f6-4234-8e4f-effab7979852-etc-kubernetes\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.828290 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.827833 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/24a7349e-c9f6-4234-8e4f-effab7979852-multus-socket-dir-parent\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.828290 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.827883 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24a7349e-c9f6-4234-8e4f-effab7979852-host-var-lib-kubelet\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.828290 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.827897 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24a7349e-c9f6-4234-8e4f-effab7979852-cnibin\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.828290 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.827921 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/24a7349e-c9f6-4234-8e4f-effab7979852-multus-socket-dir-parent\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.828290 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.827925 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/24a7349e-c9f6-4234-8e4f-effab7979852-host-run-k8s-cni-cncf-io\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.828290 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.827978 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/24a7349e-c9f6-4234-8e4f-effab7979852-host-run-k8s-cni-cncf-io\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.828290 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.827978 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24a7349e-c9f6-4234-8e4f-effab7979852-cnibin\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.829018 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.828029 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/24a7349e-c9f6-4234-8e4f-effab7979852-host-var-lib-cni-multus\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.829018 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.828048 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/99ab5d24-81bd-4f38-8812-bc1794de173a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7mwsc\" (UID: \"99ab5d24-81bd-4f38-8812-bc1794de173a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mwsc" Sep 14 11:00:55.829018 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.828099 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/24a7349e-c9f6-4234-8e4f-effab7979852-host-var-lib-cni-multus\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.829018 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.828173 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/99ab5d24-81bd-4f38-8812-bc1794de173a-registration-dir\") pod \"aws-ebs-csi-driver-node-7mwsc\" (UID: \"99ab5d24-81bd-4f38-8812-bc1794de173a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mwsc" Sep 14 11:00:55.829018 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.828200 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/99ab5d24-81bd-4f38-8812-bc1794de173a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7mwsc\" (UID: \"99ab5d24-81bd-4f38-8812-bc1794de173a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mwsc" Sep 14 11:00:55.829018 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.828216 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24a7349e-c9f6-4234-8e4f-effab7979852-cni-binary-copy\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.829018 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.828227 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/99ab5d24-81bd-4f38-8812-bc1794de173a-registration-dir\") pod \"aws-ebs-csi-driver-node-7mwsc\" (UID: \"99ab5d24-81bd-4f38-8812-bc1794de173a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mwsc" Sep 14 11:00:55.829018 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.828263 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/99ab5d24-81bd-4f38-8812-bc1794de173a-etc-selinux\") pod \"aws-ebs-csi-driver-node-7mwsc\" (UID: \"99ab5d24-81bd-4f38-8812-bc1794de173a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mwsc" Sep 14 11:00:55.829018 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.828293 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24a7349e-c9f6-4234-8e4f-effab7979852-multus-conf-dir\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.829018 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.828312 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/24a7349e-c9f6-4234-8e4f-effab7979852-hostroot\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.829018 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.828315 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/24a7349e-c9f6-4234-8e4f-effab7979852-multus-daemon-config\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.829018 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.828337 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24a7349e-c9f6-4234-8e4f-effab7979852-multus-conf-dir\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.829018 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.828350 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/99ab5d24-81bd-4f38-8812-bc1794de173a-etc-selinux\") pod \"aws-ebs-csi-driver-node-7mwsc\" (UID: \"99ab5d24-81bd-4f38-8812-bc1794de173a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mwsc" Sep 14 11:00:55.829018 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.828783 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/24a7349e-c9f6-4234-8e4f-effab7979852-multus-daemon-config\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.848363 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:55.848348 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 11:00:55.848363 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:55.848364 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 11:00:55.848480 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:55.848374 2574 projected.go:196] Error preparing data for projected volume kube-api-access-hdc5t for pod openshift-network-diagnostics/network-check-target-t6jcs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 11:00:55.848480 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:55.848418 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bb6499d-6175-4cc7-83de-a6dd85437535-kube-api-access-hdc5t podName:1bb6499d-6175-4cc7-83de-a6dd85437535 nodeName:}" failed. No retries permitted until 2026-09-14 11:00:56.348406523 +0000 UTC m=+2.147217118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hdc5t" (UniqueName: "kubernetes.io/projected/1bb6499d-6175-4cc7-83de-a6dd85437535-kube-api-access-hdc5t") pod "network-check-target-t6jcs" (UID: "1bb6499d-6175-4cc7-83de-a6dd85437535") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 11:00:55.849957 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.849940 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6qnc\" (UniqueName: \"kubernetes.io/projected/99ab5d24-81bd-4f38-8812-bc1794de173a-kube-api-access-x6qnc\") pod \"aws-ebs-csi-driver-node-7mwsc\" (UID: \"99ab5d24-81bd-4f38-8812-bc1794de173a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mwsc" Sep 14 11:00:55.850040 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.850026 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58v9z\" (UniqueName: \"kubernetes.io/projected/24a7349e-c9f6-4234-8e4f-effab7979852-kube-api-access-58v9z\") pod \"multus-hkdj4\" (UID: \"24a7349e-c9f6-4234-8e4f-effab7979852\") " pod="openshift-multus/multus-hkdj4" Sep 14 11:00:55.924704 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.924627 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xpv2v" Sep 14 11:00:55.939281 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:55.939263 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lcx2b" Sep 14 11:00:57.131456 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:57.131438 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-89vlx" Sep 14 11:00:57.144034 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:57.144019 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sfnlb" Sep 14 11:00:57.167558 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:57.167529 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-b5r9g" Sep 14 11:00:57.183836 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:57.183815 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:00:57.198965 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:57.198946 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jfdk9" Sep 14 11:00:57.206446 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:57.206431 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mwsc" Sep 14 11:00:57.229707 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:57.229693 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hkdj4" Sep 14 11:00:57.409695 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:57.409626 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfd621b6-51b0-4f2e-b1aa-ea198b732626-metrics-certs\") pod \"network-metrics-daemon-jkg6q\" (UID: \"dfd621b6-51b0-4f2e-b1aa-ea198b732626\") " pod="openshift-multus/network-metrics-daemon-jkg6q" Sep 14 11:00:57.409803 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:57.409719 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 11:00:57.409803 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:57.409767 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfd621b6-51b0-4f2e-b1aa-ea198b732626-metrics-certs podName:dfd621b6-51b0-4f2e-b1aa-ea198b732626 nodeName:}" failed. No retries permitted until 2026-09-14 11:00:58.409752172 +0000 UTC m=+3.029964673 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfd621b6-51b0-4f2e-b1aa-ea198b732626-metrics-certs") pod "network-metrics-daemon-jkg6q" (UID: "dfd621b6-51b0-4f2e-b1aa-ea198b732626") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 11:00:57.510633 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:57.510605 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07adbd4b-794d-4f1e-b566-02280ae8390a-original-pull-secret\") pod \"global-pull-secret-syncer-2sk6p\" (UID: \"07adbd4b-794d-4f1e-b566-02280ae8390a\") " pod="kube-system/global-pull-secret-syncer-2sk6p" Sep 14 11:00:57.510763 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:57.510705 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 11:00:57.510763 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:57.510750 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07adbd4b-794d-4f1e-b566-02280ae8390a-original-pull-secret podName:07adbd4b-794d-4f1e-b566-02280ae8390a nodeName:}" failed. No retries permitted until 2026-09-14 11:00:58.510737951 +0000 UTC m=+3.130950452 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/07adbd4b-794d-4f1e-b566-02280ae8390a-original-pull-secret") pod "global-pull-secret-syncer-2sk6p" (UID: "07adbd4b-794d-4f1e-b566-02280ae8390a") : object "kube-system"/"original-pull-secret" not registered Sep 14 11:00:57.610865 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:57.610834 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdc5t\" (UniqueName: \"kubernetes.io/projected/1bb6499d-6175-4cc7-83de-a6dd85437535-kube-api-access-hdc5t\") pod \"network-check-target-t6jcs\" (UID: \"1bb6499d-6175-4cc7-83de-a6dd85437535\") " pod="openshift-network-diagnostics/network-check-target-t6jcs" Sep 14 11:00:57.611068 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:57.610960 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 11:00:57.611068 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:57.610975 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 11:00:57.611068 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:57.610983 2574 projected.go:196] Error preparing data for projected volume kube-api-access-hdc5t for pod openshift-network-diagnostics/network-check-target-t6jcs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 11:00:57.611068 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:57.611027 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bb6499d-6175-4cc7-83de-a6dd85437535-kube-api-access-hdc5t podName:1bb6499d-6175-4cc7-83de-a6dd85437535 nodeName:}" failed. No retries permitted until 2026-09-14 11:00:58.611015507 +0000 UTC m=+3.231228022 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hdc5t" (UniqueName: "kubernetes.io/projected/1bb6499d-6175-4cc7-83de-a6dd85437535-kube-api-access-hdc5t") pod "network-check-target-t6jcs" (UID: "1bb6499d-6175-4cc7-83de-a6dd85437535") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 11:00:57.673249 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:57.673208 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc961b4ad_d3a5_4032_aa9a_a96c167e3df8.slice/crio-73275ca63dd39f6a3a049c7f03a84fe4e75bc0f4a160e7c241ad83595eb73662 WatchSource:0}: Error finding container 73275ca63dd39f6a3a049c7f03a84fe4e75bc0f4a160e7c241ad83595eb73662: Status 404 returned error can't find the container with id 73275ca63dd39f6a3a049c7f03a84fe4e75bc0f4a160e7c241ad83595eb73662 Sep 14 11:00:57.673600 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:57.673581 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaa56301_04e7_4389_b0e8_b39bcc58b6ad.slice/crio-ad0cb7a0be79349626a776e85aa295ab00d802e842184bc37182c4467fbf0f96 WatchSource:0}: Error finding container ad0cb7a0be79349626a776e85aa295ab00d802e842184bc37182c4467fbf0f96: Status 404 returned error can't find the container with id ad0cb7a0be79349626a776e85aa295ab00d802e842184bc37182c4467fbf0f96 Sep 14 11:00:57.682171 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:57.682155 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 11:00:57.697932 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:57.697909 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod807c9dd2_6b4e_4994_b996_6b86893e87c2.slice/crio-aaaa48f0c1fa2ce70120db646ff8234e4446aafbf668429118e26172d043f4c0 WatchSource:0}: Error finding container aaaa48f0c1fa2ce70120db646ff8234e4446aafbf668429118e26172d043f4c0: Status 404 returned error can't find the container with id aaaa48f0c1fa2ce70120db646ff8234e4446aafbf668429118e26172d043f4c0 Sep 14 11:00:57.707671 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:57.707652 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ffda2ea_6f19_4043_a160_d116acf60812.slice/crio-5737e130d42d35dc4999a1fc5289daae287445f3272c81c439e4b57a8276c1e5 WatchSource:0}: Error finding container 5737e130d42d35dc4999a1fc5289daae287445f3272c81c439e4b57a8276c1e5: Status 404 returned error can't find the container with id 5737e130d42d35dc4999a1fc5289daae287445f3272c81c439e4b57a8276c1e5 Sep 14 11:00:57.708691 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:57.708669 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 11:00:57.735055 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:57.735035 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ce405d8_5bb3_453d_8c40_cf94ed8ac8a3.slice/crio-6ec67d39f98201c2790d89a95a2ce39e3cce8f96569e78744548896f0173cb7a WatchSource:0}: Error finding container 6ec67d39f98201c2790d89a95a2ce39e3cce8f96569e78744548896f0173cb7a: Status 404 returned error can't find the container with id 6ec67d39f98201c2790d89a95a2ce39e3cce8f96569e78744548896f0173cb7a Sep 14 11:00:57.757986 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:57.757967 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod978d6d43_2346_476b_9bff_274d816f1175.slice/crio-7c4e7ef1010d3a880960c1237cc53e8f4ef9ba38a2d1df04147e2841a54231a2 WatchSource:0}: Error finding container 7c4e7ef1010d3a880960c1237cc53e8f4ef9ba38a2d1df04147e2841a54231a2: Status 404 returned error can't find the container with id 7c4e7ef1010d3a880960c1237cc53e8f4ef9ba38a2d1df04147e2841a54231a2 Sep 14 11:00:57.862302 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:57.862255 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-13 10:55:56 +0000 UTC" deadline="2028-05-26 07:28:29.690349684 +0000 UTC" Sep 14 11:00:57.862302 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:57.862294 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14876h27m31.82805935s" Sep 14 11:00:57.892572 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:57.892549 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2sk6p" Sep 14 11:00:57.892683 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:57.892666 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2sk6p" podUID="07adbd4b-794d-4f1e-b566-02280ae8390a" Sep 14 11:00:57.895841 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:57.895792 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-b5r9g" event={"ID":"5ce405d8-5bb3-453d-8c40-cf94ed8ac8a3","Type":"ContainerStarted","Data":"6ec67d39f98201c2790d89a95a2ce39e3cce8f96569e78744548896f0173cb7a"} Sep 14 11:00:57.896895 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:57.896871 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" event={"ID":"7ffda2ea-6f19-4043-a160-d116acf60812","Type":"ContainerStarted","Data":"5737e130d42d35dc4999a1fc5289daae287445f3272c81c439e4b57a8276c1e5"} Sep 14 11:00:57.897843 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:57.897824 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-89vlx" event={"ID":"807c9dd2-6b4e-4994-b996-6b86893e87c2","Type":"ContainerStarted","Data":"aaaa48f0c1fa2ce70120db646ff8234e4446aafbf668429118e26172d043f4c0"} Sep 14 11:00:57.898729 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:57.898711 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lcx2b" event={"ID":"c961b4ad-d3a5-4032-aa9a-a96c167e3df8","Type":"ContainerStarted","Data":"73275ca63dd39f6a3a049c7f03a84fe4e75bc0f4a160e7c241ad83595eb73662"} Sep 14 11:00:57.899746 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:57.899720 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jfdk9" event={"ID":"caa56301-04e7-4389-b0e8-b39bcc58b6ad","Type":"ContainerStarted","Data":"ad0cb7a0be79349626a776e85aa295ab00d802e842184bc37182c4467fbf0f96"} Sep 14 11:00:57.900666 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:57.900640 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sfnlb" event={"ID":"978d6d43-2346-476b-9bff-274d816f1175","Type":"ContainerStarted","Data":"7c4e7ef1010d3a880960c1237cc53e8f4ef9ba38a2d1df04147e2841a54231a2"} Sep 14 11:00:57.915335 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:57.915176 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3597ac5c_d3a0_42f6_ab6c_ad5cb887d70f.slice/crio-5b22ce6005faa535821a4180081f946f1b33a91bd172d05ba219cb3f978a8bd2 WatchSource:0}: Error finding container 5b22ce6005faa535821a4180081f946f1b33a91bd172d05ba219cb3f978a8bd2: Status 404 returned error can't find the container with id 5b22ce6005faa535821a4180081f946f1b33a91bd172d05ba219cb3f978a8bd2 Sep 14 11:00:57.955491 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:57.955470 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc8ab4f3c567cd8f16d2622c33e7fe29.slice/crio-ad3a3f8df923fd91294a81d6ed6b0944de3fb77d5935b39ae85e564c09d3e7f9 WatchSource:0}: Error finding container ad3a3f8df923fd91294a81d6ed6b0944de3fb77d5935b39ae85e564c09d3e7f9: Status 404 returned error can't find the container with id ad3a3f8df923fd91294a81d6ed6b0944de3fb77d5935b39ae85e564c09d3e7f9 Sep 14 11:00:58.132588 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:00:58.132552 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6be7bad72619c3572acbefbc29cd3390.slice/crio-7ebca3e510330a00acf4ad78c4f427aaf5cd253f6de3dd34e037743ef6a06331 WatchSource:0}: Error finding container 7ebca3e510330a00acf4ad78c4f427aaf5cd253f6de3dd34e037743ef6a06331: Status 404 returned error can't find the container with id 7ebca3e510330a00acf4ad78c4f427aaf5cd253f6de3dd34e037743ef6a06331 Sep 14 11:00:58.418156 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:58.418122 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfd621b6-51b0-4f2e-b1aa-ea198b732626-metrics-certs\") pod \"network-metrics-daemon-jkg6q\" (UID: \"dfd621b6-51b0-4f2e-b1aa-ea198b732626\") " pod="openshift-multus/network-metrics-daemon-jkg6q" Sep 14 11:00:58.418360 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:58.418259 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 11:00:58.418476 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:58.418378 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfd621b6-51b0-4f2e-b1aa-ea198b732626-metrics-certs podName:dfd621b6-51b0-4f2e-b1aa-ea198b732626 nodeName:}" failed. No retries permitted until 2026-09-14 11:01:00.418359977 +0000 UTC m=+5.038572481 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfd621b6-51b0-4f2e-b1aa-ea198b732626-metrics-certs") pod "network-metrics-daemon-jkg6q" (UID: "dfd621b6-51b0-4f2e-b1aa-ea198b732626") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 11:00:58.518546 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:58.518512 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07adbd4b-794d-4f1e-b566-02280ae8390a-original-pull-secret\") pod \"global-pull-secret-syncer-2sk6p\" (UID: \"07adbd4b-794d-4f1e-b566-02280ae8390a\") " pod="kube-system/global-pull-secret-syncer-2sk6p" Sep 14 11:00:58.518715 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:58.518704 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 11:00:58.518777 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:58.518772 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07adbd4b-794d-4f1e-b566-02280ae8390a-original-pull-secret podName:07adbd4b-794d-4f1e-b566-02280ae8390a nodeName:}" failed. No retries permitted until 2026-09-14 11:01:00.518744347 +0000 UTC m=+5.138956861 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/07adbd4b-794d-4f1e-b566-02280ae8390a-original-pull-secret") pod "global-pull-secret-syncer-2sk6p" (UID: "07adbd4b-794d-4f1e-b566-02280ae8390a") : object "kube-system"/"original-pull-secret" not registered Sep 14 11:00:58.619419 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:58.619386 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdc5t\" (UniqueName: \"kubernetes.io/projected/1bb6499d-6175-4cc7-83de-a6dd85437535-kube-api-access-hdc5t\") pod \"network-check-target-t6jcs\" (UID: \"1bb6499d-6175-4cc7-83de-a6dd85437535\") " pod="openshift-network-diagnostics/network-check-target-t6jcs" Sep 14 11:00:58.619679 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:58.619661 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 11:00:58.619754 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:58.619696 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 11:00:58.619754 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:58.619709 2574 projected.go:196] Error preparing data for projected volume kube-api-access-hdc5t for pod openshift-network-diagnostics/network-check-target-t6jcs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 11:00:58.619857 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:58.619765 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bb6499d-6175-4cc7-83de-a6dd85437535-kube-api-access-hdc5t podName:1bb6499d-6175-4cc7-83de-a6dd85437535 nodeName:}" failed. No retries permitted until 2026-09-14 11:01:00.61974791 +0000 UTC m=+5.239960416 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hdc5t" (UniqueName: "kubernetes.io/projected/1bb6499d-6175-4cc7-83de-a6dd85437535-kube-api-access-hdc5t") pod "network-check-target-t6jcs" (UID: "1bb6499d-6175-4cc7-83de-a6dd85437535") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 11:00:58.862487 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:58.862447 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-13 10:55:56 +0000 UTC" deadline="2028-03-18 03:59:55.834326526 +0000 UTC" Sep 14 11:00:58.862487 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:58.862485 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13216h58m56.971846034s" Sep 14 11:00:58.892905 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:58.892876 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jkg6q" Sep 14 11:00:58.893071 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:58.893018 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jkg6q" podUID="dfd621b6-51b0-4f2e-b1aa-ea198b732626" Sep 14 11:00:58.893446 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:58.893425 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6jcs" Sep 14 11:00:58.893551 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:58.893518 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6jcs" podUID="1bb6499d-6175-4cc7-83de-a6dd85437535" Sep 14 11:00:58.912109 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:58.912012 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-230.ec2.internal" event={"ID":"6be7bad72619c3572acbefbc29cd3390","Type":"ContainerStarted","Data":"7ebca3e510330a00acf4ad78c4f427aaf5cd253f6de3dd34e037743ef6a06331"} Sep 14 11:00:58.924987 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:58.919558 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-230.ec2.internal" event={"ID":"fc8ab4f3c567cd8f16d2622c33e7fe29","Type":"ContainerStarted","Data":"ad3a3f8df923fd91294a81d6ed6b0944de3fb77d5935b39ae85e564c09d3e7f9"} Sep 14 11:00:58.926489 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:58.926464 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xpv2v" event={"ID":"3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f","Type":"ContainerStarted","Data":"5b22ce6005faa535821a4180081f946f1b33a91bd172d05ba219cb3f978a8bd2"} Sep 14 11:00:59.894971 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:59.894940 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2sk6p" Sep 14 11:00:59.895429 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:00:59.895066 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2sk6p" podUID="07adbd4b-794d-4f1e-b566-02280ae8390a" Sep 14 11:00:59.932528 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:59.932491 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hkdj4" event={"ID":"24a7349e-c9f6-4234-8e4f-effab7979852","Type":"ContainerStarted","Data":"5986331c9891e3aea4ddfe4f1ef69af8181e79ebbbcdbdc25b1b74e48032d030"} Sep 14 11:00:59.934930 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:00:59.934901 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mwsc" event={"ID":"99ab5d24-81bd-4f38-8812-bc1794de173a","Type":"ContainerStarted","Data":"2809788c4341acc14445f4e3e2772ea892479f8962db58a7e01eaea1324887a6"} Sep 14 11:01:00.434755 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:00.434724 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfd621b6-51b0-4f2e-b1aa-ea198b732626-metrics-certs\") pod \"network-metrics-daemon-jkg6q\" (UID: \"dfd621b6-51b0-4f2e-b1aa-ea198b732626\") " pod="openshift-multus/network-metrics-daemon-jkg6q" Sep 14 11:01:00.434954 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:00.434934 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 11:01:00.435017 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:00.435004 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfd621b6-51b0-4f2e-b1aa-ea198b732626-metrics-certs podName:dfd621b6-51b0-4f2e-b1aa-ea198b732626 nodeName:}" failed. No retries permitted until 2026-09-14 11:01:04.434986611 +0000 UTC m=+9.055199132 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfd621b6-51b0-4f2e-b1aa-ea198b732626-metrics-certs") pod "network-metrics-daemon-jkg6q" (UID: "dfd621b6-51b0-4f2e-b1aa-ea198b732626") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 11:01:00.535696 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:00.535660 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07adbd4b-794d-4f1e-b566-02280ae8390a-original-pull-secret\") pod \"global-pull-secret-syncer-2sk6p\" (UID: \"07adbd4b-794d-4f1e-b566-02280ae8390a\") " pod="kube-system/global-pull-secret-syncer-2sk6p" Sep 14 11:01:00.535868 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:00.535814 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 11:01:00.535937 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:00.535871 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07adbd4b-794d-4f1e-b566-02280ae8390a-original-pull-secret podName:07adbd4b-794d-4f1e-b566-02280ae8390a nodeName:}" failed. No retries permitted until 2026-09-14 11:01:04.535854442 +0000 UTC m=+9.156066949 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/07adbd4b-794d-4f1e-b566-02280ae8390a-original-pull-secret") pod "global-pull-secret-syncer-2sk6p" (UID: "07adbd4b-794d-4f1e-b566-02280ae8390a") : object "kube-system"/"original-pull-secret" not registered Sep 14 11:01:00.636068 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:00.636031 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdc5t\" (UniqueName: \"kubernetes.io/projected/1bb6499d-6175-4cc7-83de-a6dd85437535-kube-api-access-hdc5t\") pod \"network-check-target-t6jcs\" (UID: \"1bb6499d-6175-4cc7-83de-a6dd85437535\") " pod="openshift-network-diagnostics/network-check-target-t6jcs" Sep 14 11:01:00.636228 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:00.636205 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 11:01:00.636228 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:00.636219 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 11:01:00.636228 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:00.636228 2574 projected.go:196] Error preparing data for projected volume kube-api-access-hdc5t for pod openshift-network-diagnostics/network-check-target-t6jcs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 11:01:00.636361 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:00.636301 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bb6499d-6175-4cc7-83de-a6dd85437535-kube-api-access-hdc5t podName:1bb6499d-6175-4cc7-83de-a6dd85437535 nodeName:}" failed. No retries permitted until 2026-09-14 11:01:04.636276666 +0000 UTC m=+9.256489179 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hdc5t" (UniqueName: "kubernetes.io/projected/1bb6499d-6175-4cc7-83de-a6dd85437535-kube-api-access-hdc5t") pod "network-check-target-t6jcs" (UID: "1bb6499d-6175-4cc7-83de-a6dd85437535") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 11:01:00.893297 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:00.893203 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6jcs" Sep 14 11:01:00.893457 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:00.893339 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6jcs" podUID="1bb6499d-6175-4cc7-83de-a6dd85437535" Sep 14 11:01:00.893876 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:00.893711 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jkg6q" Sep 14 11:01:00.893876 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:00.893821 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jkg6q" podUID="dfd621b6-51b0-4f2e-b1aa-ea198b732626" Sep 14 11:01:01.895287 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:01.893223 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2sk6p" Sep 14 11:01:01.895287 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:01.893400 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2sk6p" podUID="07adbd4b-794d-4f1e-b566-02280ae8390a" Sep 14 11:01:02.893078 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:02.892478 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jkg6q" Sep 14 11:01:02.893078 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:02.892617 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jkg6q" podUID="dfd621b6-51b0-4f2e-b1aa-ea198b732626" Sep 14 11:01:02.893078 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:02.892972 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6jcs" Sep 14 11:01:02.893078 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:02.893049 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6jcs" podUID="1bb6499d-6175-4cc7-83de-a6dd85437535" Sep 14 11:01:03.893173 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:03.892707 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2sk6p" Sep 14 11:01:03.893173 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:03.892833 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2sk6p" podUID="07adbd4b-794d-4f1e-b566-02280ae8390a" Sep 14 11:01:04.474213 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:04.473667 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfd621b6-51b0-4f2e-b1aa-ea198b732626-metrics-certs\") pod \"network-metrics-daemon-jkg6q\" (UID: \"dfd621b6-51b0-4f2e-b1aa-ea198b732626\") " pod="openshift-multus/network-metrics-daemon-jkg6q" Sep 14 11:01:04.474213 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:04.473828 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 11:01:04.474213 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:04.473884 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfd621b6-51b0-4f2e-b1aa-ea198b732626-metrics-certs podName:dfd621b6-51b0-4f2e-b1aa-ea198b732626 nodeName:}" failed. No retries permitted until 2026-09-14 11:01:12.473870563 +0000 UTC m=+17.094083085 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfd621b6-51b0-4f2e-b1aa-ea198b732626-metrics-certs") pod "network-metrics-daemon-jkg6q" (UID: "dfd621b6-51b0-4f2e-b1aa-ea198b732626") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 11:01:04.574518 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:04.574482 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07adbd4b-794d-4f1e-b566-02280ae8390a-original-pull-secret\") pod \"global-pull-secret-syncer-2sk6p\" (UID: \"07adbd4b-794d-4f1e-b566-02280ae8390a\") " pod="kube-system/global-pull-secret-syncer-2sk6p" Sep 14 11:01:04.574708 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:04.574624 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 11:01:04.574708 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:04.574699 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07adbd4b-794d-4f1e-b566-02280ae8390a-original-pull-secret podName:07adbd4b-794d-4f1e-b566-02280ae8390a nodeName:}" failed. No retries permitted until 2026-09-14 11:01:12.574677976 +0000 UTC m=+17.194890494 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/07adbd4b-794d-4f1e-b566-02280ae8390a-original-pull-secret") pod "global-pull-secret-syncer-2sk6p" (UID: "07adbd4b-794d-4f1e-b566-02280ae8390a") : object "kube-system"/"original-pull-secret" not registered Sep 14 11:01:04.675752 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:04.675708 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdc5t\" (UniqueName: \"kubernetes.io/projected/1bb6499d-6175-4cc7-83de-a6dd85437535-kube-api-access-hdc5t\") pod \"network-check-target-t6jcs\" (UID: \"1bb6499d-6175-4cc7-83de-a6dd85437535\") " pod="openshift-network-diagnostics/network-check-target-t6jcs" Sep 14 11:01:04.676396 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:04.675908 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 11:01:04.676396 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:04.675936 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 11:01:04.676396 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:04.675951 2574 projected.go:196] Error preparing data for projected volume kube-api-access-hdc5t for pod openshift-network-diagnostics/network-check-target-t6jcs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 11:01:04.676396 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:04.676014 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bb6499d-6175-4cc7-83de-a6dd85437535-kube-api-access-hdc5t podName:1bb6499d-6175-4cc7-83de-a6dd85437535 nodeName:}" failed. No retries permitted until 2026-09-14 11:01:12.675994389 +0000 UTC m=+17.296206906 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hdc5t" (UniqueName: "kubernetes.io/projected/1bb6499d-6175-4cc7-83de-a6dd85437535-kube-api-access-hdc5t") pod "network-check-target-t6jcs" (UID: "1bb6499d-6175-4cc7-83de-a6dd85437535") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 11:01:04.893145 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:04.893112 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6jcs" Sep 14 11:01:04.893380 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:04.893242 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6jcs" podUID="1bb6499d-6175-4cc7-83de-a6dd85437535" Sep 14 11:01:04.893893 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:04.893673 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jkg6q" Sep 14 11:01:04.893893 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:04.893782 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jkg6q" podUID="dfd621b6-51b0-4f2e-b1aa-ea198b732626" Sep 14 11:01:05.893827 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:05.893431 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2sk6p" Sep 14 11:01:05.893827 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:05.893557 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2sk6p" podUID="07adbd4b-794d-4f1e-b566-02280ae8390a" Sep 14 11:01:06.893155 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:06.893126 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6jcs" Sep 14 11:01:06.893370 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:06.893135 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jkg6q" Sep 14 11:01:06.893370 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:06.893238 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6jcs" podUID="1bb6499d-6175-4cc7-83de-a6dd85437535" Sep 14 11:01:06.893370 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:06.893360 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jkg6q" podUID="dfd621b6-51b0-4f2e-b1aa-ea198b732626" Sep 14 11:01:07.893124 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:07.892612 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2sk6p" Sep 14 11:01:07.893124 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:07.892746 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2sk6p" podUID="07adbd4b-794d-4f1e-b566-02280ae8390a" Sep 14 11:01:08.893058 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:08.892989 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6jcs" Sep 14 11:01:08.893058 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:08.893035 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jkg6q" Sep 14 11:01:08.893483 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:08.893107 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6jcs" podUID="1bb6499d-6175-4cc7-83de-a6dd85437535" Sep 14 11:01:08.893483 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:08.893231 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jkg6q" podUID="dfd621b6-51b0-4f2e-b1aa-ea198b732626" Sep 14 11:01:09.892614 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:09.892581 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2sk6p" Sep 14 11:01:09.892795 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:09.892701 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2sk6p" podUID="07adbd4b-794d-4f1e-b566-02280ae8390a" Sep 14 11:01:10.892287 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:10.892244 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jkg6q" Sep 14 11:01:10.892744 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:10.892242 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6jcs" Sep 14 11:01:10.892744 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:10.892378 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jkg6q" podUID="dfd621b6-51b0-4f2e-b1aa-ea198b732626" Sep 14 11:01:10.892744 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:10.892421 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6jcs" podUID="1bb6499d-6175-4cc7-83de-a6dd85437535" Sep 14 11:01:11.893155 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:11.893119 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2sk6p" Sep 14 11:01:11.893615 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:11.893240 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2sk6p" podUID="07adbd4b-794d-4f1e-b566-02280ae8390a" Sep 14 11:01:12.535435 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:12.535405 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfd621b6-51b0-4f2e-b1aa-ea198b732626-metrics-certs\") pod \"network-metrics-daemon-jkg6q\" (UID: \"dfd621b6-51b0-4f2e-b1aa-ea198b732626\") " pod="openshift-multus/network-metrics-daemon-jkg6q" Sep 14 11:01:12.535597 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:12.535527 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 11:01:12.535597 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:12.535577 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfd621b6-51b0-4f2e-b1aa-ea198b732626-metrics-certs podName:dfd621b6-51b0-4f2e-b1aa-ea198b732626 nodeName:}" failed. No retries permitted until 2026-09-14 11:01:28.535563428 +0000 UTC m=+33.155775933 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfd621b6-51b0-4f2e-b1aa-ea198b732626-metrics-certs") pod "network-metrics-daemon-jkg6q" (UID: "dfd621b6-51b0-4f2e-b1aa-ea198b732626") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 11:01:12.636557 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:12.636523 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07adbd4b-794d-4f1e-b566-02280ae8390a-original-pull-secret\") pod \"global-pull-secret-syncer-2sk6p\" (UID: \"07adbd4b-794d-4f1e-b566-02280ae8390a\") " pod="kube-system/global-pull-secret-syncer-2sk6p" Sep 14 11:01:12.636720 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:12.636683 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 11:01:12.636766 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:12.636749 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07adbd4b-794d-4f1e-b566-02280ae8390a-original-pull-secret podName:07adbd4b-794d-4f1e-b566-02280ae8390a nodeName:}" failed. No retries permitted until 2026-09-14 11:01:28.636734583 +0000 UTC m=+33.256947097 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/07adbd4b-794d-4f1e-b566-02280ae8390a-original-pull-secret") pod "global-pull-secret-syncer-2sk6p" (UID: "07adbd4b-794d-4f1e-b566-02280ae8390a") : object "kube-system"/"original-pull-secret" not registered Sep 14 11:01:12.737648 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:12.737619 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdc5t\" (UniqueName: \"kubernetes.io/projected/1bb6499d-6175-4cc7-83de-a6dd85437535-kube-api-access-hdc5t\") pod \"network-check-target-t6jcs\" (UID: \"1bb6499d-6175-4cc7-83de-a6dd85437535\") " pod="openshift-network-diagnostics/network-check-target-t6jcs" Sep 14 11:01:12.737825 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:12.737793 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 11:01:12.737825 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:12.737820 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 11:01:12.737958 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:12.737833 2574 projected.go:196] Error preparing data for projected volume kube-api-access-hdc5t for pod openshift-network-diagnostics/network-check-target-t6jcs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 11:01:12.737958 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:12.737899 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bb6499d-6175-4cc7-83de-a6dd85437535-kube-api-access-hdc5t podName:1bb6499d-6175-4cc7-83de-a6dd85437535 nodeName:}" failed. No retries permitted until 2026-09-14 11:01:28.737880564 +0000 UTC m=+33.358093085 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hdc5t" (UniqueName: "kubernetes.io/projected/1bb6499d-6175-4cc7-83de-a6dd85437535-kube-api-access-hdc5t") pod "network-check-target-t6jcs" (UID: "1bb6499d-6175-4cc7-83de-a6dd85437535") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 11:01:12.893358 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:12.893287 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jkg6q" Sep 14 11:01:12.893729 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:12.893287 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6jcs" Sep 14 11:01:12.893729 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:12.893434 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jkg6q" podUID="dfd621b6-51b0-4f2e-b1aa-ea198b732626" Sep 14 11:01:12.893729 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:12.893506 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6jcs" podUID="1bb6499d-6175-4cc7-83de-a6dd85437535" Sep 14 11:01:13.892923 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:13.892889 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2sk6p" Sep 14 11:01:13.893098 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:13.893017 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2sk6p" podUID="07adbd4b-794d-4f1e-b566-02280ae8390a" Sep 14 11:01:14.893317 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:14.893281 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jkg6q" Sep 14 11:01:14.893679 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:14.893416 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jkg6q" podUID="dfd621b6-51b0-4f2e-b1aa-ea198b732626" Sep 14 11:01:14.893679 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:14.893474 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6jcs" Sep 14 11:01:14.893679 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:14.893563 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6jcs" podUID="1bb6499d-6175-4cc7-83de-a6dd85437535" Sep 14 11:01:15.893699 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:15.893663 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2sk6p" Sep 14 11:01:15.894075 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:15.893775 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2sk6p" podUID="07adbd4b-794d-4f1e-b566-02280ae8390a" Sep 14 11:01:16.893089 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:16.893065 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6jcs" Sep 14 11:01:16.893172 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:16.893156 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6jcs" podUID="1bb6499d-6175-4cc7-83de-a6dd85437535" Sep 14 11:01:16.893234 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:16.893218 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jkg6q" Sep 14 11:01:16.893341 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:16.893325 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jkg6q" podUID="dfd621b6-51b0-4f2e-b1aa-ea198b732626" Sep 14 11:01:17.892653 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:17.892338 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2sk6p" Sep 14 11:01:17.893494 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:17.892744 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2sk6p" podUID="07adbd4b-794d-4f1e-b566-02280ae8390a" Sep 14 11:01:17.973963 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:17.973916 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sfnlb" event={"ID":"978d6d43-2346-476b-9bff-274d816f1175","Type":"ContainerStarted","Data":"1bc6fa5b457c71dbe9092b9d7daf5d505619632c5eaaefe8c7697e0066094fcb"} Sep 14 11:01:17.977418 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:17.977391 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" event={"ID":"7ffda2ea-6f19-4043-a160-d116acf60812","Type":"ContainerStarted","Data":"85bd15e483a0003ffdbdf9a527f5f5d7c9d15953106179b29f9b29a753bf3635"} Sep 14 11:01:17.977551 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:17.977468 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" event={"ID":"7ffda2ea-6f19-4043-a160-d116acf60812","Type":"ContainerStarted","Data":"7a413282438a5d54aaf3611163d290fb65d45454e430a82b46a4b605b0f8f913"} Sep 14 11:01:17.977551 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:17.977520 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" event={"ID":"7ffda2ea-6f19-4043-a160-d116acf60812","Type":"ContainerStarted","Data":"c8d62d864d9ec78733c4310a79156e17cea8bc32e525dbe7c266e23d2a872688"} Sep 14 11:01:17.977665 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:17.977558 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" event={"ID":"7ffda2ea-6f19-4043-a160-d116acf60812","Type":"ContainerStarted","Data":"07ad5ab678c4fa95ec1dd75e2a4910d6e0a3c16a54b14429b1f983cbcc233503"} Sep 14 11:01:17.977665 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:17.977589 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" event={"ID":"7ffda2ea-6f19-4043-a160-d116acf60812","Type":"ContainerStarted","Data":"6f2f2edde73cba3c0884d41abdb9d44a7b9add6c3193eeb0552fa5167ce9c21d"} Sep 14 11:01:17.977665 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:17.977631 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" event={"ID":"7ffda2ea-6f19-4043-a160-d116acf60812","Type":"ContainerStarted","Data":"9218a7ec21943073c76ae87e85f1ae8a7db8c17ada5adc5f93ed844c10ff0b95"} Sep 14 11:01:17.981473 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:17.979645 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-89vlx" event={"ID":"807c9dd2-6b4e-4994-b996-6b86893e87c2","Type":"ContainerStarted","Data":"bebd5262fbb6b1e656db1e943171e6ca16b8688005577cbb748a0fa8055ab74f"} Sep 14 11:01:17.983330 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:17.983302 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lcx2b" event={"ID":"c961b4ad-d3a5-4032-aa9a-a96c167e3df8","Type":"ContainerStarted","Data":"9f27a5cde052328f87ba5ff9e8717edbfec31b4cef3c48cc0e5e9bf217f10c9a"} Sep 14 11:01:17.984914 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:17.984889 2574 generic.go:356] "Generic (PLEG): container finished" podID="6be7bad72619c3572acbefbc29cd3390" containerID="a9c237b5995a248bcf2dd8654206d89b551bd4df78cb1dcc6c1413e1d06eb583" exitCode=0 Sep 14 11:01:17.985006 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:17.984971 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-230.ec2.internal" event={"ID":"6be7bad72619c3572acbefbc29cd3390","Type":"ContainerDied","Data":"a9c237b5995a248bcf2dd8654206d89b551bd4df78cb1dcc6c1413e1d06eb583"} Sep 14 11:01:17.986644 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:17.986614 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jfdk9" event={"ID":"caa56301-04e7-4389-b0e8-b39bcc58b6ad","Type":"ContainerStarted","Data":"48ca4eb24c3843f70c1b9a92b26d4c87ee63db782cf8ea494610340a5522076d"} Sep 14 11:01:17.988149 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:17.988117 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-230.ec2.internal" event={"ID":"fc8ab4f3c567cd8f16d2622c33e7fe29","Type":"ContainerStarted","Data":"4015bf160da99ca452bc707824a26bdb01696700272130fb194a4a5f423d84cb"} Sep 14 11:01:17.989668 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:17.989649 2574 generic.go:356] "Generic (PLEG): container finished" podID="3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f" containerID="8f1e83675a733f9f51ccf62034d27eb5b353de8c71d94efdb76754937534fe76" exitCode=0 Sep 14 11:01:17.989774 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:17.989675 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xpv2v" event={"ID":"3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f","Type":"ContainerDied","Data":"8f1e83675a733f9f51ccf62034d27eb5b353de8c71d94efdb76754937534fe76"} Sep 14 11:01:17.991414 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:17.991391 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hkdj4" event={"ID":"24a7349e-c9f6-4234-8e4f-effab7979852","Type":"ContainerStarted","Data":"b59ba55f601f374dd701a066ccb64e76d87e225a1ddd2823c8678bf7ce908565"} Sep 14 11:01:17.992585 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:17.992543 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-sfnlb" podStartSLOduration=3.054116525 podStartE2EDuration="21.992532454s" totalImagesPullingTime="18.938415929s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:01:17.992488777 +0000 UTC m=+22.612701301" watchObservedRunningTime="2026-09-14 11:01:17.992532454 +0000 UTC m=+22.612744976" Sep 14 11:01:17.993039 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:17.993019 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mwsc" event={"ID":"99ab5d24-81bd-4f38-8812-bc1794de173a","Type":"ContainerStarted","Data":"d958c60c2d8c03b7c90006fdc432ec4ab7eadb4b18fee0de45350a57e500f29f"} Sep 14 11:01:18.023135 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:18.023086 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-230.ec2.internal" podStartSLOduration=22.023071396 podStartE2EDuration="22.023071396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:01:18.022781762 +0000 UTC m=+22.642994315" watchObservedRunningTime="2026-09-14 11:01:18.023071396 +0000 UTC m=+22.643283919" Sep 14 11:01:18.063129 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:18.063085 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-89vlx" podStartSLOduration=2.995509038 podStartE2EDuration="22.063066063s" totalImagesPullingTime="19.067557025s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:01:18.043470722 +0000 UTC m=+22.663683245" watchObservedRunningTime="2026-09-14 11:01:18.063066063 +0000 UTC m=+22.683278588" Sep 14 11:01:18.063752 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:18.063720 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-hkdj4" podStartSLOduration=4.488672643 podStartE2EDuration="22.063710503s" totalImagesPullingTime="17.57503786s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:01:18.06315694 +0000 UTC m=+22.683369484" watchObservedRunningTime="2026-09-14 11:01:18.063710503 +0000 UTC m=+22.683923037" Sep 14 11:01:18.105081 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:18.105042 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-lcx2b" podStartSLOduration=4.015343382 podStartE2EDuration="23.105028406s" totalImagesPullingTime="19.089685024s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:00:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:01:18.104526214 +0000 UTC m=+22.724738738" watchObservedRunningTime="2026-09-14 11:01:18.105028406 +0000 UTC m=+22.725240928" Sep 14 11:01:18.184635 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:18.184598 2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 14 11:01:18.869072 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:18.868987 2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-14T11:01:18.184611309Z","UUID":"0ebf3852-a2ab-4582-8b57-7303ad5490c9","Handler":null,"Name":"","Endpoint":""} Sep 14 11:01:18.870354 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:18.870334 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 14 11:01:18.870472 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:18.870363 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 14 11:01:18.893086 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:18.893060 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jkg6q" Sep 14 11:01:18.893086 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:18.893081 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6jcs" Sep 14 11:01:18.893622 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:18.893150 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jkg6q" podUID="dfd621b6-51b0-4f2e-b1aa-ea198b732626" Sep 14 11:01:18.893622 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:18.893200 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6jcs" podUID="1bb6499d-6175-4cc7-83de-a6dd85437535" Sep 14 11:01:18.996394 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:18.996367 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-230.ec2.internal" event={"ID":"6be7bad72619c3572acbefbc29cd3390","Type":"ContainerStarted","Data":"9452ebc56d435f701532cc24253c531b692ab24a40413c70715f15fc4b2f32af"} Sep 14 11:01:18.997795 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:18.997769 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-b5r9g" event={"ID":"5ce405d8-5bb3-453d-8c40-cf94ed8ac8a3","Type":"ContainerStarted","Data":"035a313a10e4e3c136689664bdb0863786bcf4a2bc990725764ad681e614a3ff"} Sep 14 11:01:18.999347 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:18.999318 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mwsc" event={"ID":"99ab5d24-81bd-4f38-8812-bc1794de173a","Type":"ContainerStarted","Data":"ce36927830393f2cc5b9bde6cd7df916f84f4797e38e3a6e678b5330ceed6d43"} Sep 14 11:01:19.017091 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:19.017042 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-230.ec2.internal" podStartSLOduration=23.017028144 podStartE2EDuration="23.017028144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:01:19.015947513 +0000 UTC m=+23.636160036" watchObservedRunningTime="2026-09-14 11:01:19.017028144 +0000 UTC m=+23.637240662" Sep 14 11:01:19.017972 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:19.017934 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jfdk9" podStartSLOduration=11.519776589 podStartE2EDuration="23.017918611s" totalImagesPullingTime="11.498142022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:01:18.127616564 +0000 UTC m=+22.747829113" watchObservedRunningTime="2026-09-14 11:01:19.017918611 +0000 UTC m=+23.638131136" Sep 14 11:01:19.892583 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:19.892411 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2sk6p" Sep 14 11:01:19.892718 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:19.892696 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2sk6p" podUID="07adbd4b-794d-4f1e-b566-02280ae8390a" Sep 14 11:01:20.008637 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:20.008598 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" event={"ID":"7ffda2ea-6f19-4043-a160-d116acf60812","Type":"ContainerStarted","Data":"3918c42d5d0d4ca6a29de5621d3001c5950b920afbd4b75ff4ba0939bcb0e85d"} Sep 14 11:01:20.892790 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:20.892762 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jkg6q" Sep 14 11:01:20.893019 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:20.892765 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6jcs" Sep 14 11:01:20.893019 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:20.892896 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jkg6q" podUID="dfd621b6-51b0-4f2e-b1aa-ea198b732626" Sep 14 11:01:20.893019 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:20.892952 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6jcs" podUID="1bb6499d-6175-4cc7-83de-a6dd85437535" Sep 14 11:01:21.013600 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:21.013565 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mwsc" event={"ID":"99ab5d24-81bd-4f38-8812-bc1794de173a","Type":"ContainerStarted","Data":"fb67ecaf63340ed18a7c98c2596b291c578974ea5c4a92682f0272d0e8c3b49b"} Sep 14 11:01:21.033603 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:21.033556 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-b5r9g" podStartSLOduration=6.072297557 podStartE2EDuration="25.033539648s" totalImagesPullingTime="18.961242091s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:01:19.035479273 +0000 UTC m=+23.655691796" watchObservedRunningTime="2026-09-14 11:01:21.033539648 +0000 UTC m=+25.653752170" Sep 14 11:01:21.033813 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:21.033793 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7mwsc" podStartSLOduration=4.69250157 podStartE2EDuration="25.033786551s" totalImagesPullingTime="20.341284981s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:00:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:01:21.033130709 +0000 UTC m=+25.653343251" watchObservedRunningTime="2026-09-14 11:01:21.033786551 +0000 UTC m=+25.653999074" Sep 14 11:01:21.219128 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:21.219098 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jfdk9" Sep 14 11:01:21.219677 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:21.219651 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jfdk9" Sep 14 11:01:21.892326 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:21.892298 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2sk6p" Sep 14 11:01:21.892492 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:21.892412 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2sk6p" podUID="07adbd4b-794d-4f1e-b566-02280ae8390a" Sep 14 11:01:22.016544 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:22.016519 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jfdk9" Sep 14 11:01:22.016996 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:22.016978 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jfdk9" Sep 14 11:01:22.892590 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:22.892565 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6jcs" Sep 14 11:01:22.892738 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:22.892570 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jkg6q" Sep 14 11:01:22.892799 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:22.892654 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6jcs" podUID="1bb6499d-6175-4cc7-83de-a6dd85437535" Sep 14 11:01:22.892799 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:22.892734 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jkg6q" podUID="dfd621b6-51b0-4f2e-b1aa-ea198b732626" Sep 14 11:01:23.892630 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:23.892454 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2sk6p" Sep 14 11:01:23.893089 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:23.892719 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2sk6p" podUID="07adbd4b-794d-4f1e-b566-02280ae8390a" Sep 14 11:01:24.023889 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:24.023855 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" event={"ID":"7ffda2ea-6f19-4043-a160-d116acf60812","Type":"ContainerStarted","Data":"3fb7298247f08b40af536bfadc0d09982792df18536a05385265c8656272ec67"} Sep 14 11:01:24.024136 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:24.024117 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:01:24.025781 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:24.025755 2574 generic.go:356] "Generic (PLEG): container finished" podID="3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f" containerID="88f8c9ac9a20b0efafc7db8299ea72d2eea5d6948dfca8059e2b824c5d57450c" exitCode=0 Sep 14 11:01:24.025886 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:24.025834 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xpv2v" event={"ID":"3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f","Type":"ContainerDied","Data":"88f8c9ac9a20b0efafc7db8299ea72d2eea5d6948dfca8059e2b824c5d57450c"} Sep 14 11:01:24.040489 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:24.040472 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:01:24.063471 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:24.063434 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" podStartSLOduration=8.811531771 podStartE2EDuration="28.063421184s" totalImagesPullingTime="19.251889413s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:01:24.063124108 +0000 UTC m=+28.683336630" watchObservedRunningTime="2026-09-14 11:01:24.063421184 +0000 UTC m=+28.683633706" Sep 14 11:01:24.893237 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:24.893207 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jkg6q" Sep 14 11:01:24.893855 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:24.893207 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6jcs" Sep 14 11:01:24.893855 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:24.893338 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jkg6q" podUID="dfd621b6-51b0-4f2e-b1aa-ea198b732626" Sep 14 11:01:24.893855 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:24.893381 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6jcs" podUID="1bb6499d-6175-4cc7-83de-a6dd85437535" Sep 14 11:01:25.012437 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:25.012411 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jkg6q"] Sep 14 11:01:25.014488 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:25.014463 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t6jcs"] Sep 14 11:01:25.016774 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:25.016757 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2sk6p"] Sep 14 11:01:25.016897 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:25.016863 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2sk6p" Sep 14 11:01:25.016982 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:25.016964 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2sk6p" podUID="07adbd4b-794d-4f1e-b566-02280ae8390a" Sep 14 11:01:25.027762 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:25.027684 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6jcs" Sep 14 11:01:25.027873 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:25.027765 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6jcs" podUID="1bb6499d-6175-4cc7-83de-a6dd85437535" Sep 14 11:01:25.027873 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:25.027692 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jkg6q" Sep 14 11:01:25.028172 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:25.028141 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:01:25.028172 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:25.028142 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jkg6q" podUID="dfd621b6-51b0-4f2e-b1aa-ea198b732626" Sep 14 11:01:25.028346 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:25.028209 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:01:25.045870 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:25.045853 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:01:26.892859 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:26.892604 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6jcs" Sep 14 11:01:26.893297 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:26.892671 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jkg6q" Sep 14 11:01:26.893297 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:26.892967 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6jcs" podUID="1bb6499d-6175-4cc7-83de-a6dd85437535" Sep 14 11:01:26.893297 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:26.892671 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2sk6p" Sep 14 11:01:26.893297 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:26.893014 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jkg6q" podUID="dfd621b6-51b0-4f2e-b1aa-ea198b732626" Sep 14 11:01:26.893297 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:26.893074 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2sk6p" podUID="07adbd4b-794d-4f1e-b566-02280ae8390a" Sep 14 11:01:27.034428 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:27.034363 2574 generic.go:356] "Generic (PLEG): container finished" podID="3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f" containerID="656c5eb1ed9d12ca215e30243078065e9b5638b4967cb39a81ff4c13a0dfa46c" exitCode=0 Sep 14 11:01:27.034548 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:27.034452 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xpv2v" event={"ID":"3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f","Type":"ContainerDied","Data":"656c5eb1ed9d12ca215e30243078065e9b5638b4967cb39a81ff4c13a0dfa46c"} Sep 14 11:01:28.556408 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:28.556377 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfd621b6-51b0-4f2e-b1aa-ea198b732626-metrics-certs\") pod \"network-metrics-daemon-jkg6q\" (UID: \"dfd621b6-51b0-4f2e-b1aa-ea198b732626\") " pod="openshift-multus/network-metrics-daemon-jkg6q" Sep 14 11:01:28.556727 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:28.556503 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 11:01:28.556727 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:28.556558 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfd621b6-51b0-4f2e-b1aa-ea198b732626-metrics-certs podName:dfd621b6-51b0-4f2e-b1aa-ea198b732626 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:00.556545592 +0000 UTC m=+65.176758097 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfd621b6-51b0-4f2e-b1aa-ea198b732626-metrics-certs") pod "network-metrics-daemon-jkg6q" (UID: "dfd621b6-51b0-4f2e-b1aa-ea198b732626") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 11:01:28.657339 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:28.657312 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07adbd4b-794d-4f1e-b566-02280ae8390a-original-pull-secret\") pod \"global-pull-secret-syncer-2sk6p\" (UID: \"07adbd4b-794d-4f1e-b566-02280ae8390a\") " pod="kube-system/global-pull-secret-syncer-2sk6p" Sep 14 11:01:28.657471 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:28.657424 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 11:01:28.657507 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:28.657480 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07adbd4b-794d-4f1e-b566-02280ae8390a-original-pull-secret podName:07adbd4b-794d-4f1e-b566-02280ae8390a nodeName:}" failed. No retries permitted until 2026-09-14 11:02:00.657465423 +0000 UTC m=+65.277677929 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/07adbd4b-794d-4f1e-b566-02280ae8390a-original-pull-secret") pod "global-pull-secret-syncer-2sk6p" (UID: "07adbd4b-794d-4f1e-b566-02280ae8390a") : object "kube-system"/"original-pull-secret" not registered Sep 14 11:01:28.758049 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:28.757976 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdc5t\" (UniqueName: \"kubernetes.io/projected/1bb6499d-6175-4cc7-83de-a6dd85437535-kube-api-access-hdc5t\") pod \"network-check-target-t6jcs\" (UID: \"1bb6499d-6175-4cc7-83de-a6dd85437535\") " pod="openshift-network-diagnostics/network-check-target-t6jcs" Sep 14 11:01:28.758178 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:28.758133 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 11:01:28.758178 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:28.758154 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 11:01:28.758178 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:28.758163 2574 projected.go:196] Error preparing data for projected volume kube-api-access-hdc5t for pod openshift-network-diagnostics/network-check-target-t6jcs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 11:01:28.758284 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:28.758208 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1bb6499d-6175-4cc7-83de-a6dd85437535-kube-api-access-hdc5t podName:1bb6499d-6175-4cc7-83de-a6dd85437535 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:00.758195346 +0000 UTC m=+65.378407847 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hdc5t" (UniqueName: "kubernetes.io/projected/1bb6499d-6175-4cc7-83de-a6dd85437535-kube-api-access-hdc5t") pod "network-check-target-t6jcs" (UID: "1bb6499d-6175-4cc7-83de-a6dd85437535") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 11:01:28.893232 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:28.893203 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2sk6p" Sep 14 11:01:28.893232 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:28.893224 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6jcs" Sep 14 11:01:28.893232 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:28.893232 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jkg6q" Sep 14 11:01:28.893441 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:28.893328 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2sk6p" podUID="07adbd4b-794d-4f1e-b566-02280ae8390a" Sep 14 11:01:28.893441 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:28.893431 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t6jcs" podUID="1bb6499d-6175-4cc7-83de-a6dd85437535" Sep 14 11:01:28.893512 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:28.893497 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jkg6q" podUID="dfd621b6-51b0-4f2e-b1aa-ea198b732626" Sep 14 11:01:29.041234 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:29.041174 2574 generic.go:356] "Generic (PLEG): container finished" podID="3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f" containerID="320345c72610c73cd7b5953a4cb2b7b349cdc3d658277a9337e8c194791d5e1d" exitCode=0 Sep 14 11:01:29.041234 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:29.041226 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xpv2v" event={"ID":"3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f","Type":"ContainerDied","Data":"320345c72610c73cd7b5953a4cb2b7b349cdc3d658277a9337e8c194791d5e1d"} Sep 14 11:01:30.701895 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:30.701856 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-230.ec2.internal" event="NodeReady" Sep 14 11:01:30.702373 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:30.701988 2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 14 11:01:30.749761 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:30.749730 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-md5h2"] Sep 14 11:01:30.770716 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:30.770677 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6qbbf"] Sep 14 11:01:30.771431 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:30.771408 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-md5h2" Sep 14 11:01:30.774170 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:30.773933 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 14 11:01:30.774170 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:30.773964 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 14 11:01:30.774170 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:30.773975 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zpbn2\"" Sep 14 11:01:30.786744 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:30.786727 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-md5h2"] Sep 14 11:01:30.786854 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:30.786759 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6qbbf"] Sep 14 11:01:30.786912 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:30.786868 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6qbbf" Sep 14 11:01:30.790100 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:30.789789 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qzhb6\"" Sep 14 11:01:30.790100 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:30.789884 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 14 11:01:30.790100 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:30.790002 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 14 11:01:30.790338 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:30.790137 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 14 11:01:30.872298 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:30.872247 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc3a2da2-96cd-4397-a6f9-9e7088b04f1e-metrics-tls\") pod \"dns-default-md5h2\" (UID: \"bc3a2da2-96cd-4397-a6f9-9e7088b04f1e\") " pod="openshift-dns/dns-default-md5h2" Sep 14 11:01:30.872420 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:30.872314 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hmv4\" (UniqueName: \"kubernetes.io/projected/4e9e0945-a3ff-449a-90a6-ddf38839b670-kube-api-access-6hmv4\") pod \"ingress-canary-6qbbf\" (UID: \"4e9e0945-a3ff-449a-90a6-ddf38839b670\") " pod="openshift-ingress-canary/ingress-canary-6qbbf" Sep 14 11:01:30.872420 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:30.872340 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc3a2da2-96cd-4397-a6f9-9e7088b04f1e-tmp-dir\") pod \"dns-default-md5h2\" (UID: \"bc3a2da2-96cd-4397-a6f9-9e7088b04f1e\") " pod="openshift-dns/dns-default-md5h2" Sep 14 11:01:30.872420 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:30.872364 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rckh\" (UniqueName: \"kubernetes.io/projected/bc3a2da2-96cd-4397-a6f9-9e7088b04f1e-kube-api-access-8rckh\") pod \"dns-default-md5h2\" (UID: \"bc3a2da2-96cd-4397-a6f9-9e7088b04f1e\") " pod="openshift-dns/dns-default-md5h2" Sep 14 11:01:30.872584 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:30.872422 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e9e0945-a3ff-449a-90a6-ddf38839b670-cert\") pod \"ingress-canary-6qbbf\" (UID: \"4e9e0945-a3ff-449a-90a6-ddf38839b670\") " pod="openshift-ingress-canary/ingress-canary-6qbbf" Sep 14 11:01:30.872584 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:30.872475 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bc3a2da2-96cd-4397-a6f9-9e7088b04f1e-config-volume\") pod \"dns-default-md5h2\" (UID: \"bc3a2da2-96cd-4397-a6f9-9e7088b04f1e\") " pod="openshift-dns/dns-default-md5h2" Sep 14 11:01:30.892741 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:30.892720 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6jcs" Sep 14 11:01:30.892865 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:30.892757 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2sk6p" Sep 14 11:01:30.892865 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:30.892818 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jkg6q" Sep 14 11:01:30.896248 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:30.896231 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 14 11:01:30.896379 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:30.896322 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mckdx\"" Sep 14 11:01:30.896379 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:30.896376 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-npv4p\"" Sep 14 11:01:30.896488 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:30.896444 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 14 11:01:30.896488 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:30.896445 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 14 11:01:30.896488 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:30.896478 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 14 11:01:30.973727 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:30.973661 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e9e0945-a3ff-449a-90a6-ddf38839b670-cert\") pod \"ingress-canary-6qbbf\" (UID: \"4e9e0945-a3ff-449a-90a6-ddf38839b670\") " pod="openshift-ingress-canary/ingress-canary-6qbbf" Sep 14 11:01:30.973871 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:30.973733 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bc3a2da2-96cd-4397-a6f9-9e7088b04f1e-config-volume\") pod \"dns-default-md5h2\" (UID: \"bc3a2da2-96cd-4397-a6f9-9e7088b04f1e\") " pod="openshift-dns/dns-default-md5h2" Sep 14 11:01:30.973871 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:30.973791 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc3a2da2-96cd-4397-a6f9-9e7088b04f1e-metrics-tls\") pod \"dns-default-md5h2\" (UID: \"bc3a2da2-96cd-4397-a6f9-9e7088b04f1e\") " pod="openshift-dns/dns-default-md5h2" Sep 14 11:01:30.973871 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:30.973819 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hmv4\" (UniqueName: \"kubernetes.io/projected/4e9e0945-a3ff-449a-90a6-ddf38839b670-kube-api-access-6hmv4\") pod \"ingress-canary-6qbbf\" (UID: \"4e9e0945-a3ff-449a-90a6-ddf38839b670\") " pod="openshift-ingress-canary/ingress-canary-6qbbf" Sep 14 11:01:30.973871 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:30.973841 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc3a2da2-96cd-4397-a6f9-9e7088b04f1e-tmp-dir\") pod \"dns-default-md5h2\" (UID: \"bc3a2da2-96cd-4397-a6f9-9e7088b04f1e\") " pod="openshift-dns/dns-default-md5h2" Sep 14 11:01:30.973871 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:30.973863 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rckh\" (UniqueName: \"kubernetes.io/projected/bc3a2da2-96cd-4397-a6f9-9e7088b04f1e-kube-api-access-8rckh\") pod \"dns-default-md5h2\" (UID: \"bc3a2da2-96cd-4397-a6f9-9e7088b04f1e\") " pod="openshift-dns/dns-default-md5h2" Sep 14 11:01:30.974110 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:30.973938 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 11:01:30.974110 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:30.973997 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc3a2da2-96cd-4397-a6f9-9e7088b04f1e-metrics-tls podName:bc3a2da2-96cd-4397-a6f9-9e7088b04f1e nodeName:}" failed. No retries permitted until 2026-09-14 11:01:31.473981507 +0000 UTC m=+36.094194012 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc3a2da2-96cd-4397-a6f9-9e7088b04f1e-metrics-tls") pod "dns-default-md5h2" (UID: "bc3a2da2-96cd-4397-a6f9-9e7088b04f1e") : secret "dns-default-metrics-tls" not found Sep 14 11:01:30.974371 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:30.974346 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bc3a2da2-96cd-4397-a6f9-9e7088b04f1e-config-volume\") pod \"dns-default-md5h2\" (UID: \"bc3a2da2-96cd-4397-a6f9-9e7088b04f1e\") " pod="openshift-dns/dns-default-md5h2" Sep 14 11:01:30.974460 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:30.974436 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc3a2da2-96cd-4397-a6f9-9e7088b04f1e-tmp-dir\") pod \"dns-default-md5h2\" (UID: \"bc3a2da2-96cd-4397-a6f9-9e7088b04f1e\") " pod="openshift-dns/dns-default-md5h2" Sep 14 11:01:30.978698 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:30.978679 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e9e0945-a3ff-449a-90a6-ddf38839b670-cert\") pod \"ingress-canary-6qbbf\" (UID: \"4e9e0945-a3ff-449a-90a6-ddf38839b670\") " pod="openshift-ingress-canary/ingress-canary-6qbbf" Sep 14 11:01:30.983077 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:30.983056 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hmv4\" (UniqueName: \"kubernetes.io/projected/4e9e0945-a3ff-449a-90a6-ddf38839b670-kube-api-access-6hmv4\") pod \"ingress-canary-6qbbf\" (UID: \"4e9e0945-a3ff-449a-90a6-ddf38839b670\") " pod="openshift-ingress-canary/ingress-canary-6qbbf" Sep 14 11:01:30.983241 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:30.983219 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rckh\" (UniqueName: \"kubernetes.io/projected/bc3a2da2-96cd-4397-a6f9-9e7088b04f1e-kube-api-access-8rckh\") pod \"dns-default-md5h2\" (UID: \"bc3a2da2-96cd-4397-a6f9-9e7088b04f1e\") " pod="openshift-dns/dns-default-md5h2" Sep 14 11:01:31.097295 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:31.097251 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6qbbf" Sep 14 11:01:31.433821 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:31.433632 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6qbbf"] Sep 14 11:01:31.438129 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:01:31.438104 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e9e0945_a3ff_449a_90a6_ddf38839b670.slice/crio-3280625711eed65e2e2cf82b3c7b1339cd9a322162c00b5364e34e0ddbad7b42 WatchSource:0}: Error finding container 3280625711eed65e2e2cf82b3c7b1339cd9a322162c00b5364e34e0ddbad7b42: Status 404 returned error can't find the container with id 3280625711eed65e2e2cf82b3c7b1339cd9a322162c00b5364e34e0ddbad7b42 Sep 14 11:01:31.479972 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:31.479940 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc3a2da2-96cd-4397-a6f9-9e7088b04f1e-metrics-tls\") pod \"dns-default-md5h2\" (UID: \"bc3a2da2-96cd-4397-a6f9-9e7088b04f1e\") " pod="openshift-dns/dns-default-md5h2" Sep 14 11:01:31.480126 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:31.480099 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 11:01:31.480245 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:31.480175 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc3a2da2-96cd-4397-a6f9-9e7088b04f1e-metrics-tls podName:bc3a2da2-96cd-4397-a6f9-9e7088b04f1e nodeName:}" failed. No retries permitted until 2026-09-14 11:01:32.480155518 +0000 UTC m=+37.100368023 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc3a2da2-96cd-4397-a6f9-9e7088b04f1e-metrics-tls") pod "dns-default-md5h2" (UID: "bc3a2da2-96cd-4397-a6f9-9e7088b04f1e") : secret "dns-default-metrics-tls" not found Sep 14 11:01:32.052753 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:32.052717 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6qbbf" event={"ID":"4e9e0945-a3ff-449a-90a6-ddf38839b670","Type":"ContainerStarted","Data":"3280625711eed65e2e2cf82b3c7b1339cd9a322162c00b5364e34e0ddbad7b42"} Sep 14 11:01:32.488956 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:32.488918 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc3a2da2-96cd-4397-a6f9-9e7088b04f1e-metrics-tls\") pod \"dns-default-md5h2\" (UID: \"bc3a2da2-96cd-4397-a6f9-9e7088b04f1e\") " pod="openshift-dns/dns-default-md5h2" Sep 14 11:01:32.489121 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:32.489060 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 11:01:32.489187 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:32.489121 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc3a2da2-96cd-4397-a6f9-9e7088b04f1e-metrics-tls podName:bc3a2da2-96cd-4397-a6f9-9e7088b04f1e nodeName:}" failed. No retries permitted until 2026-09-14 11:01:34.489105923 +0000 UTC m=+39.109318433 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc3a2da2-96cd-4397-a6f9-9e7088b04f1e-metrics-tls") pod "dns-default-md5h2" (UID: "bc3a2da2-96cd-4397-a6f9-9e7088b04f1e") : secret "dns-default-metrics-tls" not found Sep 14 11:01:34.504125 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:34.504083 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc3a2da2-96cd-4397-a6f9-9e7088b04f1e-metrics-tls\") pod \"dns-default-md5h2\" (UID: \"bc3a2da2-96cd-4397-a6f9-9e7088b04f1e\") " pod="openshift-dns/dns-default-md5h2" Sep 14 11:01:34.504724 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:34.504226 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 11:01:34.504724 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:34.504311 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc3a2da2-96cd-4397-a6f9-9e7088b04f1e-metrics-tls podName:bc3a2da2-96cd-4397-a6f9-9e7088b04f1e nodeName:}" failed. No retries permitted until 2026-09-14 11:01:38.504291987 +0000 UTC m=+43.124504517 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc3a2da2-96cd-4397-a6f9-9e7088b04f1e-metrics-tls") pod "dns-default-md5h2" (UID: "bc3a2da2-96cd-4397-a6f9-9e7088b04f1e") : secret "dns-default-metrics-tls" not found Sep 14 11:01:36.062679 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:36.062639 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6qbbf" event={"ID":"4e9e0945-a3ff-449a-90a6-ddf38839b670","Type":"ContainerStarted","Data":"154532d2049187f666643503205212f730cf72fb46cbecc0506f3d3b994dc201"} Sep 14 11:01:36.065415 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:36.065390 2574 generic.go:356] "Generic (PLEG): container finished" podID="3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f" containerID="9708d8b6e6f70413b5e72ebe8753fd93103adfecade3a5630af8ba4b61f5cb45" exitCode=0 Sep 14 11:01:36.065517 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:36.065428 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xpv2v" event={"ID":"3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f","Type":"ContainerDied","Data":"9708d8b6e6f70413b5e72ebe8753fd93103adfecade3a5630af8ba4b61f5cb45"} Sep 14 11:01:36.078221 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:36.078176 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6qbbf" podStartSLOduration=2.416607836 podStartE2EDuration="6.078161428s" totalImagesPullingTime="3.661553592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:01:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:01:36.077759321 +0000 UTC m=+40.697971855" watchObservedRunningTime="2026-09-14 11:01:36.078161428 +0000 UTC m=+40.698373953" Sep 14 11:01:37.070757 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:37.070723 2574 generic.go:356] "Generic (PLEG): container finished" podID="3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f" containerID="1f142ea08e44824f3097587c424dbc2739355c3f0a515d501229b82d4b1c4235" exitCode=0 Sep 14 11:01:37.071233 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:37.070805 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xpv2v" event={"ID":"3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f","Type":"ContainerDied","Data":"1f142ea08e44824f3097587c424dbc2739355c3f0a515d501229b82d4b1c4235"} Sep 14 11:01:38.076792 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:38.076753 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xpv2v" event={"ID":"3597ac5c-d3a0-42f6-ab6c-ad5cb887d70f","Type":"ContainerStarted","Data":"6f1b0925554c31dc512a42b51cf2a5fe07f406a42adf91e3d0549205d7a47e36"} Sep 14 11:01:38.112510 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:38.112461 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xpv2v" podStartSLOduration=10.335913839 podStartE2EDuration="43.112429141s" totalImagesPullingTime="32.776515302s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:00:55 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:01:38.111928667 +0000 UTC m=+42.732141190" watchObservedRunningTime="2026-09-14 11:01:38.112429141 +0000 UTC m=+42.732641664" Sep 14 11:01:38.530369 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:38.530339 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc3a2da2-96cd-4397-a6f9-9e7088b04f1e-metrics-tls\") pod \"dns-default-md5h2\" (UID: \"bc3a2da2-96cd-4397-a6f9-9e7088b04f1e\") " pod="openshift-dns/dns-default-md5h2" Sep 14 11:01:38.530539 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:38.530482 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 11:01:38.530587 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:38.530553 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc3a2da2-96cd-4397-a6f9-9e7088b04f1e-metrics-tls podName:bc3a2da2-96cd-4397-a6f9-9e7088b04f1e nodeName:}" failed. No retries permitted until 2026-09-14 11:01:46.530537064 +0000 UTC m=+51.150749581 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc3a2da2-96cd-4397-a6f9-9e7088b04f1e-metrics-tls") pod "dns-default-md5h2" (UID: "bc3a2da2-96cd-4397-a6f9-9e7088b04f1e") : secret "dns-default-metrics-tls" not found Sep 14 11:01:46.584478 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:46.584441 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc3a2da2-96cd-4397-a6f9-9e7088b04f1e-metrics-tls\") pod \"dns-default-md5h2\" (UID: \"bc3a2da2-96cd-4397-a6f9-9e7088b04f1e\") " pod="openshift-dns/dns-default-md5h2" Sep 14 11:01:46.584953 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:46.584548 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 11:01:46.584953 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:46.584601 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc3a2da2-96cd-4397-a6f9-9e7088b04f1e-metrics-tls podName:bc3a2da2-96cd-4397-a6f9-9e7088b04f1e nodeName:}" failed. No retries permitted until 2026-09-14 11:02:02.584586703 +0000 UTC m=+67.204799208 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc3a2da2-96cd-4397-a6f9-9e7088b04f1e-metrics-tls") pod "dns-default-md5h2" (UID: "bc3a2da2-96cd-4397-a6f9-9e7088b04f1e") : secret "dns-default-metrics-tls" not found Sep 14 11:01:57.045126 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:57.045094 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xpwhs" Sep 14 11:01:58.900556 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:58.900517 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-rpq5s"] Sep 14 11:01:58.906325 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:58.906305 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rpq5s" Sep 14 11:01:58.909680 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:58.909654 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 14 11:01:58.909809 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:58.909739 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 14 11:01:58.910612 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:58.910593 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-mgrjj\"" Sep 14 11:01:58.910717 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:58.910695 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 14 11:01:58.916120 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:58.916104 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 14 11:01:58.927238 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:58.927217 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6f6c545fc8-77jgh"] Sep 14 11:01:58.931149 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:58.931133 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-rpq5s"] Sep 14 11:01:58.931226 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:58.931207 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6f6c545fc8-77jgh" Sep 14 11:01:58.935596 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:58.935579 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 14 11:01:58.935812 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:58.935798 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 14 11:01:58.936204 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:58.936187 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-7pd5c\"" Sep 14 11:01:58.936323 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:58.936188 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 14 11:01:58.936323 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:58.936187 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 14 11:01:58.936629 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:58.936587 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 14 11:01:58.938082 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:58.938064 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 14 11:01:58.970531 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:58.970511 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2baadd06-610f-4413-97c7-04f3a5c2039b-default-certificate\") pod \"router-default-6f6c545fc8-77jgh\" (UID: \"2baadd06-610f-4413-97c7-04f3a5c2039b\") " pod="openshift-ingress/router-default-6f6c545fc8-77jgh" Sep 14 11:01:58.970613 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:58.970537 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0dc7705f-2ef0-492c-8acf-2a2898a64904-config\") pod \"service-ca-operator-cf7c776b7-rpq5s\" (UID: \"0dc7705f-2ef0-492c-8acf-2a2898a64904\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rpq5s" Sep 14 11:01:58.970613 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:58.970555 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8h9f\" (UniqueName: \"kubernetes.io/projected/2baadd06-610f-4413-97c7-04f3a5c2039b-kube-api-access-q8h9f\") pod \"router-default-6f6c545fc8-77jgh\" (UID: \"2baadd06-610f-4413-97c7-04f3a5c2039b\") " pod="openshift-ingress/router-default-6f6c545fc8-77jgh" Sep 14 11:01:58.970613 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:58.970589 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2baadd06-610f-4413-97c7-04f3a5c2039b-metrics-certs\") pod \"router-default-6f6c545fc8-77jgh\" (UID: \"2baadd06-610f-4413-97c7-04f3a5c2039b\") " pod="openshift-ingress/router-default-6f6c545fc8-77jgh" Sep 14 11:01:58.970722 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:58.970633 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2baadd06-610f-4413-97c7-04f3a5c2039b-service-ca-bundle\") pod \"router-default-6f6c545fc8-77jgh\" (UID: \"2baadd06-610f-4413-97c7-04f3a5c2039b\") " pod="openshift-ingress/router-default-6f6c545fc8-77jgh" Sep 14 11:01:58.970722 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:58.970660 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx2rf\" (UniqueName: \"kubernetes.io/projected/0dc7705f-2ef0-492c-8acf-2a2898a64904-kube-api-access-mx2rf\") pod \"service-ca-operator-cf7c776b7-rpq5s\" (UID: \"0dc7705f-2ef0-492c-8acf-2a2898a64904\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rpq5s" Sep 14 11:01:58.970722 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:58.970678 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0dc7705f-2ef0-492c-8acf-2a2898a64904-serving-cert\") pod \"service-ca-operator-cf7c776b7-rpq5s\" (UID: \"0dc7705f-2ef0-492c-8acf-2a2898a64904\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rpq5s" Sep 14 11:01:58.970722 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:58.970703 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2baadd06-610f-4413-97c7-04f3a5c2039b-stats-auth\") pod \"router-default-6f6c545fc8-77jgh\" (UID: \"2baadd06-610f-4413-97c7-04f3a5c2039b\") " pod="openshift-ingress/router-default-6f6c545fc8-77jgh" Sep 14 11:01:58.985062 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:58.985040 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6f6c545fc8-77jgh"] Sep 14 11:01:59.029950 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.029928 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-c8qsb"] Sep 14 11:01:59.033454 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.033440 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lcxpq"] Sep 14 11:01:59.033574 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.033559 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-c8qsb" Sep 14 11:01:59.036464 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.036448 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-w9dj2\"" Sep 14 11:01:59.036858 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.036844 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lcxpq" Sep 14 11:01:59.039582 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.039562 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 14 11:01:59.039668 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.039570 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 14 11:01:59.039852 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.039831 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-465sp\"" Sep 14 11:01:59.039921 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.039855 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 14 11:01:59.040462 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.040370 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 14 11:01:59.044535 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.044512 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-c8qsb"] Sep 14 11:01:59.045223 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.045201 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lcxpq"] Sep 14 11:01:59.071358 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.071331 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2baadd06-610f-4413-97c7-04f3a5c2039b-metrics-certs\") pod \"router-default-6f6c545fc8-77jgh\" (UID: \"2baadd06-610f-4413-97c7-04f3a5c2039b\") " pod="openshift-ingress/router-default-6f6c545fc8-77jgh" Sep 14 11:01:59.071471 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.071361 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2baadd06-610f-4413-97c7-04f3a5c2039b-service-ca-bundle\") pod \"router-default-6f6c545fc8-77jgh\" (UID: \"2baadd06-610f-4413-97c7-04f3a5c2039b\") " pod="openshift-ingress/router-default-6f6c545fc8-77jgh" Sep 14 11:01:59.071471 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.071380 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx2rf\" (UniqueName: \"kubernetes.io/projected/0dc7705f-2ef0-492c-8acf-2a2898a64904-kube-api-access-mx2rf\") pod \"service-ca-operator-cf7c776b7-rpq5s\" (UID: \"0dc7705f-2ef0-492c-8acf-2a2898a64904\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rpq5s" Sep 14 11:01:59.071471 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.071400 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1e6a3f9d-7dd0-42f4-8d3a-a38b72eae274-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-lcxpq\" (UID: \"1e6a3f9d-7dd0-42f4-8d3a-a38b72eae274\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lcxpq" Sep 14 11:01:59.071471 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.071449 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0dc7705f-2ef0-492c-8acf-2a2898a64904-serving-cert\") pod \"service-ca-operator-cf7c776b7-rpq5s\" (UID: \"0dc7705f-2ef0-492c-8acf-2a2898a64904\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rpq5s" Sep 14 11:01:59.071664 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.071476 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1e6a3f9d-7dd0-42f4-8d3a-a38b72eae274-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-lcxpq\" (UID: \"1e6a3f9d-7dd0-42f4-8d3a-a38b72eae274\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lcxpq" Sep 14 11:01:59.071664 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.071529 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2baadd06-610f-4413-97c7-04f3a5c2039b-stats-auth\") pod \"router-default-6f6c545fc8-77jgh\" (UID: \"2baadd06-610f-4413-97c7-04f3a5c2039b\") " pod="openshift-ingress/router-default-6f6c545fc8-77jgh" Sep 14 11:01:59.071664 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:59.071537 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 14 11:01:59.071664 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:59.071559 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2baadd06-610f-4413-97c7-04f3a5c2039b-service-ca-bundle podName:2baadd06-610f-4413-97c7-04f3a5c2039b nodeName:}" failed. No retries permitted until 2026-09-14 11:01:59.571537405 +0000 UTC m=+64.191749909 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2baadd06-610f-4413-97c7-04f3a5c2039b-service-ca-bundle") pod "router-default-6f6c545fc8-77jgh" (UID: "2baadd06-610f-4413-97c7-04f3a5c2039b") : configmap references non-existent config key: service-ca.crt Sep 14 11:01:59.071664 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:59.071601 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2baadd06-610f-4413-97c7-04f3a5c2039b-metrics-certs podName:2baadd06-610f-4413-97c7-04f3a5c2039b nodeName:}" failed. No retries permitted until 2026-09-14 11:01:59.571588887 +0000 UTC m=+64.191801399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2baadd06-610f-4413-97c7-04f3a5c2039b-metrics-certs") pod "router-default-6f6c545fc8-77jgh" (UID: "2baadd06-610f-4413-97c7-04f3a5c2039b") : secret "router-metrics-certs-default" not found Sep 14 11:01:59.071664 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.071633 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk9bj\" (UniqueName: \"kubernetes.io/projected/1e6a3f9d-7dd0-42f4-8d3a-a38b72eae274-kube-api-access-bk9bj\") pod \"kube-storage-version-migrator-operator-6c8f84f897-lcxpq\" (UID: \"1e6a3f9d-7dd0-42f4-8d3a-a38b72eae274\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lcxpq" Sep 14 11:01:59.071883 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.071679 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2baadd06-610f-4413-97c7-04f3a5c2039b-default-certificate\") pod \"router-default-6f6c545fc8-77jgh\" (UID: \"2baadd06-610f-4413-97c7-04f3a5c2039b\") " pod="openshift-ingress/router-default-6f6c545fc8-77jgh" Sep 14 11:01:59.071883 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.071709 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcxqm\" (UniqueName: \"kubernetes.io/projected/a5f4b61d-1feb-41ba-9167-b503c1ef3de5-kube-api-access-tcxqm\") pod \"network-check-source-6f8b768455-c8qsb\" (UID: \"a5f4b61d-1feb-41ba-9167-b503c1ef3de5\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-c8qsb" Sep 14 11:01:59.071883 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.071746 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0dc7705f-2ef0-492c-8acf-2a2898a64904-config\") pod \"service-ca-operator-cf7c776b7-rpq5s\" (UID: \"0dc7705f-2ef0-492c-8acf-2a2898a64904\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rpq5s" Sep 14 11:01:59.071883 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.071779 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8h9f\" (UniqueName: \"kubernetes.io/projected/2baadd06-610f-4413-97c7-04f3a5c2039b-kube-api-access-q8h9f\") pod \"router-default-6f6c545fc8-77jgh\" (UID: \"2baadd06-610f-4413-97c7-04f3a5c2039b\") " pod="openshift-ingress/router-default-6f6c545fc8-77jgh" Sep 14 11:01:59.072287 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.072242 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0dc7705f-2ef0-492c-8acf-2a2898a64904-config\") pod \"service-ca-operator-cf7c776b7-rpq5s\" (UID: \"0dc7705f-2ef0-492c-8acf-2a2898a64904\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rpq5s" Sep 14 11:01:59.074786 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.074766 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2baadd06-610f-4413-97c7-04f3a5c2039b-default-certificate\") pod \"router-default-6f6c545fc8-77jgh\" (UID: \"2baadd06-610f-4413-97c7-04f3a5c2039b\") " pod="openshift-ingress/router-default-6f6c545fc8-77jgh" Sep 14 11:01:59.074883 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.074859 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0dc7705f-2ef0-492c-8acf-2a2898a64904-serving-cert\") pod \"service-ca-operator-cf7c776b7-rpq5s\" (UID: \"0dc7705f-2ef0-492c-8acf-2a2898a64904\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rpq5s" Sep 14 11:01:59.074923 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.074888 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2baadd06-610f-4413-97c7-04f3a5c2039b-stats-auth\") pod \"router-default-6f6c545fc8-77jgh\" (UID: \"2baadd06-610f-4413-97c7-04f3a5c2039b\") " pod="openshift-ingress/router-default-6f6c545fc8-77jgh" Sep 14 11:01:59.083110 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.083090 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx2rf\" (UniqueName: \"kubernetes.io/projected/0dc7705f-2ef0-492c-8acf-2a2898a64904-kube-api-access-mx2rf\") pod \"service-ca-operator-cf7c776b7-rpq5s\" (UID: \"0dc7705f-2ef0-492c-8acf-2a2898a64904\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rpq5s" Sep 14 11:01:59.083307 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.083259 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8h9f\" (UniqueName: \"kubernetes.io/projected/2baadd06-610f-4413-97c7-04f3a5c2039b-kube-api-access-q8h9f\") pod \"router-default-6f6c545fc8-77jgh\" (UID: \"2baadd06-610f-4413-97c7-04f3a5c2039b\") " pod="openshift-ingress/router-default-6f6c545fc8-77jgh" Sep 14 11:01:59.118921 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.118894 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f8c696848-hc5hp"] Sep 14 11:01:59.124587 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.124571 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f8c696848-hc5hp" Sep 14 11:01:59.127425 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.127408 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 14 11:01:59.127533 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.127500 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 14 11:01:59.129168 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.129151 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 14 11:01:59.130483 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.130433 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nd2nr\"" Sep 14 11:01:59.135454 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.135437 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 14 11:01:59.139357 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.139339 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f8c696848-hc5hp"] Sep 14 11:01:59.172404 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.172352 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcxqm\" (UniqueName: \"kubernetes.io/projected/a5f4b61d-1feb-41ba-9167-b503c1ef3de5-kube-api-access-tcxqm\") pod \"network-check-source-6f8b768455-c8qsb\" (UID: \"a5f4b61d-1feb-41ba-9167-b503c1ef3de5\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-c8qsb" Sep 14 11:01:59.172404 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.172381 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c693407b-c967-4981-950c-e7630d4dff7a-registry-certificates\") pod \"image-registry-7f8c696848-hc5hp\" (UID: \"c693407b-c967-4981-950c-e7630d4dff7a\") " pod="openshift-image-registry/image-registry-7f8c696848-hc5hp" Sep 14 11:01:59.172530 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.172406 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c693407b-c967-4981-950c-e7630d4dff7a-installation-pull-secrets\") pod \"image-registry-7f8c696848-hc5hp\" (UID: \"c693407b-c967-4981-950c-e7630d4dff7a\") " pod="openshift-image-registry/image-registry-7f8c696848-hc5hp" Sep 14 11:01:59.172530 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.172429 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c693407b-c967-4981-950c-e7630d4dff7a-trusted-ca\") pod \"image-registry-7f8c696848-hc5hp\" (UID: \"c693407b-c967-4981-950c-e7630d4dff7a\") " pod="openshift-image-registry/image-registry-7f8c696848-hc5hp" Sep 14 11:01:59.172530 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.172447 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c693407b-c967-4981-950c-e7630d4dff7a-image-registry-private-configuration\") pod \"image-registry-7f8c696848-hc5hp\" (UID: \"c693407b-c967-4981-950c-e7630d4dff7a\") " pod="openshift-image-registry/image-registry-7f8c696848-hc5hp" Sep 14 11:01:59.172641 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.172549 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1e6a3f9d-7dd0-42f4-8d3a-a38b72eae274-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-lcxpq\" (UID: \"1e6a3f9d-7dd0-42f4-8d3a-a38b72eae274\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lcxpq" Sep 14 11:01:59.172641 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.172587 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1e6a3f9d-7dd0-42f4-8d3a-a38b72eae274-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-lcxpq\" (UID: \"1e6a3f9d-7dd0-42f4-8d3a-a38b72eae274\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lcxpq" Sep 14 11:01:59.172641 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.172617 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c693407b-c967-4981-950c-e7630d4dff7a-registry-tls\") pod \"image-registry-7f8c696848-hc5hp\" (UID: \"c693407b-c967-4981-950c-e7630d4dff7a\") " pod="openshift-image-registry/image-registry-7f8c696848-hc5hp" Sep 14 11:01:59.172740 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.172683 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c693407b-c967-4981-950c-e7630d4dff7a-ca-trust-extracted\") pod \"image-registry-7f8c696848-hc5hp\" (UID: \"c693407b-c967-4981-950c-e7630d4dff7a\") " pod="openshift-image-registry/image-registry-7f8c696848-hc5hp" Sep 14 11:01:59.172740 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.172724 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c693407b-c967-4981-950c-e7630d4dff7a-bound-sa-token\") pod \"image-registry-7f8c696848-hc5hp\" (UID: \"c693407b-c967-4981-950c-e7630d4dff7a\") " pod="openshift-image-registry/image-registry-7f8c696848-hc5hp" Sep 14 11:01:59.172813 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.172745 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk9bj\" (UniqueName: \"kubernetes.io/projected/1e6a3f9d-7dd0-42f4-8d3a-a38b72eae274-kube-api-access-bk9bj\") pod \"kube-storage-version-migrator-operator-6c8f84f897-lcxpq\" (UID: \"1e6a3f9d-7dd0-42f4-8d3a-a38b72eae274\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lcxpq" Sep 14 11:01:59.172813 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.172777 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjf6s\" (UniqueName: \"kubernetes.io/projected/c693407b-c967-4981-950c-e7630d4dff7a-kube-api-access-mjf6s\") pod \"image-registry-7f8c696848-hc5hp\" (UID: \"c693407b-c967-4981-950c-e7630d4dff7a\") " pod="openshift-image-registry/image-registry-7f8c696848-hc5hp" Sep 14 11:01:59.173161 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.173146 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1e6a3f9d-7dd0-42f4-8d3a-a38b72eae274-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-lcxpq\" (UID: \"1e6a3f9d-7dd0-42f4-8d3a-a38b72eae274\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lcxpq" Sep 14 11:01:59.174668 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.174652 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1e6a3f9d-7dd0-42f4-8d3a-a38b72eae274-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-lcxpq\" (UID: \"1e6a3f9d-7dd0-42f4-8d3a-a38b72eae274\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lcxpq" Sep 14 11:01:59.184885 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.184862 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 14 11:01:59.185429 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.185415 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk9bj\" (UniqueName: \"kubernetes.io/projected/1e6a3f9d-7dd0-42f4-8d3a-a38b72eae274-kube-api-access-bk9bj\") pod \"kube-storage-version-migrator-operator-6c8f84f897-lcxpq\" (UID: \"1e6a3f9d-7dd0-42f4-8d3a-a38b72eae274\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lcxpq" Sep 14 11:01:59.192948 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.192935 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 14 11:01:59.203902 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.203881 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcxqm\" (UniqueName: \"kubernetes.io/projected/a5f4b61d-1feb-41ba-9167-b503c1ef3de5-kube-api-access-tcxqm\") pod \"network-check-source-6f8b768455-c8qsb\" (UID: \"a5f4b61d-1feb-41ba-9167-b503c1ef3de5\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-c8qsb" Sep 14 11:01:59.215191 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.215177 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rpq5s" Sep 14 11:01:59.273943 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.273914 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c693407b-c967-4981-950c-e7630d4dff7a-registry-tls\") pod \"image-registry-7f8c696848-hc5hp\" (UID: \"c693407b-c967-4981-950c-e7630d4dff7a\") " pod="openshift-image-registry/image-registry-7f8c696848-hc5hp" Sep 14 11:01:59.274035 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.273959 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c693407b-c967-4981-950c-e7630d4dff7a-ca-trust-extracted\") pod \"image-registry-7f8c696848-hc5hp\" (UID: \"c693407b-c967-4981-950c-e7630d4dff7a\") " pod="openshift-image-registry/image-registry-7f8c696848-hc5hp" Sep 14 11:01:59.274035 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.274008 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c693407b-c967-4981-950c-e7630d4dff7a-bound-sa-token\") pod \"image-registry-7f8c696848-hc5hp\" (UID: \"c693407b-c967-4981-950c-e7630d4dff7a\") " pod="openshift-image-registry/image-registry-7f8c696848-hc5hp" Sep 14 11:01:59.274104 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.274062 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjf6s\" (UniqueName: \"kubernetes.io/projected/c693407b-c967-4981-950c-e7630d4dff7a-kube-api-access-mjf6s\") pod \"image-registry-7f8c696848-hc5hp\" (UID: \"c693407b-c967-4981-950c-e7630d4dff7a\") " pod="openshift-image-registry/image-registry-7f8c696848-hc5hp" Sep 14 11:01:59.274104 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.274097 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c693407b-c967-4981-950c-e7630d4dff7a-registry-certificates\") pod \"image-registry-7f8c696848-hc5hp\" (UID: \"c693407b-c967-4981-950c-e7630d4dff7a\") " pod="openshift-image-registry/image-registry-7f8c696848-hc5hp" Sep 14 11:01:59.274177 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.274124 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c693407b-c967-4981-950c-e7630d4dff7a-installation-pull-secrets\") pod \"image-registry-7f8c696848-hc5hp\" (UID: \"c693407b-c967-4981-950c-e7630d4dff7a\") " pod="openshift-image-registry/image-registry-7f8c696848-hc5hp" Sep 14 11:01:59.274177 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.274160 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c693407b-c967-4981-950c-e7630d4dff7a-trusted-ca\") pod \"image-registry-7f8c696848-hc5hp\" (UID: \"c693407b-c967-4981-950c-e7630d4dff7a\") " pod="openshift-image-registry/image-registry-7f8c696848-hc5hp" Sep 14 11:01:59.274382 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.274189 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c693407b-c967-4981-950c-e7630d4dff7a-image-registry-private-configuration\") pod \"image-registry-7f8c696848-hc5hp\" (UID: \"c693407b-c967-4981-950c-e7630d4dff7a\") " pod="openshift-image-registry/image-registry-7f8c696848-hc5hp" Sep 14 11:01:59.274418 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:59.274389 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 11:01:59.274418 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:59.274405 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f8c696848-hc5hp: secret "image-registry-tls" not found Sep 14 11:01:59.274497 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:59.274464 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c693407b-c967-4981-950c-e7630d4dff7a-registry-tls podName:c693407b-c967-4981-950c-e7630d4dff7a nodeName:}" failed. No retries permitted until 2026-09-14 11:01:59.774443797 +0000 UTC m=+64.394656319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c693407b-c967-4981-950c-e7630d4dff7a-registry-tls") pod "image-registry-7f8c696848-hc5hp" (UID: "c693407b-c967-4981-950c-e7630d4dff7a") : secret "image-registry-tls" not found Sep 14 11:01:59.275336 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.275250 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c693407b-c967-4981-950c-e7630d4dff7a-registry-certificates\") pod \"image-registry-7f8c696848-hc5hp\" (UID: \"c693407b-c967-4981-950c-e7630d4dff7a\") " pod="openshift-image-registry/image-registry-7f8c696848-hc5hp" Sep 14 11:01:59.275445 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.275431 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c693407b-c967-4981-950c-e7630d4dff7a-ca-trust-extracted\") pod \"image-registry-7f8c696848-hc5hp\" (UID: \"c693407b-c967-4981-950c-e7630d4dff7a\") " pod="openshift-image-registry/image-registry-7f8c696848-hc5hp" Sep 14 11:01:59.276156 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.276130 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c693407b-c967-4981-950c-e7630d4dff7a-trusted-ca\") pod \"image-registry-7f8c696848-hc5hp\" (UID: \"c693407b-c967-4981-950c-e7630d4dff7a\") " pod="openshift-image-registry/image-registry-7f8c696848-hc5hp" Sep 14 11:01:59.277684 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.277658 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c693407b-c967-4981-950c-e7630d4dff7a-installation-pull-secrets\") pod \"image-registry-7f8c696848-hc5hp\" (UID: \"c693407b-c967-4981-950c-e7630d4dff7a\") " pod="openshift-image-registry/image-registry-7f8c696848-hc5hp" Sep 14 11:01:59.278440 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.278139 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c693407b-c967-4981-950c-e7630d4dff7a-image-registry-private-configuration\") pod \"image-registry-7f8c696848-hc5hp\" (UID: \"c693407b-c967-4981-950c-e7630d4dff7a\") " pod="openshift-image-registry/image-registry-7f8c696848-hc5hp" Sep 14 11:01:59.282988 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.282970 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c693407b-c967-4981-950c-e7630d4dff7a-bound-sa-token\") pod \"image-registry-7f8c696848-hc5hp\" (UID: \"c693407b-c967-4981-950c-e7630d4dff7a\") " pod="openshift-image-registry/image-registry-7f8c696848-hc5hp" Sep 14 11:01:59.283824 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.283806 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjf6s\" (UniqueName: \"kubernetes.io/projected/c693407b-c967-4981-950c-e7630d4dff7a-kube-api-access-mjf6s\") pod \"image-registry-7f8c696848-hc5hp\" (UID: \"c693407b-c967-4981-950c-e7630d4dff7a\") " pod="openshift-image-registry/image-registry-7f8c696848-hc5hp" Sep 14 11:01:59.343639 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.343614 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-c8qsb" Sep 14 11:01:59.349579 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.349561 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lcxpq" Sep 14 11:01:59.486133 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.486108 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-rpq5s"] Sep 14 11:01:59.488765 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:01:59.488742 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dc7705f_2ef0_492c_8acf_2a2898a64904.slice/crio-e77c3f985e3bcd36e3d4e163726a42ad8d8775e1b0e9a9fb3299538c5d64111a WatchSource:0}: Error finding container e77c3f985e3bcd36e3d4e163726a42ad8d8775e1b0e9a9fb3299538c5d64111a: Status 404 returned error can't find the container with id e77c3f985e3bcd36e3d4e163726a42ad8d8775e1b0e9a9fb3299538c5d64111a Sep 14 11:01:59.576026 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.575997 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2baadd06-610f-4413-97c7-04f3a5c2039b-metrics-certs\") pod \"router-default-6f6c545fc8-77jgh\" (UID: \"2baadd06-610f-4413-97c7-04f3a5c2039b\") " pod="openshift-ingress/router-default-6f6c545fc8-77jgh" Sep 14 11:01:59.576122 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.576031 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2baadd06-610f-4413-97c7-04f3a5c2039b-service-ca-bundle\") pod \"router-default-6f6c545fc8-77jgh\" (UID: \"2baadd06-610f-4413-97c7-04f3a5c2039b\") " pod="openshift-ingress/router-default-6f6c545fc8-77jgh" Sep 14 11:01:59.576167 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:59.576140 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 14 11:01:59.576216 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:59.576205 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2baadd06-610f-4413-97c7-04f3a5c2039b-service-ca-bundle podName:2baadd06-610f-4413-97c7-04f3a5c2039b nodeName:}" failed. No retries permitted until 2026-09-14 11:02:00.576186463 +0000 UTC m=+65.196398985 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2baadd06-610f-4413-97c7-04f3a5c2039b-service-ca-bundle") pod "router-default-6f6c545fc8-77jgh" (UID: "2baadd06-610f-4413-97c7-04f3a5c2039b") : configmap references non-existent config key: service-ca.crt Sep 14 11:01:59.576261 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:59.576227 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2baadd06-610f-4413-97c7-04f3a5c2039b-metrics-certs podName:2baadd06-610f-4413-97c7-04f3a5c2039b nodeName:}" failed. No retries permitted until 2026-09-14 11:02:00.576219403 +0000 UTC m=+65.196431917 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2baadd06-610f-4413-97c7-04f3a5c2039b-metrics-certs") pod "router-default-6f6c545fc8-77jgh" (UID: "2baadd06-610f-4413-97c7-04f3a5c2039b") : secret "router-metrics-certs-default" not found Sep 14 11:01:59.623834 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.623798 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-c8qsb"] Sep 14 11:01:59.624888 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.624857 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lcxpq"] Sep 14 11:01:59.627239 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:01:59.627216 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e6a3f9d_7dd0_42f4_8d3a_a38b72eae274.slice/crio-964b502808d7cf913758e93e74ce560f2bdcff680703c257f754fdf1c67958bc WatchSource:0}: Error finding container 964b502808d7cf913758e93e74ce560f2bdcff680703c257f754fdf1c67958bc: Status 404 returned error can't find the container with id 964b502808d7cf913758e93e74ce560f2bdcff680703c257f754fdf1c67958bc Sep 14 11:01:59.627858 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:01:59.627836 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5f4b61d_1feb_41ba_9167_b503c1ef3de5.slice/crio-31b58313cede06322f3e9d015fcbc47887e7275beafe8a4b6095bafc22f491ed WatchSource:0}: Error finding container 31b58313cede06322f3e9d015fcbc47887e7275beafe8a4b6095bafc22f491ed: Status 404 returned error can't find the container with id 31b58313cede06322f3e9d015fcbc47887e7275beafe8a4b6095bafc22f491ed Sep 14 11:01:59.777482 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:01:59.777415 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c693407b-c967-4981-950c-e7630d4dff7a-registry-tls\") pod \"image-registry-7f8c696848-hc5hp\" (UID: \"c693407b-c967-4981-950c-e7630d4dff7a\") " pod="openshift-image-registry/image-registry-7f8c696848-hc5hp" Sep 14 11:01:59.777590 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:59.777570 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 11:01:59.777637 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:59.777592 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f8c696848-hc5hp: secret "image-registry-tls" not found Sep 14 11:01:59.777668 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:01:59.777642 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c693407b-c967-4981-950c-e7630d4dff7a-registry-tls podName:c693407b-c967-4981-950c-e7630d4dff7a nodeName:}" failed. No retries permitted until 2026-09-14 11:02:00.777628183 +0000 UTC m=+65.397840683 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c693407b-c967-4981-950c-e7630d4dff7a-registry-tls") pod "image-registry-7f8c696848-hc5hp" (UID: "c693407b-c967-4981-950c-e7630d4dff7a") : secret "image-registry-tls" not found Sep 14 11:02:00.127724 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:00.127648 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-c8qsb" event={"ID":"a5f4b61d-1feb-41ba-9167-b503c1ef3de5","Type":"ContainerStarted","Data":"31b58313cede06322f3e9d015fcbc47887e7275beafe8a4b6095bafc22f491ed"} Sep 14 11:02:00.129103 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:00.129067 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lcxpq" event={"ID":"1e6a3f9d-7dd0-42f4-8d3a-a38b72eae274","Type":"ContainerStarted","Data":"964b502808d7cf913758e93e74ce560f2bdcff680703c257f754fdf1c67958bc"} Sep 14 11:02:00.130900 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:00.130871 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rpq5s" event={"ID":"0dc7705f-2ef0-492c-8acf-2a2898a64904","Type":"ContainerStarted","Data":"e77c3f985e3bcd36e3d4e163726a42ad8d8775e1b0e9a9fb3299538c5d64111a"} Sep 14 11:02:00.586118 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:00.586077 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfd621b6-51b0-4f2e-b1aa-ea198b732626-metrics-certs\") pod \"network-metrics-daemon-jkg6q\" (UID: \"dfd621b6-51b0-4f2e-b1aa-ea198b732626\") " pod="openshift-multus/network-metrics-daemon-jkg6q" Sep 14 11:02:00.586118 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:00.586121 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2baadd06-610f-4413-97c7-04f3a5c2039b-metrics-certs\") pod \"router-default-6f6c545fc8-77jgh\" (UID: \"2baadd06-610f-4413-97c7-04f3a5c2039b\") " pod="openshift-ingress/router-default-6f6c545fc8-77jgh" Sep 14 11:02:00.586453 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:00.586154 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2baadd06-610f-4413-97c7-04f3a5c2039b-service-ca-bundle\") pod \"router-default-6f6c545fc8-77jgh\" (UID: \"2baadd06-610f-4413-97c7-04f3a5c2039b\") " pod="openshift-ingress/router-default-6f6c545fc8-77jgh" Sep 14 11:02:00.586453 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:00.586351 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2baadd06-610f-4413-97c7-04f3a5c2039b-service-ca-bundle podName:2baadd06-610f-4413-97c7-04f3a5c2039b nodeName:}" failed. No retries permitted until 2026-09-14 11:02:02.586331961 +0000 UTC m=+67.206544463 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2baadd06-610f-4413-97c7-04f3a5c2039b-service-ca-bundle") pod "router-default-6f6c545fc8-77jgh" (UID: "2baadd06-610f-4413-97c7-04f3a5c2039b") : configmap references non-existent config key: service-ca.crt Sep 14 11:02:00.586792 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:00.586772 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 14 11:02:00.586891 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:00.586827 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2baadd06-610f-4413-97c7-04f3a5c2039b-metrics-certs podName:2baadd06-610f-4413-97c7-04f3a5c2039b nodeName:}" failed. No retries permitted until 2026-09-14 11:02:02.586811159 +0000 UTC m=+67.207023660 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2baadd06-610f-4413-97c7-04f3a5c2039b-metrics-certs") pod "router-default-6f6c545fc8-77jgh" (UID: "2baadd06-610f-4413-97c7-04f3a5c2039b") : secret "router-metrics-certs-default" not found Sep 14 11:02:00.588825 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:00.588801 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 14 11:02:00.596800 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:00.596776 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 14 11:02:00.596909 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:00.596845 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfd621b6-51b0-4f2e-b1aa-ea198b732626-metrics-certs podName:dfd621b6-51b0-4f2e-b1aa-ea198b732626 nodeName:}" failed. No retries permitted until 2026-09-14 11:03:04.596827751 +0000 UTC m=+129.217040266 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfd621b6-51b0-4f2e-b1aa-ea198b732626-metrics-certs") pod "network-metrics-daemon-jkg6q" (UID: "dfd621b6-51b0-4f2e-b1aa-ea198b732626") : secret "metrics-daemon-secret" not found Sep 14 11:02:00.686777 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:00.686738 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07adbd4b-794d-4f1e-b566-02280ae8390a-original-pull-secret\") pod \"global-pull-secret-syncer-2sk6p\" (UID: \"07adbd4b-794d-4f1e-b566-02280ae8390a\") " pod="kube-system/global-pull-secret-syncer-2sk6p" Sep 14 11:02:00.689492 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:00.689463 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 14 11:02:00.701532 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:00.701505 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07adbd4b-794d-4f1e-b566-02280ae8390a-original-pull-secret\") pod \"global-pull-secret-syncer-2sk6p\" (UID: \"07adbd4b-794d-4f1e-b566-02280ae8390a\") " pod="kube-system/global-pull-secret-syncer-2sk6p" Sep 14 11:02:00.787585 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:00.787552 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c693407b-c967-4981-950c-e7630d4dff7a-registry-tls\") pod \"image-registry-7f8c696848-hc5hp\" (UID: \"c693407b-c967-4981-950c-e7630d4dff7a\") " pod="openshift-image-registry/image-registry-7f8c696848-hc5hp" Sep 14 11:02:00.787719 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:00.787641 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdc5t\" (UniqueName: \"kubernetes.io/projected/1bb6499d-6175-4cc7-83de-a6dd85437535-kube-api-access-hdc5t\") pod \"network-check-target-t6jcs\" (UID: \"1bb6499d-6175-4cc7-83de-a6dd85437535\") " pod="openshift-network-diagnostics/network-check-target-t6jcs" Sep 14 11:02:00.788178 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:00.788159 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 11:02:00.788178 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:00.788180 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f8c696848-hc5hp: secret "image-registry-tls" not found Sep 14 11:02:00.788371 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:00.788244 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c693407b-c967-4981-950c-e7630d4dff7a-registry-tls podName:c693407b-c967-4981-950c-e7630d4dff7a nodeName:}" failed. No retries permitted until 2026-09-14 11:02:02.788225301 +0000 UTC m=+67.408437815 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c693407b-c967-4981-950c-e7630d4dff7a-registry-tls") pod "image-registry-7f8c696848-hc5hp" (UID: "c693407b-c967-4981-950c-e7630d4dff7a") : secret "image-registry-tls" not found Sep 14 11:02:00.791222 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:00.791178 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdc5t\" (UniqueName: \"kubernetes.io/projected/1bb6499d-6175-4cc7-83de-a6dd85437535-kube-api-access-hdc5t\") pod \"network-check-target-t6jcs\" (UID: \"1bb6499d-6175-4cc7-83de-a6dd85437535\") " pod="openshift-network-diagnostics/network-check-target-t6jcs" Sep 14 11:02:00.907515 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:00.907433 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mckdx\"" Sep 14 11:02:00.910782 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:00.910755 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2sk6p" Sep 14 11:02:00.915387 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:00.915363 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t6jcs" Sep 14 11:02:01.204415 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:01.204375 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t6jcs"] Sep 14 11:02:01.207677 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:01.207639 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2sk6p"] Sep 14 11:02:02.141664 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:02.141630 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2sk6p" event={"ID":"07adbd4b-794d-4f1e-b566-02280ae8390a","Type":"ContainerStarted","Data":"4dc2687603e9d5abf9367916e65008f2fba43d98c1bc8cd05d1e36e852d24965"} Sep 14 11:02:02.142751 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:02.142731 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t6jcs" event={"ID":"1bb6499d-6175-4cc7-83de-a6dd85437535","Type":"ContainerStarted","Data":"319ac1fc7d4c2b6887925908b7a300b2bb89f810bb12d5913bc135b80173a361"} Sep 14 11:02:02.604945 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:02.604905 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc3a2da2-96cd-4397-a6f9-9e7088b04f1e-metrics-tls\") pod \"dns-default-md5h2\" (UID: \"bc3a2da2-96cd-4397-a6f9-9e7088b04f1e\") " pod="openshift-dns/dns-default-md5h2" Sep 14 11:02:02.605385 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:02.604988 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2baadd06-610f-4413-97c7-04f3a5c2039b-metrics-certs\") pod \"router-default-6f6c545fc8-77jgh\" (UID: \"2baadd06-610f-4413-97c7-04f3a5c2039b\") " pod="openshift-ingress/router-default-6f6c545fc8-77jgh" Sep 14 11:02:02.605385 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:02.605024 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2baadd06-610f-4413-97c7-04f3a5c2039b-service-ca-bundle\") pod \"router-default-6f6c545fc8-77jgh\" (UID: \"2baadd06-610f-4413-97c7-04f3a5c2039b\") " pod="openshift-ingress/router-default-6f6c545fc8-77jgh" Sep 14 11:02:02.605385 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:02.605066 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 11:02:02.605385 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:02.605146 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc3a2da2-96cd-4397-a6f9-9e7088b04f1e-metrics-tls podName:bc3a2da2-96cd-4397-a6f9-9e7088b04f1e nodeName:}" failed. No retries permitted until 2026-09-14 11:02:34.605126291 +0000 UTC m=+99.225338812 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc3a2da2-96cd-4397-a6f9-9e7088b04f1e-metrics-tls") pod "dns-default-md5h2" (UID: "bc3a2da2-96cd-4397-a6f9-9e7088b04f1e") : secret "dns-default-metrics-tls" not found Sep 14 11:02:02.605385 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:02.605163 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2baadd06-610f-4413-97c7-04f3a5c2039b-service-ca-bundle podName:2baadd06-610f-4413-97c7-04f3a5c2039b nodeName:}" failed. No retries permitted until 2026-09-14 11:02:06.605153011 +0000 UTC m=+71.225365517 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2baadd06-610f-4413-97c7-04f3a5c2039b-service-ca-bundle") pod "router-default-6f6c545fc8-77jgh" (UID: "2baadd06-610f-4413-97c7-04f3a5c2039b") : configmap references non-existent config key: service-ca.crt Sep 14 11:02:02.605385 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:02.605166 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 14 11:02:02.605385 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:02.605251 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2baadd06-610f-4413-97c7-04f3a5c2039b-metrics-certs podName:2baadd06-610f-4413-97c7-04f3a5c2039b nodeName:}" failed. No retries permitted until 2026-09-14 11:02:06.605231428 +0000 UTC m=+71.225443942 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2baadd06-610f-4413-97c7-04f3a5c2039b-metrics-certs") pod "router-default-6f6c545fc8-77jgh" (UID: "2baadd06-610f-4413-97c7-04f3a5c2039b") : secret "router-metrics-certs-default" not found Sep 14 11:02:02.807804 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:02.807761 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c693407b-c967-4981-950c-e7630d4dff7a-registry-tls\") pod \"image-registry-7f8c696848-hc5hp\" (UID: \"c693407b-c967-4981-950c-e7630d4dff7a\") " pod="openshift-image-registry/image-registry-7f8c696848-hc5hp" Sep 14 11:02:02.807968 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:02.807897 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 11:02:02.807968 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:02.807920 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f8c696848-hc5hp: secret "image-registry-tls" not found Sep 14 11:02:02.808099 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:02.807987 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c693407b-c967-4981-950c-e7630d4dff7a-registry-tls podName:c693407b-c967-4981-950c-e7630d4dff7a nodeName:}" failed. No retries permitted until 2026-09-14 11:02:06.807967074 +0000 UTC m=+71.428179589 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c693407b-c967-4981-950c-e7630d4dff7a-registry-tls") pod "image-registry-7f8c696848-hc5hp" (UID: "c693407b-c967-4981-950c-e7630d4dff7a") : secret "image-registry-tls" not found Sep 14 11:02:04.152528 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:04.152428 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rpq5s" event={"ID":"0dc7705f-2ef0-492c-8acf-2a2898a64904","Type":"ContainerStarted","Data":"fb57b1c920251bec353444f333603f5d60cb6b6f8d26b75b38281f7d94881672"} Sep 14 11:02:04.154944 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:04.154470 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t6jcs" event={"ID":"1bb6499d-6175-4cc7-83de-a6dd85437535","Type":"ContainerStarted","Data":"ad9af41185723be5ae757f4c15e94a70f0fba540d18944cf6802518ecbe2a130"} Sep 14 11:02:04.154944 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:04.154896 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-t6jcs" Sep 14 11:02:04.156848 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:04.156808 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-c8qsb" event={"ID":"a5f4b61d-1feb-41ba-9167-b503c1ef3de5","Type":"ContainerStarted","Data":"a52a7cbb4ab3279957a70ee97210646047f60610a3c216d267b75a3c91e27bea"} Sep 14 11:02:04.159121 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:04.159088 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lcxpq" event={"ID":"1e6a3f9d-7dd0-42f4-8d3a-a38b72eae274","Type":"ContainerStarted","Data":"e4e44e3f6c669328884783fd35cd7bf3082ca9a6fff95a51b1b6bc8384a16f6c"} Sep 14 11:02:04.170774 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:04.170719 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rpq5s" podStartSLOduration=2.522791111 podStartE2EDuration="6.170701804s" totalImagesPullingTime="3.647910693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:01:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:04.169238293 +0000 UTC m=+68.789450837" watchObservedRunningTime="2026-09-14 11:02:04.170701804 +0000 UTC m=+68.790914329" Sep 14 11:02:04.208327 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:04.208246 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-t6jcs" podStartSLOduration=66.441161053 podStartE2EDuration="1m8.208224427s" totalImagesPullingTime="1.767063374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:04.190898266 +0000 UTC m=+68.811110820" watchObservedRunningTime="2026-09-14 11:02:04.208224427 +0000 UTC m=+68.828437016" Sep 14 11:02:04.227299 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:04.226943 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-c8qsb" podStartSLOduration=1.710994292 podStartE2EDuration="5.226926483s" totalImagesPullingTime="3.515932191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:01:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:04.226811496 +0000 UTC m=+68.847024045" watchObservedRunningTime="2026-09-14 11:02:04.226926483 +0000 UTC m=+68.847139005" Sep 14 11:02:04.227299 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:04.227063 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lcxpq" podStartSLOduration=1.721073141 podStartE2EDuration="5.227055008s" totalImagesPullingTime="3.505981867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:01:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:04.209348053 +0000 UTC m=+68.829560580" watchObservedRunningTime="2026-09-14 11:02:04.227055008 +0000 UTC m=+68.847267532" Sep 14 11:02:06.122490 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:06.122452 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bt4kc"] Sep 14 11:02:06.154195 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:06.154158 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bt4kc"] Sep 14 11:02:06.154379 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:06.154291 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bt4kc" Sep 14 11:02:06.157093 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:06.157044 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 14 11:02:06.157235 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:06.157095 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nzc8q\"" Sep 14 11:02:06.157741 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:06.157422 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 14 11:02:06.157861 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:06.157835 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 14 11:02:06.157925 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:06.157889 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 14 11:02:06.237894 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:06.237859 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbsql\" (UniqueName: \"kubernetes.io/projected/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-kube-api-access-tbsql\") pod \"insights-runtime-extractor-bt4kc\" (UID: \"8ad7a744-ff5a-49f7-9b99-e2bd045686c7\") " pod="openshift-insights/insights-runtime-extractor-bt4kc" Sep 14 11:02:06.238081 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:06.237901 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bt4kc\" (UID: \"8ad7a744-ff5a-49f7-9b99-e2bd045686c7\") " pod="openshift-insights/insights-runtime-extractor-bt4kc" Sep 14 11:02:06.238081 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:06.237931 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bt4kc\" (UID: \"8ad7a744-ff5a-49f7-9b99-e2bd045686c7\") " pod="openshift-insights/insights-runtime-extractor-bt4kc" Sep 14 11:02:06.238081 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:06.238003 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-data-volume\") pod \"insights-runtime-extractor-bt4kc\" (UID: \"8ad7a744-ff5a-49f7-9b99-e2bd045686c7\") " pod="openshift-insights/insights-runtime-extractor-bt4kc" Sep 14 11:02:06.238244 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:06.238108 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-crio-socket\") pod \"insights-runtime-extractor-bt4kc\" (UID: \"8ad7a744-ff5a-49f7-9b99-e2bd045686c7\") " pod="openshift-insights/insights-runtime-extractor-bt4kc" Sep 14 11:02:06.339559 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:06.339520 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbsql\" (UniqueName: \"kubernetes.io/projected/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-kube-api-access-tbsql\") pod \"insights-runtime-extractor-bt4kc\" (UID: \"8ad7a744-ff5a-49f7-9b99-e2bd045686c7\") " pod="openshift-insights/insights-runtime-extractor-bt4kc" Sep 14 11:02:06.339559 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:06.339565 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bt4kc\" (UID: \"8ad7a744-ff5a-49f7-9b99-e2bd045686c7\") " pod="openshift-insights/insights-runtime-extractor-bt4kc" Sep 14 11:02:06.339809 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:06.339591 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bt4kc\" (UID: \"8ad7a744-ff5a-49f7-9b99-e2bd045686c7\") " pod="openshift-insights/insights-runtime-extractor-bt4kc" Sep 14 11:02:06.339809 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:06.339651 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-data-volume\") pod \"insights-runtime-extractor-bt4kc\" (UID: \"8ad7a744-ff5a-49f7-9b99-e2bd045686c7\") " pod="openshift-insights/insights-runtime-extractor-bt4kc" Sep 14 11:02:06.339809 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:06.339660 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 14 11:02:06.339809 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:06.339680 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-crio-socket\") pod \"insights-runtime-extractor-bt4kc\" (UID: \"8ad7a744-ff5a-49f7-9b99-e2bd045686c7\") " pod="openshift-insights/insights-runtime-extractor-bt4kc" Sep 14 11:02:06.339809 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:06.339737 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-kube-rbac-proxy-cm podName:8ad7a744-ff5a-49f7-9b99-e2bd045686c7 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:06.839717056 +0000 UTC m=+71.459929557 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bt4kc" (UID: "8ad7a744-ff5a-49f7-9b99-e2bd045686c7") : configmap "kube-rbac-proxy" not found Sep 14 11:02:06.339809 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:06.339794 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 14 11:02:06.340147 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:06.339882 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-insights-runtime-extractor-tls podName:8ad7a744-ff5a-49f7-9b99-e2bd045686c7 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:06.839865194 +0000 UTC m=+71.460077698 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bt4kc" (UID: "8ad7a744-ff5a-49f7-9b99-e2bd045686c7") : secret "insights-runtime-extractor-tls" not found Sep 14 11:02:06.340147 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:06.339795 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-crio-socket\") pod \"insights-runtime-extractor-bt4kc\" (UID: \"8ad7a744-ff5a-49f7-9b99-e2bd045686c7\") " pod="openshift-insights/insights-runtime-extractor-bt4kc" Sep 14 11:02:06.340147 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:06.340112 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-data-volume\") pod \"insights-runtime-extractor-bt4kc\" (UID: \"8ad7a744-ff5a-49f7-9b99-e2bd045686c7\") " pod="openshift-insights/insights-runtime-extractor-bt4kc" Sep 14 11:02:06.354877 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:06.354851 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbsql\" (UniqueName: \"kubernetes.io/projected/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-kube-api-access-tbsql\") pod \"insights-runtime-extractor-bt4kc\" (UID: \"8ad7a744-ff5a-49f7-9b99-e2bd045686c7\") " pod="openshift-insights/insights-runtime-extractor-bt4kc" Sep 14 11:02:06.641995 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:06.641956 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2baadd06-610f-4413-97c7-04f3a5c2039b-service-ca-bundle\") pod \"router-default-6f6c545fc8-77jgh\" (UID: \"2baadd06-610f-4413-97c7-04f3a5c2039b\") " pod="openshift-ingress/router-default-6f6c545fc8-77jgh" Sep 14 11:02:06.642181 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:06.642044 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2baadd06-610f-4413-97c7-04f3a5c2039b-metrics-certs\") pod \"router-default-6f6c545fc8-77jgh\" (UID: \"2baadd06-610f-4413-97c7-04f3a5c2039b\") " pod="openshift-ingress/router-default-6f6c545fc8-77jgh" Sep 14 11:02:06.642181 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:06.642166 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2baadd06-610f-4413-97c7-04f3a5c2039b-service-ca-bundle podName:2baadd06-610f-4413-97c7-04f3a5c2039b nodeName:}" failed. No retries permitted until 2026-09-14 11:02:14.642142679 +0000 UTC m=+79.262355185 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2baadd06-610f-4413-97c7-04f3a5c2039b-service-ca-bundle") pod "router-default-6f6c545fc8-77jgh" (UID: "2baadd06-610f-4413-97c7-04f3a5c2039b") : configmap references non-existent config key: service-ca.crt Sep 14 11:02:06.642364 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:06.642205 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 14 11:02:06.642364 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:06.642326 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2baadd06-610f-4413-97c7-04f3a5c2039b-metrics-certs podName:2baadd06-610f-4413-97c7-04f3a5c2039b nodeName:}" failed. No retries permitted until 2026-09-14 11:02:14.64230834 +0000 UTC m=+79.262520842 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2baadd06-610f-4413-97c7-04f3a5c2039b-metrics-certs") pod "router-default-6f6c545fc8-77jgh" (UID: "2baadd06-610f-4413-97c7-04f3a5c2039b") : secret "router-metrics-certs-default" not found Sep 14 11:02:06.844039 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:06.844006 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bt4kc\" (UID: \"8ad7a744-ff5a-49f7-9b99-e2bd045686c7\") " pod="openshift-insights/insights-runtime-extractor-bt4kc" Sep 14 11:02:06.844039 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:06.844042 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bt4kc\" (UID: \"8ad7a744-ff5a-49f7-9b99-e2bd045686c7\") " pod="openshift-insights/insights-runtime-extractor-bt4kc" Sep 14 11:02:06.844292 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:06.844074 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c693407b-c967-4981-950c-e7630d4dff7a-registry-tls\") pod \"image-registry-7f8c696848-hc5hp\" (UID: \"c693407b-c967-4981-950c-e7630d4dff7a\") " pod="openshift-image-registry/image-registry-7f8c696848-hc5hp" Sep 14 11:02:06.844292 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:06.844141 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 14 11:02:06.844292 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:06.844170 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 14 11:02:06.844292 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:06.844174 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 11:02:06.844292 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:06.844192 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f8c696848-hc5hp: secret "image-registry-tls" not found Sep 14 11:02:06.844292 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:06.844219 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-kube-rbac-proxy-cm podName:8ad7a744-ff5a-49f7-9b99-e2bd045686c7 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:07.844200454 +0000 UTC m=+72.464412968 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bt4kc" (UID: "8ad7a744-ff5a-49f7-9b99-e2bd045686c7") : configmap "kube-rbac-proxy" not found Sep 14 11:02:06.844292 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:06.844236 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c693407b-c967-4981-950c-e7630d4dff7a-registry-tls podName:c693407b-c967-4981-950c-e7630d4dff7a nodeName:}" failed. No retries permitted until 2026-09-14 11:02:14.844225445 +0000 UTC m=+79.464437963 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c693407b-c967-4981-950c-e7630d4dff7a-registry-tls") pod "image-registry-7f8c696848-hc5hp" (UID: "c693407b-c967-4981-950c-e7630d4dff7a") : secret "image-registry-tls" not found Sep 14 11:02:06.844292 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:06.844250 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-insights-runtime-extractor-tls podName:8ad7a744-ff5a-49f7-9b99-e2bd045686c7 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:07.844243943 +0000 UTC m=+72.464456446 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bt4kc" (UID: "8ad7a744-ff5a-49f7-9b99-e2bd045686c7") : secret "insights-runtime-extractor-tls" not found Sep 14 11:02:07.170211 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:07.170124 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2sk6p" event={"ID":"07adbd4b-794d-4f1e-b566-02280ae8390a","Type":"ContainerStarted","Data":"5973b35e3676da5f13a30636edb1a72d7bbbb5d1e1b91b07176b122a1bac651c"} Sep 14 11:02:07.192031 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:07.191987 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2sk6p" podStartSLOduration=65.921818444 podStartE2EDuration="1m11.191968727s" totalImagesPullingTime="5.270150283s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:07.190952643 +0000 UTC m=+71.811165185" watchObservedRunningTime="2026-09-14 11:02:07.191968727 +0000 UTC m=+71.812181252" Sep 14 11:02:07.852845 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:07.852811 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bt4kc\" (UID: \"8ad7a744-ff5a-49f7-9b99-e2bd045686c7\") " pod="openshift-insights/insights-runtime-extractor-bt4kc" Sep 14 11:02:07.852845 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:07.852849 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bt4kc\" (UID: \"8ad7a744-ff5a-49f7-9b99-e2bd045686c7\") " pod="openshift-insights/insights-runtime-extractor-bt4kc" Sep 14 11:02:07.853132 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:07.852956 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 14 11:02:07.853132 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:07.852998 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 14 11:02:07.853132 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:07.853024 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-kube-rbac-proxy-cm podName:8ad7a744-ff5a-49f7-9b99-e2bd045686c7 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:09.853006777 +0000 UTC m=+74.473219295 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bt4kc" (UID: "8ad7a744-ff5a-49f7-9b99-e2bd045686c7") : configmap "kube-rbac-proxy" not found Sep 14 11:02:07.853132 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:07.853043 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-insights-runtime-extractor-tls podName:8ad7a744-ff5a-49f7-9b99-e2bd045686c7 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:09.853034113 +0000 UTC m=+74.473246614 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bt4kc" (UID: "8ad7a744-ff5a-49f7-9b99-e2bd045686c7") : secret "insights-runtime-extractor-tls" not found Sep 14 11:02:08.146249 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:08.146180 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lcx2b_c961b4ad-d3a5-4032-aa9a-a96c167e3df8/dns-node-resolver/0.log" Sep 14 11:02:09.346846 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:09.346819 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sfnlb_978d6d43-2346-476b-9bff-274d816f1175/node-ca/0.log" Sep 14 11:02:09.747045 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:09.747017 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6qbbf_4e9e0945-a3ff-449a-90a6-ddf38839b670/serve-healthcheck-canary/0.log" Sep 14 11:02:09.867815 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:09.867766 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bt4kc\" (UID: \"8ad7a744-ff5a-49f7-9b99-e2bd045686c7\") " pod="openshift-insights/insights-runtime-extractor-bt4kc" Sep 14 11:02:09.867815 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:09.867816 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bt4kc\" (UID: \"8ad7a744-ff5a-49f7-9b99-e2bd045686c7\") " pod="openshift-insights/insights-runtime-extractor-bt4kc" Sep 14 11:02:09.868052 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:09.867911 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 14 11:02:09.868052 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:09.867942 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 14 11:02:09.868052 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:09.867990 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-kube-rbac-proxy-cm podName:8ad7a744-ff5a-49f7-9b99-e2bd045686c7 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:13.867970713 +0000 UTC m=+78.488183214 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bt4kc" (UID: "8ad7a744-ff5a-49f7-9b99-e2bd045686c7") : configmap "kube-rbac-proxy" not found Sep 14 11:02:09.868052 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:09.868009 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-insights-runtime-extractor-tls podName:8ad7a744-ff5a-49f7-9b99-e2bd045686c7 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:13.867998245 +0000 UTC m=+78.488210767 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bt4kc" (UID: "8ad7a744-ff5a-49f7-9b99-e2bd045686c7") : secret "insights-runtime-extractor-tls" not found Sep 14 11:02:13.898722 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:13.898693 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bt4kc\" (UID: \"8ad7a744-ff5a-49f7-9b99-e2bd045686c7\") " pod="openshift-insights/insights-runtime-extractor-bt4kc" Sep 14 11:02:13.898722 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:13.898728 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bt4kc\" (UID: \"8ad7a744-ff5a-49f7-9b99-e2bd045686c7\") " pod="openshift-insights/insights-runtime-extractor-bt4kc" Sep 14 11:02:13.899256 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:13.898819 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 14 11:02:13.899256 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:13.898833 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 14 11:02:13.899256 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:13.898890 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-insights-runtime-extractor-tls podName:8ad7a744-ff5a-49f7-9b99-e2bd045686c7 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:21.898876594 +0000 UTC m=+86.519089108 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bt4kc" (UID: "8ad7a744-ff5a-49f7-9b99-e2bd045686c7") : secret "insights-runtime-extractor-tls" not found Sep 14 11:02:13.899256 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:13.898902 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-kube-rbac-proxy-cm podName:8ad7a744-ff5a-49f7-9b99-e2bd045686c7 nodeName:}" failed. No retries permitted until 2026-09-14 11:02:21.898896425 +0000 UTC m=+86.519108925 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bt4kc" (UID: "8ad7a744-ff5a-49f7-9b99-e2bd045686c7") : configmap "kube-rbac-proxy" not found Sep 14 11:02:14.706459 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:14.706422 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2baadd06-610f-4413-97c7-04f3a5c2039b-metrics-certs\") pod \"router-default-6f6c545fc8-77jgh\" (UID: \"2baadd06-610f-4413-97c7-04f3a5c2039b\") " pod="openshift-ingress/router-default-6f6c545fc8-77jgh" Sep 14 11:02:14.706634 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:14.706505 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2baadd06-610f-4413-97c7-04f3a5c2039b-service-ca-bundle\") pod \"router-default-6f6c545fc8-77jgh\" (UID: \"2baadd06-610f-4413-97c7-04f3a5c2039b\") " pod="openshift-ingress/router-default-6f6c545fc8-77jgh" Sep 14 11:02:14.706634 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:14.706568 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 14 11:02:14.706634 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:14.706616 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2baadd06-610f-4413-97c7-04f3a5c2039b-service-ca-bundle podName:2baadd06-610f-4413-97c7-04f3a5c2039b nodeName:}" failed. No retries permitted until 2026-09-14 11:02:30.706603944 +0000 UTC m=+95.326816458 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2baadd06-610f-4413-97c7-04f3a5c2039b-service-ca-bundle") pod "router-default-6f6c545fc8-77jgh" (UID: "2baadd06-610f-4413-97c7-04f3a5c2039b") : configmap references non-existent config key: service-ca.crt Sep 14 11:02:14.706634 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:14.706631 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2baadd06-610f-4413-97c7-04f3a5c2039b-metrics-certs podName:2baadd06-610f-4413-97c7-04f3a5c2039b nodeName:}" failed. No retries permitted until 2026-09-14 11:02:30.706625329 +0000 UTC m=+95.326837830 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2baadd06-610f-4413-97c7-04f3a5c2039b-metrics-certs") pod "router-default-6f6c545fc8-77jgh" (UID: "2baadd06-610f-4413-97c7-04f3a5c2039b") : secret "router-metrics-certs-default" not found Sep 14 11:02:14.908117 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:14.908085 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c693407b-c967-4981-950c-e7630d4dff7a-registry-tls\") pod \"image-registry-7f8c696848-hc5hp\" (UID: \"c693407b-c967-4981-950c-e7630d4dff7a\") " pod="openshift-image-registry/image-registry-7f8c696848-hc5hp" Sep 14 11:02:14.910637 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:14.910617 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c693407b-c967-4981-950c-e7630d4dff7a-registry-tls\") pod \"image-registry-7f8c696848-hc5hp\" (UID: \"c693407b-c967-4981-950c-e7630d4dff7a\") " pod="openshift-image-registry/image-registry-7f8c696848-hc5hp" Sep 14 11:02:15.033409 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:15.033342 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f8c696848-hc5hp" Sep 14 11:02:15.330429 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:02:15.330402 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc693407b_c967_4981_950c_e7630d4dff7a.slice/crio-e493ec37d496460566126d81b9e6087a466abd57dd8c9d7c53d261e92e458288 WatchSource:0}: Error finding container e493ec37d496460566126d81b9e6087a466abd57dd8c9d7c53d261e92e458288: Status 404 returned error can't find the container with id e493ec37d496460566126d81b9e6087a466abd57dd8c9d7c53d261e92e458288 Sep 14 11:02:15.333514 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:15.333488 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f8c696848-hc5hp"] Sep 14 11:02:16.200438 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:16.200398 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f8c696848-hc5hp" event={"ID":"c693407b-c967-4981-950c-e7630d4dff7a","Type":"ContainerStarted","Data":"bab6a89802165696756eab8719eaf0d0db539f4e7a98215ac999c9a690c70aaa"} Sep 14 11:02:16.200796 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:16.200456 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f8c696848-hc5hp" event={"ID":"c693407b-c967-4981-950c-e7630d4dff7a","Type":"ContainerStarted","Data":"e493ec37d496460566126d81b9e6087a466abd57dd8c9d7c53d261e92e458288"} Sep 14 11:02:16.200796 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:16.200501 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f8c696848-hc5hp" Sep 14 11:02:16.221253 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:16.221204 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7f8c696848-hc5hp" podStartSLOduration=17.221191617 podStartE2EDuration="17.221191617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:01:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:16.219899732 +0000 UTC m=+80.840112265" watchObservedRunningTime="2026-09-14 11:02:16.221191617 +0000 UTC m=+80.841404140" Sep 14 11:02:21.965439 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:21.965401 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bt4kc\" (UID: \"8ad7a744-ff5a-49f7-9b99-e2bd045686c7\") " pod="openshift-insights/insights-runtime-extractor-bt4kc" Sep 14 11:02:21.965439 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:21.965443 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bt4kc\" (UID: \"8ad7a744-ff5a-49f7-9b99-e2bd045686c7\") " pod="openshift-insights/insights-runtime-extractor-bt4kc" Sep 14 11:02:21.966022 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:21.966003 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bt4kc\" (UID: \"8ad7a744-ff5a-49f7-9b99-e2bd045686c7\") " pod="openshift-insights/insights-runtime-extractor-bt4kc" Sep 14 11:02:21.967987 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:21.967971 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8ad7a744-ff5a-49f7-9b99-e2bd045686c7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bt4kc\" (UID: \"8ad7a744-ff5a-49f7-9b99-e2bd045686c7\") " pod="openshift-insights/insights-runtime-extractor-bt4kc" Sep 14 11:02:22.065783 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:22.065752 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bt4kc" Sep 14 11:02:22.349951 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:22.349890 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bt4kc"] Sep 14 11:02:22.353073 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:02:22.353049 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ad7a744_ff5a_49f7_9b99_e2bd045686c7.slice/crio-53a436e7d0bedd5fca37c71cb9b97c0b18a61aa63dc5be07d3cf326bd861f827 WatchSource:0}: Error finding container 53a436e7d0bedd5fca37c71cb9b97c0b18a61aa63dc5be07d3cf326bd861f827: Status 404 returned error can't find the container with id 53a436e7d0bedd5fca37c71cb9b97c0b18a61aa63dc5be07d3cf326bd861f827 Sep 14 11:02:23.225757 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:23.225723 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bt4kc" event={"ID":"8ad7a744-ff5a-49f7-9b99-e2bd045686c7","Type":"ContainerStarted","Data":"ac12191c9b53acd2ef49afff40d8df4a740c0575a9b90b189c12c782b6b76395"} Sep 14 11:02:23.226100 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:23.225786 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bt4kc" event={"ID":"8ad7a744-ff5a-49f7-9b99-e2bd045686c7","Type":"ContainerStarted","Data":"53a436e7d0bedd5fca37c71cb9b97c0b18a61aa63dc5be07d3cf326bd861f827"} Sep 14 11:02:24.231440 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:24.231400 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bt4kc" event={"ID":"8ad7a744-ff5a-49f7-9b99-e2bd045686c7","Type":"ContainerStarted","Data":"ab510adfb6db3d3674e870a9974e940a4ce0e0a320c7fb944b8ef60850fbf23d"} Sep 14 11:02:25.236662 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:25.236613 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bt4kc" event={"ID":"8ad7a744-ff5a-49f7-9b99-e2bd045686c7","Type":"ContainerStarted","Data":"79a0ede6e7f4d83a7cd0970548136fefa9b7e7805841e72f9422c94b977b8895"} Sep 14 11:02:25.258645 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:25.258596 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bt4kc" podStartSLOduration=17.082458564 podStartE2EDuration="19.258582357s" totalImagesPullingTime="2.176123793s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:02:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:25.257125423 +0000 UTC m=+89.877337965" watchObservedRunningTime="2026-09-14 11:02:25.258582357 +0000 UTC m=+89.878794879" Sep 14 11:02:27.309194 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:27.309162 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f8c696848-hc5hp"] Sep 14 11:02:27.351214 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:27.351185 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-9rw9d"] Sep 14 11:02:27.354192 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:27.354174 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-9rw9d" Sep 14 11:02:27.358834 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:27.358811 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 14 11:02:27.358834 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:27.358826 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 14 11:02:27.359020 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:27.358903 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-xf5js\"" Sep 14 11:02:27.384143 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:27.384117 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-9rw9d"] Sep 14 11:02:27.384932 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:27.384911 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-79cb87f565-swrl2"] Sep 14 11:02:27.387891 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:27.387745 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79cb87f565-swrl2" Sep 14 11:02:27.421304 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:27.421258 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79cb87f565-swrl2"] Sep 14 11:02:27.507635 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:27.507609 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7d61446d-2f48-452e-bcc9-3f3495fc1544-bound-sa-token\") pod \"image-registry-79cb87f565-swrl2\" (UID: \"7d61446d-2f48-452e-bcc9-3f3495fc1544\") " pod="openshift-image-registry/image-registry-79cb87f565-swrl2" Sep 14 11:02:27.507755 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:27.507642 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7d61446d-2f48-452e-bcc9-3f3495fc1544-ca-trust-extracted\") pod \"image-registry-79cb87f565-swrl2\" (UID: \"7d61446d-2f48-452e-bcc9-3f3495fc1544\") " pod="openshift-image-registry/image-registry-79cb87f565-swrl2" Sep 14 11:02:27.507755 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:27.507661 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6w68\" (UniqueName: \"kubernetes.io/projected/3b0d9445-30eb-4097-9f1c-1fb34c292559-kube-api-access-m6w68\") pod \"downloads-76f99584f4-9rw9d\" (UID: \"3b0d9445-30eb-4097-9f1c-1fb34c292559\") " pod="openshift-console/downloads-76f99584f4-9rw9d" Sep 14 11:02:27.507755 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:27.507684 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7d61446d-2f48-452e-bcc9-3f3495fc1544-registry-tls\") pod \"image-registry-79cb87f565-swrl2\" (UID: \"7d61446d-2f48-452e-bcc9-3f3495fc1544\") " pod="openshift-image-registry/image-registry-79cb87f565-swrl2" Sep 14 11:02:27.507755 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:27.507702 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7d61446d-2f48-452e-bcc9-3f3495fc1544-registry-certificates\") pod \"image-registry-79cb87f565-swrl2\" (UID: \"7d61446d-2f48-452e-bcc9-3f3495fc1544\") " pod="openshift-image-registry/image-registry-79cb87f565-swrl2" Sep 14 11:02:27.507755 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:27.507738 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7d61446d-2f48-452e-bcc9-3f3495fc1544-installation-pull-secrets\") pod \"image-registry-79cb87f565-swrl2\" (UID: \"7d61446d-2f48-452e-bcc9-3f3495fc1544\") " pod="openshift-image-registry/image-registry-79cb87f565-swrl2" Sep 14 11:02:27.507951 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:27.507767 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjgv8\" (UniqueName: \"kubernetes.io/projected/7d61446d-2f48-452e-bcc9-3f3495fc1544-kube-api-access-fjgv8\") pod \"image-registry-79cb87f565-swrl2\" (UID: \"7d61446d-2f48-452e-bcc9-3f3495fc1544\") " pod="openshift-image-registry/image-registry-79cb87f565-swrl2" Sep 14 11:02:27.507951 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:27.507805 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7d61446d-2f48-452e-bcc9-3f3495fc1544-image-registry-private-configuration\") pod \"image-registry-79cb87f565-swrl2\" (UID: \"7d61446d-2f48-452e-bcc9-3f3495fc1544\") " pod="openshift-image-registry/image-registry-79cb87f565-swrl2" Sep 14 11:02:27.507951 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:27.507854 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7d61446d-2f48-452e-bcc9-3f3495fc1544-trusted-ca\") pod \"image-registry-79cb87f565-swrl2\" (UID: \"7d61446d-2f48-452e-bcc9-3f3495fc1544\") " pod="openshift-image-registry/image-registry-79cb87f565-swrl2" Sep 14 11:02:27.608952 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:27.608887 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7d61446d-2f48-452e-bcc9-3f3495fc1544-trusted-ca\") pod \"image-registry-79cb87f565-swrl2\" (UID: \"7d61446d-2f48-452e-bcc9-3f3495fc1544\") " pod="openshift-image-registry/image-registry-79cb87f565-swrl2" Sep 14 11:02:27.608952 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:27.608949 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7d61446d-2f48-452e-bcc9-3f3495fc1544-bound-sa-token\") pod \"image-registry-79cb87f565-swrl2\" (UID: \"7d61446d-2f48-452e-bcc9-3f3495fc1544\") " pod="openshift-image-registry/image-registry-79cb87f565-swrl2" Sep 14 11:02:27.609107 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:27.608985 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7d61446d-2f48-452e-bcc9-3f3495fc1544-ca-trust-extracted\") pod \"image-registry-79cb87f565-swrl2\" (UID: \"7d61446d-2f48-452e-bcc9-3f3495fc1544\") " pod="openshift-image-registry/image-registry-79cb87f565-swrl2" Sep 14 11:02:27.609107 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:27.609010 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6w68\" (UniqueName: \"kubernetes.io/projected/3b0d9445-30eb-4097-9f1c-1fb34c292559-kube-api-access-m6w68\") pod \"downloads-76f99584f4-9rw9d\" (UID: \"3b0d9445-30eb-4097-9f1c-1fb34c292559\") " pod="openshift-console/downloads-76f99584f4-9rw9d" Sep 14 11:02:27.609107 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:27.609033 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7d61446d-2f48-452e-bcc9-3f3495fc1544-registry-tls\") pod \"image-registry-79cb87f565-swrl2\" (UID: \"7d61446d-2f48-452e-bcc9-3f3495fc1544\") " pod="openshift-image-registry/image-registry-79cb87f565-swrl2" Sep 14 11:02:27.609448 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:27.609329 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7d61446d-2f48-452e-bcc9-3f3495fc1544-registry-certificates\") pod \"image-registry-79cb87f565-swrl2\" (UID: \"7d61446d-2f48-452e-bcc9-3f3495fc1544\") " pod="openshift-image-registry/image-registry-79cb87f565-swrl2" Sep 14 11:02:27.609535 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:27.609519 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7d61446d-2f48-452e-bcc9-3f3495fc1544-installation-pull-secrets\") pod \"image-registry-79cb87f565-swrl2\" (UID: \"7d61446d-2f48-452e-bcc9-3f3495fc1544\") " pod="openshift-image-registry/image-registry-79cb87f565-swrl2" Sep 14 11:02:27.609729 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:27.609570 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjgv8\" (UniqueName: \"kubernetes.io/projected/7d61446d-2f48-452e-bcc9-3f3495fc1544-kube-api-access-fjgv8\") pod \"image-registry-79cb87f565-swrl2\" (UID: \"7d61446d-2f48-452e-bcc9-3f3495fc1544\") " pod="openshift-image-registry/image-registry-79cb87f565-swrl2" Sep 14 11:02:27.609792 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:27.609738 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7d61446d-2f48-452e-bcc9-3f3495fc1544-ca-trust-extracted\") pod \"image-registry-79cb87f565-swrl2\" (UID: \"7d61446d-2f48-452e-bcc9-3f3495fc1544\") " pod="openshift-image-registry/image-registry-79cb87f565-swrl2" Sep 14 11:02:27.609792 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:27.609757 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7d61446d-2f48-452e-bcc9-3f3495fc1544-image-registry-private-configuration\") pod \"image-registry-79cb87f565-swrl2\" (UID: \"7d61446d-2f48-452e-bcc9-3f3495fc1544\") " pod="openshift-image-registry/image-registry-79cb87f565-swrl2" Sep 14 11:02:27.610368 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:27.610334 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7d61446d-2f48-452e-bcc9-3f3495fc1544-trusted-ca\") pod \"image-registry-79cb87f565-swrl2\" (UID: \"7d61446d-2f48-452e-bcc9-3f3495fc1544\") " pod="openshift-image-registry/image-registry-79cb87f565-swrl2" Sep 14 11:02:27.612222 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:27.612200 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7d61446d-2f48-452e-bcc9-3f3495fc1544-registry-certificates\") pod \"image-registry-79cb87f565-swrl2\" (UID: \"7d61446d-2f48-452e-bcc9-3f3495fc1544\") " pod="openshift-image-registry/image-registry-79cb87f565-swrl2" Sep 14 11:02:27.614691 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:27.614668 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7d61446d-2f48-452e-bcc9-3f3495fc1544-image-registry-private-configuration\") pod \"image-registry-79cb87f565-swrl2\" (UID: \"7d61446d-2f48-452e-bcc9-3f3495fc1544\") " pod="openshift-image-registry/image-registry-79cb87f565-swrl2" Sep 14 11:02:27.614797 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:27.614752 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7d61446d-2f48-452e-bcc9-3f3495fc1544-registry-tls\") pod \"image-registry-79cb87f565-swrl2\" (UID: \"7d61446d-2f48-452e-bcc9-3f3495fc1544\") " pod="openshift-image-registry/image-registry-79cb87f565-swrl2" Sep 14 11:02:27.615957 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:27.615937 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7d61446d-2f48-452e-bcc9-3f3495fc1544-installation-pull-secrets\") pod \"image-registry-79cb87f565-swrl2\" (UID: \"7d61446d-2f48-452e-bcc9-3f3495fc1544\") " pod="openshift-image-registry/image-registry-79cb87f565-swrl2" Sep 14 11:02:27.624875 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:27.624856 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7d61446d-2f48-452e-bcc9-3f3495fc1544-bound-sa-token\") pod \"image-registry-79cb87f565-swrl2\" (UID: \"7d61446d-2f48-452e-bcc9-3f3495fc1544\") " pod="openshift-image-registry/image-registry-79cb87f565-swrl2" Sep 14 11:02:27.626880 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:27.626863 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6w68\" (UniqueName: \"kubernetes.io/projected/3b0d9445-30eb-4097-9f1c-1fb34c292559-kube-api-access-m6w68\") pod \"downloads-76f99584f4-9rw9d\" (UID: \"3b0d9445-30eb-4097-9f1c-1fb34c292559\") " pod="openshift-console/downloads-76f99584f4-9rw9d" Sep 14 11:02:27.635918 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:27.635899 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjgv8\" (UniqueName: \"kubernetes.io/projected/7d61446d-2f48-452e-bcc9-3f3495fc1544-kube-api-access-fjgv8\") pod \"image-registry-79cb87f565-swrl2\" (UID: \"7d61446d-2f48-452e-bcc9-3f3495fc1544\") " pod="openshift-image-registry/image-registry-79cb87f565-swrl2" Sep 14 11:02:27.662764 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:27.662736 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-9rw9d" Sep 14 11:02:27.696814 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:27.696788 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79cb87f565-swrl2" Sep 14 11:02:27.965818 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:27.965791 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-9rw9d"] Sep 14 11:02:27.968747 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:02:27.968723 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b0d9445_30eb_4097_9f1c_1fb34c292559.slice/crio-fc44d738253d38125b9dec84281a0c4b166d714636f7b3f0c5e11f1dc810a9bf WatchSource:0}: Error finding container fc44d738253d38125b9dec84281a0c4b166d714636f7b3f0c5e11f1dc810a9bf: Status 404 returned error can't find the container with id fc44d738253d38125b9dec84281a0c4b166d714636f7b3f0c5e11f1dc810a9bf Sep 14 11:02:28.001160 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:28.001137 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79cb87f565-swrl2"] Sep 14 11:02:28.004230 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:02:28.004208 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d61446d_2f48_452e_bcc9_3f3495fc1544.slice/crio-42b631e85568dd2c4ac73ad3e63ccfbe4bb3d0bac7a487ee8ac7e87faf0d6311 WatchSource:0}: Error finding container 42b631e85568dd2c4ac73ad3e63ccfbe4bb3d0bac7a487ee8ac7e87faf0d6311: Status 404 returned error can't find the container with id 42b631e85568dd2c4ac73ad3e63ccfbe4bb3d0bac7a487ee8ac7e87faf0d6311 Sep 14 11:02:28.248512 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:28.248431 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79cb87f565-swrl2" event={"ID":"7d61446d-2f48-452e-bcc9-3f3495fc1544","Type":"ContainerStarted","Data":"e26104cc2bd8c88e50ba9da3e3b1cec68bc2c82c73b1a227f42d39f53a04b505"} Sep 14 11:02:28.248512 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:28.248485 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79cb87f565-swrl2" event={"ID":"7d61446d-2f48-452e-bcc9-3f3495fc1544","Type":"ContainerStarted","Data":"42b631e85568dd2c4ac73ad3e63ccfbe4bb3d0bac7a487ee8ac7e87faf0d6311"} Sep 14 11:02:28.248718 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:28.248564 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-79cb87f565-swrl2" Sep 14 11:02:28.249740 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:28.249719 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-9rw9d" event={"ID":"3b0d9445-30eb-4097-9f1c-1fb34c292559","Type":"ContainerStarted","Data":"fc44d738253d38125b9dec84281a0c4b166d714636f7b3f0c5e11f1dc810a9bf"} Sep 14 11:02:28.279864 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:28.279820 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-79cb87f565-swrl2" podStartSLOduration=1.279803474 podStartE2EDuration="1.279803474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:02:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:28.278465272 +0000 UTC m=+92.898677811" watchObservedRunningTime="2026-09-14 11:02:28.279803474 +0000 UTC m=+92.900015996" Sep 14 11:02:30.736153 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:30.736118 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2baadd06-610f-4413-97c7-04f3a5c2039b-service-ca-bundle\") pod \"router-default-6f6c545fc8-77jgh\" (UID: \"2baadd06-610f-4413-97c7-04f3a5c2039b\") " pod="openshift-ingress/router-default-6f6c545fc8-77jgh" Sep 14 11:02:30.736647 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:30.736187 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2baadd06-610f-4413-97c7-04f3a5c2039b-metrics-certs\") pod \"router-default-6f6c545fc8-77jgh\" (UID: \"2baadd06-610f-4413-97c7-04f3a5c2039b\") " pod="openshift-ingress/router-default-6f6c545fc8-77jgh" Sep 14 11:02:30.736806 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:30.736774 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2baadd06-610f-4413-97c7-04f3a5c2039b-service-ca-bundle\") pod \"router-default-6f6c545fc8-77jgh\" (UID: \"2baadd06-610f-4413-97c7-04f3a5c2039b\") " pod="openshift-ingress/router-default-6f6c545fc8-77jgh" Sep 14 11:02:30.738912 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:30.738891 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2baadd06-610f-4413-97c7-04f3a5c2039b-metrics-certs\") pod \"router-default-6f6c545fc8-77jgh\" (UID: \"2baadd06-610f-4413-97c7-04f3a5c2039b\") " pod="openshift-ingress/router-default-6f6c545fc8-77jgh" Sep 14 11:02:31.038746 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:31.038669 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6f6c545fc8-77jgh" Sep 14 11:02:31.335595 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:31.335514 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6f6c545fc8-77jgh"] Sep 14 11:02:31.337936 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:02:31.337907 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2baadd06_610f_4413_97c7_04f3a5c2039b.slice/crio-23c186e40944a8b05760049f2bc7a5c256b7aa167607a83801889751692ebc38 WatchSource:0}: Error finding container 23c186e40944a8b05760049f2bc7a5c256b7aa167607a83801889751692ebc38: Status 404 returned error can't find the container with id 23c186e40944a8b05760049f2bc7a5c256b7aa167607a83801889751692ebc38 Sep 14 11:02:32.268960 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:32.268925 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6f6c545fc8-77jgh" event={"ID":"2baadd06-610f-4413-97c7-04f3a5c2039b","Type":"ContainerStarted","Data":"b6c63b64ad3706750c134960767d96eec4296a0044cb4cb715894e26b8ad02e2"} Sep 14 11:02:32.269411 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:32.268977 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6f6c545fc8-77jgh" event={"ID":"2baadd06-610f-4413-97c7-04f3a5c2039b","Type":"ContainerStarted","Data":"23c186e40944a8b05760049f2bc7a5c256b7aa167607a83801889751692ebc38"} Sep 14 11:02:32.292953 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:32.292910 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6f6c545fc8-77jgh" podStartSLOduration=34.292895519 podStartE2EDuration="34.292895519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:01:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:32.290795642 +0000 UTC m=+96.911008183" watchObservedRunningTime="2026-09-14 11:02:32.292895519 +0000 UTC m=+96.913108043" Sep 14 11:02:33.039810 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:33.039765 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6f6c545fc8-77jgh" Sep 14 11:02:33.042741 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:33.042716 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6f6c545fc8-77jgh" Sep 14 11:02:33.272859 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:33.272824 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6f6c545fc8-77jgh" Sep 14 11:02:33.274105 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:33.274084 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6f6c545fc8-77jgh" Sep 14 11:02:34.671782 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:34.671745 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc3a2da2-96cd-4397-a6f9-9e7088b04f1e-metrics-tls\") pod \"dns-default-md5h2\" (UID: \"bc3a2da2-96cd-4397-a6f9-9e7088b04f1e\") " pod="openshift-dns/dns-default-md5h2" Sep 14 11:02:34.675740 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:34.675580 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc3a2da2-96cd-4397-a6f9-9e7088b04f1e-metrics-tls\") pod \"dns-default-md5h2\" (UID: \"bc3a2da2-96cd-4397-a6f9-9e7088b04f1e\") " pod="openshift-dns/dns-default-md5h2" Sep 14 11:02:34.685806 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:34.685784 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zpbn2\"" Sep 14 11:02:34.693536 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:34.693514 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-md5h2" Sep 14 11:02:35.018410 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:35.018382 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-md5h2"] Sep 14 11:02:35.022052 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:02:35.022019 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc3a2da2_96cd_4397_a6f9_9e7088b04f1e.slice/crio-a02976d46b81a1226a8798965d1366e41d97c09b39e12f2db12606306f3d67bd WatchSource:0}: Error finding container a02976d46b81a1226a8798965d1366e41d97c09b39e12f2db12606306f3d67bd: Status 404 returned error can't find the container with id a02976d46b81a1226a8798965d1366e41d97c09b39e12f2db12606306f3d67bd Sep 14 11:02:35.282644 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:35.282554 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-md5h2" event={"ID":"bc3a2da2-96cd-4397-a6f9-9e7088b04f1e","Type":"ContainerStarted","Data":"a02976d46b81a1226a8798965d1366e41d97c09b39e12f2db12606306f3d67bd"} Sep 14 11:02:36.167540 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:36.167504 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-t6jcs" Sep 14 11:02:37.315569 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:37.315544 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7f8c696848-hc5hp" Sep 14 11:02:41.375055 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:41.375022 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5db4d47bf7-8w6fr"] Sep 14 11:02:41.378768 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:41.378746 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5db4d47bf7-8w6fr" Sep 14 11:02:41.381897 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:41.381865 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 14 11:02:41.382007 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:41.381914 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 14 11:02:41.382795 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:41.382778 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-77gsx\"" Sep 14 11:02:41.382904 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:41.382801 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 14 11:02:41.382904 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:41.382855 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 14 11:02:41.382904 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:41.382780 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 14 11:02:41.386579 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:41.386515 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 14 11:02:41.397261 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:41.397239 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5db4d47bf7-8w6fr"] Sep 14 11:02:41.421514 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:41.421486 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e7cead7b-bcec-4b90-bf0a-55057d5f39a6-service-ca\") pod \"console-5db4d47bf7-8w6fr\" (UID: \"e7cead7b-bcec-4b90-bf0a-55057d5f39a6\") " pod="openshift-console/console-5db4d47bf7-8w6fr" Sep 14 11:02:41.421620 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:41.421517 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7cead7b-bcec-4b90-bf0a-55057d5f39a6-trusted-ca-bundle\") pod \"console-5db4d47bf7-8w6fr\" (UID: \"e7cead7b-bcec-4b90-bf0a-55057d5f39a6\") " pod="openshift-console/console-5db4d47bf7-8w6fr" Sep 14 11:02:41.421620 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:41.421539 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gwgh\" (UniqueName: \"kubernetes.io/projected/e7cead7b-bcec-4b90-bf0a-55057d5f39a6-kube-api-access-2gwgh\") pod \"console-5db4d47bf7-8w6fr\" (UID: \"e7cead7b-bcec-4b90-bf0a-55057d5f39a6\") " pod="openshift-console/console-5db4d47bf7-8w6fr" Sep 14 11:02:41.421620 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:41.421612 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e7cead7b-bcec-4b90-bf0a-55057d5f39a6-console-serving-cert\") pod \"console-5db4d47bf7-8w6fr\" (UID: \"e7cead7b-bcec-4b90-bf0a-55057d5f39a6\") " pod="openshift-console/console-5db4d47bf7-8w6fr" Sep 14 11:02:41.421780 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:41.421643 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e7cead7b-bcec-4b90-bf0a-55057d5f39a6-console-config\") pod \"console-5db4d47bf7-8w6fr\" (UID: \"e7cead7b-bcec-4b90-bf0a-55057d5f39a6\") " pod="openshift-console/console-5db4d47bf7-8w6fr" Sep 14 11:02:41.421780 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:41.421685 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e7cead7b-bcec-4b90-bf0a-55057d5f39a6-console-oauth-config\") pod \"console-5db4d47bf7-8w6fr\" (UID: \"e7cead7b-bcec-4b90-bf0a-55057d5f39a6\") " pod="openshift-console/console-5db4d47bf7-8w6fr" Sep 14 11:02:41.421780 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:41.421703 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e7cead7b-bcec-4b90-bf0a-55057d5f39a6-oauth-serving-cert\") pod \"console-5db4d47bf7-8w6fr\" (UID: \"e7cead7b-bcec-4b90-bf0a-55057d5f39a6\") " pod="openshift-console/console-5db4d47bf7-8w6fr" Sep 14 11:02:41.522414 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:41.522376 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gwgh\" (UniqueName: \"kubernetes.io/projected/e7cead7b-bcec-4b90-bf0a-55057d5f39a6-kube-api-access-2gwgh\") pod \"console-5db4d47bf7-8w6fr\" (UID: \"e7cead7b-bcec-4b90-bf0a-55057d5f39a6\") " pod="openshift-console/console-5db4d47bf7-8w6fr" Sep 14 11:02:41.522593 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:41.522442 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e7cead7b-bcec-4b90-bf0a-55057d5f39a6-console-serving-cert\") pod \"console-5db4d47bf7-8w6fr\" (UID: \"e7cead7b-bcec-4b90-bf0a-55057d5f39a6\") " pod="openshift-console/console-5db4d47bf7-8w6fr" Sep 14 11:02:41.522593 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:41.522472 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e7cead7b-bcec-4b90-bf0a-55057d5f39a6-console-config\") pod \"console-5db4d47bf7-8w6fr\" (UID: \"e7cead7b-bcec-4b90-bf0a-55057d5f39a6\") " pod="openshift-console/console-5db4d47bf7-8w6fr" Sep 14 11:02:41.522593 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:41.522526 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e7cead7b-bcec-4b90-bf0a-55057d5f39a6-console-oauth-config\") pod \"console-5db4d47bf7-8w6fr\" (UID: \"e7cead7b-bcec-4b90-bf0a-55057d5f39a6\") " pod="openshift-console/console-5db4d47bf7-8w6fr" Sep 14 11:02:41.522593 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:41.522549 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e7cead7b-bcec-4b90-bf0a-55057d5f39a6-oauth-serving-cert\") pod \"console-5db4d47bf7-8w6fr\" (UID: \"e7cead7b-bcec-4b90-bf0a-55057d5f39a6\") " pod="openshift-console/console-5db4d47bf7-8w6fr" Sep 14 11:02:41.523008 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:41.522975 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e7cead7b-bcec-4b90-bf0a-55057d5f39a6-service-ca\") pod \"console-5db4d47bf7-8w6fr\" (UID: \"e7cead7b-bcec-4b90-bf0a-55057d5f39a6\") " pod="openshift-console/console-5db4d47bf7-8w6fr" Sep 14 11:02:41.523123 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:41.523024 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7cead7b-bcec-4b90-bf0a-55057d5f39a6-trusted-ca-bundle\") pod \"console-5db4d47bf7-8w6fr\" (UID: \"e7cead7b-bcec-4b90-bf0a-55057d5f39a6\") " pod="openshift-console/console-5db4d47bf7-8w6fr" Sep 14 11:02:41.523399 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:41.523378 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e7cead7b-bcec-4b90-bf0a-55057d5f39a6-console-config\") pod \"console-5db4d47bf7-8w6fr\" (UID: \"e7cead7b-bcec-4b90-bf0a-55057d5f39a6\") " pod="openshift-console/console-5db4d47bf7-8w6fr" Sep 14 11:02:41.523547 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:41.523457 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e7cead7b-bcec-4b90-bf0a-55057d5f39a6-oauth-serving-cert\") pod \"console-5db4d47bf7-8w6fr\" (UID: \"e7cead7b-bcec-4b90-bf0a-55057d5f39a6\") " pod="openshift-console/console-5db4d47bf7-8w6fr" Sep 14 11:02:41.523949 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:41.523924 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e7cead7b-bcec-4b90-bf0a-55057d5f39a6-service-ca\") pod \"console-5db4d47bf7-8w6fr\" (UID: \"e7cead7b-bcec-4b90-bf0a-55057d5f39a6\") " pod="openshift-console/console-5db4d47bf7-8w6fr" Sep 14 11:02:41.524142 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:41.524110 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7cead7b-bcec-4b90-bf0a-55057d5f39a6-trusted-ca-bundle\") pod \"console-5db4d47bf7-8w6fr\" (UID: \"e7cead7b-bcec-4b90-bf0a-55057d5f39a6\") " pod="openshift-console/console-5db4d47bf7-8w6fr" Sep 14 11:02:41.525535 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:41.525514 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e7cead7b-bcec-4b90-bf0a-55057d5f39a6-console-oauth-config\") pod \"console-5db4d47bf7-8w6fr\" (UID: \"e7cead7b-bcec-4b90-bf0a-55057d5f39a6\") " pod="openshift-console/console-5db4d47bf7-8w6fr" Sep 14 11:02:41.525906 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:41.525885 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e7cead7b-bcec-4b90-bf0a-55057d5f39a6-console-serving-cert\") pod \"console-5db4d47bf7-8w6fr\" (UID: \"e7cead7b-bcec-4b90-bf0a-55057d5f39a6\") " pod="openshift-console/console-5db4d47bf7-8w6fr" Sep 14 11:02:41.535017 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:41.534997 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gwgh\" (UniqueName: \"kubernetes.io/projected/e7cead7b-bcec-4b90-bf0a-55057d5f39a6-kube-api-access-2gwgh\") pod \"console-5db4d47bf7-8w6fr\" (UID: \"e7cead7b-bcec-4b90-bf0a-55057d5f39a6\") " pod="openshift-console/console-5db4d47bf7-8w6fr" Sep 14 11:02:41.690200 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:41.690164 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5db4d47bf7-8w6fr" Sep 14 11:02:44.318690 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:44.318606 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-md5h2" event={"ID":"bc3a2da2-96cd-4397-a6f9-9e7088b04f1e","Type":"ContainerStarted","Data":"70e47e10426c52d54112d8f0ec9366c1f89d12e6995ac4f4162920fa197469fc"} Sep 14 11:02:44.322509 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:44.322477 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-9rw9d" event={"ID":"3b0d9445-30eb-4097-9f1c-1fb34c292559","Type":"ContainerStarted","Data":"7486f7ceb43b47dfddc9ac576f0c3e12ceaf9736871ffab1e0824bd43f183daf"} Sep 14 11:02:44.322755 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:44.322736 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-9rw9d" Sep 14 11:02:44.323924 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:44.323899 2574 patch_prober.go:28] interesting pod/downloads-76f99584f4-9rw9d container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.14:8080/\": dial tcp 10.134.0.14:8080: connect: connection refused" start-of-body= Sep 14 11:02:44.324019 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:44.323946 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-76f99584f4-9rw9d" podUID="3b0d9445-30eb-4097-9f1c-1fb34c292559" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.14:8080/\": dial tcp 10.134.0.14:8080: connect: connection refused" Sep 14 11:02:44.342918 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:44.342896 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-k25g8"] Sep 14 11:02:44.346065 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:44.346047 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-k25g8" Sep 14 11:02:44.348163 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:44.348145 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-56dhl\"" Sep 14 11:02:44.348594 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:44.348580 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 14 11:02:44.348645 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:44.348636 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 14 11:02:44.348835 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:44.348820 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 14 11:02:44.349103 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:44.349086 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 14 11:02:44.349185 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:44.349172 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 14 11:02:44.349616 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:44.349603 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 14 11:02:44.354469 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:44.354425 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-9rw9d" podStartSLOduration=1.080578498 podStartE2EDuration="17.354412233s" totalImagesPullingTime="16.273833735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:02:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:44.353422777 +0000 UTC m=+108.973635321" watchObservedRunningTime="2026-09-14 11:02:44.354412233 +0000 UTC m=+108.974624756" Sep 14 11:02:44.427089 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:44.427056 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5db4d47bf7-8w6fr"] Sep 14 11:02:44.430828 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:02:44.430785 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7cead7b_bcec_4b90_bf0a_55057d5f39a6.slice/crio-01d84375536f1a6d08c978051697ee5136c8299a76b72920e0a4722c25e9d2a6 WatchSource:0}: Error finding container 01d84375536f1a6d08c978051697ee5136c8299a76b72920e0a4722c25e9d2a6: Status 404 returned error can't find the container with id 01d84375536f1a6d08c978051697ee5136c8299a76b72920e0a4722c25e9d2a6 Sep 14 11:02:44.447190 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:44.447162 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6337f168-6fbc-4c83-ab67-9476b93b697c-root\") pod \"node-exporter-k25g8\" (UID: \"6337f168-6fbc-4c83-ab67-9476b93b697c\") " pod="openshift-monitoring/node-exporter-k25g8" Sep 14 11:02:44.447483 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:44.447462 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6337f168-6fbc-4c83-ab67-9476b93b697c-node-exporter-textfile\") pod \"node-exporter-k25g8\" (UID: \"6337f168-6fbc-4c83-ab67-9476b93b697c\") " pod="openshift-monitoring/node-exporter-k25g8" Sep 14 11:02:44.447646 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:44.447633 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6337f168-6fbc-4c83-ab67-9476b93b697c-node-exporter-tls\") pod \"node-exporter-k25g8\" (UID: \"6337f168-6fbc-4c83-ab67-9476b93b697c\") " pod="openshift-monitoring/node-exporter-k25g8" Sep 14 11:02:44.447789 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:44.447778 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6337f168-6fbc-4c83-ab67-9476b93b697c-metrics-client-ca\") pod \"node-exporter-k25g8\" (UID: \"6337f168-6fbc-4c83-ab67-9476b93b697c\") " pod="openshift-monitoring/node-exporter-k25g8" Sep 14 11:02:44.447879 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:44.447866 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9szwt\" (UniqueName: \"kubernetes.io/projected/6337f168-6fbc-4c83-ab67-9476b93b697c-kube-api-access-9szwt\") pod \"node-exporter-k25g8\" (UID: \"6337f168-6fbc-4c83-ab67-9476b93b697c\") " pod="openshift-monitoring/node-exporter-k25g8" Sep 14 11:02:44.448010 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:44.447963 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6337f168-6fbc-4c83-ab67-9476b93b697c-node-exporter-wtmp\") pod \"node-exporter-k25g8\" (UID: \"6337f168-6fbc-4c83-ab67-9476b93b697c\") " pod="openshift-monitoring/node-exporter-k25g8" Sep 14 11:02:44.448109 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:44.448096 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6337f168-6fbc-4c83-ab67-9476b93b697c-node-exporter-accelerators-collector-config\") pod \"node-exporter-k25g8\" (UID: \"6337f168-6fbc-4c83-ab67-9476b93b697c\") " pod="openshift-monitoring/node-exporter-k25g8" Sep 14 11:02:44.448257 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:44.448218 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6337f168-6fbc-4c83-ab67-9476b93b697c-sys\") pod \"node-exporter-k25g8\" (UID: \"6337f168-6fbc-4c83-ab67-9476b93b697c\") " pod="openshift-monitoring/node-exporter-k25g8" Sep 14 11:02:44.448333 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:44.448255 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6337f168-6fbc-4c83-ab67-9476b93b697c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k25g8\" (UID: \"6337f168-6fbc-4c83-ab67-9476b93b697c\") " pod="openshift-monitoring/node-exporter-k25g8" Sep 14 11:02:44.548714 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:44.548678 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6337f168-6fbc-4c83-ab67-9476b93b697c-node-exporter-textfile\") pod \"node-exporter-k25g8\" (UID: \"6337f168-6fbc-4c83-ab67-9476b93b697c\") " pod="openshift-monitoring/node-exporter-k25g8" Sep 14 11:02:44.548889 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:44.548726 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6337f168-6fbc-4c83-ab67-9476b93b697c-node-exporter-tls\") pod \"node-exporter-k25g8\" (UID: \"6337f168-6fbc-4c83-ab67-9476b93b697c\") " pod="openshift-monitoring/node-exporter-k25g8" Sep 14 11:02:44.548889 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:44.548756 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6337f168-6fbc-4c83-ab67-9476b93b697c-metrics-client-ca\") pod \"node-exporter-k25g8\" (UID: \"6337f168-6fbc-4c83-ab67-9476b93b697c\") " pod="openshift-monitoring/node-exporter-k25g8" Sep 14 11:02:44.548889 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:44.548782 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9szwt\" (UniqueName: \"kubernetes.io/projected/6337f168-6fbc-4c83-ab67-9476b93b697c-kube-api-access-9szwt\") pod \"node-exporter-k25g8\" (UID: \"6337f168-6fbc-4c83-ab67-9476b93b697c\") " pod="openshift-monitoring/node-exporter-k25g8" Sep 14 11:02:44.548889 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:44.548848 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6337f168-6fbc-4c83-ab67-9476b93b697c-node-exporter-wtmp\") pod \"node-exporter-k25g8\" (UID: \"6337f168-6fbc-4c83-ab67-9476b93b697c\") " pod="openshift-monitoring/node-exporter-k25g8" Sep 14 11:02:44.549085 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:44.548883 2574 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 14 11:02:44.549085 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:44.548982 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6337f168-6fbc-4c83-ab67-9476b93b697c-node-exporter-tls podName:6337f168-6fbc-4c83-ab67-9476b93b697c nodeName:}" failed. No retries permitted until 2026-09-14 11:02:45.048958885 +0000 UTC m=+109.669171387 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/6337f168-6fbc-4c83-ab67-9476b93b697c-node-exporter-tls") pod "node-exporter-k25g8" (UID: "6337f168-6fbc-4c83-ab67-9476b93b697c") : secret "node-exporter-tls" not found Sep 14 11:02:44.549209 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:44.548891 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6337f168-6fbc-4c83-ab67-9476b93b697c-node-exporter-accelerators-collector-config\") pod \"node-exporter-k25g8\" (UID: \"6337f168-6fbc-4c83-ab67-9476b93b697c\") " pod="openshift-monitoring/node-exporter-k25g8" Sep 14 11:02:44.549209 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:44.549146 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6337f168-6fbc-4c83-ab67-9476b93b697c-sys\") pod \"node-exporter-k25g8\" (UID: \"6337f168-6fbc-4c83-ab67-9476b93b697c\") " pod="openshift-monitoring/node-exporter-k25g8" Sep 14 11:02:44.549209 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:44.549156 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6337f168-6fbc-4c83-ab67-9476b93b697c-node-exporter-textfile\") pod \"node-exporter-k25g8\" (UID: \"6337f168-6fbc-4c83-ab67-9476b93b697c\") " pod="openshift-monitoring/node-exporter-k25g8" Sep 14 11:02:44.549209 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:44.549177 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6337f168-6fbc-4c83-ab67-9476b93b697c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k25g8\" (UID: \"6337f168-6fbc-4c83-ab67-9476b93b697c\") " pod="openshift-monitoring/node-exporter-k25g8" Sep 14 11:02:44.549437 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:44.549237 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6337f168-6fbc-4c83-ab67-9476b93b697c-sys\") pod \"node-exporter-k25g8\" (UID: \"6337f168-6fbc-4c83-ab67-9476b93b697c\") " pod="openshift-monitoring/node-exporter-k25g8" Sep 14 11:02:44.549437 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:44.549251 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6337f168-6fbc-4c83-ab67-9476b93b697c-root\") pod \"node-exporter-k25g8\" (UID: \"6337f168-6fbc-4c83-ab67-9476b93b697c\") " pod="openshift-monitoring/node-exporter-k25g8" Sep 14 11:02:44.549437 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:44.549355 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6337f168-6fbc-4c83-ab67-9476b93b697c-root\") pod \"node-exporter-k25g8\" (UID: \"6337f168-6fbc-4c83-ab67-9476b93b697c\") " pod="openshift-monitoring/node-exporter-k25g8" Sep 14 11:02:44.549437 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:44.549397 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6337f168-6fbc-4c83-ab67-9476b93b697c-node-exporter-wtmp\") pod \"node-exporter-k25g8\" (UID: \"6337f168-6fbc-4c83-ab67-9476b93b697c\") " pod="openshift-monitoring/node-exporter-k25g8" Sep 14 11:02:44.549623 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:44.549520 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6337f168-6fbc-4c83-ab67-9476b93b697c-node-exporter-accelerators-collector-config\") pod \"node-exporter-k25g8\" (UID: \"6337f168-6fbc-4c83-ab67-9476b93b697c\") " pod="openshift-monitoring/node-exporter-k25g8" Sep 14 11:02:44.549623 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:44.549537 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6337f168-6fbc-4c83-ab67-9476b93b697c-metrics-client-ca\") pod \"node-exporter-k25g8\" (UID: \"6337f168-6fbc-4c83-ab67-9476b93b697c\") " pod="openshift-monitoring/node-exporter-k25g8" Sep 14 11:02:44.551876 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:44.551852 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6337f168-6fbc-4c83-ab67-9476b93b697c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k25g8\" (UID: \"6337f168-6fbc-4c83-ab67-9476b93b697c\") " pod="openshift-monitoring/node-exporter-k25g8" Sep 14 11:02:44.566430 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:44.566403 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9szwt\" (UniqueName: \"kubernetes.io/projected/6337f168-6fbc-4c83-ab67-9476b93b697c-kube-api-access-9szwt\") pod \"node-exporter-k25g8\" (UID: \"6337f168-6fbc-4c83-ab67-9476b93b697c\") " pod="openshift-monitoring/node-exporter-k25g8" Sep 14 11:02:45.054656 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.054619 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6337f168-6fbc-4c83-ab67-9476b93b697c-node-exporter-tls\") pod \"node-exporter-k25g8\" (UID: \"6337f168-6fbc-4c83-ab67-9476b93b697c\") " pod="openshift-monitoring/node-exporter-k25g8" Sep 14 11:02:45.058466 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.058440 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6337f168-6fbc-4c83-ab67-9476b93b697c-node-exporter-tls\") pod \"node-exporter-k25g8\" (UID: \"6337f168-6fbc-4c83-ab67-9476b93b697c\") " pod="openshift-monitoring/node-exporter-k25g8" Sep 14 11:02:45.261318 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.261283 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-k25g8" Sep 14 11:02:45.275903 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:02:45.275791 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6337f168_6fbc_4c83_ab67_9476b93b697c.slice/crio-674cce47e6741c0c550bbd2c6dce50ba7cd62d5f26aa580fdcf330b12c3e3494 WatchSource:0}: Error finding container 674cce47e6741c0c550bbd2c6dce50ba7cd62d5f26aa580fdcf330b12c3e3494: Status 404 returned error can't find the container with id 674cce47e6741c0c550bbd2c6dce50ba7cd62d5f26aa580fdcf330b12c3e3494 Sep 14 11:02:45.329717 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.329637 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-md5h2" event={"ID":"bc3a2da2-96cd-4397-a6f9-9e7088b04f1e","Type":"ContainerStarted","Data":"a3c61adbf540bf0941a29f40652132fc94d430a71b375a1c10887947ea5286ad"} Sep 14 11:02:45.330164 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.329739 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-md5h2" Sep 14 11:02:45.331519 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.331492 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5db4d47bf7-8w6fr" event={"ID":"e7cead7b-bcec-4b90-bf0a-55057d5f39a6","Type":"ContainerStarted","Data":"01d84375536f1a6d08c978051697ee5136c8299a76b72920e0a4722c25e9d2a6"} Sep 14 11:02:45.333202 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.333178 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k25g8" event={"ID":"6337f168-6fbc-4c83-ab67-9476b93b697c","Type":"ContainerStarted","Data":"674cce47e6741c0c550bbd2c6dce50ba7cd62d5f26aa580fdcf330b12c3e3494"} Sep 14 11:02:45.345575 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.345485 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-9rw9d" Sep 14 11:02:45.360210 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.360127 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-md5h2" podStartSLOduration=66.239888743 podStartE2EDuration="1m15.360108766s" totalImagesPullingTime="9.120220023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:01:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:45.359412134 +0000 UTC m=+109.979624678" watchObservedRunningTime="2026-09-14 11:02:45.360108766 +0000 UTC m=+109.980321289" Sep 14 11:02:45.432440 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.430485 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 14 11:02:45.438299 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.435802 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 14 11:02:45.443457 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.443425 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 14 11:02:45.443771 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.443744 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zhwsq\"" Sep 14 11:02:45.448907 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.444011 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 14 11:02:45.448907 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.444136 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 14 11:02:45.448907 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.444243 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 14 11:02:45.448907 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.444369 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 14 11:02:45.448907 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.444499 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 14 11:02:45.448907 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.444710 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 14 11:02:45.448907 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.444753 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 14 11:02:45.448907 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.448687 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 14 11:02:45.455895 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.455837 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 14 11:02:45.458522 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.458494 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6406ef1e-5938-48da-8c40-54d4c0e717e9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6406ef1e-5938-48da-8c40-54d4c0e717e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 11:02:45.458636 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.458550 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6406ef1e-5938-48da-8c40-54d4c0e717e9-config-volume\") pod \"alertmanager-main-0\" (UID: \"6406ef1e-5938-48da-8c40-54d4c0e717e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 11:02:45.458636 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.458579 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6406ef1e-5938-48da-8c40-54d4c0e717e9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6406ef1e-5938-48da-8c40-54d4c0e717e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 11:02:45.458636 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.458615 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6406ef1e-5938-48da-8c40-54d4c0e717e9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6406ef1e-5938-48da-8c40-54d4c0e717e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 11:02:45.458795 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.458645 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6406ef1e-5938-48da-8c40-54d4c0e717e9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6406ef1e-5938-48da-8c40-54d4c0e717e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 11:02:45.458795 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.458678 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6406ef1e-5938-48da-8c40-54d4c0e717e9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6406ef1e-5938-48da-8c40-54d4c0e717e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 11:02:45.458795 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.458723 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6406ef1e-5938-48da-8c40-54d4c0e717e9-config-out\") pod \"alertmanager-main-0\" (UID: \"6406ef1e-5938-48da-8c40-54d4c0e717e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 11:02:45.458795 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.458751 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6406ef1e-5938-48da-8c40-54d4c0e717e9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6406ef1e-5938-48da-8c40-54d4c0e717e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 11:02:45.458795 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.458781 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6406ef1e-5938-48da-8c40-54d4c0e717e9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6406ef1e-5938-48da-8c40-54d4c0e717e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 11:02:45.459047 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.458804 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6406ef1e-5938-48da-8c40-54d4c0e717e9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6406ef1e-5938-48da-8c40-54d4c0e717e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 11:02:45.459047 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.458879 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6406ef1e-5938-48da-8c40-54d4c0e717e9-web-config\") pod \"alertmanager-main-0\" (UID: \"6406ef1e-5938-48da-8c40-54d4c0e717e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 11:02:45.459047 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.458908 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mms9\" (UniqueName: \"kubernetes.io/projected/6406ef1e-5938-48da-8c40-54d4c0e717e9-kube-api-access-8mms9\") pod \"alertmanager-main-0\" (UID: \"6406ef1e-5938-48da-8c40-54d4c0e717e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 11:02:45.459047 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.458943 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6406ef1e-5938-48da-8c40-54d4c0e717e9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6406ef1e-5938-48da-8c40-54d4c0e717e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 11:02:45.560326 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.559586 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mms9\" (UniqueName: \"kubernetes.io/projected/6406ef1e-5938-48da-8c40-54d4c0e717e9-kube-api-access-8mms9\") pod \"alertmanager-main-0\" (UID: \"6406ef1e-5938-48da-8c40-54d4c0e717e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 11:02:45.560326 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.559633 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6406ef1e-5938-48da-8c40-54d4c0e717e9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6406ef1e-5938-48da-8c40-54d4c0e717e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 11:02:45.560326 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.559686 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6406ef1e-5938-48da-8c40-54d4c0e717e9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6406ef1e-5938-48da-8c40-54d4c0e717e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 11:02:45.560326 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.559718 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6406ef1e-5938-48da-8c40-54d4c0e717e9-config-volume\") pod \"alertmanager-main-0\" (UID: \"6406ef1e-5938-48da-8c40-54d4c0e717e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 11:02:45.560326 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.559741 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6406ef1e-5938-48da-8c40-54d4c0e717e9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6406ef1e-5938-48da-8c40-54d4c0e717e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 11:02:45.560326 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.559774 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6406ef1e-5938-48da-8c40-54d4c0e717e9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6406ef1e-5938-48da-8c40-54d4c0e717e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 11:02:45.560326 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.559797 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6406ef1e-5938-48da-8c40-54d4c0e717e9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6406ef1e-5938-48da-8c40-54d4c0e717e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 11:02:45.560326 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.559825 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6406ef1e-5938-48da-8c40-54d4c0e717e9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6406ef1e-5938-48da-8c40-54d4c0e717e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 11:02:45.560326 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.559873 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6406ef1e-5938-48da-8c40-54d4c0e717e9-config-out\") pod \"alertmanager-main-0\" (UID: \"6406ef1e-5938-48da-8c40-54d4c0e717e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 11:02:45.560326 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.559900 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6406ef1e-5938-48da-8c40-54d4c0e717e9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6406ef1e-5938-48da-8c40-54d4c0e717e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 11:02:45.560326 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.559930 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6406ef1e-5938-48da-8c40-54d4c0e717e9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6406ef1e-5938-48da-8c40-54d4c0e717e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 11:02:45.560326 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.559955 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6406ef1e-5938-48da-8c40-54d4c0e717e9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6406ef1e-5938-48da-8c40-54d4c0e717e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 11:02:45.560326 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.560003 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6406ef1e-5938-48da-8c40-54d4c0e717e9-web-config\") pod \"alertmanager-main-0\" (UID: \"6406ef1e-5938-48da-8c40-54d4c0e717e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 11:02:45.562441 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.562397 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6406ef1e-5938-48da-8c40-54d4c0e717e9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6406ef1e-5938-48da-8c40-54d4c0e717e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 11:02:45.562863 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.562838 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6406ef1e-5938-48da-8c40-54d4c0e717e9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6406ef1e-5938-48da-8c40-54d4c0e717e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 11:02:45.563844 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.563817 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6406ef1e-5938-48da-8c40-54d4c0e717e9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6406ef1e-5938-48da-8c40-54d4c0e717e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 11:02:45.564302 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.564282 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6406ef1e-5938-48da-8c40-54d4c0e717e9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6406ef1e-5938-48da-8c40-54d4c0e717e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 11:02:45.569349 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.569078 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6406ef1e-5938-48da-8c40-54d4c0e717e9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6406ef1e-5938-48da-8c40-54d4c0e717e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 11:02:45.571472 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.571431 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mms9\" (UniqueName: \"kubernetes.io/projected/6406ef1e-5938-48da-8c40-54d4c0e717e9-kube-api-access-8mms9\") pod \"alertmanager-main-0\" (UID: \"6406ef1e-5938-48da-8c40-54d4c0e717e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 11:02:45.577672 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.577623 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6406ef1e-5938-48da-8c40-54d4c0e717e9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6406ef1e-5938-48da-8c40-54d4c0e717e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 11:02:45.577838 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.577785 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6406ef1e-5938-48da-8c40-54d4c0e717e9-config-out\") pod \"alertmanager-main-0\" (UID: \"6406ef1e-5938-48da-8c40-54d4c0e717e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 11:02:45.578454 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.578076 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6406ef1e-5938-48da-8c40-54d4c0e717e9-config-volume\") pod \"alertmanager-main-0\" (UID: \"6406ef1e-5938-48da-8c40-54d4c0e717e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 11:02:45.578454 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.578410 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6406ef1e-5938-48da-8c40-54d4c0e717e9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6406ef1e-5938-48da-8c40-54d4c0e717e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 11:02:45.579057 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.579015 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6406ef1e-5938-48da-8c40-54d4c0e717e9-web-config\") pod \"alertmanager-main-0\" (UID: \"6406ef1e-5938-48da-8c40-54d4c0e717e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 11:02:45.579144 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.579115 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6406ef1e-5938-48da-8c40-54d4c0e717e9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6406ef1e-5938-48da-8c40-54d4c0e717e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 11:02:45.581479 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.581427 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6406ef1e-5938-48da-8c40-54d4c0e717e9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6406ef1e-5938-48da-8c40-54d4c0e717e9\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 11:02:45.756696 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:45.756663 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 14 11:02:46.075832 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:46.075787 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 14 11:02:46.078215 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:02:46.078181 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6406ef1e_5938_48da_8c40_54d4c0e717e9.slice/crio-e71179525768704164feedf2cd5eee21ffdcfc1cd1b9080899a3e945d4ff86b5 WatchSource:0}: Error finding container e71179525768704164feedf2cd5eee21ffdcfc1cd1b9080899a3e945d4ff86b5: Status 404 returned error can't find the container with id e71179525768704164feedf2cd5eee21ffdcfc1cd1b9080899a3e945d4ff86b5 Sep 14 11:02:46.338692 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:46.338605 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6406ef1e-5938-48da-8c40-54d4c0e717e9","Type":"ContainerStarted","Data":"e71179525768704164feedf2cd5eee21ffdcfc1cd1b9080899a3e945d4ff86b5"} Sep 14 11:02:48.707476 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:48.707441 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5cf5d68bb4-nb8ds"] Sep 14 11:02:48.727164 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:48.727115 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5cf5d68bb4-nb8ds"] Sep 14 11:02:48.727324 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:48.727303 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5cf5d68bb4-nb8ds" Sep 14 11:02:48.733776 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:48.733411 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 14 11:02:48.733776 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:48.733497 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 14 11:02:48.733776 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:48.733546 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-4h6ftipsje1cn\"" Sep 14 11:02:48.733776 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:48.733635 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-4jl45\"" Sep 14 11:02:48.734053 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:48.733915 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 14 11:02:48.734053 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:48.733967 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 14 11:02:48.791561 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:48.791522 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/92a227c2-cbea-4fc3-a721-b7fba16c0705-audit-log\") pod \"metrics-server-5cf5d68bb4-nb8ds\" (UID: \"92a227c2-cbea-4fc3-a721-b7fba16c0705\") " pod="openshift-monitoring/metrics-server-5cf5d68bb4-nb8ds" Sep 14 11:02:48.791705 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:48.791582 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/92a227c2-cbea-4fc3-a721-b7fba16c0705-secret-metrics-server-client-certs\") pod \"metrics-server-5cf5d68bb4-nb8ds\" (UID: \"92a227c2-cbea-4fc3-a721-b7fba16c0705\") " pod="openshift-monitoring/metrics-server-5cf5d68bb4-nb8ds" Sep 14 11:02:48.791705 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:48.791611 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/92a227c2-cbea-4fc3-a721-b7fba16c0705-secret-metrics-server-tls\") pod \"metrics-server-5cf5d68bb4-nb8ds\" (UID: \"92a227c2-cbea-4fc3-a721-b7fba16c0705\") " pod="openshift-monitoring/metrics-server-5cf5d68bb4-nb8ds" Sep 14 11:02:48.791705 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:48.791635 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92a227c2-cbea-4fc3-a721-b7fba16c0705-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5cf5d68bb4-nb8ds\" (UID: \"92a227c2-cbea-4fc3-a721-b7fba16c0705\") " pod="openshift-monitoring/metrics-server-5cf5d68bb4-nb8ds" Sep 14 11:02:48.791705 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:48.791672 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/92a227c2-cbea-4fc3-a721-b7fba16c0705-metrics-server-audit-profiles\") pod \"metrics-server-5cf5d68bb4-nb8ds\" (UID: \"92a227c2-cbea-4fc3-a721-b7fba16c0705\") " pod="openshift-monitoring/metrics-server-5cf5d68bb4-nb8ds" Sep 14 11:02:48.791705 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:48.791701 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwzr4\" (UniqueName: \"kubernetes.io/projected/92a227c2-cbea-4fc3-a721-b7fba16c0705-kube-api-access-dwzr4\") pod \"metrics-server-5cf5d68bb4-nb8ds\" (UID: \"92a227c2-cbea-4fc3-a721-b7fba16c0705\") " pod="openshift-monitoring/metrics-server-5cf5d68bb4-nb8ds" Sep 14 11:02:48.791958 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:48.791799 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/92a227c2-cbea-4fc3-a721-b7fba16c0705-client-ca-bundle\") pod \"metrics-server-5cf5d68bb4-nb8ds\" (UID: \"92a227c2-cbea-4fc3-a721-b7fba16c0705\") " pod="openshift-monitoring/metrics-server-5cf5d68bb4-nb8ds" Sep 14 11:02:48.892862 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:48.892824 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/92a227c2-cbea-4fc3-a721-b7fba16c0705-client-ca-bundle\") pod \"metrics-server-5cf5d68bb4-nb8ds\" (UID: \"92a227c2-cbea-4fc3-a721-b7fba16c0705\") " pod="openshift-monitoring/metrics-server-5cf5d68bb4-nb8ds" Sep 14 11:02:48.893045 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:48.892904 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/92a227c2-cbea-4fc3-a721-b7fba16c0705-audit-log\") pod \"metrics-server-5cf5d68bb4-nb8ds\" (UID: \"92a227c2-cbea-4fc3-a721-b7fba16c0705\") " pod="openshift-monitoring/metrics-server-5cf5d68bb4-nb8ds" Sep 14 11:02:48.893045 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:48.892954 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/92a227c2-cbea-4fc3-a721-b7fba16c0705-secret-metrics-server-client-certs\") pod \"metrics-server-5cf5d68bb4-nb8ds\" (UID: \"92a227c2-cbea-4fc3-a721-b7fba16c0705\") " pod="openshift-monitoring/metrics-server-5cf5d68bb4-nb8ds" Sep 14 11:02:48.893045 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:48.892978 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/92a227c2-cbea-4fc3-a721-b7fba16c0705-secret-metrics-server-tls\") pod \"metrics-server-5cf5d68bb4-nb8ds\" (UID: \"92a227c2-cbea-4fc3-a721-b7fba16c0705\") " pod="openshift-monitoring/metrics-server-5cf5d68bb4-nb8ds" Sep 14 11:02:48.893045 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:48.893002 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92a227c2-cbea-4fc3-a721-b7fba16c0705-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5cf5d68bb4-nb8ds\" (UID: \"92a227c2-cbea-4fc3-a721-b7fba16c0705\") " pod="openshift-monitoring/metrics-server-5cf5d68bb4-nb8ds" Sep 14 11:02:48.893260 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:48.893049 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/92a227c2-cbea-4fc3-a721-b7fba16c0705-metrics-server-audit-profiles\") pod \"metrics-server-5cf5d68bb4-nb8ds\" (UID: \"92a227c2-cbea-4fc3-a721-b7fba16c0705\") " pod="openshift-monitoring/metrics-server-5cf5d68bb4-nb8ds" Sep 14 11:02:48.893260 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:48.893089 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwzr4\" (UniqueName: \"kubernetes.io/projected/92a227c2-cbea-4fc3-a721-b7fba16c0705-kube-api-access-dwzr4\") pod \"metrics-server-5cf5d68bb4-nb8ds\" (UID: \"92a227c2-cbea-4fc3-a721-b7fba16c0705\") " pod="openshift-monitoring/metrics-server-5cf5d68bb4-nb8ds" Sep 14 11:02:48.893397 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:48.893372 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/92a227c2-cbea-4fc3-a721-b7fba16c0705-audit-log\") pod \"metrics-server-5cf5d68bb4-nb8ds\" (UID: \"92a227c2-cbea-4fc3-a721-b7fba16c0705\") " pod="openshift-monitoring/metrics-server-5cf5d68bb4-nb8ds" Sep 14 11:02:48.894119 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:48.894088 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92a227c2-cbea-4fc3-a721-b7fba16c0705-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5cf5d68bb4-nb8ds\" (UID: \"92a227c2-cbea-4fc3-a721-b7fba16c0705\") " pod="openshift-monitoring/metrics-server-5cf5d68bb4-nb8ds" Sep 14 11:02:48.894697 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:48.894650 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/92a227c2-cbea-4fc3-a721-b7fba16c0705-metrics-server-audit-profiles\") pod \"metrics-server-5cf5d68bb4-nb8ds\" (UID: \"92a227c2-cbea-4fc3-a721-b7fba16c0705\") " pod="openshift-monitoring/metrics-server-5cf5d68bb4-nb8ds" Sep 14 11:02:48.896232 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:48.896204 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/92a227c2-cbea-4fc3-a721-b7fba16c0705-client-ca-bundle\") pod \"metrics-server-5cf5d68bb4-nb8ds\" (UID: \"92a227c2-cbea-4fc3-a721-b7fba16c0705\") " pod="openshift-monitoring/metrics-server-5cf5d68bb4-nb8ds" Sep 14 11:02:48.897774 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:48.897733 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/92a227c2-cbea-4fc3-a721-b7fba16c0705-secret-metrics-server-tls\") pod \"metrics-server-5cf5d68bb4-nb8ds\" (UID: \"92a227c2-cbea-4fc3-a721-b7fba16c0705\") " pod="openshift-monitoring/metrics-server-5cf5d68bb4-nb8ds" Sep 14 11:02:48.897871 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:48.897821 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/92a227c2-cbea-4fc3-a721-b7fba16c0705-secret-metrics-server-client-certs\") pod \"metrics-server-5cf5d68bb4-nb8ds\" (UID: \"92a227c2-cbea-4fc3-a721-b7fba16c0705\") " pod="openshift-monitoring/metrics-server-5cf5d68bb4-nb8ds" Sep 14 11:02:48.904018 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:48.903976 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwzr4\" (UniqueName: \"kubernetes.io/projected/92a227c2-cbea-4fc3-a721-b7fba16c0705-kube-api-access-dwzr4\") pod \"metrics-server-5cf5d68bb4-nb8ds\" (UID: \"92a227c2-cbea-4fc3-a721-b7fba16c0705\") " pod="openshift-monitoring/metrics-server-5cf5d68bb4-nb8ds" Sep 14 11:02:49.047304 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.046983 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5cf5d68bb4-nb8ds" Sep 14 11:02:49.099535 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.099469 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-b6zt5"] Sep 14 11:02:49.113561 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.113529 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-b6zt5" Sep 14 11:02:49.122636 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.115867 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-b6zt5"] Sep 14 11:02:49.122636 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.117499 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 14 11:02:49.122636 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.117562 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-4st2j\"" Sep 14 11:02:49.196430 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.196397 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a3079479-adb1-43e3-9d0b-589673000bef-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-b6zt5\" (UID: \"a3079479-adb1-43e3-9d0b-589673000bef\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-b6zt5" Sep 14 11:02:49.196587 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.196453 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj2pr\" (UniqueName: \"kubernetes.io/projected/a3079479-adb1-43e3-9d0b-589673000bef-kube-api-access-zj2pr\") pod \"monitoring-plugin-5d85778b9d-b6zt5\" (UID: \"a3079479-adb1-43e3-9d0b-589673000bef\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-b6zt5" Sep 14 11:02:49.259283 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.259239 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-79cb87f565-swrl2" Sep 14 11:02:49.297014 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.296984 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a3079479-adb1-43e3-9d0b-589673000bef-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-b6zt5\" (UID: \"a3079479-adb1-43e3-9d0b-589673000bef\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-b6zt5" Sep 14 11:02:49.297014 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.297016 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj2pr\" (UniqueName: \"kubernetes.io/projected/a3079479-adb1-43e3-9d0b-589673000bef-kube-api-access-zj2pr\") pod \"monitoring-plugin-5d85778b9d-b6zt5\" (UID: \"a3079479-adb1-43e3-9d0b-589673000bef\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-b6zt5" Sep 14 11:02:49.297221 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:49.297154 2574 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Sep 14 11:02:49.297296 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:02:49.297236 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3079479-adb1-43e3-9d0b-589673000bef-monitoring-plugin-cert podName:a3079479-adb1-43e3-9d0b-589673000bef nodeName:}" failed. No retries permitted until 2026-09-14 11:02:49.797215222 +0000 UTC m=+114.417427732 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/a3079479-adb1-43e3-9d0b-589673000bef-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-b6zt5" (UID: "a3079479-adb1-43e3-9d0b-589673000bef") : secret "monitoring-plugin-cert" not found Sep 14 11:02:49.315863 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.315808 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj2pr\" (UniqueName: \"kubernetes.io/projected/a3079479-adb1-43e3-9d0b-589673000bef-kube-api-access-zj2pr\") pod \"monitoring-plugin-5d85778b9d-b6zt5\" (UID: \"a3079479-adb1-43e3-9d0b-589673000bef\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-b6zt5" Sep 14 11:02:49.362163 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.362124 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5db4d47bf7-8w6fr" event={"ID":"e7cead7b-bcec-4b90-bf0a-55057d5f39a6","Type":"ContainerStarted","Data":"0cfce2d1b01e705c15d851fb25d017c2652c97187649aabeb7efade74ebb4ece"} Sep 14 11:02:49.403953 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.403905 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5db4d47bf7-8w6fr" podStartSLOduration=4.602837268 podStartE2EDuration="8.403888425s" totalImagesPullingTime="3.801051157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:02:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:49.402704026 +0000 UTC m=+114.022916578" watchObservedRunningTime="2026-09-14 11:02:49.403888425 +0000 UTC m=+114.024100947" Sep 14 11:02:49.634951 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.634918 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-85fdf9d4cc-sk57l"] Sep 14 11:02:49.663604 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.657715 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-85fdf9d4cc-sk57l"] Sep 14 11:02:49.663604 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.657837 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-85fdf9d4cc-sk57l" Sep 14 11:02:49.663604 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.660634 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 14 11:02:49.663604 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.660837 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 14 11:02:49.663604 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.661255 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 14 11:02:49.663604 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.661349 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-mx2hq\"" Sep 14 11:02:49.663604 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.661478 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 14 11:02:49.663604 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.661776 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 14 11:02:49.668458 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.668436 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 14 11:02:49.701643 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.701613 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ql9s\" (UniqueName: \"kubernetes.io/projected/560befdf-25ba-4c18-8946-30f75f0e53b1-kube-api-access-4ql9s\") pod \"telemeter-client-85fdf9d4cc-sk57l\" (UID: \"560befdf-25ba-4c18-8946-30f75f0e53b1\") " pod="openshift-monitoring/telemeter-client-85fdf9d4cc-sk57l" Sep 14 11:02:49.701762 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.701659 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/560befdf-25ba-4c18-8946-30f75f0e53b1-metrics-client-ca\") pod \"telemeter-client-85fdf9d4cc-sk57l\" (UID: \"560befdf-25ba-4c18-8946-30f75f0e53b1\") " pod="openshift-monitoring/telemeter-client-85fdf9d4cc-sk57l" Sep 14 11:02:49.701762 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.701708 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/560befdf-25ba-4c18-8946-30f75f0e53b1-secret-telemeter-client\") pod \"telemeter-client-85fdf9d4cc-sk57l\" (UID: \"560befdf-25ba-4c18-8946-30f75f0e53b1\") " pod="openshift-monitoring/telemeter-client-85fdf9d4cc-sk57l" Sep 14 11:02:49.701762 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.701753 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/560befdf-25ba-4c18-8946-30f75f0e53b1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-85fdf9d4cc-sk57l\" (UID: \"560befdf-25ba-4c18-8946-30f75f0e53b1\") " pod="openshift-monitoring/telemeter-client-85fdf9d4cc-sk57l" Sep 14 11:02:49.701935 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.701826 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/560befdf-25ba-4c18-8946-30f75f0e53b1-telemeter-client-tls\") pod \"telemeter-client-85fdf9d4cc-sk57l\" (UID: \"560befdf-25ba-4c18-8946-30f75f0e53b1\") " pod="openshift-monitoring/telemeter-client-85fdf9d4cc-sk57l" Sep 14 11:02:49.701935 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.701855 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/560befdf-25ba-4c18-8946-30f75f0e53b1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-85fdf9d4cc-sk57l\" (UID: \"560befdf-25ba-4c18-8946-30f75f0e53b1\") " pod="openshift-monitoring/telemeter-client-85fdf9d4cc-sk57l" Sep 14 11:02:49.701935 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.701907 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/560befdf-25ba-4c18-8946-30f75f0e53b1-serving-certs-ca-bundle\") pod \"telemeter-client-85fdf9d4cc-sk57l\" (UID: \"560befdf-25ba-4c18-8946-30f75f0e53b1\") " pod="openshift-monitoring/telemeter-client-85fdf9d4cc-sk57l" Sep 14 11:02:49.702089 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.701945 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/560befdf-25ba-4c18-8946-30f75f0e53b1-federate-client-tls\") pod \"telemeter-client-85fdf9d4cc-sk57l\" (UID: \"560befdf-25ba-4c18-8946-30f75f0e53b1\") " pod="openshift-monitoring/telemeter-client-85fdf9d4cc-sk57l" Sep 14 11:02:49.803808 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.802597 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/560befdf-25ba-4c18-8946-30f75f0e53b1-serving-certs-ca-bundle\") pod \"telemeter-client-85fdf9d4cc-sk57l\" (UID: \"560befdf-25ba-4c18-8946-30f75f0e53b1\") " pod="openshift-monitoring/telemeter-client-85fdf9d4cc-sk57l" Sep 14 11:02:49.803808 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.802648 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/560befdf-25ba-4c18-8946-30f75f0e53b1-federate-client-tls\") pod \"telemeter-client-85fdf9d4cc-sk57l\" (UID: \"560befdf-25ba-4c18-8946-30f75f0e53b1\") " pod="openshift-monitoring/telemeter-client-85fdf9d4cc-sk57l" Sep 14 11:02:49.803808 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.802678 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ql9s\" (UniqueName: \"kubernetes.io/projected/560befdf-25ba-4c18-8946-30f75f0e53b1-kube-api-access-4ql9s\") pod \"telemeter-client-85fdf9d4cc-sk57l\" (UID: \"560befdf-25ba-4c18-8946-30f75f0e53b1\") " pod="openshift-monitoring/telemeter-client-85fdf9d4cc-sk57l" Sep 14 11:02:49.803808 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.802706 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/560befdf-25ba-4c18-8946-30f75f0e53b1-metrics-client-ca\") pod \"telemeter-client-85fdf9d4cc-sk57l\" (UID: \"560befdf-25ba-4c18-8946-30f75f0e53b1\") " pod="openshift-monitoring/telemeter-client-85fdf9d4cc-sk57l" Sep 14 11:02:49.803808 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.802760 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/560befdf-25ba-4c18-8946-30f75f0e53b1-secret-telemeter-client\") pod \"telemeter-client-85fdf9d4cc-sk57l\" (UID: \"560befdf-25ba-4c18-8946-30f75f0e53b1\") " pod="openshift-monitoring/telemeter-client-85fdf9d4cc-sk57l" Sep 14 11:02:49.803808 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.802807 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/560befdf-25ba-4c18-8946-30f75f0e53b1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-85fdf9d4cc-sk57l\" (UID: \"560befdf-25ba-4c18-8946-30f75f0e53b1\") " pod="openshift-monitoring/telemeter-client-85fdf9d4cc-sk57l" Sep 14 11:02:49.803808 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.802851 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a3079479-adb1-43e3-9d0b-589673000bef-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-b6zt5\" (UID: \"a3079479-adb1-43e3-9d0b-589673000bef\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-b6zt5" Sep 14 11:02:49.803808 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.802904 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/560befdf-25ba-4c18-8946-30f75f0e53b1-telemeter-client-tls\") pod \"telemeter-client-85fdf9d4cc-sk57l\" (UID: \"560befdf-25ba-4c18-8946-30f75f0e53b1\") " pod="openshift-monitoring/telemeter-client-85fdf9d4cc-sk57l" Sep 14 11:02:49.803808 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.802936 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/560befdf-25ba-4c18-8946-30f75f0e53b1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-85fdf9d4cc-sk57l\" (UID: \"560befdf-25ba-4c18-8946-30f75f0e53b1\") " pod="openshift-monitoring/telemeter-client-85fdf9d4cc-sk57l" Sep 14 11:02:49.804636 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.803869 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/560befdf-25ba-4c18-8946-30f75f0e53b1-serving-certs-ca-bundle\") pod \"telemeter-client-85fdf9d4cc-sk57l\" (UID: \"560befdf-25ba-4c18-8946-30f75f0e53b1\") " pod="openshift-monitoring/telemeter-client-85fdf9d4cc-sk57l" Sep 14 11:02:49.805147 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.805098 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/560befdf-25ba-4c18-8946-30f75f0e53b1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-85fdf9d4cc-sk57l\" (UID: \"560befdf-25ba-4c18-8946-30f75f0e53b1\") " pod="openshift-monitoring/telemeter-client-85fdf9d4cc-sk57l" Sep 14 11:02:49.805449 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.805426 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/560befdf-25ba-4c18-8946-30f75f0e53b1-metrics-client-ca\") pod \"telemeter-client-85fdf9d4cc-sk57l\" (UID: \"560befdf-25ba-4c18-8946-30f75f0e53b1\") " pod="openshift-monitoring/telemeter-client-85fdf9d4cc-sk57l" Sep 14 11:02:49.805892 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.805870 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/560befdf-25ba-4c18-8946-30f75f0e53b1-secret-telemeter-client\") pod \"telemeter-client-85fdf9d4cc-sk57l\" (UID: \"560befdf-25ba-4c18-8946-30f75f0e53b1\") " pod="openshift-monitoring/telemeter-client-85fdf9d4cc-sk57l" Sep 14 11:02:49.807502 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.807474 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/560befdf-25ba-4c18-8946-30f75f0e53b1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-85fdf9d4cc-sk57l\" (UID: \"560befdf-25ba-4c18-8946-30f75f0e53b1\") " pod="openshift-monitoring/telemeter-client-85fdf9d4cc-sk57l" Sep 14 11:02:49.808305 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.808257 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a3079479-adb1-43e3-9d0b-589673000bef-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-b6zt5\" (UID: \"a3079479-adb1-43e3-9d0b-589673000bef\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-b6zt5" Sep 14 11:02:49.808732 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.808710 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/560befdf-25ba-4c18-8946-30f75f0e53b1-federate-client-tls\") pod \"telemeter-client-85fdf9d4cc-sk57l\" (UID: \"560befdf-25ba-4c18-8946-30f75f0e53b1\") " pod="openshift-monitoring/telemeter-client-85fdf9d4cc-sk57l" Sep 14 11:02:49.810001 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.809975 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/560befdf-25ba-4c18-8946-30f75f0e53b1-telemeter-client-tls\") pod \"telemeter-client-85fdf9d4cc-sk57l\" (UID: \"560befdf-25ba-4c18-8946-30f75f0e53b1\") " pod="openshift-monitoring/telemeter-client-85fdf9d4cc-sk57l" Sep 14 11:02:49.818768 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.818732 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ql9s\" (UniqueName: \"kubernetes.io/projected/560befdf-25ba-4c18-8946-30f75f0e53b1-kube-api-access-4ql9s\") pod \"telemeter-client-85fdf9d4cc-sk57l\" (UID: \"560befdf-25ba-4c18-8946-30f75f0e53b1\") " pod="openshift-monitoring/telemeter-client-85fdf9d4cc-sk57l" Sep 14 11:02:49.959126 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.959098 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5cf5d68bb4-nb8ds"] Sep 14 11:02:49.964189 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:02:49.964163 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92a227c2_cbea_4fc3_a721_b7fba16c0705.slice/crio-dec0b3bb235eb4b53f1098281d939265254edc602ee0505d806a75608248d4a3 WatchSource:0}: Error finding container dec0b3bb235eb4b53f1098281d939265254edc602ee0505d806a75608248d4a3: Status 404 returned error can't find the container with id dec0b3bb235eb4b53f1098281d939265254edc602ee0505d806a75608248d4a3 Sep 14 11:02:49.988907 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:49.988884 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-85fdf9d4cc-sk57l" Sep 14 11:02:50.034709 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:50.033405 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-b6zt5" Sep 14 11:02:50.282810 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:50.282738 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-85fdf9d4cc-sk57l"] Sep 14 11:02:50.285299 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:02:50.285251 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod560befdf_25ba_4c18_8946_30f75f0e53b1.slice/crio-1638d4b6c639a718a844ac9b453a67922c0935fd63a4aa42138b9c4890eec7a5 WatchSource:0}: Error finding container 1638d4b6c639a718a844ac9b453a67922c0935fd63a4aa42138b9c4890eec7a5: Status 404 returned error can't find the container with id 1638d4b6c639a718a844ac9b453a67922c0935fd63a4aa42138b9c4890eec7a5 Sep 14 11:02:50.334995 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:50.334969 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-b6zt5"] Sep 14 11:02:50.339529 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:02:50.339499 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3079479_adb1_43e3_9d0b_589673000bef.slice/crio-4103c086d8e5c31dda9885b972df466dbbfb12b17e5c1f9d3e622a2c89b5a04f WatchSource:0}: Error finding container 4103c086d8e5c31dda9885b972df466dbbfb12b17e5c1f9d3e622a2c89b5a04f: Status 404 returned error can't find the container with id 4103c086d8e5c31dda9885b972df466dbbfb12b17e5c1f9d3e622a2c89b5a04f Sep 14 11:02:50.373780 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:50.373745 2574 generic.go:356] "Generic (PLEG): container finished" podID="6337f168-6fbc-4c83-ab67-9476b93b697c" containerID="1fb410762a09e62ea3e3abe4d9e527a29e93486a8ce38ba5f2a28a920d04267a" exitCode=0 Sep 14 11:02:50.373886 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:50.373841 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k25g8" event={"ID":"6337f168-6fbc-4c83-ab67-9476b93b697c","Type":"ContainerDied","Data":"1fb410762a09e62ea3e3abe4d9e527a29e93486a8ce38ba5f2a28a920d04267a"} Sep 14 11:02:50.375301 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:50.375254 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-b6zt5" event={"ID":"a3079479-adb1-43e3-9d0b-589673000bef","Type":"ContainerStarted","Data":"4103c086d8e5c31dda9885b972df466dbbfb12b17e5c1f9d3e622a2c89b5a04f"} Sep 14 11:02:50.377170 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:50.377149 2574 generic.go:356] "Generic (PLEG): container finished" podID="6406ef1e-5938-48da-8c40-54d4c0e717e9" containerID="8bf13091b7ab858c6a44d6d390b879938f8e88bc23779eabc563329b7b622e90" exitCode=0 Sep 14 11:02:50.377301 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:50.377227 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6406ef1e-5938-48da-8c40-54d4c0e717e9","Type":"ContainerDied","Data":"8bf13091b7ab858c6a44d6d390b879938f8e88bc23779eabc563329b7b622e90"} Sep 14 11:02:50.379158 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:50.379127 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-85fdf9d4cc-sk57l" event={"ID":"560befdf-25ba-4c18-8946-30f75f0e53b1","Type":"ContainerStarted","Data":"1638d4b6c639a718a844ac9b453a67922c0935fd63a4aa42138b9c4890eec7a5"} Sep 14 11:02:50.380563 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:50.380540 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5cf5d68bb4-nb8ds" event={"ID":"92a227c2-cbea-4fc3-a721-b7fba16c0705","Type":"ContainerStarted","Data":"dec0b3bb235eb4b53f1098281d939265254edc602ee0505d806a75608248d4a3"} Sep 14 11:02:50.941073 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:50.938948 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 14 11:02:50.958781 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:50.958751 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:50.967609 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:50.966880 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 14 11:02:50.967609 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:50.967090 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-ttctr\"" Sep 14 11:02:50.967609 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:50.967129 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 14 11:02:50.967609 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:50.967261 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 14 11:02:50.967609 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:50.967430 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 14 11:02:50.967609 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:50.967478 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 14 11:02:50.967981 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:50.967675 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 14 11:02:50.968300 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:50.968113 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 14 11:02:50.969703 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:50.969475 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7mlmallehfrak\"" Sep 14 11:02:50.972326 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:50.971957 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 14 11:02:50.974538 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:50.974508 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 14 11:02:50.977291 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:50.974721 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 14 11:02:50.977291 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:50.976792 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 14 11:02:50.977291 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:50.977008 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 14 11:02:50.977483 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:50.977409 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 14 11:02:50.981158 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:50.981139 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 14 11:02:51.016299 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.015968 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53829fff-bda5-4556-885c-6eb9465ecb04-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.016299 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.016007 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53829fff-bda5-4556-885c-6eb9465ecb04-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.016299 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.016033 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnz85\" (UniqueName: \"kubernetes.io/projected/53829fff-bda5-4556-885c-6eb9465ecb04-kube-api-access-wnz85\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.016299 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.016064 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/53829fff-bda5-4556-885c-6eb9465ecb04-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.016299 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.016091 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/53829fff-bda5-4556-885c-6eb9465ecb04-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.016299 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.016116 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/53829fff-bda5-4556-885c-6eb9465ecb04-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.016299 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.016167 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53829fff-bda5-4556-885c-6eb9465ecb04-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.016299 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.016192 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/53829fff-bda5-4556-885c-6eb9465ecb04-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.016299 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.016230 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/53829fff-bda5-4556-885c-6eb9465ecb04-web-config\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.016299 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.016254 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/53829fff-bda5-4556-885c-6eb9465ecb04-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.017873 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.016978 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/53829fff-bda5-4556-885c-6eb9465ecb04-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.017873 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.017031 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/53829fff-bda5-4556-885c-6eb9465ecb04-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.017873 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.017083 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/53829fff-bda5-4556-885c-6eb9465ecb04-config\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.017873 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.017133 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/53829fff-bda5-4556-885c-6eb9465ecb04-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.017873 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.017166 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/53829fff-bda5-4556-885c-6eb9465ecb04-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.017873 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.017195 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53829fff-bda5-4556-885c-6eb9465ecb04-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.017873 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.017246 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/53829fff-bda5-4556-885c-6eb9465ecb04-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.017873 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.017298 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/53829fff-bda5-4556-885c-6eb9465ecb04-config-out\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.017873 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.017331 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/53829fff-bda5-4556-885c-6eb9465ecb04-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.017873 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.017354 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/53829fff-bda5-4556-885c-6eb9465ecb04-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.017873 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.017442 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 14 11:02:51.119347 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.118490 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/53829fff-bda5-4556-885c-6eb9465ecb04-config\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.119347 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.118556 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/53829fff-bda5-4556-885c-6eb9465ecb04-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.119347 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.118588 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/53829fff-bda5-4556-885c-6eb9465ecb04-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.119347 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.118623 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53829fff-bda5-4556-885c-6eb9465ecb04-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.119347 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.118659 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/53829fff-bda5-4556-885c-6eb9465ecb04-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.119347 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.118688 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/53829fff-bda5-4556-885c-6eb9465ecb04-config-out\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.119347 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.118717 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/53829fff-bda5-4556-885c-6eb9465ecb04-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.119347 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.118744 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/53829fff-bda5-4556-885c-6eb9465ecb04-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.119347 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.118781 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53829fff-bda5-4556-885c-6eb9465ecb04-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.119347 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.118803 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53829fff-bda5-4556-885c-6eb9465ecb04-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.119347 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.118827 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnz85\" (UniqueName: \"kubernetes.io/projected/53829fff-bda5-4556-885c-6eb9465ecb04-kube-api-access-wnz85\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.119347 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.118858 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/53829fff-bda5-4556-885c-6eb9465ecb04-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.119347 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.118890 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/53829fff-bda5-4556-885c-6eb9465ecb04-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.119347 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.118911 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/53829fff-bda5-4556-885c-6eb9465ecb04-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.119347 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.118957 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53829fff-bda5-4556-885c-6eb9465ecb04-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.119347 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.118984 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/53829fff-bda5-4556-885c-6eb9465ecb04-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.119347 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.119022 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/53829fff-bda5-4556-885c-6eb9465ecb04-web-config\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.120368 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.119043 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/53829fff-bda5-4556-885c-6eb9465ecb04-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.120368 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.119093 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/53829fff-bda5-4556-885c-6eb9465ecb04-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.120368 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.119120 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/53829fff-bda5-4556-885c-6eb9465ecb04-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.120368 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.120095 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53829fff-bda5-4556-885c-6eb9465ecb04-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.122176 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.122145 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/53829fff-bda5-4556-885c-6eb9465ecb04-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.123662 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.122505 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53829fff-bda5-4556-885c-6eb9465ecb04-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.123662 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.123344 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53829fff-bda5-4556-885c-6eb9465ecb04-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.127833 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.127138 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53829fff-bda5-4556-885c-6eb9465ecb04-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.127833 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.127207 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/53829fff-bda5-4556-885c-6eb9465ecb04-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.129386 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.129292 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/53829fff-bda5-4556-885c-6eb9465ecb04-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.130008 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.129986 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/53829fff-bda5-4556-885c-6eb9465ecb04-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.131914 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.130524 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/53829fff-bda5-4556-885c-6eb9465ecb04-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.132071 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.130541 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/53829fff-bda5-4556-885c-6eb9465ecb04-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.137454 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.137410 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/53829fff-bda5-4556-885c-6eb9465ecb04-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.137966 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.137927 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/53829fff-bda5-4556-885c-6eb9465ecb04-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.139009 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.138499 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/53829fff-bda5-4556-885c-6eb9465ecb04-web-config\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.143019 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.142978 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/53829fff-bda5-4556-885c-6eb9465ecb04-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.143626 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.143572 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/53829fff-bda5-4556-885c-6eb9465ecb04-config\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.144192 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.144150 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/53829fff-bda5-4556-885c-6eb9465ecb04-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.144896 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.144858 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/53829fff-bda5-4556-885c-6eb9465ecb04-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.145242 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.145200 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/53829fff-bda5-4556-885c-6eb9465ecb04-config-out\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.149640 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.149600 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/53829fff-bda5-4556-885c-6eb9465ecb04-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.154927 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.154888 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnz85\" (UniqueName: \"kubernetes.io/projected/53829fff-bda5-4556-885c-6eb9465ecb04-kube-api-access-wnz85\") pod \"prometheus-k8s-0\" (UID: \"53829fff-bda5-4556-885c-6eb9465ecb04\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.275924 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.275570 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:02:51.407232 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.406948 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k25g8" event={"ID":"6337f168-6fbc-4c83-ab67-9476b93b697c","Type":"ContainerStarted","Data":"e52fda7265c38ab89b52438fd3e5c82b988c9af44824d0609c95f73e6271b27b"} Sep 14 11:02:51.407232 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.407008 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k25g8" event={"ID":"6337f168-6fbc-4c83-ab67-9476b93b697c","Type":"ContainerStarted","Data":"a5184cbf50eae198707ca822e4cec8cbf4f00b078489d366539a730ba450e6f1"} Sep 14 11:02:51.612906 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.612635 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-k25g8" podStartSLOduration=3.156935717 podStartE2EDuration="7.612616815s" totalImagesPullingTime="4.376055966s" totalInitContainerRuntime="79.625132ms" isStatefulPod=true podCreationTimestamp="2026-09-14 11:02:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:51.436228493 +0000 UTC m=+116.056441033" watchObservedRunningTime="2026-09-14 11:02:51.612616815 +0000 UTC m=+116.232829403" Sep 14 11:02:51.613446 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.613364 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 14 11:02:51.691584 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.691527 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5db4d47bf7-8w6fr" Sep 14 11:02:51.691948 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.691799 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5db4d47bf7-8w6fr" Sep 14 11:02:51.699512 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:51.699448 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5db4d47bf7-8w6fr" Sep 14 11:02:52.329362 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:52.329243 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7f8c696848-hc5hp" podUID="c693407b-c967-4981-950c-e7630d4dff7a" containerName="registry" containerID="cri-o://bab6a89802165696756eab8719eaf0d0db539f4e7a98215ac999c9a690c70aaa" gracePeriod=30 Sep 14 11:02:52.417669 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:52.417643 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5db4d47bf7-8w6fr" Sep 14 11:02:53.417671 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:53.417621 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"53829fff-bda5-4556-885c-6eb9465ecb04","Type":"ContainerStarted","Data":"6c3fcf23a1c68c01ab7b0dca2b8b8655eee90196bc7add00e690ac405e58aadc"} Sep 14 11:02:54.422239 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:54.422166 2574 generic.go:356] "Generic (PLEG): container finished" podID="c693407b-c967-4981-950c-e7630d4dff7a" containerID="bab6a89802165696756eab8719eaf0d0db539f4e7a98215ac999c9a690c70aaa" exitCode=0 Sep 14 11:02:54.422239 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:54.422214 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f8c696848-hc5hp" event={"ID":"c693407b-c967-4981-950c-e7630d4dff7a","Type":"ContainerDied","Data":"bab6a89802165696756eab8719eaf0d0db539f4e7a98215ac999c9a690c70aaa"} Sep 14 11:02:54.496407 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:54.496386 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f8c696848-hc5hp" Sep 14 11:02:54.556170 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:54.556143 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c693407b-c967-4981-950c-e7630d4dff7a-image-registry-private-configuration\") pod \"c693407b-c967-4981-950c-e7630d4dff7a\" (UID: \"c693407b-c967-4981-950c-e7630d4dff7a\") " Sep 14 11:02:54.556291 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:54.556181 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c693407b-c967-4981-950c-e7630d4dff7a-trusted-ca\") pod \"c693407b-c967-4981-950c-e7630d4dff7a\" (UID: \"c693407b-c967-4981-950c-e7630d4dff7a\") " Sep 14 11:02:54.556291 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:54.556205 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c693407b-c967-4981-950c-e7630d4dff7a-bound-sa-token\") pod \"c693407b-c967-4981-950c-e7630d4dff7a\" (UID: \"c693407b-c967-4981-950c-e7630d4dff7a\") " Sep 14 11:02:54.556291 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:54.556234 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c693407b-c967-4981-950c-e7630d4dff7a-registry-certificates\") pod \"c693407b-c967-4981-950c-e7630d4dff7a\" (UID: \"c693407b-c967-4981-950c-e7630d4dff7a\") " Sep 14 11:02:54.556291 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:54.556261 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c693407b-c967-4981-950c-e7630d4dff7a-registry-tls\") pod \"c693407b-c967-4981-950c-e7630d4dff7a\" (UID: \"c693407b-c967-4981-950c-e7630d4dff7a\") " Sep 14 11:02:54.556497 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:54.556318 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c693407b-c967-4981-950c-e7630d4dff7a-ca-trust-extracted\") pod \"c693407b-c967-4981-950c-e7630d4dff7a\" (UID: \"c693407b-c967-4981-950c-e7630d4dff7a\") " Sep 14 11:02:54.556497 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:54.556341 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjf6s\" (UniqueName: \"kubernetes.io/projected/c693407b-c967-4981-950c-e7630d4dff7a-kube-api-access-mjf6s\") pod \"c693407b-c967-4981-950c-e7630d4dff7a\" (UID: \"c693407b-c967-4981-950c-e7630d4dff7a\") " Sep 14 11:02:54.556497 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:54.556418 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c693407b-c967-4981-950c-e7630d4dff7a-installation-pull-secrets\") pod \"c693407b-c967-4981-950c-e7630d4dff7a\" (UID: \"c693407b-c967-4981-950c-e7630d4dff7a\") " Sep 14 11:02:54.556642 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:54.556608 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c693407b-c967-4981-950c-e7630d4dff7a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c693407b-c967-4981-950c-e7630d4dff7a" (UID: "c693407b-c967-4981-950c-e7630d4dff7a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:02:54.558626 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:54.556776 2574 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c693407b-c967-4981-950c-e7630d4dff7a-registry-certificates\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:02:54.558626 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:54.557160 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c693407b-c967-4981-950c-e7630d4dff7a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c693407b-c967-4981-950c-e7630d4dff7a" (UID: "c693407b-c967-4981-950c-e7630d4dff7a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:02:54.559557 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:54.559420 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c693407b-c967-4981-950c-e7630d4dff7a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c693407b-c967-4981-950c-e7630d4dff7a" (UID: "c693407b-c967-4981-950c-e7630d4dff7a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:02:54.559837 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:54.559801 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c693407b-c967-4981-950c-e7630d4dff7a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c693407b-c967-4981-950c-e7630d4dff7a" (UID: "c693407b-c967-4981-950c-e7630d4dff7a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:02:54.559920 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:54.559897 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c693407b-c967-4981-950c-e7630d4dff7a-kube-api-access-mjf6s" (OuterVolumeSpecName: "kube-api-access-mjf6s") pod "c693407b-c967-4981-950c-e7630d4dff7a" (UID: "c693407b-c967-4981-950c-e7630d4dff7a"). InnerVolumeSpecName "kube-api-access-mjf6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:02:54.560016 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:54.559994 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c693407b-c967-4981-950c-e7630d4dff7a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c693407b-c967-4981-950c-e7630d4dff7a" (UID: "c693407b-c967-4981-950c-e7630d4dff7a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:02:54.560734 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:54.560714 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c693407b-c967-4981-950c-e7630d4dff7a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c693407b-c967-4981-950c-e7630d4dff7a" (UID: "c693407b-c967-4981-950c-e7630d4dff7a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:02:54.568352 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:54.568327 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c693407b-c967-4981-950c-e7630d4dff7a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c693407b-c967-4981-950c-e7630d4dff7a" (UID: "c693407b-c967-4981-950c-e7630d4dff7a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:02:54.657657 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:54.657571 2574 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c693407b-c967-4981-950c-e7630d4dff7a-ca-trust-extracted\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:02:54.657657 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:54.657600 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjf6s\" (UniqueName: \"kubernetes.io/projected/c693407b-c967-4981-950c-e7630d4dff7a-kube-api-access-mjf6s\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:02:54.657657 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:54.657615 2574 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c693407b-c967-4981-950c-e7630d4dff7a-installation-pull-secrets\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:02:54.657657 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:54.657631 2574 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c693407b-c967-4981-950c-e7630d4dff7a-image-registry-private-configuration\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:02:54.657657 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:54.657646 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c693407b-c967-4981-950c-e7630d4dff7a-trusted-ca\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:02:54.657657 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:54.657659 2574 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c693407b-c967-4981-950c-e7630d4dff7a-bound-sa-token\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:02:54.657657 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:54.657671 2574 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c693407b-c967-4981-950c-e7630d4dff7a-registry-tls\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:02:55.351105 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:55.351076 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-md5h2" Sep 14 11:02:55.428066 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:55.428031 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-b6zt5" event={"ID":"a3079479-adb1-43e3-9d0b-589673000bef","Type":"ContainerStarted","Data":"e552545a92c66c1e589259e08b2c2330bfcd97ae01332a527f4b8e4b723532e6"} Sep 14 11:02:55.428561 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:55.428253 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-b6zt5" Sep 14 11:02:55.430745 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:55.430620 2574 generic.go:356] "Generic (PLEG): container finished" podID="53829fff-bda5-4556-885c-6eb9465ecb04" containerID="6fb6c56db093b28447081a9be7f3a8b88b5d0fe269dd4ae871ea9c67daf75475" exitCode=0 Sep 14 11:02:55.430745 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:55.430698 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"53829fff-bda5-4556-885c-6eb9465ecb04","Type":"ContainerDied","Data":"6fb6c56db093b28447081a9be7f3a8b88b5d0fe269dd4ae871ea9c67daf75475"} Sep 14 11:02:55.434123 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:55.434103 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-b6zt5" Sep 14 11:02:55.434846 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:55.434824 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6406ef1e-5938-48da-8c40-54d4c0e717e9","Type":"ContainerStarted","Data":"d9d68bb88958a1ffb95db245b4d012bdbee54c857b1355f7635a04668333d53b"} Sep 14 11:02:55.434939 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:55.434879 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6406ef1e-5938-48da-8c40-54d4c0e717e9","Type":"ContainerStarted","Data":"e26c6aae42308a7f0a46d5a90b660a88a208d85eeab38d59965f8b64c585e5d3"} Sep 14 11:02:55.434939 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:55.434913 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6406ef1e-5938-48da-8c40-54d4c0e717e9","Type":"ContainerStarted","Data":"cfcc428c3061863d69a583268fd41d338dffc02d86087783f33b761157856956"} Sep 14 11:02:55.435026 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:55.434947 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6406ef1e-5938-48da-8c40-54d4c0e717e9","Type":"ContainerStarted","Data":"a0e0289a6e19baaa3b36cdad98299aea4ea6505a334df64769a294bf6066d772"} Sep 14 11:02:55.435026 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:55.434981 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6406ef1e-5938-48da-8c40-54d4c0e717e9","Type":"ContainerStarted","Data":"7d8f47312e7be1613cedf5161a68de1fc0f78aec110e45939c4d8451637b5d79"} Sep 14 11:02:55.437293 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:55.437244 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-85fdf9d4cc-sk57l" event={"ID":"560befdf-25ba-4c18-8946-30f75f0e53b1","Type":"ContainerStarted","Data":"374524407b98456494a102f4061e3f74aa08320d06a71173b5c3694fd6fb5001"} Sep 14 11:02:55.437411 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:55.437313 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-85fdf9d4cc-sk57l" event={"ID":"560befdf-25ba-4c18-8946-30f75f0e53b1","Type":"ContainerStarted","Data":"1f00d79327533ea23c48ff92207ae877ea153e2aec08cdddfe3694170c925bf3"} Sep 14 11:02:55.437411 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:55.437342 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-85fdf9d4cc-sk57l" event={"ID":"560befdf-25ba-4c18-8946-30f75f0e53b1","Type":"ContainerStarted","Data":"80d0ed4ff021355ba92b95361ff617d7c1e6df8a4cbdda744f5c739919ce7a31"} Sep 14 11:02:55.438801 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:55.438786 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f8c696848-hc5hp" Sep 14 11:02:55.438801 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:55.438790 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f8c696848-hc5hp" event={"ID":"c693407b-c967-4981-950c-e7630d4dff7a","Type":"ContainerDied","Data":"e493ec37d496460566126d81b9e6087a466abd57dd8c9d7c53d261e92e458288"} Sep 14 11:02:55.438938 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:55.438827 2574 scope.go:117] "RemoveContainer" containerID="bab6a89802165696756eab8719eaf0d0db539f4e7a98215ac999c9a690c70aaa" Sep 14 11:02:55.440551 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:55.440525 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5cf5d68bb4-nb8ds" event={"ID":"92a227c2-cbea-4fc3-a721-b7fba16c0705","Type":"ContainerStarted","Data":"46ef684bd8875b7a4b22417959a4b195b8093ae88c1f4e2932f0836d6239f562"} Sep 14 11:02:55.461823 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:55.461781 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-b6zt5" podStartSLOduration=2.346150008 podStartE2EDuration="6.461764072s" totalImagesPullingTime="4.115614064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:02:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:55.460231142 +0000 UTC m=+120.080443666" watchObservedRunningTime="2026-09-14 11:02:55.461764072 +0000 UTC m=+120.081976598" Sep 14 11:02:55.483311 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:55.483259 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f8c696848-hc5hp"] Sep 14 11:02:55.487152 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:55.487133 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7f8c696848-hc5hp"] Sep 14 11:02:55.577773 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:55.577726 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5cf5d68bb4-nb8ds" podStartSLOduration=3.087354441 podStartE2EDuration="7.57771151s" totalImagesPullingTime="4.490357069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:02:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:55.57743998 +0000 UTC m=+120.197652504" watchObservedRunningTime="2026-09-14 11:02:55.57771151 +0000 UTC m=+120.197924033" Sep 14 11:02:55.625098 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:55.625037 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-85fdf9d4cc-sk57l" podStartSLOduration=2.034591048 podStartE2EDuration="6.625022673s" totalImagesPullingTime="4.590431625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:02:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:55.624006247 +0000 UTC m=+120.244218788" watchObservedRunningTime="2026-09-14 11:02:55.625022673 +0000 UTC m=+120.245235197" Sep 14 11:02:55.899170 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:55.899141 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c693407b-c967-4981-950c-e7630d4dff7a" path="/var/lib/kubelet/pods/c693407b-c967-4981-950c-e7630d4dff7a/volumes" Sep 14 11:02:57.453334 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:57.453295 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6406ef1e-5938-48da-8c40-54d4c0e717e9","Type":"ContainerStarted","Data":"2a295946fc17a59e16767e640769f75155591ea2f9b46c0a1dc316b82abc225f"} Sep 14 11:02:57.497894 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:02:57.497722 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.4823877100000002 podStartE2EDuration="12.497704482s" totalImagesPullingTime="8.99215822s" totalInitContainerRuntime="23.158552ms" isStatefulPod=false podCreationTimestamp="2026-09-14 11:02:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:02:57.496874799 +0000 UTC m=+122.117087333" watchObservedRunningTime="2026-09-14 11:02:57.497704482 +0000 UTC m=+122.117917004" Sep 14 11:03:00.466235 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:00.466203 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"53829fff-bda5-4556-885c-6eb9465ecb04","Type":"ContainerStarted","Data":"f11d40eefe9a8a481653e98f6f26a9ee90f9137654b7b05dbd45b16652f5715a"} Sep 14 11:03:00.466595 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:00.466278 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"53829fff-bda5-4556-885c-6eb9465ecb04","Type":"ContainerStarted","Data":"8ad15cb0c938b12bca4c4111659f99e20c8c8bdfaf4f0588f5742f9bad6fdca1"} Sep 14 11:03:03.483787 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:03.483738 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"53829fff-bda5-4556-885c-6eb9465ecb04","Type":"ContainerStarted","Data":"b2786cc2bdf079d6f293009c1bc1b32a3f3bf80159d9d9a2104d9dc6dc685c89"} Sep 14 11:03:03.484316 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:03.483825 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"53829fff-bda5-4556-885c-6eb9465ecb04","Type":"ContainerStarted","Data":"89a073ce9d37a8aac0854d9d955883ce2a867b0043efdc8c13e36d33467af2e9"} Sep 14 11:03:03.484316 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:03.483870 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"53829fff-bda5-4556-885c-6eb9465ecb04","Type":"ContainerStarted","Data":"bd168c305840084155d28a2cbbd945de4cad762d7c554f23b3f0614a89a2834a"} Sep 14 11:03:03.484316 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:03.483955 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"53829fff-bda5-4556-885c-6eb9465ecb04","Type":"ContainerStarted","Data":"602708c7b2f73a629dd6f7064bbea5bc1a44b19299e9192d4f5d5dee3e3a76ae"} Sep 14 11:03:03.521037 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:03.519443 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=6.326400313 podStartE2EDuration="13.519422927s" totalImagesPullingTime="7.167067926s" totalInitContainerRuntime="25.954688ms" isStatefulPod=false podCreationTimestamp="2026-09-14 11:02:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:03:03.518937076 +0000 UTC m=+128.139149618" watchObservedRunningTime="2026-09-14 11:03:03.519422927 +0000 UTC m=+128.139635448" Sep 14 11:03:04.650412 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:04.650373 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfd621b6-51b0-4f2e-b1aa-ea198b732626-metrics-certs\") pod \"network-metrics-daemon-jkg6q\" (UID: \"dfd621b6-51b0-4f2e-b1aa-ea198b732626\") " pod="openshift-multus/network-metrics-daemon-jkg6q" Sep 14 11:03:04.652894 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:04.652868 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfd621b6-51b0-4f2e-b1aa-ea198b732626-metrics-certs\") pod \"network-metrics-daemon-jkg6q\" (UID: \"dfd621b6-51b0-4f2e-b1aa-ea198b732626\") " pod="openshift-multus/network-metrics-daemon-jkg6q" Sep 14 11:03:04.819442 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:04.819413 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-npv4p\"" Sep 14 11:03:04.827825 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:04.827801 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jkg6q" Sep 14 11:03:05.126212 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:05.126129 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jkg6q"] Sep 14 11:03:05.129700 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:03:05.129671 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd621b6_51b0_4f2e_b1aa_ea198b732626.slice/crio-a28170b68f57399bdb0bc3e3486551c543e1b340b2f236ec7dcff212219578d7 WatchSource:0}: Error finding container a28170b68f57399bdb0bc3e3486551c543e1b340b2f236ec7dcff212219578d7: Status 404 returned error can't find the container with id a28170b68f57399bdb0bc3e3486551c543e1b340b2f236ec7dcff212219578d7 Sep 14 11:03:05.492344 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:05.492311 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jkg6q" event={"ID":"dfd621b6-51b0-4f2e-b1aa-ea198b732626","Type":"ContainerStarted","Data":"a28170b68f57399bdb0bc3e3486551c543e1b340b2f236ec7dcff212219578d7"} Sep 14 11:03:06.276729 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:06.276694 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:03:07.501704 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:07.501660 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jkg6q" event={"ID":"dfd621b6-51b0-4f2e-b1aa-ea198b732626","Type":"ContainerStarted","Data":"9bdf3dfbaf608a9343023ae69786be52c2d2eceae9f3d62711ab98a9eac5529f"} Sep 14 11:03:07.502134 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:07.501751 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jkg6q" event={"ID":"dfd621b6-51b0-4f2e-b1aa-ea198b732626","Type":"ContainerStarted","Data":"7fa618d169ed125f8a73011176198cb8ecf81c5d45369c3daf846a66d1e6579e"} Sep 14 11:03:07.521924 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:07.521877 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jkg6q" podStartSLOduration=130.113433255 podStartE2EDuration="2m11.521862603s" totalImagesPullingTime="1.408429348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:03:07.518733215 +0000 UTC m=+132.138945768" watchObservedRunningTime="2026-09-14 11:03:07.521862603 +0000 UTC m=+132.142075126" Sep 14 11:03:07.607691 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:07.607658 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5db4d47bf7-8w6fr"] Sep 14 11:03:09.047790 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:09.047754 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5cf5d68bb4-nb8ds" Sep 14 11:03:09.048169 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:09.047823 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5cf5d68bb4-nb8ds" Sep 14 11:03:09.512005 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:09.511973 2574 generic.go:356] "Generic (PLEG): container finished" podID="0dc7705f-2ef0-492c-8acf-2a2898a64904" containerID="fb57b1c920251bec353444f333603f5d60cb6b6f8d26b75b38281f7d94881672" exitCode=0 Sep 14 11:03:09.512155 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:09.512046 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rpq5s" event={"ID":"0dc7705f-2ef0-492c-8acf-2a2898a64904","Type":"ContainerDied","Data":"fb57b1c920251bec353444f333603f5d60cb6b6f8d26b75b38281f7d94881672"} Sep 14 11:03:09.512356 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:09.512344 2574 scope.go:117] "RemoveContainer" containerID="fb57b1c920251bec353444f333603f5d60cb6b6f8d26b75b38281f7d94881672" Sep 14 11:03:10.518737 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:10.518699 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-rpq5s" event={"ID":"0dc7705f-2ef0-492c-8acf-2a2898a64904","Type":"ContainerStarted","Data":"d98bd0017fd0198ded4b38cb24100429c571020424bf5bb56147102ae712116b"} Sep 14 11:03:29.060863 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:29.059206 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5cf5d68bb4-nb8ds" Sep 14 11:03:29.070130 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:29.070106 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5cf5d68bb4-nb8ds" Sep 14 11:03:29.604102 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:29.604058 2574 generic.go:356] "Generic (PLEG): container finished" podID="1e6a3f9d-7dd0-42f4-8d3a-a38b72eae274" containerID="e4e44e3f6c669328884783fd35cd7bf3082ca9a6fff95a51b1b6bc8384a16f6c" exitCode=0 Sep 14 11:03:29.604417 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:29.604386 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lcxpq" event={"ID":"1e6a3f9d-7dd0-42f4-8d3a-a38b72eae274","Type":"ContainerDied","Data":"e4e44e3f6c669328884783fd35cd7bf3082ca9a6fff95a51b1b6bc8384a16f6c"} Sep 14 11:03:29.605798 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:29.605206 2574 scope.go:117] "RemoveContainer" containerID="e4e44e3f6c669328884783fd35cd7bf3082ca9a6fff95a51b1b6bc8384a16f6c" Sep 14 11:03:30.613420 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:30.613384 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-lcxpq" event={"ID":"1e6a3f9d-7dd0-42f4-8d3a-a38b72eae274","Type":"ContainerStarted","Data":"f6e8c6f95146a446da62a9989e19f9994aa4e060e4ac5b48616c81b7d1e953e2"} Sep 14 11:03:32.627933 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:32.627896 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5db4d47bf7-8w6fr" podUID="e7cead7b-bcec-4b90-bf0a-55057d5f39a6" containerName="console" containerID="cri-o://0cfce2d1b01e705c15d851fb25d017c2652c97187649aabeb7efade74ebb4ece" gracePeriod=15 Sep 14 11:03:32.906128 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:32.906078 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5db4d47bf7-8w6fr" Sep 14 11:03:32.986751 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:32.986724 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e7cead7b-bcec-4b90-bf0a-55057d5f39a6-console-config\") pod \"e7cead7b-bcec-4b90-bf0a-55057d5f39a6\" (UID: \"e7cead7b-bcec-4b90-bf0a-55057d5f39a6\") " Sep 14 11:03:32.986868 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:32.986758 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7cead7b-bcec-4b90-bf0a-55057d5f39a6-trusted-ca-bundle\") pod \"e7cead7b-bcec-4b90-bf0a-55057d5f39a6\" (UID: \"e7cead7b-bcec-4b90-bf0a-55057d5f39a6\") " Sep 14 11:03:32.986868 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:32.986813 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e7cead7b-bcec-4b90-bf0a-55057d5f39a6-service-ca\") pod \"e7cead7b-bcec-4b90-bf0a-55057d5f39a6\" (UID: \"e7cead7b-bcec-4b90-bf0a-55057d5f39a6\") " Sep 14 11:03:32.986868 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:32.986831 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e7cead7b-bcec-4b90-bf0a-55057d5f39a6-console-serving-cert\") pod \"e7cead7b-bcec-4b90-bf0a-55057d5f39a6\" (UID: \"e7cead7b-bcec-4b90-bf0a-55057d5f39a6\") " Sep 14 11:03:32.987009 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:32.986872 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e7cead7b-bcec-4b90-bf0a-55057d5f39a6-oauth-serving-cert\") pod \"e7cead7b-bcec-4b90-bf0a-55057d5f39a6\" (UID: \"e7cead7b-bcec-4b90-bf0a-55057d5f39a6\") " Sep 14 11:03:32.987009 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:32.986905 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gwgh\" (UniqueName: \"kubernetes.io/projected/e7cead7b-bcec-4b90-bf0a-55057d5f39a6-kube-api-access-2gwgh\") pod \"e7cead7b-bcec-4b90-bf0a-55057d5f39a6\" (UID: \"e7cead7b-bcec-4b90-bf0a-55057d5f39a6\") " Sep 14 11:03:32.987009 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:32.986949 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e7cead7b-bcec-4b90-bf0a-55057d5f39a6-console-oauth-config\") pod \"e7cead7b-bcec-4b90-bf0a-55057d5f39a6\" (UID: \"e7cead7b-bcec-4b90-bf0a-55057d5f39a6\") " Sep 14 11:03:32.987158 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:32.987070 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7cead7b-bcec-4b90-bf0a-55057d5f39a6-console-config" (OuterVolumeSpecName: "console-config") pod "e7cead7b-bcec-4b90-bf0a-55057d5f39a6" (UID: "e7cead7b-bcec-4b90-bf0a-55057d5f39a6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:03:32.987247 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:32.987224 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7cead7b-bcec-4b90-bf0a-55057d5f39a6-service-ca" (OuterVolumeSpecName: "service-ca") pod "e7cead7b-bcec-4b90-bf0a-55057d5f39a6" (UID: "e7cead7b-bcec-4b90-bf0a-55057d5f39a6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:03:32.987345 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:32.987260 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7cead7b-bcec-4b90-bf0a-55057d5f39a6-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e7cead7b-bcec-4b90-bf0a-55057d5f39a6" (UID: "e7cead7b-bcec-4b90-bf0a-55057d5f39a6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:03:32.987345 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:32.987318 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7cead7b-bcec-4b90-bf0a-55057d5f39a6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e7cead7b-bcec-4b90-bf0a-55057d5f39a6" (UID: "e7cead7b-bcec-4b90-bf0a-55057d5f39a6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:03:32.987497 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:32.987480 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e7cead7b-bcec-4b90-bf0a-55057d5f39a6-service-ca\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:03:32.987555 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:32.987505 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e7cead7b-bcec-4b90-bf0a-55057d5f39a6-oauth-serving-cert\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:03:32.987555 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:32.987520 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e7cead7b-bcec-4b90-bf0a-55057d5f39a6-console-config\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:03:32.987555 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:32.987534 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e7cead7b-bcec-4b90-bf0a-55057d5f39a6-trusted-ca-bundle\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:03:32.989442 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:32.989416 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7cead7b-bcec-4b90-bf0a-55057d5f39a6-kube-api-access-2gwgh" (OuterVolumeSpecName: "kube-api-access-2gwgh") pod "e7cead7b-bcec-4b90-bf0a-55057d5f39a6" (UID: "e7cead7b-bcec-4b90-bf0a-55057d5f39a6"). InnerVolumeSpecName "kube-api-access-2gwgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:03:32.989442 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:32.989430 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7cead7b-bcec-4b90-bf0a-55057d5f39a6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e7cead7b-bcec-4b90-bf0a-55057d5f39a6" (UID: "e7cead7b-bcec-4b90-bf0a-55057d5f39a6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:03:32.989590 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:32.989514 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7cead7b-bcec-4b90-bf0a-55057d5f39a6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e7cead7b-bcec-4b90-bf0a-55057d5f39a6" (UID: "e7cead7b-bcec-4b90-bf0a-55057d5f39a6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:03:33.088215 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:33.088182 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e7cead7b-bcec-4b90-bf0a-55057d5f39a6-console-serving-cert\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:03:33.088215 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:33.088212 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gwgh\" (UniqueName: \"kubernetes.io/projected/e7cead7b-bcec-4b90-bf0a-55057d5f39a6-kube-api-access-2gwgh\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:03:33.088395 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:33.088225 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e7cead7b-bcec-4b90-bf0a-55057d5f39a6-console-oauth-config\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:03:33.627967 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:33.627928 2574 generic.go:356] "Generic (PLEG): container finished" podID="e7cead7b-bcec-4b90-bf0a-55057d5f39a6" containerID="0cfce2d1b01e705c15d851fb25d017c2652c97187649aabeb7efade74ebb4ece" exitCode=0 Sep 14 11:03:33.628363 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:33.627978 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5db4d47bf7-8w6fr" Sep 14 11:03:33.628363 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:33.628019 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5db4d47bf7-8w6fr" event={"ID":"e7cead7b-bcec-4b90-bf0a-55057d5f39a6","Type":"ContainerDied","Data":"0cfce2d1b01e705c15d851fb25d017c2652c97187649aabeb7efade74ebb4ece"} Sep 14 11:03:33.628363 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:33.628080 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5db4d47bf7-8w6fr" event={"ID":"e7cead7b-bcec-4b90-bf0a-55057d5f39a6","Type":"ContainerDied","Data":"01d84375536f1a6d08c978051697ee5136c8299a76b72920e0a4722c25e9d2a6"} Sep 14 11:03:33.628363 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:33.628103 2574 scope.go:117] "RemoveContainer" containerID="0cfce2d1b01e705c15d851fb25d017c2652c97187649aabeb7efade74ebb4ece" Sep 14 11:03:33.640135 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:33.640121 2574 scope.go:117] "RemoveContainer" containerID="0cfce2d1b01e705c15d851fb25d017c2652c97187649aabeb7efade74ebb4ece" Sep 14 11:03:33.640447 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:03:33.640426 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cfce2d1b01e705c15d851fb25d017c2652c97187649aabeb7efade74ebb4ece\": container with ID starting with 0cfce2d1b01e705c15d851fb25d017c2652c97187649aabeb7efade74ebb4ece not found: ID does not exist" containerID="0cfce2d1b01e705c15d851fb25d017c2652c97187649aabeb7efade74ebb4ece" Sep 14 11:03:33.640504 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:33.640454 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cfce2d1b01e705c15d851fb25d017c2652c97187649aabeb7efade74ebb4ece"} err="failed to get container status \"0cfce2d1b01e705c15d851fb25d017c2652c97187649aabeb7efade74ebb4ece\": rpc error: code = NotFound desc = could not find container \"0cfce2d1b01e705c15d851fb25d017c2652c97187649aabeb7efade74ebb4ece\": container with ID starting with 0cfce2d1b01e705c15d851fb25d017c2652c97187649aabeb7efade74ebb4ece not found: ID does not exist" Sep 14 11:03:33.648902 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:33.648880 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5db4d47bf7-8w6fr"] Sep 14 11:03:33.656537 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:33.656517 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5db4d47bf7-8w6fr"] Sep 14 11:03:33.898048 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:33.897991 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7cead7b-bcec-4b90-bf0a-55057d5f39a6" path="/var/lib/kubelet/pods/e7cead7b-bcec-4b90-bf0a-55057d5f39a6/volumes" Sep 14 11:03:51.276149 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:51.276045 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:03:51.292424 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:51.292399 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:03:51.715719 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:03:51.715691 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 11:05:55.820431 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:05:55.816048 2574 kubelet.go:1648] "Image garbage collection succeeded" Sep 14 11:06:00.324101 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:00.324059 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f85645c65-z2tnm"] Sep 14 11:06:00.326547 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:00.324523 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c693407b-c967-4981-950c-e7630d4dff7a" containerName="registry" Sep 14 11:06:00.326547 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:00.324540 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c693407b-c967-4981-950c-e7630d4dff7a" containerName="registry" Sep 14 11:06:00.326547 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:00.324569 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7cead7b-bcec-4b90-bf0a-55057d5f39a6" containerName="console" Sep 14 11:06:00.326547 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:00.324574 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7cead7b-bcec-4b90-bf0a-55057d5f39a6" containerName="console" Sep 14 11:06:00.326547 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:00.324626 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7cead7b-bcec-4b90-bf0a-55057d5f39a6" containerName="console" Sep 14 11:06:00.326547 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:00.324633 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c693407b-c967-4981-950c-e7630d4dff7a" containerName="registry" Sep 14 11:06:00.327533 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:00.327513 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-z2tnm" Sep 14 11:06:00.330211 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:00.330188 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 14 11:06:00.330211 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:00.330191 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wrjmp\"" Sep 14 11:06:00.330402 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:00.330384 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 14 11:06:00.330456 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:00.330387 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 14 11:06:00.330500 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:00.330445 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 14 11:06:00.356576 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:00.356547 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f85645c65-z2tnm"] Sep 14 11:06:00.495447 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:00.495387 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/415f2f7b-4196-4d3d-b4ee-e237110a54bf-tmp\") pod \"opendatahub-operator-controller-manager-6f85645c65-z2tnm\" (UID: \"415f2f7b-4196-4d3d-b4ee-e237110a54bf\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-z2tnm" Sep 14 11:06:00.495644 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:00.495502 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/415f2f7b-4196-4d3d-b4ee-e237110a54bf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f85645c65-z2tnm\" (UID: \"415f2f7b-4196-4d3d-b4ee-e237110a54bf\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-z2tnm" Sep 14 11:06:00.495644 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:00.495630 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlgpt\" (UniqueName: \"kubernetes.io/projected/415f2f7b-4196-4d3d-b4ee-e237110a54bf-kube-api-access-xlgpt\") pod \"opendatahub-operator-controller-manager-6f85645c65-z2tnm\" (UID: \"415f2f7b-4196-4d3d-b4ee-e237110a54bf\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-z2tnm" Sep 14 11:06:00.495753 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:00.495695 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/415f2f7b-4196-4d3d-b4ee-e237110a54bf-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f85645c65-z2tnm\" (UID: \"415f2f7b-4196-4d3d-b4ee-e237110a54bf\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-z2tnm" Sep 14 11:06:00.495753 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:00.495745 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/415f2f7b-4196-4d3d-b4ee-e237110a54bf-manifests\") pod \"opendatahub-operator-controller-manager-6f85645c65-z2tnm\" (UID: \"415f2f7b-4196-4d3d-b4ee-e237110a54bf\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-z2tnm" Sep 14 11:06:00.596659 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:00.596562 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlgpt\" (UniqueName: \"kubernetes.io/projected/415f2f7b-4196-4d3d-b4ee-e237110a54bf-kube-api-access-xlgpt\") pod \"opendatahub-operator-controller-manager-6f85645c65-z2tnm\" (UID: \"415f2f7b-4196-4d3d-b4ee-e237110a54bf\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-z2tnm" Sep 14 11:06:00.596659 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:00.596623 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/415f2f7b-4196-4d3d-b4ee-e237110a54bf-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f85645c65-z2tnm\" (UID: \"415f2f7b-4196-4d3d-b4ee-e237110a54bf\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-z2tnm" Sep 14 11:06:00.596888 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:00.596664 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/415f2f7b-4196-4d3d-b4ee-e237110a54bf-manifests\") pod \"opendatahub-operator-controller-manager-6f85645c65-z2tnm\" (UID: \"415f2f7b-4196-4d3d-b4ee-e237110a54bf\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-z2tnm" Sep 14 11:06:00.596888 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:00.596695 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/415f2f7b-4196-4d3d-b4ee-e237110a54bf-tmp\") pod \"opendatahub-operator-controller-manager-6f85645c65-z2tnm\" (UID: \"415f2f7b-4196-4d3d-b4ee-e237110a54bf\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-z2tnm" Sep 14 11:06:00.596888 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:00.596727 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/415f2f7b-4196-4d3d-b4ee-e237110a54bf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f85645c65-z2tnm\" (UID: \"415f2f7b-4196-4d3d-b4ee-e237110a54bf\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-z2tnm" Sep 14 11:06:00.597193 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:00.597156 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/415f2f7b-4196-4d3d-b4ee-e237110a54bf-manifests\") pod \"opendatahub-operator-controller-manager-6f85645c65-z2tnm\" (UID: \"415f2f7b-4196-4d3d-b4ee-e237110a54bf\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-z2tnm" Sep 14 11:06:00.599164 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:00.599138 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/415f2f7b-4196-4d3d-b4ee-e237110a54bf-tmp\") pod \"opendatahub-operator-controller-manager-6f85645c65-z2tnm\" (UID: \"415f2f7b-4196-4d3d-b4ee-e237110a54bf\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-z2tnm" Sep 14 11:06:00.599336 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:00.599313 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/415f2f7b-4196-4d3d-b4ee-e237110a54bf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f85645c65-z2tnm\" (UID: \"415f2f7b-4196-4d3d-b4ee-e237110a54bf\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-z2tnm" Sep 14 11:06:00.599437 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:00.599422 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/415f2f7b-4196-4d3d-b4ee-e237110a54bf-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f85645c65-z2tnm\" (UID: \"415f2f7b-4196-4d3d-b4ee-e237110a54bf\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-z2tnm" Sep 14 11:06:00.620768 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:00.620741 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlgpt\" (UniqueName: \"kubernetes.io/projected/415f2f7b-4196-4d3d-b4ee-e237110a54bf-kube-api-access-xlgpt\") pod \"opendatahub-operator-controller-manager-6f85645c65-z2tnm\" (UID: \"415f2f7b-4196-4d3d-b4ee-e237110a54bf\") " pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-z2tnm" Sep 14 11:06:00.637587 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:00.637562 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-z2tnm" Sep 14 11:06:00.922135 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:00.922111 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f85645c65-z2tnm"] Sep 14 11:06:00.922466 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:06:00.922441 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod415f2f7b_4196_4d3d_b4ee_e237110a54bf.slice/crio-e1977412ed5afc19104ee64075b0de2149f8ba21f9676c5784eae5947b4fd851 WatchSource:0}: Error finding container e1977412ed5afc19104ee64075b0de2149f8ba21f9676c5784eae5947b4fd851: Status 404 returned error can't find the container with id e1977412ed5afc19104ee64075b0de2149f8ba21f9676c5784eae5947b4fd851 Sep 14 11:06:00.924190 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:00.924173 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 11:06:01.203777 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:01.203744 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-z2tnm" event={"ID":"415f2f7b-4196-4d3d-b4ee-e237110a54bf","Type":"ContainerStarted","Data":"e1977412ed5afc19104ee64075b0de2149f8ba21f9676c5784eae5947b4fd851"} Sep 14 11:06:04.219152 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:04.219114 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-z2tnm" event={"ID":"415f2f7b-4196-4d3d-b4ee-e237110a54bf","Type":"ContainerStarted","Data":"e81c9e45c758ff738ad6714606697729c3d640c439e8dedcd1fcd9d5125cdf47"} Sep 14 11:06:05.225187 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:05.225151 2574 generic.go:356] "Generic (PLEG): container finished" podID="415f2f7b-4196-4d3d-b4ee-e237110a54bf" containerID="e81c9e45c758ff738ad6714606697729c3d640c439e8dedcd1fcd9d5125cdf47" exitCode=0 Sep 14 11:06:05.225640 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:05.225242 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-z2tnm" event={"ID":"415f2f7b-4196-4d3d-b4ee-e237110a54bf","Type":"ContainerDied","Data":"e81c9e45c758ff738ad6714606697729c3d640c439e8dedcd1fcd9d5125cdf47"} Sep 14 11:06:06.239082 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:06.239040 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-z2tnm" event={"ID":"415f2f7b-4196-4d3d-b4ee-e237110a54bf","Type":"ContainerStarted","Data":"039cedaaa4789bd3b3051129d07736e5a692a686e9dbb8c359847364ca313658"} Sep 14 11:06:06.239618 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:06.239199 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-z2tnm" Sep 14 11:06:06.316339 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:06.316278 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-z2tnm" podStartSLOduration=1.793956499 podStartE2EDuration="6.316248579s" totalImagesPullingTime="3.464919714s" totalInitContainerRuntime="1.057372366s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:06:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:06:06.313476034 +0000 UTC m=+310.933688578" watchObservedRunningTime="2026-09-14 11:06:06.316248579 +0000 UTC m=+310.936461169" Sep 14 11:06:17.246604 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:17.246570 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6f85645c65-z2tnm" Sep 14 11:06:21.129115 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:21.129080 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5f95fc98f9-kzb94"] Sep 14 11:06:21.132469 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:21.132451 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5f95fc98f9-kzb94" Sep 14 11:06:21.138727 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:21.138410 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-vft58\"" Sep 14 11:06:21.142488 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:21.142469 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 14 11:06:21.142587 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:21.142567 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 14 11:06:21.178235 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:21.178205 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5f95fc98f9-kzb94"] Sep 14 11:06:21.264047 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:21.264014 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9aa22333-c697-43d6-ac9f-eb93f64b8801-tmp\") pod \"kube-auth-proxy-5f95fc98f9-kzb94\" (UID: \"9aa22333-c697-43d6-ac9f-eb93f64b8801\") " pod="openshift-ingress/kube-auth-proxy-5f95fc98f9-kzb94" Sep 14 11:06:21.264225 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:21.264074 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl9fc\" (UniqueName: \"kubernetes.io/projected/9aa22333-c697-43d6-ac9f-eb93f64b8801-kube-api-access-gl9fc\") pod \"kube-auth-proxy-5f95fc98f9-kzb94\" (UID: \"9aa22333-c697-43d6-ac9f-eb93f64b8801\") " pod="openshift-ingress/kube-auth-proxy-5f95fc98f9-kzb94" Sep 14 11:06:21.264225 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:21.264137 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9aa22333-c697-43d6-ac9f-eb93f64b8801-tls-certs\") pod \"kube-auth-proxy-5f95fc98f9-kzb94\" (UID: \"9aa22333-c697-43d6-ac9f-eb93f64b8801\") " pod="openshift-ingress/kube-auth-proxy-5f95fc98f9-kzb94" Sep 14 11:06:21.364839 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:21.364800 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9aa22333-c697-43d6-ac9f-eb93f64b8801-tls-certs\") pod \"kube-auth-proxy-5f95fc98f9-kzb94\" (UID: \"9aa22333-c697-43d6-ac9f-eb93f64b8801\") " pod="openshift-ingress/kube-auth-proxy-5f95fc98f9-kzb94" Sep 14 11:06:21.365032 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:21.364889 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9aa22333-c697-43d6-ac9f-eb93f64b8801-tmp\") pod \"kube-auth-proxy-5f95fc98f9-kzb94\" (UID: \"9aa22333-c697-43d6-ac9f-eb93f64b8801\") " pod="openshift-ingress/kube-auth-proxy-5f95fc98f9-kzb94" Sep 14 11:06:21.365032 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:21.364945 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl9fc\" (UniqueName: \"kubernetes.io/projected/9aa22333-c697-43d6-ac9f-eb93f64b8801-kube-api-access-gl9fc\") pod \"kube-auth-proxy-5f95fc98f9-kzb94\" (UID: \"9aa22333-c697-43d6-ac9f-eb93f64b8801\") " pod="openshift-ingress/kube-auth-proxy-5f95fc98f9-kzb94" Sep 14 11:06:21.365032 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:06:21.364967 2574 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Sep 14 11:06:21.365231 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:06:21.365058 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9aa22333-c697-43d6-ac9f-eb93f64b8801-tls-certs podName:9aa22333-c697-43d6-ac9f-eb93f64b8801 nodeName:}" failed. No retries permitted until 2026-09-14 11:06:21.865036731 +0000 UTC m=+326.485249252 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/9aa22333-c697-43d6-ac9f-eb93f64b8801-tls-certs") pod "kube-auth-proxy-5f95fc98f9-kzb94" (UID: "9aa22333-c697-43d6-ac9f-eb93f64b8801") : secret "kube-auth-proxy-tls" not found Sep 14 11:06:21.367497 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:21.367465 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9aa22333-c697-43d6-ac9f-eb93f64b8801-tmp\") pod \"kube-auth-proxy-5f95fc98f9-kzb94\" (UID: \"9aa22333-c697-43d6-ac9f-eb93f64b8801\") " pod="openshift-ingress/kube-auth-proxy-5f95fc98f9-kzb94" Sep 14 11:06:21.389050 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:21.388995 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl9fc\" (UniqueName: \"kubernetes.io/projected/9aa22333-c697-43d6-ac9f-eb93f64b8801-kube-api-access-gl9fc\") pod \"kube-auth-proxy-5f95fc98f9-kzb94\" (UID: \"9aa22333-c697-43d6-ac9f-eb93f64b8801\") " pod="openshift-ingress/kube-auth-proxy-5f95fc98f9-kzb94" Sep 14 11:06:21.869975 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:21.869936 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9aa22333-c697-43d6-ac9f-eb93f64b8801-tls-certs\") pod \"kube-auth-proxy-5f95fc98f9-kzb94\" (UID: \"9aa22333-c697-43d6-ac9f-eb93f64b8801\") " pod="openshift-ingress/kube-auth-proxy-5f95fc98f9-kzb94" Sep 14 11:06:21.872551 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:21.872522 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9aa22333-c697-43d6-ac9f-eb93f64b8801-tls-certs\") pod \"kube-auth-proxy-5f95fc98f9-kzb94\" (UID: \"9aa22333-c697-43d6-ac9f-eb93f64b8801\") " pod="openshift-ingress/kube-auth-proxy-5f95fc98f9-kzb94" Sep 14 11:06:22.041860 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:22.041814 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5f95fc98f9-kzb94" Sep 14 11:06:22.343590 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:22.343556 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5f95fc98f9-kzb94"] Sep 14 11:06:22.344149 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:06:22.344127 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aa22333_c697_43d6_ac9f_eb93f64b8801.slice/crio-2f16cfa9f0f9b61fb14f86ed740c26797c087c1e5cb2928b863538ed856acea2 WatchSource:0}: Error finding container 2f16cfa9f0f9b61fb14f86ed740c26797c087c1e5cb2928b863538ed856acea2: Status 404 returned error can't find the container with id 2f16cfa9f0f9b61fb14f86ed740c26797c087c1e5cb2928b863538ed856acea2 Sep 14 11:06:22.966684 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:22.966652 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-45rkt"] Sep 14 11:06:22.971151 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:22.971133 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-45rkt" Sep 14 11:06:22.975772 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:22.975752 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 14 11:06:22.976058 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:22.976039 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-qd8g6\"" Sep 14 11:06:22.976514 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:22.976501 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 14 11:06:22.976609 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:22.976505 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 14 11:06:22.976730 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:22.976713 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 14 11:06:22.989964 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:22.989940 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-45rkt"] Sep 14 11:06:23.081350 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:23.081319 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-45rkt\" (UID: \"76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-45rkt" Sep 14 11:06:23.081529 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:23.081361 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-45rkt\" (UID: \"76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-45rkt" Sep 14 11:06:23.081529 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:23.081391 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-45rkt\" (UID: \"76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-45rkt" Sep 14 11:06:23.081529 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:23.081410 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-45rkt\" (UID: \"76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-45rkt" Sep 14 11:06:23.081529 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:23.081455 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-45rkt\" (UID: \"76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-45rkt" Sep 14 11:06:23.081529 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:23.081499 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-45rkt\" (UID: \"76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-45rkt" Sep 14 11:06:23.081529 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:23.081527 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z62z8\" (UniqueName: \"kubernetes.io/projected/76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e-kube-api-access-z62z8\") pod \"istiod-openshift-gateway-7b7995d5cd-45rkt\" (UID: \"76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-45rkt" Sep 14 11:06:23.182146 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:23.182107 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-45rkt\" (UID: \"76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-45rkt" Sep 14 11:06:23.182340 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:23.182164 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-45rkt\" (UID: \"76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-45rkt" Sep 14 11:06:23.182340 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:23.182196 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-45rkt\" (UID: \"76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-45rkt" Sep 14 11:06:23.182340 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:23.182213 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-45rkt\" (UID: \"76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-45rkt" Sep 14 11:06:23.182340 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:23.182232 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-45rkt\" (UID: \"76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-45rkt" Sep 14 11:06:23.182340 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:23.182255 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-45rkt\" (UID: \"76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-45rkt" Sep 14 11:06:23.182340 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:23.182320 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z62z8\" (UniqueName: \"kubernetes.io/projected/76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e-kube-api-access-z62z8\") pod \"istiod-openshift-gateway-7b7995d5cd-45rkt\" (UID: \"76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-45rkt" Sep 14 11:06:23.183035 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:23.182963 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-45rkt\" (UID: \"76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-45rkt" Sep 14 11:06:23.184767 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:23.184741 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-45rkt\" (UID: \"76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-45rkt" Sep 14 11:06:23.184767 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:23.184762 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-45rkt\" (UID: \"76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-45rkt" Sep 14 11:06:23.184933 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:23.184895 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-45rkt\" (UID: \"76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-45rkt" Sep 14 11:06:23.184933 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:23.184893 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-45rkt\" (UID: \"76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-45rkt" Sep 14 11:06:23.211561 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:23.211536 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-45rkt\" (UID: \"76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-45rkt" Sep 14 11:06:23.213425 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:23.213407 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z62z8\" (UniqueName: \"kubernetes.io/projected/76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e-kube-api-access-z62z8\") pod \"istiod-openshift-gateway-7b7995d5cd-45rkt\" (UID: \"76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-45rkt" Sep 14 11:06:23.281023 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:23.280947 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-45rkt" Sep 14 11:06:23.311633 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:23.311599 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5f95fc98f9-kzb94" event={"ID":"9aa22333-c697-43d6-ac9f-eb93f64b8801","Type":"ContainerStarted","Data":"2f16cfa9f0f9b61fb14f86ed740c26797c087c1e5cb2928b863538ed856acea2"} Sep 14 11:06:23.576169 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:23.576141 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-45rkt"] Sep 14 11:06:23.578343 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:06:23.578313 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76d7fe61_ea90_4c33_9c71_bfd93f4b0f9e.slice/crio-fcbced27bdd7dddf22eced70bf58615b238b997a4dfa9b2155fa048fec808aff WatchSource:0}: Error finding container fcbced27bdd7dddf22eced70bf58615b238b997a4dfa9b2155fa048fec808aff: Status 404 returned error can't find the container with id fcbced27bdd7dddf22eced70bf58615b238b997a4dfa9b2155fa048fec808aff Sep 14 11:06:24.319659 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:24.319614 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-45rkt" event={"ID":"76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e","Type":"ContainerStarted","Data":"fcbced27bdd7dddf22eced70bf58615b238b997a4dfa9b2155fa048fec808aff"} Sep 14 11:06:27.340052 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:27.340013 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 14 11:06:27.340356 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:27.340081 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 14 11:06:28.340465 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:28.340428 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-45rkt" event={"ID":"76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e","Type":"ContainerStarted","Data":"24d8a77f30d26edba6b08833a64488dbf07aa2be5d13bd5923d9b832d08c2244"} Sep 14 11:06:28.340927 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:28.340621 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-45rkt" Sep 14 11:06:28.342107 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:28.342078 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5f95fc98f9-kzb94" event={"ID":"9aa22333-c697-43d6-ac9f-eb93f64b8801","Type":"ContainerStarted","Data":"c2b293b7765eb634c6928683e4ddd3d74f54f00e923d11546a21cf68884718a0"} Sep 14 11:06:28.342664 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:28.342641 2574 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-45rkt container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 14 11:06:28.342761 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:28.342688 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-45rkt" podUID="76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 14 11:06:28.375805 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:28.375757 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-45rkt" podStartSLOduration=2.616644413 podStartE2EDuration="6.375742785s" totalImagesPullingTime="3.759098372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:06:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:06:28.371629423 +0000 UTC m=+332.991841957" watchObservedRunningTime="2026-09-14 11:06:28.375742785 +0000 UTC m=+332.995955307" Sep 14 11:06:28.395889 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:28.395837 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5f95fc98f9-kzb94" podStartSLOduration=2.355286421 podStartE2EDuration="7.395823331s" totalImagesPullingTime="5.04053691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:06:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:06:28.394955287 +0000 UTC m=+333.015167836" watchObservedRunningTime="2026-09-14 11:06:28.395823331 +0000 UTC m=+333.016035854" Sep 14 11:06:29.347502 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:29.347473 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-45rkt" Sep 14 11:06:31.726864 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:31.726830 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-j8jfw"] Sep 14 11:06:31.731590 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:31.731568 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-j8jfw" Sep 14 11:06:31.738303 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:31.738160 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 14 11:06:31.738724 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:31.738701 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-lkhqb\"" Sep 14 11:06:31.744863 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:31.744838 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-j8jfw"] Sep 14 11:06:31.861571 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:31.861534 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07d207b3-1e62-4a62-b090-3e56e9c10715-cert\") pod \"odh-model-controller-7998c67dc4-j8jfw\" (UID: \"07d207b3-1e62-4a62-b090-3e56e9c10715\") " pod="opendatahub/odh-model-controller-7998c67dc4-j8jfw" Sep 14 11:06:31.861713 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:31.861581 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48s85\" (UniqueName: \"kubernetes.io/projected/07d207b3-1e62-4a62-b090-3e56e9c10715-kube-api-access-48s85\") pod \"odh-model-controller-7998c67dc4-j8jfw\" (UID: \"07d207b3-1e62-4a62-b090-3e56e9c10715\") " pod="opendatahub/odh-model-controller-7998c67dc4-j8jfw" Sep 14 11:06:31.962865 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:31.962813 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07d207b3-1e62-4a62-b090-3e56e9c10715-cert\") pod \"odh-model-controller-7998c67dc4-j8jfw\" (UID: \"07d207b3-1e62-4a62-b090-3e56e9c10715\") " pod="opendatahub/odh-model-controller-7998c67dc4-j8jfw" Sep 14 11:06:31.962865 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:31.962870 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48s85\" (UniqueName: \"kubernetes.io/projected/07d207b3-1e62-4a62-b090-3e56e9c10715-kube-api-access-48s85\") pod \"odh-model-controller-7998c67dc4-j8jfw\" (UID: \"07d207b3-1e62-4a62-b090-3e56e9c10715\") " pod="opendatahub/odh-model-controller-7998c67dc4-j8jfw" Sep 14 11:06:31.963117 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:06:31.962981 2574 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 14 11:06:31.963117 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:06:31.963056 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07d207b3-1e62-4a62-b090-3e56e9c10715-cert podName:07d207b3-1e62-4a62-b090-3e56e9c10715 nodeName:}" failed. No retries permitted until 2026-09-14 11:06:32.463036308 +0000 UTC m=+337.083248823 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/07d207b3-1e62-4a62-b090-3e56e9c10715-cert") pod "odh-model-controller-7998c67dc4-j8jfw" (UID: "07d207b3-1e62-4a62-b090-3e56e9c10715") : secret "odh-model-controller-webhook-cert" not found Sep 14 11:06:31.977728 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:31.977675 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48s85\" (UniqueName: \"kubernetes.io/projected/07d207b3-1e62-4a62-b090-3e56e9c10715-kube-api-access-48s85\") pod \"odh-model-controller-7998c67dc4-j8jfw\" (UID: \"07d207b3-1e62-4a62-b090-3e56e9c10715\") " pod="opendatahub/odh-model-controller-7998c67dc4-j8jfw" Sep 14 11:06:32.298078 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:32.298004 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-txc6g"] Sep 14 11:06:32.301545 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:32.301522 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-txc6g" Sep 14 11:06:32.304137 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:32.304114 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 14 11:06:32.304260 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:32.304137 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-8zhh4\"" Sep 14 11:06:32.315830 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:32.315803 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-txc6g"] Sep 14 11:06:32.367072 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:32.367037 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2da68c9f-c790-4560-85bd-43dc1233bf8b-tls-certs\") pod \"model-serving-api-54d89bd5fc-txc6g\" (UID: \"2da68c9f-c790-4560-85bd-43dc1233bf8b\") " pod="opendatahub/model-serving-api-54d89bd5fc-txc6g" Sep 14 11:06:32.367293 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:32.367092 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szzwc\" (UniqueName: \"kubernetes.io/projected/2da68c9f-c790-4560-85bd-43dc1233bf8b-kube-api-access-szzwc\") pod \"model-serving-api-54d89bd5fc-txc6g\" (UID: \"2da68c9f-c790-4560-85bd-43dc1233bf8b\") " pod="opendatahub/model-serving-api-54d89bd5fc-txc6g" Sep 14 11:06:32.468545 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:32.468513 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2da68c9f-c790-4560-85bd-43dc1233bf8b-tls-certs\") pod \"model-serving-api-54d89bd5fc-txc6g\" (UID: \"2da68c9f-c790-4560-85bd-43dc1233bf8b\") " pod="opendatahub/model-serving-api-54d89bd5fc-txc6g" Sep 14 11:06:32.468738 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:32.468561 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szzwc\" (UniqueName: \"kubernetes.io/projected/2da68c9f-c790-4560-85bd-43dc1233bf8b-kube-api-access-szzwc\") pod \"model-serving-api-54d89bd5fc-txc6g\" (UID: \"2da68c9f-c790-4560-85bd-43dc1233bf8b\") " pod="opendatahub/model-serving-api-54d89bd5fc-txc6g" Sep 14 11:06:32.468738 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:32.468601 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07d207b3-1e62-4a62-b090-3e56e9c10715-cert\") pod \"odh-model-controller-7998c67dc4-j8jfw\" (UID: \"07d207b3-1e62-4a62-b090-3e56e9c10715\") " pod="opendatahub/odh-model-controller-7998c67dc4-j8jfw" Sep 14 11:06:32.471170 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:32.471144 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07d207b3-1e62-4a62-b090-3e56e9c10715-cert\") pod \"odh-model-controller-7998c67dc4-j8jfw\" (UID: \"07d207b3-1e62-4a62-b090-3e56e9c10715\") " pod="opendatahub/odh-model-controller-7998c67dc4-j8jfw" Sep 14 11:06:32.471170 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:32.471158 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2da68c9f-c790-4560-85bd-43dc1233bf8b-tls-certs\") pod \"model-serving-api-54d89bd5fc-txc6g\" (UID: \"2da68c9f-c790-4560-85bd-43dc1233bf8b\") " pod="opendatahub/model-serving-api-54d89bd5fc-txc6g" Sep 14 11:06:32.478104 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:32.478084 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szzwc\" (UniqueName: \"kubernetes.io/projected/2da68c9f-c790-4560-85bd-43dc1233bf8b-kube-api-access-szzwc\") pod \"model-serving-api-54d89bd5fc-txc6g\" (UID: \"2da68c9f-c790-4560-85bd-43dc1233bf8b\") " pod="opendatahub/model-serving-api-54d89bd5fc-txc6g" Sep 14 11:06:32.612555 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:32.612457 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-txc6g" Sep 14 11:06:32.641887 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:32.641852 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-j8jfw" Sep 14 11:06:32.922445 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:32.922405 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-txc6g"] Sep 14 11:06:32.923592 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:06:32.923566 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2da68c9f_c790_4560_85bd_43dc1233bf8b.slice/crio-911767bd15e800cf8445582b856e63d00922881b9908d0aa86fc584f3b69952f WatchSource:0}: Error finding container 911767bd15e800cf8445582b856e63d00922881b9908d0aa86fc584f3b69952f: Status 404 returned error can't find the container with id 911767bd15e800cf8445582b856e63d00922881b9908d0aa86fc584f3b69952f Sep 14 11:06:32.939731 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:32.939703 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-j8jfw"] Sep 14 11:06:32.941959 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:06:32.941933 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07d207b3_1e62_4a62_b090_3e56e9c10715.slice/crio-0fa2f61e37e2a95c260e95f419d2fe902a1aae977e0c00a55191287276962a1e WatchSource:0}: Error finding container 0fa2f61e37e2a95c260e95f419d2fe902a1aae977e0c00a55191287276962a1e: Status 404 returned error can't find the container with id 0fa2f61e37e2a95c260e95f419d2fe902a1aae977e0c00a55191287276962a1e Sep 14 11:06:33.367200 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:33.367159 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-j8jfw" event={"ID":"07d207b3-1e62-4a62-b090-3e56e9c10715","Type":"ContainerStarted","Data":"0fa2f61e37e2a95c260e95f419d2fe902a1aae977e0c00a55191287276962a1e"} Sep 14 11:06:33.368494 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:33.368468 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-txc6g" event={"ID":"2da68c9f-c790-4560-85bd-43dc1233bf8b","Type":"ContainerStarted","Data":"911767bd15e800cf8445582b856e63d00922881b9908d0aa86fc584f3b69952f"} Sep 14 11:06:36.386450 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:36.386402 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-j8jfw" event={"ID":"07d207b3-1e62-4a62-b090-3e56e9c10715","Type":"ContainerStarted","Data":"8bf1129f40497e19666668064e80be5819b874ae58d41972ac075f86efb441d9"} Sep 14 11:06:36.386896 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:36.386715 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-j8jfw" Sep 14 11:06:36.408645 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:36.408588 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-j8jfw" podStartSLOduration=2.627269105 podStartE2EDuration="5.408569312s" totalImagesPullingTime="2.781300207s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:06:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:06:36.406763411 +0000 UTC m=+341.026975949" watchObservedRunningTime="2026-09-14 11:06:36.408569312 +0000 UTC m=+341.028781836" Sep 14 11:06:37.393728 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:37.393631 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-txc6g" event={"ID":"2da68c9f-c790-4560-85bd-43dc1233bf8b","Type":"ContainerStarted","Data":"cb14d00c94853ca6a9b9220080341c8abb364d601db963314cd547239d260377"} Sep 14 11:06:37.394138 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:37.393822 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-txc6g" Sep 14 11:06:37.414907 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:37.414862 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-txc6g" podStartSLOduration=1.315812072 podStartE2EDuration="5.414847943s" totalImagesPullingTime="4.099035871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:06:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:06:37.413399791 +0000 UTC m=+342.033612339" watchObservedRunningTime="2026-09-14 11:06:37.414847943 +0000 UTC m=+342.035060468" Sep 14 11:06:39.745488 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:39.745454 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-86plh"] Sep 14 11:06:39.748992 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:39.748965 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-86plh" Sep 14 11:06:39.769319 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:39.769297 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-86plh"] Sep 14 11:06:39.837461 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:39.837431 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/477e2e5e-7397-409e-9d1c-b35652715697-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-86plh\" (UID: \"477e2e5e-7397-409e-9d1c-b35652715697\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-86plh" Sep 14 11:06:39.837579 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:39.837476 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/477e2e5e-7397-409e-9d1c-b35652715697-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-86plh\" (UID: \"477e2e5e-7397-409e-9d1c-b35652715697\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-86plh" Sep 14 11:06:39.837579 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:39.837548 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/477e2e5e-7397-409e-9d1c-b35652715697-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-86plh\" (UID: \"477e2e5e-7397-409e-9d1c-b35652715697\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-86plh" Sep 14 11:06:39.837579 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:39.837571 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/477e2e5e-7397-409e-9d1c-b35652715697-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-86plh\" (UID: \"477e2e5e-7397-409e-9d1c-b35652715697\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-86plh" Sep 14 11:06:39.837682 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:39.837596 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/477e2e5e-7397-409e-9d1c-b35652715697-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-86plh\" (UID: \"477e2e5e-7397-409e-9d1c-b35652715697\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-86plh" Sep 14 11:06:39.837682 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:39.837613 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/477e2e5e-7397-409e-9d1c-b35652715697-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-86plh\" (UID: \"477e2e5e-7397-409e-9d1c-b35652715697\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-86plh" Sep 14 11:06:39.837682 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:39.837644 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84pc8\" (UniqueName: \"kubernetes.io/projected/477e2e5e-7397-409e-9d1c-b35652715697-kube-api-access-84pc8\") pod \"istiod-openshift-gateway-5677ccb49b-86plh\" (UID: \"477e2e5e-7397-409e-9d1c-b35652715697\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-86plh" Sep 14 11:06:39.938366 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:39.938340 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/477e2e5e-7397-409e-9d1c-b35652715697-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-86plh\" (UID: \"477e2e5e-7397-409e-9d1c-b35652715697\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-86plh" Sep 14 11:06:39.938468 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:39.938388 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/477e2e5e-7397-409e-9d1c-b35652715697-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-86plh\" (UID: \"477e2e5e-7397-409e-9d1c-b35652715697\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-86plh" Sep 14 11:06:39.938468 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:39.938409 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/477e2e5e-7397-409e-9d1c-b35652715697-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-86plh\" (UID: \"477e2e5e-7397-409e-9d1c-b35652715697\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-86plh" Sep 14 11:06:39.938662 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:39.938643 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/477e2e5e-7397-409e-9d1c-b35652715697-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-86plh\" (UID: \"477e2e5e-7397-409e-9d1c-b35652715697\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-86plh" Sep 14 11:06:39.938705 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:39.938674 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/477e2e5e-7397-409e-9d1c-b35652715697-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-86plh\" (UID: \"477e2e5e-7397-409e-9d1c-b35652715697\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-86plh" Sep 14 11:06:39.938759 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:39.938714 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84pc8\" (UniqueName: \"kubernetes.io/projected/477e2e5e-7397-409e-9d1c-b35652715697-kube-api-access-84pc8\") pod \"istiod-openshift-gateway-5677ccb49b-86plh\" (UID: \"477e2e5e-7397-409e-9d1c-b35652715697\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-86plh" Sep 14 11:06:39.938759 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:39.938741 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/477e2e5e-7397-409e-9d1c-b35652715697-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-86plh\" (UID: \"477e2e5e-7397-409e-9d1c-b35652715697\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-86plh" Sep 14 11:06:39.939416 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:39.939392 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/477e2e5e-7397-409e-9d1c-b35652715697-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-86plh\" (UID: \"477e2e5e-7397-409e-9d1c-b35652715697\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-86plh" Sep 14 11:06:39.941108 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:39.941086 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/477e2e5e-7397-409e-9d1c-b35652715697-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-86plh\" (UID: \"477e2e5e-7397-409e-9d1c-b35652715697\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-86plh" Sep 14 11:06:39.941366 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:39.941343 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/477e2e5e-7397-409e-9d1c-b35652715697-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-86plh\" (UID: \"477e2e5e-7397-409e-9d1c-b35652715697\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-86plh" Sep 14 11:06:39.941441 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:39.941352 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/477e2e5e-7397-409e-9d1c-b35652715697-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-86plh\" (UID: \"477e2e5e-7397-409e-9d1c-b35652715697\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-86plh" Sep 14 11:06:39.941482 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:39.941436 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/477e2e5e-7397-409e-9d1c-b35652715697-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-86plh\" (UID: \"477e2e5e-7397-409e-9d1c-b35652715697\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-86plh" Sep 14 11:06:39.946816 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:39.946792 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/477e2e5e-7397-409e-9d1c-b35652715697-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-86plh\" (UID: \"477e2e5e-7397-409e-9d1c-b35652715697\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-86plh" Sep 14 11:06:39.946929 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:39.946905 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84pc8\" (UniqueName: \"kubernetes.io/projected/477e2e5e-7397-409e-9d1c-b35652715697-kube-api-access-84pc8\") pod \"istiod-openshift-gateway-5677ccb49b-86plh\" (UID: \"477e2e5e-7397-409e-9d1c-b35652715697\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-86plh" Sep 14 11:06:40.057845 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:40.057779 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-86plh" Sep 14 11:06:40.392660 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:40.392586 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-86plh"] Sep 14 11:06:40.395973 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:06:40.395947 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod477e2e5e_7397_409e_9d1c_b35652715697.slice/crio-ee218a8e90b94ad24c3b5adf4dec2098ed59decfeeeb72366c4c5483c1456ea5 WatchSource:0}: Error finding container ee218a8e90b94ad24c3b5adf4dec2098ed59decfeeeb72366c4c5483c1456ea5: Status 404 returned error can't find the container with id ee218a8e90b94ad24c3b5adf4dec2098ed59decfeeeb72366c4c5483c1456ea5 Sep 14 11:06:40.399191 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:40.399163 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 14 11:06:40.399290 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:40.399224 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 14 11:06:40.410031 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:40.409994 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-86plh" event={"ID":"477e2e5e-7397-409e-9d1c-b35652715697","Type":"ContainerStarted","Data":"ee218a8e90b94ad24c3b5adf4dec2098ed59decfeeeb72366c4c5483c1456ea5"} Sep 14 11:06:41.416245 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:41.416205 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-86plh" event={"ID":"477e2e5e-7397-409e-9d1c-b35652715697","Type":"ContainerStarted","Data":"38c6e7db098d9e2a80317f1e1f9c345b4bebc9ae98056d77e2cea6c2819a0ff6"} Sep 14 11:06:41.416728 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:41.416420 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-86plh" Sep 14 11:06:41.417898 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:41.417877 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-86plh" Sep 14 11:06:41.462480 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:41.462428 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-86plh" podStartSLOduration=2.462413407 podStartE2EDuration="2.462413407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:06:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:06:41.459076024 +0000 UTC m=+346.079288548" watchObservedRunningTime="2026-09-14 11:06:41.462413407 +0000 UTC m=+346.082625932" Sep 14 11:06:41.613926 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:41.613899 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-45rkt"] Sep 14 11:06:41.614117 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:41.614082 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-45rkt" podUID="76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e" containerName="discovery" containerID="cri-o://24d8a77f30d26edba6b08833a64488dbf07aa2be5d13bd5923d9b832d08c2244" gracePeriod=30 Sep 14 11:06:41.861901 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:41.861878 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-45rkt" Sep 14 11:06:41.957221 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:41.957198 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e-istio-csr-dns-cert\") pod \"76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e\" (UID: \"76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e\") " Sep 14 11:06:41.957399 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:41.957256 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e-cacerts\") pod \"76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e\" (UID: \"76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e\") " Sep 14 11:06:41.957399 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:41.957337 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z62z8\" (UniqueName: \"kubernetes.io/projected/76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e-kube-api-access-z62z8\") pod \"76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e\" (UID: \"76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e\") " Sep 14 11:06:41.957517 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:41.957399 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e-istio-kubeconfig\") pod \"76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e\" (UID: \"76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e\") " Sep 14 11:06:41.957517 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:41.957438 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e-local-certs\") pod \"76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e\" (UID: \"76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e\") " Sep 14 11:06:41.957517 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:41.957461 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e-istio-csr-ca-configmap\") pod \"76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e\" (UID: \"76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e\") " Sep 14 11:06:41.957667 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:41.957631 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e-istio-token\") pod \"76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e\" (UID: \"76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e\") " Sep 14 11:06:41.957862 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:41.957830 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e" (UID: "76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:06:41.958059 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:41.957996 2574 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e-istio-csr-ca-configmap\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:06:41.959879 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:41.959855 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e-local-certs" (OuterVolumeSpecName: "local-certs") pod "76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e" (UID: "76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:06:41.959879 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:41.959861 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e" (UID: "76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:06:41.960134 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:41.960113 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e" (UID: "76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:06:41.960283 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:41.960247 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e-kube-api-access-z62z8" (OuterVolumeSpecName: "kube-api-access-z62z8") pod "76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e" (UID: "76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e"). InnerVolumeSpecName "kube-api-access-z62z8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:06:41.960283 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:41.960244 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e-istio-token" (OuterVolumeSpecName: "istio-token") pod "76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e" (UID: "76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:06:41.960378 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:41.960285 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e-cacerts" (OuterVolumeSpecName: "cacerts") pod "76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e" (UID: "76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:06:42.059191 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:42.059108 2574 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e-istio-kubeconfig\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:06:42.059414 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:42.059204 2574 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e-local-certs\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:06:42.059414 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:42.059215 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e-istio-token\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:06:42.059414 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:42.059223 2574 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e-istio-csr-dns-cert\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:06:42.059414 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:42.059231 2574 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e-cacerts\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:06:42.059414 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:42.059241 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z62z8\" (UniqueName: \"kubernetes.io/projected/76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e-kube-api-access-z62z8\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:06:42.422417 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:42.422326 2574 generic.go:356] "Generic (PLEG): container finished" podID="76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e" containerID="24d8a77f30d26edba6b08833a64488dbf07aa2be5d13bd5923d9b832d08c2244" exitCode=0 Sep 14 11:06:42.422417 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:42.422373 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-45rkt" Sep 14 11:06:42.422869 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:42.422412 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-45rkt" event={"ID":"76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e","Type":"ContainerDied","Data":"24d8a77f30d26edba6b08833a64488dbf07aa2be5d13bd5923d9b832d08c2244"} Sep 14 11:06:42.422869 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:42.422463 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-45rkt" event={"ID":"76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e","Type":"ContainerDied","Data":"fcbced27bdd7dddf22eced70bf58615b238b997a4dfa9b2155fa048fec808aff"} Sep 14 11:06:42.422869 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:42.422482 2574 scope.go:117] "RemoveContainer" containerID="24d8a77f30d26edba6b08833a64488dbf07aa2be5d13bd5923d9b832d08c2244" Sep 14 11:06:42.431669 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:42.431649 2574 scope.go:117] "RemoveContainer" containerID="24d8a77f30d26edba6b08833a64488dbf07aa2be5d13bd5923d9b832d08c2244" Sep 14 11:06:42.431931 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:06:42.431914 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24d8a77f30d26edba6b08833a64488dbf07aa2be5d13bd5923d9b832d08c2244\": container with ID starting with 24d8a77f30d26edba6b08833a64488dbf07aa2be5d13bd5923d9b832d08c2244 not found: ID does not exist" containerID="24d8a77f30d26edba6b08833a64488dbf07aa2be5d13bd5923d9b832d08c2244" Sep 14 11:06:42.431982 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:42.431940 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24d8a77f30d26edba6b08833a64488dbf07aa2be5d13bd5923d9b832d08c2244"} err="failed to get container status \"24d8a77f30d26edba6b08833a64488dbf07aa2be5d13bd5923d9b832d08c2244\": rpc error: code = NotFound desc = could not find container \"24d8a77f30d26edba6b08833a64488dbf07aa2be5d13bd5923d9b832d08c2244\": container with ID starting with 24d8a77f30d26edba6b08833a64488dbf07aa2be5d13bd5923d9b832d08c2244 not found: ID does not exist" Sep 14 11:06:42.459742 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:42.459722 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-45rkt"] Sep 14 11:06:42.465142 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:42.465122 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-45rkt"] Sep 14 11:06:43.903801 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:43.903747 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e" path="/var/lib/kubelet/pods/76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e/volumes" Sep 14 11:06:47.395806 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:47.395727 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-j8jfw" Sep 14 11:06:48.403590 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:06:48.403560 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-txc6g" Sep 14 11:07:45.078173 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:07:45.078138 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-78c6d7c756-btn66"] Sep 14 11:07:45.078592 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:07:45.078506 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e" containerName="discovery" Sep 14 11:07:45.078592 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:07:45.078518 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e" containerName="discovery" Sep 14 11:07:45.078592 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:07:45.078567 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="76d7fe61-ea90-4c33-9c71-bfd93f4b0f9e" containerName="discovery" Sep 14 11:07:45.081646 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:07:45.081628 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78c6d7c756-btn66" Sep 14 11:07:45.084879 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:07:45.084849 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 14 11:07:45.085019 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:07:45.084905 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 14 11:07:45.085848 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:07:45.085828 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 14 11:07:45.087686 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:07:45.087667 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-77gsx\"" Sep 14 11:07:45.087799 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:07:45.087690 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 14 11:07:45.087799 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:07:45.087692 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 14 11:07:45.098354 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:07:45.098333 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78c6d7c756-btn66"] Sep 14 11:07:45.106513 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:07:45.106491 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 14 11:07:45.181520 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:07:45.181493 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e64b36d-e059-413e-bca9-8d05a5850fd1-service-ca\") pod \"console-78c6d7c756-btn66\" (UID: \"8e64b36d-e059-413e-bca9-8d05a5850fd1\") " pod="openshift-console/console-78c6d7c756-btn66" Sep 14 11:07:45.181621 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:07:45.181541 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e64b36d-e059-413e-bca9-8d05a5850fd1-trusted-ca-bundle\") pod \"console-78c6d7c756-btn66\" (UID: \"8e64b36d-e059-413e-bca9-8d05a5850fd1\") " pod="openshift-console/console-78c6d7c756-btn66" Sep 14 11:07:45.181621 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:07:45.181575 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e64b36d-e059-413e-bca9-8d05a5850fd1-console-oauth-config\") pod \"console-78c6d7c756-btn66\" (UID: \"8e64b36d-e059-413e-bca9-8d05a5850fd1\") " pod="openshift-console/console-78c6d7c756-btn66" Sep 14 11:07:45.181706 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:07:45.181686 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psjkv\" (UniqueName: \"kubernetes.io/projected/8e64b36d-e059-413e-bca9-8d05a5850fd1-kube-api-access-psjkv\") pod \"console-78c6d7c756-btn66\" (UID: \"8e64b36d-e059-413e-bca9-8d05a5850fd1\") " pod="openshift-console/console-78c6d7c756-btn66" Sep 14 11:07:45.181746 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:07:45.181721 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e64b36d-e059-413e-bca9-8d05a5850fd1-console-serving-cert\") pod \"console-78c6d7c756-btn66\" (UID: \"8e64b36d-e059-413e-bca9-8d05a5850fd1\") " pod="openshift-console/console-78c6d7c756-btn66" Sep 14 11:07:45.181779 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:07:45.181757 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e64b36d-e059-413e-bca9-8d05a5850fd1-oauth-serving-cert\") pod \"console-78c6d7c756-btn66\" (UID: \"8e64b36d-e059-413e-bca9-8d05a5850fd1\") " pod="openshift-console/console-78c6d7c756-btn66" Sep 14 11:07:45.181811 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:07:45.181787 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e64b36d-e059-413e-bca9-8d05a5850fd1-console-config\") pod \"console-78c6d7c756-btn66\" (UID: \"8e64b36d-e059-413e-bca9-8d05a5850fd1\") " pod="openshift-console/console-78c6d7c756-btn66" Sep 14 11:07:45.282968 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:07:45.282929 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psjkv\" (UniqueName: \"kubernetes.io/projected/8e64b36d-e059-413e-bca9-8d05a5850fd1-kube-api-access-psjkv\") pod \"console-78c6d7c756-btn66\" (UID: \"8e64b36d-e059-413e-bca9-8d05a5850fd1\") " pod="openshift-console/console-78c6d7c756-btn66" Sep 14 11:07:45.282968 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:07:45.282976 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e64b36d-e059-413e-bca9-8d05a5850fd1-console-serving-cert\") pod \"console-78c6d7c756-btn66\" (UID: \"8e64b36d-e059-413e-bca9-8d05a5850fd1\") " pod="openshift-console/console-78c6d7c756-btn66" Sep 14 11:07:45.283146 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:07:45.283000 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e64b36d-e059-413e-bca9-8d05a5850fd1-oauth-serving-cert\") pod \"console-78c6d7c756-btn66\" (UID: \"8e64b36d-e059-413e-bca9-8d05a5850fd1\") " pod="openshift-console/console-78c6d7c756-btn66" Sep 14 11:07:45.283146 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:07:45.283023 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e64b36d-e059-413e-bca9-8d05a5850fd1-console-config\") pod \"console-78c6d7c756-btn66\" (UID: \"8e64b36d-e059-413e-bca9-8d05a5850fd1\") " pod="openshift-console/console-78c6d7c756-btn66" Sep 14 11:07:45.283146 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:07:45.283067 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e64b36d-e059-413e-bca9-8d05a5850fd1-service-ca\") pod \"console-78c6d7c756-btn66\" (UID: \"8e64b36d-e059-413e-bca9-8d05a5850fd1\") " pod="openshift-console/console-78c6d7c756-btn66" Sep 14 11:07:45.283250 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:07:45.283212 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e64b36d-e059-413e-bca9-8d05a5850fd1-trusted-ca-bundle\") pod \"console-78c6d7c756-btn66\" (UID: \"8e64b36d-e059-413e-bca9-8d05a5850fd1\") " pod="openshift-console/console-78c6d7c756-btn66" Sep 14 11:07:45.283338 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:07:45.283261 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e64b36d-e059-413e-bca9-8d05a5850fd1-console-oauth-config\") pod \"console-78c6d7c756-btn66\" (UID: \"8e64b36d-e059-413e-bca9-8d05a5850fd1\") " pod="openshift-console/console-78c6d7c756-btn66" Sep 14 11:07:45.283882 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:07:45.283857 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e64b36d-e059-413e-bca9-8d05a5850fd1-oauth-serving-cert\") pod \"console-78c6d7c756-btn66\" (UID: \"8e64b36d-e059-413e-bca9-8d05a5850fd1\") " pod="openshift-console/console-78c6d7c756-btn66" Sep 14 11:07:45.283975 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:07:45.283890 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e64b36d-e059-413e-bca9-8d05a5850fd1-console-config\") pod \"console-78c6d7c756-btn66\" (UID: \"8e64b36d-e059-413e-bca9-8d05a5850fd1\") " pod="openshift-console/console-78c6d7c756-btn66" Sep 14 11:07:45.283975 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:07:45.283857 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e64b36d-e059-413e-bca9-8d05a5850fd1-service-ca\") pod \"console-78c6d7c756-btn66\" (UID: \"8e64b36d-e059-413e-bca9-8d05a5850fd1\") " pod="openshift-console/console-78c6d7c756-btn66" Sep 14 11:07:45.283975 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:07:45.283924 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e64b36d-e059-413e-bca9-8d05a5850fd1-trusted-ca-bundle\") pod \"console-78c6d7c756-btn66\" (UID: \"8e64b36d-e059-413e-bca9-8d05a5850fd1\") " pod="openshift-console/console-78c6d7c756-btn66" Sep 14 11:07:45.285626 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:07:45.285609 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e64b36d-e059-413e-bca9-8d05a5850fd1-console-serving-cert\") pod \"console-78c6d7c756-btn66\" (UID: \"8e64b36d-e059-413e-bca9-8d05a5850fd1\") " pod="openshift-console/console-78c6d7c756-btn66" Sep 14 11:07:45.286013 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:07:45.285992 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e64b36d-e059-413e-bca9-8d05a5850fd1-console-oauth-config\") pod \"console-78c6d7c756-btn66\" (UID: \"8e64b36d-e059-413e-bca9-8d05a5850fd1\") " pod="openshift-console/console-78c6d7c756-btn66" Sep 14 11:07:45.292774 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:07:45.292753 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psjkv\" (UniqueName: \"kubernetes.io/projected/8e64b36d-e059-413e-bca9-8d05a5850fd1-kube-api-access-psjkv\") pod \"console-78c6d7c756-btn66\" (UID: \"8e64b36d-e059-413e-bca9-8d05a5850fd1\") " pod="openshift-console/console-78c6d7c756-btn66" Sep 14 11:07:45.390830 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:07:45.390776 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78c6d7c756-btn66" Sep 14 11:07:45.677603 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:07:45.677574 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78c6d7c756-btn66"] Sep 14 11:07:45.683545 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:07:45.683504 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e64b36d_e059_413e_bca9_8d05a5850fd1.slice/crio-f58fede434b07249a4233ffceb87efbd0fe64c0d7f031e77b25e0d6ec5ad8754 WatchSource:0}: Error finding container f58fede434b07249a4233ffceb87efbd0fe64c0d7f031e77b25e0d6ec5ad8754: Status 404 returned error can't find the container with id f58fede434b07249a4233ffceb87efbd0fe64c0d7f031e77b25e0d6ec5ad8754 Sep 14 11:07:45.702855 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:07:45.702826 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78c6d7c756-btn66" event={"ID":"8e64b36d-e059-413e-bca9-8d05a5850fd1","Type":"ContainerStarted","Data":"f58fede434b07249a4233ffceb87efbd0fe64c0d7f031e77b25e0d6ec5ad8754"} Sep 14 11:07:46.709721 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:07:46.709691 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78c6d7c756-btn66" event={"ID":"8e64b36d-e059-413e-bca9-8d05a5850fd1","Type":"ContainerStarted","Data":"66605c7f1b09533068f179234a06c147a002c553d3d849ffefbacef81c5804b0"} Sep 14 11:07:46.738129 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:07:46.738084 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-78c6d7c756-btn66" podStartSLOduration=1.738066258 podStartE2EDuration="1.738066258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:07:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:07:46.733904275 +0000 UTC m=+411.354116817" watchObservedRunningTime="2026-09-14 11:07:46.738066258 +0000 UTC m=+411.358278782" Sep 14 11:07:55.390954 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:07:55.390918 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-78c6d7c756-btn66" Sep 14 11:07:55.391408 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:07:55.390980 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78c6d7c756-btn66" Sep 14 11:07:55.395784 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:07:55.395762 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-78c6d7c756-btn66" Sep 14 11:07:55.754350 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:07:55.754324 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-78c6d7c756-btn66" Sep 14 11:08:06.700015 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:06.699984 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-zvw7m"] Sep 14 11:08:06.703621 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:06.703599 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-zvw7m" Sep 14 11:08:06.706378 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:06.706238 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 14 11:08:06.706378 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:06.706293 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-hwdsh\"" Sep 14 11:08:06.706560 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:06.706525 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 14 11:08:06.723064 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:06.723039 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-zvw7m"] Sep 14 11:08:06.767681 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:06.767651 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlhx8\" (UniqueName: \"kubernetes.io/projected/66f6a622-980a-4372-b50c-fc633812faa9-kube-api-access-dlhx8\") pod \"authorino-operator-76b478bd78-zvw7m\" (UID: \"66f6a622-980a-4372-b50c-fc633812faa9\") " pod="kuadrant-system/authorino-operator-76b478bd78-zvw7m" Sep 14 11:08:06.868191 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:06.868166 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlhx8\" (UniqueName: \"kubernetes.io/projected/66f6a622-980a-4372-b50c-fc633812faa9-kube-api-access-dlhx8\") pod \"authorino-operator-76b478bd78-zvw7m\" (UID: \"66f6a622-980a-4372-b50c-fc633812faa9\") " pod="kuadrant-system/authorino-operator-76b478bd78-zvw7m" Sep 14 11:08:06.883751 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:06.880972 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlhx8\" (UniqueName: \"kubernetes.io/projected/66f6a622-980a-4372-b50c-fc633812faa9-kube-api-access-dlhx8\") pod \"authorino-operator-76b478bd78-zvw7m\" (UID: \"66f6a622-980a-4372-b50c-fc633812faa9\") " pod="kuadrant-system/authorino-operator-76b478bd78-zvw7m" Sep 14 11:08:07.013543 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:07.013489 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-zvw7m" Sep 14 11:08:07.296955 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:07.296923 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-zvw7m"] Sep 14 11:08:07.299790 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:08:07.299766 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66f6a622_980a_4372_b50c_fc633812faa9.slice/crio-376d73e14fea433f98f18959434136874fbe0d224c696fdc2cc849b57481fa13 WatchSource:0}: Error finding container 376d73e14fea433f98f18959434136874fbe0d224c696fdc2cc849b57481fa13: Status 404 returned error can't find the container with id 376d73e14fea433f98f18959434136874fbe0d224c696fdc2cc849b57481fa13 Sep 14 11:08:07.806868 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:07.806830 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-zvw7m" event={"ID":"66f6a622-980a-4372-b50c-fc633812faa9","Type":"ContainerStarted","Data":"376d73e14fea433f98f18959434136874fbe0d224c696fdc2cc849b57481fa13"} Sep 14 11:08:08.934999 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:08.934969 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-k2ktp"] Sep 14 11:08:08.943890 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:08.943865 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-k2ktp" Sep 14 11:08:08.946781 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:08.946751 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-jhh5n\"" Sep 14 11:08:08.952649 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:08.952384 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-k2ktp"] Sep 14 11:08:08.989401 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:08.989370 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dwwq\" (UniqueName: \"kubernetes.io/projected/bb86fca2-45d8-46a9-a38b-85cc037b0d7f-kube-api-access-2dwwq\") pod \"limitador-operator-controller-manager-74495c69f8-k2ktp\" (UID: \"bb86fca2-45d8-46a9-a38b-85cc037b0d7f\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-k2ktp" Sep 14 11:08:09.090210 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:09.090175 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dwwq\" (UniqueName: \"kubernetes.io/projected/bb86fca2-45d8-46a9-a38b-85cc037b0d7f-kube-api-access-2dwwq\") pod \"limitador-operator-controller-manager-74495c69f8-k2ktp\" (UID: \"bb86fca2-45d8-46a9-a38b-85cc037b0d7f\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-k2ktp" Sep 14 11:08:09.103881 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:09.103851 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dwwq\" (UniqueName: \"kubernetes.io/projected/bb86fca2-45d8-46a9-a38b-85cc037b0d7f-kube-api-access-2dwwq\") pod \"limitador-operator-controller-manager-74495c69f8-k2ktp\" (UID: \"bb86fca2-45d8-46a9-a38b-85cc037b0d7f\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-k2ktp" Sep 14 11:08:09.257243 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:09.257164 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-k2ktp" Sep 14 11:08:09.820194 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:09.820169 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-zvw7m" event={"ID":"66f6a622-980a-4372-b50c-fc633812faa9","Type":"ContainerStarted","Data":"82fdb0d0065c741d49fbbd6a70fd7e419f42fa61979bae896871a676c2923507"} Sep 14 11:08:09.820339 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:09.820321 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-zvw7m" Sep 14 11:08:09.850016 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:09.849977 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-zvw7m" podStartSLOduration=1.4111747989999999 podStartE2EDuration="3.849963475s" totalImagesPullingTime="2.438788676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:08:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:08:09.849453697 +0000 UTC m=+434.469666240" watchObservedRunningTime="2026-09-14 11:08:09.849963475 +0000 UTC m=+434.470175998" Sep 14 11:08:09.957967 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:09.957941 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-k2ktp"] Sep 14 11:08:09.960311 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:08:09.960254 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb86fca2_45d8_46a9_a38b_85cc037b0d7f.slice/crio-07a543b72545d371d5cac309597d556bb4bb70b65229c608504fc9f5c02e5c3a WatchSource:0}: Error finding container 07a543b72545d371d5cac309597d556bb4bb70b65229c608504fc9f5c02e5c3a: Status 404 returned error can't find the container with id 07a543b72545d371d5cac309597d556bb4bb70b65229c608504fc9f5c02e5c3a Sep 14 11:08:10.825793 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:10.825756 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-k2ktp" event={"ID":"bb86fca2-45d8-46a9-a38b-85cc037b0d7f","Type":"ContainerStarted","Data":"07a543b72545d371d5cac309597d556bb4bb70b65229c608504fc9f5c02e5c3a"} Sep 14 11:08:13.842084 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:13.842043 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-k2ktp" event={"ID":"bb86fca2-45d8-46a9-a38b-85cc037b0d7f","Type":"ContainerStarted","Data":"e9289001319d35fb698a258b77c5ccf55e4ce5c5c170cd9a03d810f873b7fc8b"} Sep 14 11:08:13.842563 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:13.842180 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-k2ktp" Sep 14 11:08:13.871012 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:13.870958 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-k2ktp" podStartSLOduration=2.9918342 podStartE2EDuration="5.870941155s" totalImagesPullingTime="2.879106955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:08:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:08:13.869776218 +0000 UTC m=+438.489988767" watchObservedRunningTime="2026-09-14 11:08:13.870941155 +0000 UTC m=+438.491153678" Sep 14 11:08:20.827529 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:20.827460 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-zvw7m" Sep 14 11:08:24.848519 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:24.848484 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-k2ktp" Sep 14 11:08:42.151559 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:42.151528 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-k2ktp"] Sep 14 11:08:42.152124 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:42.151806 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-k2ktp" podUID="bb86fca2-45d8-46a9-a38b-85cc037b0d7f" containerName="manager" containerID="cri-o://e9289001319d35fb698a258b77c5ccf55e4ce5c5c170cd9a03d810f873b7fc8b" gracePeriod=2 Sep 14 11:08:42.172743 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:42.172701 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-k2ktp"] Sep 14 11:08:42.190601 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:42.190570 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-m29tk"] Sep 14 11:08:42.190967 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:42.190939 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb86fca2-45d8-46a9-a38b-85cc037b0d7f" containerName="manager" Sep 14 11:08:42.190967 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:42.190954 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb86fca2-45d8-46a9-a38b-85cc037b0d7f" containerName="manager" Sep 14 11:08:42.191098 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:42.191024 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb86fca2-45d8-46a9-a38b-85cc037b0d7f" containerName="manager" Sep 14 11:08:42.194115 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:42.194093 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m29tk" Sep 14 11:08:42.197596 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:08:42.197569 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-k2ktp\" is forbidden: User \"system:node:ip-10-0-137-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-230.ec2.internal' and this object" podUID="bb86fca2-45d8-46a9-a38b-85cc037b0d7f" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-k2ktp" Sep 14 11:08:42.205937 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:42.205913 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-m29tk"] Sep 14 11:08:42.288254 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:42.288220 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btx6h\" (UniqueName: \"kubernetes.io/projected/30723dd6-3820-45a5-b47a-3ba0942e29b5-kube-api-access-btx6h\") pod \"limitador-operator-controller-manager-74495c69f8-m29tk\" (UID: \"30723dd6-3820-45a5-b47a-3ba0942e29b5\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m29tk" Sep 14 11:08:42.387117 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:42.387090 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-k2ktp" Sep 14 11:08:42.389131 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:42.389112 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dwwq\" (UniqueName: \"kubernetes.io/projected/bb86fca2-45d8-46a9-a38b-85cc037b0d7f-kube-api-access-2dwwq\") pod \"bb86fca2-45d8-46a9-a38b-85cc037b0d7f\" (UID: \"bb86fca2-45d8-46a9-a38b-85cc037b0d7f\") " Sep 14 11:08:42.389324 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:42.389304 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btx6h\" (UniqueName: \"kubernetes.io/projected/30723dd6-3820-45a5-b47a-3ba0942e29b5-kube-api-access-btx6h\") pod \"limitador-operator-controller-manager-74495c69f8-m29tk\" (UID: \"30723dd6-3820-45a5-b47a-3ba0942e29b5\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m29tk" Sep 14 11:08:42.389400 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:08:42.389316 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-k2ktp\" is forbidden: User \"system:node:ip-10-0-137-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-230.ec2.internal' and this object" podUID="bb86fca2-45d8-46a9-a38b-85cc037b0d7f" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-k2ktp" Sep 14 11:08:42.391356 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:42.391332 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb86fca2-45d8-46a9-a38b-85cc037b0d7f-kube-api-access-2dwwq" (OuterVolumeSpecName: "kube-api-access-2dwwq") pod "bb86fca2-45d8-46a9-a38b-85cc037b0d7f" (UID: "bb86fca2-45d8-46a9-a38b-85cc037b0d7f"). InnerVolumeSpecName "kube-api-access-2dwwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:08:42.402026 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:42.401978 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btx6h\" (UniqueName: \"kubernetes.io/projected/30723dd6-3820-45a5-b47a-3ba0942e29b5-kube-api-access-btx6h\") pod \"limitador-operator-controller-manager-74495c69f8-m29tk\" (UID: \"30723dd6-3820-45a5-b47a-3ba0942e29b5\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m29tk" Sep 14 11:08:42.490000 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:42.489973 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dwwq\" (UniqueName: \"kubernetes.io/projected/bb86fca2-45d8-46a9-a38b-85cc037b0d7f-kube-api-access-2dwwq\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:08:42.524864 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:42.524843 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m29tk" Sep 14 11:08:42.814896 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:08:42.814865 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30723dd6_3820_45a5_b47a_3ba0942e29b5.slice/crio-06e203dc9e2443b340bbb9584767dc1ab012936670aeb904c62d2c35f276aaed WatchSource:0}: Error finding container 06e203dc9e2443b340bbb9584767dc1ab012936670aeb904c62d2c35f276aaed: Status 404 returned error can't find the container with id 06e203dc9e2443b340bbb9584767dc1ab012936670aeb904c62d2c35f276aaed Sep 14 11:08:42.816117 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:42.816092 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-m29tk"] Sep 14 11:08:42.973925 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:42.973893 2574 generic.go:356] "Generic (PLEG): container finished" podID="bb86fca2-45d8-46a9-a38b-85cc037b0d7f" containerID="e9289001319d35fb698a258b77c5ccf55e4ce5c5c170cd9a03d810f873b7fc8b" exitCode=0 Sep 14 11:08:42.974087 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:42.973943 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-k2ktp" Sep 14 11:08:42.974087 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:42.973952 2574 scope.go:117] "RemoveContainer" containerID="e9289001319d35fb698a258b77c5ccf55e4ce5c5c170cd9a03d810f873b7fc8b" Sep 14 11:08:42.975816 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:42.975789 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m29tk" event={"ID":"30723dd6-3820-45a5-b47a-3ba0942e29b5","Type":"ContainerStarted","Data":"4f4fba78dc0ba0a746ca9b5aa8379b1f417061930c85504e905783ad6bb9f85e"} Sep 14 11:08:42.975932 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:42.975852 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m29tk" Sep 14 11:08:42.975932 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:42.975880 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m29tk" event={"ID":"30723dd6-3820-45a5-b47a-3ba0942e29b5","Type":"ContainerStarted","Data":"06e203dc9e2443b340bbb9584767dc1ab012936670aeb904c62d2c35f276aaed"} Sep 14 11:08:42.976374 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:08:42.976352 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-k2ktp\" is forbidden: User \"system:node:ip-10-0-137-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-230.ec2.internal' and this object" podUID="bb86fca2-45d8-46a9-a38b-85cc037b0d7f" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-k2ktp" Sep 14 11:08:42.978434 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:08:42.978406 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-k2ktp\" is forbidden: User \"system:node:ip-10-0-137-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-230.ec2.internal' and this object" podUID="bb86fca2-45d8-46a9-a38b-85cc037b0d7f" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-k2ktp" Sep 14 11:08:42.982376 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:42.982347 2574 scope.go:117] "RemoveContainer" containerID="e9289001319d35fb698a258b77c5ccf55e4ce5c5c170cd9a03d810f873b7fc8b" Sep 14 11:08:42.982634 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:08:42.982617 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9289001319d35fb698a258b77c5ccf55e4ce5c5c170cd9a03d810f873b7fc8b\": container with ID starting with e9289001319d35fb698a258b77c5ccf55e4ce5c5c170cd9a03d810f873b7fc8b not found: ID does not exist" containerID="e9289001319d35fb698a258b77c5ccf55e4ce5c5c170cd9a03d810f873b7fc8b" Sep 14 11:08:42.982687 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:42.982642 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9289001319d35fb698a258b77c5ccf55e4ce5c5c170cd9a03d810f873b7fc8b"} err="failed to get container status \"e9289001319d35fb698a258b77c5ccf55e4ce5c5c170cd9a03d810f873b7fc8b\": rpc error: code = NotFound desc = could not find container \"e9289001319d35fb698a258b77c5ccf55e4ce5c5c170cd9a03d810f873b7fc8b\": container with ID starting with e9289001319d35fb698a258b77c5ccf55e4ce5c5c170cd9a03d810f873b7fc8b not found: ID does not exist" Sep 14 11:08:43.000618 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:43.000577 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m29tk" podStartSLOduration=1.000533766 podStartE2EDuration="1.000533766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:08:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:08:42.999994297 +0000 UTC m=+467.620206831" watchObservedRunningTime="2026-09-14 11:08:43.000533766 +0000 UTC m=+467.620746290" Sep 14 11:08:43.002762 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:08:43.002729 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-k2ktp\" is forbidden: User \"system:node:ip-10-0-137-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-230.ec2.internal' and this object" podUID="bb86fca2-45d8-46a9-a38b-85cc037b0d7f" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-k2ktp" Sep 14 11:08:43.899053 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:43.899020 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb86fca2-45d8-46a9-a38b-85cc037b0d7f" path="/var/lib/kubelet/pods/bb86fca2-45d8-46a9-a38b-85cc037b0d7f/volumes" Sep 14 11:08:53.983876 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:08:53.983838 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m29tk" Sep 14 11:09:47.785872 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:09:47.785786 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-lvtpn"] Sep 14 11:09:47.791665 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:09:47.791644 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-lvtpn" Sep 14 11:09:47.794717 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:09:47.794690 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-sysctl-allowlist\"" Sep 14 11:09:47.875800 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:09:47.875764 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnb82\" (UniqueName: \"kubernetes.io/projected/c53d6119-f15e-42a3-96e5-48c0c2caa38c-kube-api-access-mnb82\") pod \"cni-sysctl-allowlist-ds-lvtpn\" (UID: \"c53d6119-f15e-42a3-96e5-48c0c2caa38c\") " pod="openshift-multus/cni-sysctl-allowlist-ds-lvtpn" Sep 14 11:09:47.875959 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:09:47.875816 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/c53d6119-f15e-42a3-96e5-48c0c2caa38c-ready\") pod \"cni-sysctl-allowlist-ds-lvtpn\" (UID: \"c53d6119-f15e-42a3-96e5-48c0c2caa38c\") " pod="openshift-multus/cni-sysctl-allowlist-ds-lvtpn" Sep 14 11:09:47.875959 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:09:47.875838 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c53d6119-f15e-42a3-96e5-48c0c2caa38c-tuning-conf-dir\") pod \"cni-sysctl-allowlist-ds-lvtpn\" (UID: \"c53d6119-f15e-42a3-96e5-48c0c2caa38c\") " pod="openshift-multus/cni-sysctl-allowlist-ds-lvtpn" Sep 14 11:09:47.875959 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:09:47.875920 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c53d6119-f15e-42a3-96e5-48c0c2caa38c-cni-sysctl-allowlist\") pod \"cni-sysctl-allowlist-ds-lvtpn\" (UID: \"c53d6119-f15e-42a3-96e5-48c0c2caa38c\") " pod="openshift-multus/cni-sysctl-allowlist-ds-lvtpn" Sep 14 11:09:47.977062 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:09:47.977015 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnb82\" (UniqueName: \"kubernetes.io/projected/c53d6119-f15e-42a3-96e5-48c0c2caa38c-kube-api-access-mnb82\") pod \"cni-sysctl-allowlist-ds-lvtpn\" (UID: \"c53d6119-f15e-42a3-96e5-48c0c2caa38c\") " pod="openshift-multus/cni-sysctl-allowlist-ds-lvtpn" Sep 14 11:09:47.977290 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:09:47.977100 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/c53d6119-f15e-42a3-96e5-48c0c2caa38c-ready\") pod \"cni-sysctl-allowlist-ds-lvtpn\" (UID: \"c53d6119-f15e-42a3-96e5-48c0c2caa38c\") " pod="openshift-multus/cni-sysctl-allowlist-ds-lvtpn" Sep 14 11:09:47.977290 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:09:47.977129 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c53d6119-f15e-42a3-96e5-48c0c2caa38c-tuning-conf-dir\") pod \"cni-sysctl-allowlist-ds-lvtpn\" (UID: \"c53d6119-f15e-42a3-96e5-48c0c2caa38c\") " pod="openshift-multus/cni-sysctl-allowlist-ds-lvtpn" Sep 14 11:09:47.977290 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:09:47.977173 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c53d6119-f15e-42a3-96e5-48c0c2caa38c-cni-sysctl-allowlist\") pod \"cni-sysctl-allowlist-ds-lvtpn\" (UID: \"c53d6119-f15e-42a3-96e5-48c0c2caa38c\") " pod="openshift-multus/cni-sysctl-allowlist-ds-lvtpn" Sep 14 11:09:47.977821 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:09:47.977792 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c53d6119-f15e-42a3-96e5-48c0c2caa38c-cni-sysctl-allowlist\") pod \"cni-sysctl-allowlist-ds-lvtpn\" (UID: \"c53d6119-f15e-42a3-96e5-48c0c2caa38c\") " pod="openshift-multus/cni-sysctl-allowlist-ds-lvtpn" Sep 14 11:09:47.978380 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:09:47.978355 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/c53d6119-f15e-42a3-96e5-48c0c2caa38c-ready\") pod \"cni-sysctl-allowlist-ds-lvtpn\" (UID: \"c53d6119-f15e-42a3-96e5-48c0c2caa38c\") " pod="openshift-multus/cni-sysctl-allowlist-ds-lvtpn" Sep 14 11:09:47.978480 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:09:47.978436 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c53d6119-f15e-42a3-96e5-48c0c2caa38c-tuning-conf-dir\") pod \"cni-sysctl-allowlist-ds-lvtpn\" (UID: \"c53d6119-f15e-42a3-96e5-48c0c2caa38c\") " pod="openshift-multus/cni-sysctl-allowlist-ds-lvtpn" Sep 14 11:09:47.991465 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:09:47.991427 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnb82\" (UniqueName: \"kubernetes.io/projected/c53d6119-f15e-42a3-96e5-48c0c2caa38c-kube-api-access-mnb82\") pod \"cni-sysctl-allowlist-ds-lvtpn\" (UID: \"c53d6119-f15e-42a3-96e5-48c0c2caa38c\") " pod="openshift-multus/cni-sysctl-allowlist-ds-lvtpn" Sep 14 11:09:48.100556 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:09:48.100466 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-lvtpn" Sep 14 11:09:48.112425 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:09:48.112395 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc53d6119_f15e_42a3_96e5_48c0c2caa38c.slice/crio-3eb6a81528611767a49c377771b640eda169918bdd08d0b2b20a7da34cdd4f21 WatchSource:0}: Error finding container 3eb6a81528611767a49c377771b640eda169918bdd08d0b2b20a7da34cdd4f21: Status 404 returned error can't find the container with id 3eb6a81528611767a49c377771b640eda169918bdd08d0b2b20a7da34cdd4f21 Sep 14 11:09:48.285740 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:09:48.285704 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-lvtpn" event={"ID":"c53d6119-f15e-42a3-96e5-48c0c2caa38c","Type":"ContainerStarted","Data":"9f1a90734c95e2e60272736b7eed3dc5b8bfa6ed68810cf1a4613c48b3ae95db"} Sep 14 11:09:48.285941 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:09:48.285765 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-lvtpn" event={"ID":"c53d6119-f15e-42a3-96e5-48c0c2caa38c","Type":"ContainerStarted","Data":"3eb6a81528611767a49c377771b640eda169918bdd08d0b2b20a7da34cdd4f21"} Sep 14 11:09:48.285941 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:09:48.285933 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-multus/cni-sysctl-allowlist-ds-lvtpn" Sep 14 11:09:48.317062 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:09:48.317011 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/cni-sysctl-allowlist-ds-lvtpn" podStartSLOduration=1.316994594 podStartE2EDuration="1.316994594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:09:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:09:48.316759492 +0000 UTC m=+532.936972025" watchObservedRunningTime="2026-09-14 11:09:48.316994594 +0000 UTC m=+532.937207140" Sep 14 11:09:49.110704 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:09:49.110665 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lhvl5"] Sep 14 11:09:49.114253 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:09:49.114230 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-lhvl5" Sep 14 11:09:49.117185 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:09:49.117166 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 14 11:09:49.117309 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:09:49.117168 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7khrq\"" Sep 14 11:09:49.126756 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:09:49.126735 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lhvl5"] Sep 14 11:09:49.188878 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:09:49.188845 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1f71825b-72cc-44b7-9030-7d2387f091e3-config-file\") pod \"limitador-limitador-8588749977-lhvl5\" (UID: \"1f71825b-72cc-44b7-9030-7d2387f091e3\") " pod="kuadrant-system/limitador-limitador-8588749977-lhvl5" Sep 14 11:09:49.189037 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:09:49.188888 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6sg2\" (UniqueName: \"kubernetes.io/projected/1f71825b-72cc-44b7-9030-7d2387f091e3-kube-api-access-m6sg2\") pod \"limitador-limitador-8588749977-lhvl5\" (UID: \"1f71825b-72cc-44b7-9030-7d2387f091e3\") " pod="kuadrant-system/limitador-limitador-8588749977-lhvl5" Sep 14 11:09:49.197078 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:09:49.197048 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lhvl5"] Sep 14 11:09:49.290004 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:09:49.289975 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1f71825b-72cc-44b7-9030-7d2387f091e3-config-file\") pod \"limitador-limitador-8588749977-lhvl5\" (UID: \"1f71825b-72cc-44b7-9030-7d2387f091e3\") " pod="kuadrant-system/limitador-limitador-8588749977-lhvl5" Sep 14 11:09:49.290250 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:09:49.290020 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6sg2\" (UniqueName: \"kubernetes.io/projected/1f71825b-72cc-44b7-9030-7d2387f091e3-kube-api-access-m6sg2\") pod \"limitador-limitador-8588749977-lhvl5\" (UID: \"1f71825b-72cc-44b7-9030-7d2387f091e3\") " pod="kuadrant-system/limitador-limitador-8588749977-lhvl5" Sep 14 11:09:49.290752 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:09:49.290727 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1f71825b-72cc-44b7-9030-7d2387f091e3-config-file\") pod \"limitador-limitador-8588749977-lhvl5\" (UID: \"1f71825b-72cc-44b7-9030-7d2387f091e3\") " pod="kuadrant-system/limitador-limitador-8588749977-lhvl5" Sep 14 11:09:49.301570 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:09:49.301539 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6sg2\" (UniqueName: \"kubernetes.io/projected/1f71825b-72cc-44b7-9030-7d2387f091e3-kube-api-access-m6sg2\") pod \"limitador-limitador-8588749977-lhvl5\" (UID: \"1f71825b-72cc-44b7-9030-7d2387f091e3\") " pod="kuadrant-system/limitador-limitador-8588749977-lhvl5" Sep 14 11:09:49.302049 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:09:49.302010 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-multus/cni-sysctl-allowlist-ds-lvtpn" Sep 14 11:09:49.426211 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:09:49.426175 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-lhvl5" Sep 14 11:09:49.705548 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:09:49.705524 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lhvl5"] Sep 14 11:09:49.707672 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:09:49.707646 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f71825b_72cc_44b7_9030_7d2387f091e3.slice/crio-eed434140a0b479d747f4e7ef4b75bc2f03f2ee2a4badbd9386f2b0555e614c0 WatchSource:0}: Error finding container eed434140a0b479d747f4e7ef4b75bc2f03f2ee2a4badbd9386f2b0555e614c0: Status 404 returned error can't find the container with id eed434140a0b479d747f4e7ef4b75bc2f03f2ee2a4badbd9386f2b0555e614c0 Sep 14 11:09:50.297076 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:09:50.297038 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-lhvl5" event={"ID":"1f71825b-72cc-44b7-9030-7d2387f091e3","Type":"ContainerStarted","Data":"eed434140a0b479d747f4e7ef4b75bc2f03f2ee2a4badbd9386f2b0555e614c0"} Sep 14 11:09:50.454180 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:09:50.454147 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-lvtpn"] Sep 14 11:09:51.302851 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:09:51.302766 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-lhvl5" event={"ID":"1f71825b-72cc-44b7-9030-7d2387f091e3","Type":"ContainerStarted","Data":"198da4bd369c452de56557be650f775a3351b254735c6d963e40afe215f60ea5"} Sep 14 11:09:51.303249 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:09:51.302877 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-multus/cni-sysctl-allowlist-ds-lvtpn" podUID="c53d6119-f15e-42a3-96e5-48c0c2caa38c" containerName="kube-multus-additional-cni-plugins" containerID="cri-o://9f1a90734c95e2e60272736b7eed3dc5b8bfa6ed68810cf1a4613c48b3ae95db" gracePeriod=30 Sep 14 11:09:51.303249 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:09:51.302977 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-lhvl5" Sep 14 11:09:59.292903 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:09:59.292854 2574 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="9f1a90734c95e2e60272736b7eed3dc5b8bfa6ed68810cf1a4613c48b3ae95db" cmd=["/bin/bash","-c","test -f /ready/ready"] Sep 14 11:09:59.293882 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:09:59.293846 2574 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="9f1a90734c95e2e60272736b7eed3dc5b8bfa6ed68810cf1a4613c48b3ae95db" cmd=["/bin/bash","-c","test -f /ready/ready"] Sep 14 11:09:59.294743 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:09:59.294707 2574 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="9f1a90734c95e2e60272736b7eed3dc5b8bfa6ed68810cf1a4613c48b3ae95db" cmd=["/bin/bash","-c","test -f /ready/ready"] Sep 14 11:09:59.294838 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:09:59.294753 2574 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-multus/cni-sysctl-allowlist-ds-lvtpn" podUID="c53d6119-f15e-42a3-96e5-48c0c2caa38c" containerName="kube-multus-additional-cni-plugins" probeResult="unknown" Sep 14 11:10:02.308672 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:02.308642 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-lhvl5" Sep 14 11:10:02.330046 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:02.329995 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-lhvl5" podStartSLOduration=12.106595531 podStartE2EDuration="13.329978106s" totalImagesPullingTime="1.223382575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:09:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:09:51.367095122 +0000 UTC m=+535.987307646" watchObservedRunningTime="2026-09-14 11:10:02.329978106 +0000 UTC m=+546.950190630" Sep 14 11:10:03.722607 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:03.722570 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lhvl5"] Sep 14 11:10:03.723046 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:03.722757 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-lhvl5" podUID="1f71825b-72cc-44b7-9030-7d2387f091e3" containerName="limitador" containerID="cri-o://198da4bd369c452de56557be650f775a3351b254735c6d963e40afe215f60ea5" gracePeriod=30 Sep 14 11:10:04.264841 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:04.264820 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-lhvl5" Sep 14 11:10:04.332990 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:04.332892 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1f71825b-72cc-44b7-9030-7d2387f091e3-config-file\") pod \"1f71825b-72cc-44b7-9030-7d2387f091e3\" (UID: \"1f71825b-72cc-44b7-9030-7d2387f091e3\") " Sep 14 11:10:04.332990 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:04.332957 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6sg2\" (UniqueName: \"kubernetes.io/projected/1f71825b-72cc-44b7-9030-7d2387f091e3-kube-api-access-m6sg2\") pod \"1f71825b-72cc-44b7-9030-7d2387f091e3\" (UID: \"1f71825b-72cc-44b7-9030-7d2387f091e3\") " Sep 14 11:10:04.333228 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:04.333204 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f71825b-72cc-44b7-9030-7d2387f091e3-config-file" (OuterVolumeSpecName: "config-file") pod "1f71825b-72cc-44b7-9030-7d2387f091e3" (UID: "1f71825b-72cc-44b7-9030-7d2387f091e3"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:10:04.335136 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:04.335116 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f71825b-72cc-44b7-9030-7d2387f091e3-kube-api-access-m6sg2" (OuterVolumeSpecName: "kube-api-access-m6sg2") pod "1f71825b-72cc-44b7-9030-7d2387f091e3" (UID: "1f71825b-72cc-44b7-9030-7d2387f091e3"). InnerVolumeSpecName "kube-api-access-m6sg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:10:04.365562 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:04.365532 2574 generic.go:356] "Generic (PLEG): container finished" podID="1f71825b-72cc-44b7-9030-7d2387f091e3" containerID="198da4bd369c452de56557be650f775a3351b254735c6d963e40afe215f60ea5" exitCode=0 Sep 14 11:10:04.365695 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:04.365571 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-lhvl5" Sep 14 11:10:04.365695 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:04.365626 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-lhvl5" event={"ID":"1f71825b-72cc-44b7-9030-7d2387f091e3","Type":"ContainerDied","Data":"198da4bd369c452de56557be650f775a3351b254735c6d963e40afe215f60ea5"} Sep 14 11:10:04.365695 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:04.365676 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-lhvl5" event={"ID":"1f71825b-72cc-44b7-9030-7d2387f091e3","Type":"ContainerDied","Data":"eed434140a0b479d747f4e7ef4b75bc2f03f2ee2a4badbd9386f2b0555e614c0"} Sep 14 11:10:04.365844 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:04.365699 2574 scope.go:117] "RemoveContainer" containerID="198da4bd369c452de56557be650f775a3351b254735c6d963e40afe215f60ea5" Sep 14 11:10:04.374643 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:04.374629 2574 scope.go:117] "RemoveContainer" containerID="198da4bd369c452de56557be650f775a3351b254735c6d963e40afe215f60ea5" Sep 14 11:10:04.374898 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:10:04.374882 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"198da4bd369c452de56557be650f775a3351b254735c6d963e40afe215f60ea5\": container with ID starting with 198da4bd369c452de56557be650f775a3351b254735c6d963e40afe215f60ea5 not found: ID does not exist" containerID="198da4bd369c452de56557be650f775a3351b254735c6d963e40afe215f60ea5" Sep 14 11:10:04.374937 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:04.374906 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"198da4bd369c452de56557be650f775a3351b254735c6d963e40afe215f60ea5"} err="failed to get container status \"198da4bd369c452de56557be650f775a3351b254735c6d963e40afe215f60ea5\": rpc error: code = NotFound desc = could not find container \"198da4bd369c452de56557be650f775a3351b254735c6d963e40afe215f60ea5\": container with ID starting with 198da4bd369c452de56557be650f775a3351b254735c6d963e40afe215f60ea5 not found: ID does not exist" Sep 14 11:10:04.387042 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:04.387013 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lhvl5"] Sep 14 11:10:04.389645 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:04.389624 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lhvl5"] Sep 14 11:10:04.434020 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:04.433983 2574 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1f71825b-72cc-44b7-9030-7d2387f091e3-config-file\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:10:04.434020 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:04.434014 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6sg2\" (UniqueName: \"kubernetes.io/projected/1f71825b-72cc-44b7-9030-7d2387f091e3-kube-api-access-m6sg2\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:10:05.472120 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:05.472084 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-hfwtg"] Sep 14 11:10:05.472540 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:05.472476 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f71825b-72cc-44b7-9030-7d2387f091e3" containerName="limitador" Sep 14 11:10:05.472540 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:05.472489 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f71825b-72cc-44b7-9030-7d2387f091e3" containerName="limitador" Sep 14 11:10:05.472607 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:05.472554 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f71825b-72cc-44b7-9030-7d2387f091e3" containerName="limitador" Sep 14 11:10:05.476991 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:05.476972 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hfwtg" Sep 14 11:10:05.480157 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:05.480136 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 14 11:10:05.480564 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:05.480549 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 11:10:05.480993 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:05.480974 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-n79lm\"" Sep 14 11:10:05.481210 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:05.481194 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 14 11:10:05.489838 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:05.489813 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-hfwtg"] Sep 14 11:10:05.543931 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:05.543892 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5677dd72-2c2c-4ef0-877c-fd3cde06194e-data\") pod \"postgres-69fb4594cf-hfwtg\" (UID: \"5677dd72-2c2c-4ef0-877c-fd3cde06194e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hfwtg" Sep 14 11:10:05.544116 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:05.543943 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsv9s\" (UniqueName: \"kubernetes.io/projected/5677dd72-2c2c-4ef0-877c-fd3cde06194e-kube-api-access-vsv9s\") pod \"postgres-69fb4594cf-hfwtg\" (UID: \"5677dd72-2c2c-4ef0-877c-fd3cde06194e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hfwtg" Sep 14 11:10:05.645042 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:05.645004 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5677dd72-2c2c-4ef0-877c-fd3cde06194e-data\") pod \"postgres-69fb4594cf-hfwtg\" (UID: \"5677dd72-2c2c-4ef0-877c-fd3cde06194e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hfwtg" Sep 14 11:10:05.645212 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:05.645055 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsv9s\" (UniqueName: \"kubernetes.io/projected/5677dd72-2c2c-4ef0-877c-fd3cde06194e-kube-api-access-vsv9s\") pod \"postgres-69fb4594cf-hfwtg\" (UID: \"5677dd72-2c2c-4ef0-877c-fd3cde06194e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hfwtg" Sep 14 11:10:05.645448 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:05.645424 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5677dd72-2c2c-4ef0-877c-fd3cde06194e-data\") pod \"postgres-69fb4594cf-hfwtg\" (UID: \"5677dd72-2c2c-4ef0-877c-fd3cde06194e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hfwtg" Sep 14 11:10:05.654329 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:05.654308 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsv9s\" (UniqueName: \"kubernetes.io/projected/5677dd72-2c2c-4ef0-877c-fd3cde06194e-kube-api-access-vsv9s\") pod \"postgres-69fb4594cf-hfwtg\" (UID: \"5677dd72-2c2c-4ef0-877c-fd3cde06194e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-hfwtg" Sep 14 11:10:05.788182 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:05.788099 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hfwtg" Sep 14 11:10:05.899257 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:05.899227 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f71825b-72cc-44b7-9030-7d2387f091e3" path="/var/lib/kubelet/pods/1f71825b-72cc-44b7-9030-7d2387f091e3/volumes" Sep 14 11:10:06.060520 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:06.060492 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-hfwtg"] Sep 14 11:10:06.062096 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:10:06.062072 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5677dd72_2c2c_4ef0_877c_fd3cde06194e.slice/crio-5cc48e48050c8b1c5975b6be312f191a16e20b91fd046e77a2ccee435f65023c WatchSource:0}: Error finding container 5cc48e48050c8b1c5975b6be312f191a16e20b91fd046e77a2ccee435f65023c: Status 404 returned error can't find the container with id 5cc48e48050c8b1c5975b6be312f191a16e20b91fd046e77a2ccee435f65023c Sep 14 11:10:06.377661 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:06.377572 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hfwtg" event={"ID":"5677dd72-2c2c-4ef0-877c-fd3cde06194e","Type":"ContainerStarted","Data":"5cc48e48050c8b1c5975b6be312f191a16e20b91fd046e77a2ccee435f65023c"} Sep 14 11:10:09.293114 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:10:09.293057 2574 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="9f1a90734c95e2e60272736b7eed3dc5b8bfa6ed68810cf1a4613c48b3ae95db" cmd=["/bin/bash","-c","test -f /ready/ready"] Sep 14 11:10:09.294187 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:10:09.294151 2574 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="9f1a90734c95e2e60272736b7eed3dc5b8bfa6ed68810cf1a4613c48b3ae95db" cmd=["/bin/bash","-c","test -f /ready/ready"] Sep 14 11:10:09.295094 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:10:09.295061 2574 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="9f1a90734c95e2e60272736b7eed3dc5b8bfa6ed68810cf1a4613c48b3ae95db" cmd=["/bin/bash","-c","test -f /ready/ready"] Sep 14 11:10:09.295183 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:10:09.295111 2574 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-multus/cni-sysctl-allowlist-ds-lvtpn" podUID="c53d6119-f15e-42a3-96e5-48c0c2caa38c" containerName="kube-multus-additional-cni-plugins" probeResult="unknown" Sep 14 11:10:11.414351 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:11.414246 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hfwtg" event={"ID":"5677dd72-2c2c-4ef0-877c-fd3cde06194e","Type":"ContainerStarted","Data":"a6e2df644ebf26a2c1b984e5973689a512061f766223a2b27fbc86829a6594e8"} Sep 14 11:10:11.414705 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:11.414396 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hfwtg" Sep 14 11:10:17.450986 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:17.450961 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hfwtg" Sep 14 11:10:17.480753 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:17.480704 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-hfwtg" podStartSLOduration=7.598465055 podStartE2EDuration="12.480688222s" totalImagesPullingTime="4.882223167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:10:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:10:11.438526329 +0000 UTC m=+556.058738875" watchObservedRunningTime="2026-09-14 11:10:17.480688222 +0000 UTC m=+562.100900736" Sep 14 11:10:19.293311 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:10:19.293250 2574 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="9f1a90734c95e2e60272736b7eed3dc5b8bfa6ed68810cf1a4613c48b3ae95db" cmd=["/bin/bash","-c","test -f /ready/ready"] Sep 14 11:10:19.294260 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:10:19.294229 2574 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="9f1a90734c95e2e60272736b7eed3dc5b8bfa6ed68810cf1a4613c48b3ae95db" cmd=["/bin/bash","-c","test -f /ready/ready"] Sep 14 11:10:19.295143 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:10:19.295116 2574 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="9f1a90734c95e2e60272736b7eed3dc5b8bfa6ed68810cf1a4613c48b3ae95db" cmd=["/bin/bash","-c","test -f /ready/ready"] Sep 14 11:10:19.295231 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:10:19.295153 2574 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-multus/cni-sysctl-allowlist-ds-lvtpn" podUID="c53d6119-f15e-42a3-96e5-48c0c2caa38c" containerName="kube-multus-additional-cni-plugins" probeResult="unknown" Sep 14 11:10:21.393305 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:21.393261 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_cni-sysctl-allowlist-ds-lvtpn_c53d6119-f15e-42a3-96e5-48c0c2caa38c/kube-multus-additional-cni-plugins/0.log" Sep 14 11:10:21.395731 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:21.395712 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-lvtpn" Sep 14 11:10:21.465732 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:21.465707 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_cni-sysctl-allowlist-ds-lvtpn_c53d6119-f15e-42a3-96e5-48c0c2caa38c/kube-multus-additional-cni-plugins/0.log" Sep 14 11:10:21.465904 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:21.465749 2574 generic.go:356] "Generic (PLEG): container finished" podID="c53d6119-f15e-42a3-96e5-48c0c2caa38c" containerID="9f1a90734c95e2e60272736b7eed3dc5b8bfa6ed68810cf1a4613c48b3ae95db" exitCode=137 Sep 14 11:10:21.465904 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:21.465803 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-lvtpn" Sep 14 11:10:21.465904 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:21.465839 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-lvtpn" event={"ID":"c53d6119-f15e-42a3-96e5-48c0c2caa38c","Type":"ContainerDied","Data":"9f1a90734c95e2e60272736b7eed3dc5b8bfa6ed68810cf1a4613c48b3ae95db"} Sep 14 11:10:21.465904 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:21.465897 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-lvtpn" event={"ID":"c53d6119-f15e-42a3-96e5-48c0c2caa38c","Type":"ContainerDied","Data":"3eb6a81528611767a49c377771b640eda169918bdd08d0b2b20a7da34cdd4f21"} Sep 14 11:10:21.466044 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:21.465917 2574 scope.go:117] "RemoveContainer" containerID="9f1a90734c95e2e60272736b7eed3dc5b8bfa6ed68810cf1a4613c48b3ae95db" Sep 14 11:10:21.475187 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:21.475167 2574 scope.go:117] "RemoveContainer" containerID="9f1a90734c95e2e60272736b7eed3dc5b8bfa6ed68810cf1a4613c48b3ae95db" Sep 14 11:10:21.475476 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:10:21.475457 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f1a90734c95e2e60272736b7eed3dc5b8bfa6ed68810cf1a4613c48b3ae95db\": container with ID starting with 9f1a90734c95e2e60272736b7eed3dc5b8bfa6ed68810cf1a4613c48b3ae95db not found: ID does not exist" containerID="9f1a90734c95e2e60272736b7eed3dc5b8bfa6ed68810cf1a4613c48b3ae95db" Sep 14 11:10:21.475519 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:21.475485 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f1a90734c95e2e60272736b7eed3dc5b8bfa6ed68810cf1a4613c48b3ae95db"} err="failed to get container status \"9f1a90734c95e2e60272736b7eed3dc5b8bfa6ed68810cf1a4613c48b3ae95db\": rpc error: code = NotFound desc = could not find container \"9f1a90734c95e2e60272736b7eed3dc5b8bfa6ed68810cf1a4613c48b3ae95db\": container with ID starting with 9f1a90734c95e2e60272736b7eed3dc5b8bfa6ed68810cf1a4613c48b3ae95db not found: ID does not exist" Sep 14 11:10:21.499805 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:21.499783 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c53d6119-f15e-42a3-96e5-48c0c2caa38c-cni-sysctl-allowlist\") pod \"c53d6119-f15e-42a3-96e5-48c0c2caa38c\" (UID: \"c53d6119-f15e-42a3-96e5-48c0c2caa38c\") " Sep 14 11:10:21.499877 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:21.499823 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnb82\" (UniqueName: \"kubernetes.io/projected/c53d6119-f15e-42a3-96e5-48c0c2caa38c-kube-api-access-mnb82\") pod \"c53d6119-f15e-42a3-96e5-48c0c2caa38c\" (UID: \"c53d6119-f15e-42a3-96e5-48c0c2caa38c\") " Sep 14 11:10:21.499877 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:21.499869 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/c53d6119-f15e-42a3-96e5-48c0c2caa38c-ready\") pod \"c53d6119-f15e-42a3-96e5-48c0c2caa38c\" (UID: \"c53d6119-f15e-42a3-96e5-48c0c2caa38c\") " Sep 14 11:10:21.499962 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:21.499896 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c53d6119-f15e-42a3-96e5-48c0c2caa38c-tuning-conf-dir\") pod \"c53d6119-f15e-42a3-96e5-48c0c2caa38c\" (UID: \"c53d6119-f15e-42a3-96e5-48c0c2caa38c\") " Sep 14 11:10:21.500072 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:21.500053 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/c53d6119-f15e-42a3-96e5-48c0c2caa38c-tuning-conf-dir" (OuterVolumeSpecName: "tuning-conf-dir") pod "c53d6119-f15e-42a3-96e5-48c0c2caa38c" (UID: "c53d6119-f15e-42a3-96e5-48c0c2caa38c"). InnerVolumeSpecName "tuning-conf-dir". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Sep 14 11:10:21.500137 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:21.500099 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c53d6119-f15e-42a3-96e5-48c0c2caa38c-cni-sysctl-allowlist" (OuterVolumeSpecName: "cni-sysctl-allowlist") pod "c53d6119-f15e-42a3-96e5-48c0c2caa38c" (UID: "c53d6119-f15e-42a3-96e5-48c0c2caa38c"). InnerVolumeSpecName "cni-sysctl-allowlist". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:10:21.500190 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:21.500152 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c53d6119-f15e-42a3-96e5-48c0c2caa38c-ready" (OuterVolumeSpecName: "ready") pod "c53d6119-f15e-42a3-96e5-48c0c2caa38c" (UID: "c53d6119-f15e-42a3-96e5-48c0c2caa38c"). InnerVolumeSpecName "ready". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:10:21.500190 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:21.500167 2574 reconciler_common.go:299] "Volume detached for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c53d6119-f15e-42a3-96e5-48c0c2caa38c-tuning-conf-dir\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:10:21.500190 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:21.500180 2574 reconciler_common.go:299] "Volume detached for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c53d6119-f15e-42a3-96e5-48c0c2caa38c-cni-sysctl-allowlist\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:10:21.502085 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:21.502062 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c53d6119-f15e-42a3-96e5-48c0c2caa38c-kube-api-access-mnb82" (OuterVolumeSpecName: "kube-api-access-mnb82") pod "c53d6119-f15e-42a3-96e5-48c0c2caa38c" (UID: "c53d6119-f15e-42a3-96e5-48c0c2caa38c"). InnerVolumeSpecName "kube-api-access-mnb82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:10:21.601514 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:21.601431 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnb82\" (UniqueName: \"kubernetes.io/projected/c53d6119-f15e-42a3-96e5-48c0c2caa38c-kube-api-access-mnb82\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:10:21.601514 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:21.601464 2574 reconciler_common.go:299] "Volume detached for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/c53d6119-f15e-42a3-96e5-48c0c2caa38c-ready\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:10:21.788999 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:21.788967 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-lvtpn"] Sep 14 11:10:21.790501 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:21.790483 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-lvtpn"] Sep 14 11:10:21.898738 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:21.898648 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c53d6119-f15e-42a3-96e5-48c0c2caa38c" path="/var/lib/kubelet/pods/c53d6119-f15e-42a3-96e5-48c0c2caa38c/volumes" Sep 14 11:10:24.846126 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:24.846097 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6c6f69b9fc-pwdqk"] Sep 14 11:10:24.846517 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:24.846492 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c53d6119-f15e-42a3-96e5-48c0c2caa38c" containerName="kube-multus-additional-cni-plugins" Sep 14 11:10:24.846517 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:24.846504 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c53d6119-f15e-42a3-96e5-48c0c2caa38c" containerName="kube-multus-additional-cni-plugins" Sep 14 11:10:24.846581 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:24.846567 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c53d6119-f15e-42a3-96e5-48c0c2caa38c" containerName="kube-multus-additional-cni-plugins" Sep 14 11:10:24.851205 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:24.851184 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c6f69b9fc-pwdqk" Sep 14 11:10:24.855329 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:24.855287 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 14 11:10:24.855581 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:24.855560 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 14 11:10:24.855686 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:24.855580 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-txcjg\"" Sep 14 11:10:24.857308 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:24.857291 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 14 11:10:24.865470 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:24.865447 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c6f69b9fc-pwdqk"] Sep 14 11:10:24.930200 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:24.930177 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0feb1a4b-cffe-4f4a-981b-ec171c84b972-metrics-certs\") pod \"maas-controller-6c6f69b9fc-pwdqk\" (UID: \"0feb1a4b-cffe-4f4a-981b-ec171c84b972\") " pod="opendatahub/maas-controller-6c6f69b9fc-pwdqk" Sep 14 11:10:24.930346 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:24.930223 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5ghj\" (UniqueName: \"kubernetes.io/projected/0feb1a4b-cffe-4f4a-981b-ec171c84b972-kube-api-access-l5ghj\") pod \"maas-controller-6c6f69b9fc-pwdqk\" (UID: \"0feb1a4b-cffe-4f4a-981b-ec171c84b972\") " pod="opendatahub/maas-controller-6c6f69b9fc-pwdqk" Sep 14 11:10:24.930346 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:24.930257 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0feb1a4b-cffe-4f4a-981b-ec171c84b972-webhook-cert\") pod \"maas-controller-6c6f69b9fc-pwdqk\" (UID: \"0feb1a4b-cffe-4f4a-981b-ec171c84b972\") " pod="opendatahub/maas-controller-6c6f69b9fc-pwdqk" Sep 14 11:10:25.030902 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:25.030875 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0feb1a4b-cffe-4f4a-981b-ec171c84b972-metrics-certs\") pod \"maas-controller-6c6f69b9fc-pwdqk\" (UID: \"0feb1a4b-cffe-4f4a-981b-ec171c84b972\") " pod="opendatahub/maas-controller-6c6f69b9fc-pwdqk" Sep 14 11:10:25.031015 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:25.030925 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5ghj\" (UniqueName: \"kubernetes.io/projected/0feb1a4b-cffe-4f4a-981b-ec171c84b972-kube-api-access-l5ghj\") pod \"maas-controller-6c6f69b9fc-pwdqk\" (UID: \"0feb1a4b-cffe-4f4a-981b-ec171c84b972\") " pod="opendatahub/maas-controller-6c6f69b9fc-pwdqk" Sep 14 11:10:25.031071 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:25.031046 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0feb1a4b-cffe-4f4a-981b-ec171c84b972-webhook-cert\") pod \"maas-controller-6c6f69b9fc-pwdqk\" (UID: \"0feb1a4b-cffe-4f4a-981b-ec171c84b972\") " pod="opendatahub/maas-controller-6c6f69b9fc-pwdqk" Sep 14 11:10:25.033466 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:25.033444 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0feb1a4b-cffe-4f4a-981b-ec171c84b972-metrics-certs\") pod \"maas-controller-6c6f69b9fc-pwdqk\" (UID: \"0feb1a4b-cffe-4f4a-981b-ec171c84b972\") " pod="opendatahub/maas-controller-6c6f69b9fc-pwdqk" Sep 14 11:10:25.033568 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:25.033550 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0feb1a4b-cffe-4f4a-981b-ec171c84b972-webhook-cert\") pod \"maas-controller-6c6f69b9fc-pwdqk\" (UID: \"0feb1a4b-cffe-4f4a-981b-ec171c84b972\") " pod="opendatahub/maas-controller-6c6f69b9fc-pwdqk" Sep 14 11:10:25.108991 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:25.108927 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5ghj\" (UniqueName: \"kubernetes.io/projected/0feb1a4b-cffe-4f4a-981b-ec171c84b972-kube-api-access-l5ghj\") pod \"maas-controller-6c6f69b9fc-pwdqk\" (UID: \"0feb1a4b-cffe-4f4a-981b-ec171c84b972\") " pod="opendatahub/maas-controller-6c6f69b9fc-pwdqk" Sep 14 11:10:25.122242 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:25.122218 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-796b4cc5fc-c4gnz"] Sep 14 11:10:25.125753 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:25.125733 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-796b4cc5fc-c4gnz" Sep 14 11:10:25.141608 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:25.141590 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-796b4cc5fc-c4gnz"] Sep 14 11:10:25.163026 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:25.163003 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c6f69b9fc-pwdqk" Sep 14 11:10:25.233248 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:25.233217 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v2mn\" (UniqueName: \"kubernetes.io/projected/4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc-kube-api-access-6v2mn\") pod \"maas-controller-796b4cc5fc-c4gnz\" (UID: \"4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc\") " pod="opendatahub/maas-controller-796b4cc5fc-c4gnz" Sep 14 11:10:25.233383 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:25.233348 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc-webhook-cert\") pod \"maas-controller-796b4cc5fc-c4gnz\" (UID: \"4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc\") " pod="opendatahub/maas-controller-796b4cc5fc-c4gnz" Sep 14 11:10:25.233780 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:25.233383 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc-metrics-certs\") pod \"maas-controller-796b4cc5fc-c4gnz\" (UID: \"4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc\") " pod="opendatahub/maas-controller-796b4cc5fc-c4gnz" Sep 14 11:10:25.334115 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:25.334079 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v2mn\" (UniqueName: \"kubernetes.io/projected/4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc-kube-api-access-6v2mn\") pod \"maas-controller-796b4cc5fc-c4gnz\" (UID: \"4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc\") " pod="opendatahub/maas-controller-796b4cc5fc-c4gnz" Sep 14 11:10:25.334332 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:25.334142 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc-webhook-cert\") pod \"maas-controller-796b4cc5fc-c4gnz\" (UID: \"4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc\") " pod="opendatahub/maas-controller-796b4cc5fc-c4gnz" Sep 14 11:10:25.334332 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:25.334159 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc-metrics-certs\") pod \"maas-controller-796b4cc5fc-c4gnz\" (UID: \"4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc\") " pod="opendatahub/maas-controller-796b4cc5fc-c4gnz" Sep 14 11:10:25.336891 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:25.336870 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc-webhook-cert\") pod \"maas-controller-796b4cc5fc-c4gnz\" (UID: \"4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc\") " pod="opendatahub/maas-controller-796b4cc5fc-c4gnz" Sep 14 11:10:25.336983 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:25.336870 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc-metrics-certs\") pod \"maas-controller-796b4cc5fc-c4gnz\" (UID: \"4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc\") " pod="opendatahub/maas-controller-796b4cc5fc-c4gnz" Sep 14 11:10:25.355933 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:25.355908 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v2mn\" (UniqueName: \"kubernetes.io/projected/4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc-kube-api-access-6v2mn\") pod \"maas-controller-796b4cc5fc-c4gnz\" (UID: \"4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc\") " pod="opendatahub/maas-controller-796b4cc5fc-c4gnz" Sep 14 11:10:25.435723 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:25.435698 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-796b4cc5fc-c4gnz" Sep 14 11:10:25.479201 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:25.479173 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6c6f69b9fc-pwdqk"] Sep 14 11:10:25.481031 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:10:25.481002 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0feb1a4b_cffe_4f4a_981b_ec171c84b972.slice/crio-575af0dcdce454bbfdd7da9354be038acada246cfe481f3d107ca4f48ee34ab5 WatchSource:0}: Error finding container 575af0dcdce454bbfdd7da9354be038acada246cfe481f3d107ca4f48ee34ab5: Status 404 returned error can't find the container with id 575af0dcdce454bbfdd7da9354be038acada246cfe481f3d107ca4f48ee34ab5 Sep 14 11:10:25.491839 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:25.491803 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c6f69b9fc-pwdqk" event={"ID":"0feb1a4b-cffe-4f4a-981b-ec171c84b972","Type":"ContainerStarted","Data":"575af0dcdce454bbfdd7da9354be038acada246cfe481f3d107ca4f48ee34ab5"} Sep 14 11:10:25.727851 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:25.727746 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-796b4cc5fc-c4gnz"] Sep 14 11:10:25.730156 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:10:25.730115 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4099bb96_fd23_4dc5_a9b7_fb37cd82c5dc.slice/crio-198a0fb93c42e3133295b360f32928919fc3197e6244b79708a9656cb2d16f96 WatchSource:0}: Error finding container 198a0fb93c42e3133295b360f32928919fc3197e6244b79708a9656cb2d16f96: Status 404 returned error can't find the container with id 198a0fb93c42e3133295b360f32928919fc3197e6244b79708a9656cb2d16f96 Sep 14 11:10:26.500680 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:26.500640 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-796b4cc5fc-c4gnz" event={"ID":"4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc","Type":"ContainerStarted","Data":"198a0fb93c42e3133295b360f32928919fc3197e6244b79708a9656cb2d16f96"} Sep 14 11:10:28.514502 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:28.514465 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c6f69b9fc-pwdqk" event={"ID":"0feb1a4b-cffe-4f4a-981b-ec171c84b972","Type":"ContainerStarted","Data":"27d4a3e4f44235c543e8dbb3c7eb7317f58565d046751bcf958b4941dfcbaedd"} Sep 14 11:10:28.514924 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:28.514573 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6c6f69b9fc-pwdqk" Sep 14 11:10:28.516008 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:28.515980 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-796b4cc5fc-c4gnz" event={"ID":"4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc","Type":"ContainerStarted","Data":"91b2bbe1a0d85e1ddb20b2dc3b9f6f5ec9077b5848ae581ee2f024e8098f9dcb"} Sep 14 11:10:28.516136 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:28.516122 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-796b4cc5fc-c4gnz" Sep 14 11:10:28.630667 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:28.630571 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6c6f69b9fc-pwdqk" podStartSLOduration=1.777863407 podStartE2EDuration="4.630554203s" totalImagesPullingTime="2.852690796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:10:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:10:28.629092227 +0000 UTC m=+573.249304776" watchObservedRunningTime="2026-09-14 11:10:28.630554203 +0000 UTC m=+573.250766725" Sep 14 11:10:28.687895 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:28.687844 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-796b4cc5fc-c4gnz" podStartSLOduration=1.09463915 podStartE2EDuration="3.687828637s" totalImagesPullingTime="2.593189487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:10:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:10:28.68529591 +0000 UTC m=+573.305508441" watchObservedRunningTime="2026-09-14 11:10:28.687828637 +0000 UTC m=+573.308041160" Sep 14 11:10:39.522950 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:39.522921 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6c6f69b9fc-pwdqk" Sep 14 11:10:39.523378 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:39.522991 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-796b4cc5fc-c4gnz" Sep 14 11:10:39.645386 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:39.645349 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6c6f69b9fc-pwdqk"] Sep 14 11:10:39.645607 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:39.645578 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6c6f69b9fc-pwdqk" podUID="0feb1a4b-cffe-4f4a-981b-ec171c84b972" containerName="manager" containerID="cri-o://27d4a3e4f44235c543e8dbb3c7eb7317f58565d046751bcf958b4941dfcbaedd" gracePeriod=30 Sep 14 11:10:39.899340 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:39.899314 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c6f69b9fc-pwdqk" Sep 14 11:10:40.079547 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:40.079453 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5ghj\" (UniqueName: \"kubernetes.io/projected/0feb1a4b-cffe-4f4a-981b-ec171c84b972-kube-api-access-l5ghj\") pod \"0feb1a4b-cffe-4f4a-981b-ec171c84b972\" (UID: \"0feb1a4b-cffe-4f4a-981b-ec171c84b972\") " Sep 14 11:10:40.079547 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:40.079504 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0feb1a4b-cffe-4f4a-981b-ec171c84b972-webhook-cert\") pod \"0feb1a4b-cffe-4f4a-981b-ec171c84b972\" (UID: \"0feb1a4b-cffe-4f4a-981b-ec171c84b972\") " Sep 14 11:10:40.079779 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:40.079579 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0feb1a4b-cffe-4f4a-981b-ec171c84b972-metrics-certs\") pod \"0feb1a4b-cffe-4f4a-981b-ec171c84b972\" (UID: \"0feb1a4b-cffe-4f4a-981b-ec171c84b972\") " Sep 14 11:10:40.081959 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:40.081931 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0feb1a4b-cffe-4f4a-981b-ec171c84b972-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0feb1a4b-cffe-4f4a-981b-ec171c84b972" (UID: "0feb1a4b-cffe-4f4a-981b-ec171c84b972"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:10:40.081959 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:40.081947 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0feb1a4b-cffe-4f4a-981b-ec171c84b972-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "0feb1a4b-cffe-4f4a-981b-ec171c84b972" (UID: "0feb1a4b-cffe-4f4a-981b-ec171c84b972"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:10:40.082102 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:40.082089 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0feb1a4b-cffe-4f4a-981b-ec171c84b972-kube-api-access-l5ghj" (OuterVolumeSpecName: "kube-api-access-l5ghj") pod "0feb1a4b-cffe-4f4a-981b-ec171c84b972" (UID: "0feb1a4b-cffe-4f4a-981b-ec171c84b972"). InnerVolumeSpecName "kube-api-access-l5ghj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:10:40.180663 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:40.180629 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0feb1a4b-cffe-4f4a-981b-ec171c84b972-metrics-certs\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:10:40.180663 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:40.180658 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5ghj\" (UniqueName: \"kubernetes.io/projected/0feb1a4b-cffe-4f4a-981b-ec171c84b972-kube-api-access-l5ghj\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:10:40.180663 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:40.180668 2574 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0feb1a4b-cffe-4f4a-981b-ec171c84b972-webhook-cert\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:10:40.581669 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:40.581633 2574 generic.go:356] "Generic (PLEG): container finished" podID="0feb1a4b-cffe-4f4a-981b-ec171c84b972" containerID="27d4a3e4f44235c543e8dbb3c7eb7317f58565d046751bcf958b4941dfcbaedd" exitCode=0 Sep 14 11:10:40.582060 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:40.581689 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c6f69b9fc-pwdqk" event={"ID":"0feb1a4b-cffe-4f4a-981b-ec171c84b972","Type":"ContainerDied","Data":"27d4a3e4f44235c543e8dbb3c7eb7317f58565d046751bcf958b4941dfcbaedd"} Sep 14 11:10:40.582060 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:40.581752 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6c6f69b9fc-pwdqk" event={"ID":"0feb1a4b-cffe-4f4a-981b-ec171c84b972","Type":"ContainerDied","Data":"575af0dcdce454bbfdd7da9354be038acada246cfe481f3d107ca4f48ee34ab5"} Sep 14 11:10:40.582060 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:40.581689 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6c6f69b9fc-pwdqk" Sep 14 11:10:40.582060 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:40.581773 2574 scope.go:117] "RemoveContainer" containerID="27d4a3e4f44235c543e8dbb3c7eb7317f58565d046751bcf958b4941dfcbaedd" Sep 14 11:10:40.591094 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:40.591074 2574 scope.go:117] "RemoveContainer" containerID="27d4a3e4f44235c543e8dbb3c7eb7317f58565d046751bcf958b4941dfcbaedd" Sep 14 11:10:40.591401 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:10:40.591381 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27d4a3e4f44235c543e8dbb3c7eb7317f58565d046751bcf958b4941dfcbaedd\": container with ID starting with 27d4a3e4f44235c543e8dbb3c7eb7317f58565d046751bcf958b4941dfcbaedd not found: ID does not exist" containerID="27d4a3e4f44235c543e8dbb3c7eb7317f58565d046751bcf958b4941dfcbaedd" Sep 14 11:10:40.591459 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:40.591410 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27d4a3e4f44235c543e8dbb3c7eb7317f58565d046751bcf958b4941dfcbaedd"} err="failed to get container status \"27d4a3e4f44235c543e8dbb3c7eb7317f58565d046751bcf958b4941dfcbaedd\": rpc error: code = NotFound desc = could not find container \"27d4a3e4f44235c543e8dbb3c7eb7317f58565d046751bcf958b4941dfcbaedd\": container with ID starting with 27d4a3e4f44235c543e8dbb3c7eb7317f58565d046751bcf958b4941dfcbaedd not found: ID does not exist" Sep 14 11:10:40.604729 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:40.604702 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6c6f69b9fc-pwdqk"] Sep 14 11:10:40.611308 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:40.611287 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6c6f69b9fc-pwdqk"] Sep 14 11:10:41.898913 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:10:41.898869 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0feb1a4b-cffe-4f4a-981b-ec171c84b972" path="/var/lib/kubelet/pods/0feb1a4b-cffe-4f4a-981b-ec171c84b972/volumes" Sep 14 11:11:03.184221 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:03.184183 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-8f99c9866-4pd64"] Sep 14 11:11:03.189805 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:03.184787 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0feb1a4b-cffe-4f4a-981b-ec171c84b972" containerName="manager" Sep 14 11:11:03.189805 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:03.184805 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0feb1a4b-cffe-4f4a-981b-ec171c84b972" containerName="manager" Sep 14 11:11:03.189805 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:03.184902 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0feb1a4b-cffe-4f4a-981b-ec171c84b972" containerName="manager" Sep 14 11:11:03.189939 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:03.189924 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8f99c9866-4pd64" Sep 14 11:11:03.204442 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:03.204420 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 14 11:11:03.204554 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:03.204421 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 14 11:11:03.204926 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:03.204912 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-w7q69\"" Sep 14 11:11:03.213911 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:03.213884 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8f99c9866-4pd64"] Sep 14 11:11:03.293031 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:03.292986 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be24615b-72c5-4376-a5b3-9f51d6df14d4-metrics-certs\") pod \"maas-api-8f99c9866-4pd64\" (UID: \"be24615b-72c5-4376-a5b3-9f51d6df14d4\") " pod="odh-ai-gateway-infra/maas-api-8f99c9866-4pd64" Sep 14 11:11:03.293230 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:03.293097 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnp6s\" (UniqueName: \"kubernetes.io/projected/be24615b-72c5-4376-a5b3-9f51d6df14d4-kube-api-access-cnp6s\") pod \"maas-api-8f99c9866-4pd64\" (UID: \"be24615b-72c5-4376-a5b3-9f51d6df14d4\") " pod="odh-ai-gateway-infra/maas-api-8f99c9866-4pd64" Sep 14 11:11:03.293230 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:03.293165 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be24615b-72c5-4376-a5b3-9f51d6df14d4-maas-api-tls\") pod \"maas-api-8f99c9866-4pd64\" (UID: \"be24615b-72c5-4376-a5b3-9f51d6df14d4\") " pod="odh-ai-gateway-infra/maas-api-8f99c9866-4pd64" Sep 14 11:11:03.293230 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:03.293214 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be24615b-72c5-4376-a5b3-9f51d6df14d4-openshift-service-ca\") pod \"maas-api-8f99c9866-4pd64\" (UID: \"be24615b-72c5-4376-a5b3-9f51d6df14d4\") " pod="odh-ai-gateway-infra/maas-api-8f99c9866-4pd64" Sep 14 11:11:03.394648 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:03.394607 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be24615b-72c5-4376-a5b3-9f51d6df14d4-maas-api-tls\") pod \"maas-api-8f99c9866-4pd64\" (UID: \"be24615b-72c5-4376-a5b3-9f51d6df14d4\") " pod="odh-ai-gateway-infra/maas-api-8f99c9866-4pd64" Sep 14 11:11:03.394648 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:03.394653 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be24615b-72c5-4376-a5b3-9f51d6df14d4-openshift-service-ca\") pod \"maas-api-8f99c9866-4pd64\" (UID: \"be24615b-72c5-4376-a5b3-9f51d6df14d4\") " pod="odh-ai-gateway-infra/maas-api-8f99c9866-4pd64" Sep 14 11:11:03.394927 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:03.394731 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be24615b-72c5-4376-a5b3-9f51d6df14d4-metrics-certs\") pod \"maas-api-8f99c9866-4pd64\" (UID: \"be24615b-72c5-4376-a5b3-9f51d6df14d4\") " pod="odh-ai-gateway-infra/maas-api-8f99c9866-4pd64" Sep 14 11:11:03.394927 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:03.394765 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnp6s\" (UniqueName: \"kubernetes.io/projected/be24615b-72c5-4376-a5b3-9f51d6df14d4-kube-api-access-cnp6s\") pod \"maas-api-8f99c9866-4pd64\" (UID: \"be24615b-72c5-4376-a5b3-9f51d6df14d4\") " pod="odh-ai-gateway-infra/maas-api-8f99c9866-4pd64" Sep 14 11:11:03.395384 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:03.395364 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be24615b-72c5-4376-a5b3-9f51d6df14d4-openshift-service-ca\") pod \"maas-api-8f99c9866-4pd64\" (UID: \"be24615b-72c5-4376-a5b3-9f51d6df14d4\") " pod="odh-ai-gateway-infra/maas-api-8f99c9866-4pd64" Sep 14 11:11:03.397255 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:03.397234 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be24615b-72c5-4376-a5b3-9f51d6df14d4-metrics-certs\") pod \"maas-api-8f99c9866-4pd64\" (UID: \"be24615b-72c5-4376-a5b3-9f51d6df14d4\") " pod="odh-ai-gateway-infra/maas-api-8f99c9866-4pd64" Sep 14 11:11:03.397380 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:03.397362 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be24615b-72c5-4376-a5b3-9f51d6df14d4-maas-api-tls\") pod \"maas-api-8f99c9866-4pd64\" (UID: \"be24615b-72c5-4376-a5b3-9f51d6df14d4\") " pod="odh-ai-gateway-infra/maas-api-8f99c9866-4pd64" Sep 14 11:11:03.423654 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:03.423625 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnp6s\" (UniqueName: \"kubernetes.io/projected/be24615b-72c5-4376-a5b3-9f51d6df14d4-kube-api-access-cnp6s\") pod \"maas-api-8f99c9866-4pd64\" (UID: \"be24615b-72c5-4376-a5b3-9f51d6df14d4\") " pod="odh-ai-gateway-infra/maas-api-8f99c9866-4pd64" Sep 14 11:11:03.500990 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:03.500883 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8f99c9866-4pd64" Sep 14 11:11:03.784946 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:03.784856 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8f99c9866-4pd64"] Sep 14 11:11:03.787688 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:11:03.787662 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe24615b_72c5_4376_a5b3_9f51d6df14d4.slice/crio-b255199da343040d0d676fbd8bc53197564ea0ec509c7ce8ee8e302c90c219a1 WatchSource:0}: Error finding container b255199da343040d0d676fbd8bc53197564ea0ec509c7ce8ee8e302c90c219a1: Status 404 returned error can't find the container with id b255199da343040d0d676fbd8bc53197564ea0ec509c7ce8ee8e302c90c219a1 Sep 14 11:11:03.789126 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:03.789108 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 11:11:04.204967 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:04.204931 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-p5srl"] Sep 14 11:11:04.209489 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:04.209471 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-p5srl" Sep 14 11:11:04.220098 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:04.219609 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-nknzl\"" Sep 14 11:11:04.220098 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:04.219983 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 14 11:11:04.232256 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:04.232228 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-p5srl"] Sep 14 11:11:04.270530 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:04.270499 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-mvnvh"] Sep 14 11:11:04.274397 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:04.274377 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-mvnvh" Sep 14 11:11:04.295576 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:04.295552 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-mvnvh"] Sep 14 11:11:04.302453 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:04.302427 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6ecef0e-9b32-41ab-8452-618836054a5a-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-p5srl\" (UID: \"a6ecef0e-9b32-41ab-8452-618836054a5a\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-p5srl" Sep 14 11:11:04.302585 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:04.302545 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhnkr\" (UniqueName: \"kubernetes.io/projected/a6ecef0e-9b32-41ab-8452-618836054a5a-kube-api-access-qhnkr\") pod \"payload-pre-processing-59fbc47bf8-p5srl\" (UID: \"a6ecef0e-9b32-41ab-8452-618836054a5a\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-p5srl" Sep 14 11:11:04.403690 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:04.403642 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4eea439-1943-4f74-9067-d1bb67a438f1-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-mvnvh\" (UID: \"d4eea439-1943-4f74-9067-d1bb67a438f1\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-mvnvh" Sep 14 11:11:04.403862 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:04.403701 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhnkr\" (UniqueName: \"kubernetes.io/projected/a6ecef0e-9b32-41ab-8452-618836054a5a-kube-api-access-qhnkr\") pod \"payload-pre-processing-59fbc47bf8-p5srl\" (UID: \"a6ecef0e-9b32-41ab-8452-618836054a5a\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-p5srl" Sep 14 11:11:04.403862 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:04.403782 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6ecef0e-9b32-41ab-8452-618836054a5a-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-p5srl\" (UID: \"a6ecef0e-9b32-41ab-8452-618836054a5a\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-p5srl" Sep 14 11:11:04.403862 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:04.403803 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grlr4\" (UniqueName: \"kubernetes.io/projected/d4eea439-1943-4f74-9067-d1bb67a438f1-kube-api-access-grlr4\") pod \"payload-processing-5b54b7fbb7-mvnvh\" (UID: \"d4eea439-1943-4f74-9067-d1bb67a438f1\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-mvnvh" Sep 14 11:11:04.404481 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:04.404462 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6ecef0e-9b32-41ab-8452-618836054a5a-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-p5srl\" (UID: \"a6ecef0e-9b32-41ab-8452-618836054a5a\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-p5srl" Sep 14 11:11:04.422235 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:04.422206 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhnkr\" (UniqueName: \"kubernetes.io/projected/a6ecef0e-9b32-41ab-8452-618836054a5a-kube-api-access-qhnkr\") pod \"payload-pre-processing-59fbc47bf8-p5srl\" (UID: \"a6ecef0e-9b32-41ab-8452-618836054a5a\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-p5srl" Sep 14 11:11:04.505174 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:04.505085 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4eea439-1943-4f74-9067-d1bb67a438f1-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-mvnvh\" (UID: \"d4eea439-1943-4f74-9067-d1bb67a438f1\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-mvnvh" Sep 14 11:11:04.505174 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:04.505160 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grlr4\" (UniqueName: \"kubernetes.io/projected/d4eea439-1943-4f74-9067-d1bb67a438f1-kube-api-access-grlr4\") pod \"payload-processing-5b54b7fbb7-mvnvh\" (UID: \"d4eea439-1943-4f74-9067-d1bb67a438f1\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-mvnvh" Sep 14 11:11:04.506122 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:04.506094 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4eea439-1943-4f74-9067-d1bb67a438f1-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-mvnvh\" (UID: \"d4eea439-1943-4f74-9067-d1bb67a438f1\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-mvnvh" Sep 14 11:11:04.514708 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:04.514681 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grlr4\" (UniqueName: \"kubernetes.io/projected/d4eea439-1943-4f74-9067-d1bb67a438f1-kube-api-access-grlr4\") pod \"payload-processing-5b54b7fbb7-mvnvh\" (UID: \"d4eea439-1943-4f74-9067-d1bb67a438f1\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-mvnvh" Sep 14 11:11:04.520569 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:04.520551 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-p5srl" Sep 14 11:11:04.584674 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:04.584639 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-mvnvh" Sep 14 11:11:04.700180 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:04.700146 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8f99c9866-4pd64" event={"ID":"be24615b-72c5-4376-a5b3-9f51d6df14d4","Type":"ContainerStarted","Data":"b255199da343040d0d676fbd8bc53197564ea0ec509c7ce8ee8e302c90c219a1"} Sep 14 11:11:04.809406 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:04.809367 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-p5srl"] Sep 14 11:11:04.811246 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:11:04.811224 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6ecef0e_9b32_41ab_8452_618836054a5a.slice/crio-2b75cf15097991d491825006760a72274db38fce6cfc23f7151e0ce5c93ef324 WatchSource:0}: Error finding container 2b75cf15097991d491825006760a72274db38fce6cfc23f7151e0ce5c93ef324: Status 404 returned error can't find the container with id 2b75cf15097991d491825006760a72274db38fce6cfc23f7151e0ce5c93ef324 Sep 14 11:11:04.887749 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:04.887726 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-mvnvh"] Sep 14 11:11:04.889636 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:11:04.889610 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4eea439_1943_4f74_9067_d1bb67a438f1.slice/crio-e53f14ea97cd31494187675b42fd0659546c4244eee43f55e692e2cbd736ef42 WatchSource:0}: Error finding container e53f14ea97cd31494187675b42fd0659546c4244eee43f55e692e2cbd736ef42: Status 404 returned error can't find the container with id e53f14ea97cd31494187675b42fd0659546c4244eee43f55e692e2cbd736ef42 Sep 14 11:11:05.711949 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:05.711911 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-p5srl" event={"ID":"a6ecef0e-9b32-41ab-8452-618836054a5a","Type":"ContainerStarted","Data":"2b75cf15097991d491825006760a72274db38fce6cfc23f7151e0ce5c93ef324"} Sep 14 11:11:05.713681 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:05.713651 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-mvnvh" event={"ID":"d4eea439-1943-4f74-9067-d1bb67a438f1","Type":"ContainerStarted","Data":"e53f14ea97cd31494187675b42fd0659546c4244eee43f55e692e2cbd736ef42"} Sep 14 11:11:08.740707 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:08.740670 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8f99c9866-4pd64" event={"ID":"be24615b-72c5-4376-a5b3-9f51d6df14d4","Type":"ContainerStarted","Data":"7052c65d76d01ce53900f65f7d0382a2306e1b6f6d332e21b85733ceef97d6a8"} Sep 14 11:11:08.741212 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:08.740792 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-8f99c9866-4pd64" Sep 14 11:11:08.742520 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:08.742496 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-p5srl" event={"ID":"a6ecef0e-9b32-41ab-8452-618836054a5a","Type":"ContainerStarted","Data":"832adebcaab81e535149cbac234553b4b3aa4158772804d9764a86ed00e13d91"} Sep 14 11:11:08.742632 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:08.742582 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-p5srl" Sep 14 11:11:08.744092 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:08.744068 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-mvnvh" event={"ID":"d4eea439-1943-4f74-9067-d1bb67a438f1","Type":"ContainerStarted","Data":"42a6b72ac9c32f868e40c827d4e7d527ca25ac7421d63bee5e3f31fc7293f7c1"} Sep 14 11:11:08.744219 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:08.744205 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-mvnvh" Sep 14 11:11:08.761998 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:08.761955 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-8f99c9866-4pd64" podStartSLOduration=1.514937924 podStartE2EDuration="5.76193956s" totalImagesPullingTime="4.247001636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:11:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:11:08.760180491 +0000 UTC m=+613.380393052" watchObservedRunningTime="2026-09-14 11:11:08.76193956 +0000 UTC m=+613.382152084" Sep 14 11:11:08.784498 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:08.784422 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-p5srl" podStartSLOduration=1.554986851 podStartE2EDuration="4.784406062s" totalImagesPullingTime="3.229419211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:11:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:11:08.782129575 +0000 UTC m=+613.402342100" watchObservedRunningTime="2026-09-14 11:11:08.784406062 +0000 UTC m=+613.404618585" Sep 14 11:11:08.809449 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:08.809398 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-mvnvh" podStartSLOduration=1.664936753 podStartE2EDuration="4.809378383s" totalImagesPullingTime="3.14444163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:11:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:11:08.80695161 +0000 UTC m=+613.427164158" watchObservedRunningTime="2026-09-14 11:11:08.809378383 +0000 UTC m=+613.429590908" Sep 14 11:11:14.755759 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:14.755729 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-8f99c9866-4pd64" Sep 14 11:11:19.752216 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:19.752134 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-p5srl" Sep 14 11:11:19.752625 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:19.752223 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-mvnvh" Sep 14 11:11:45.910903 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:45.910871 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp"] Sep 14 11:11:45.917245 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:45.917225 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp" Sep 14 11:11:45.923102 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:45.922955 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6r78c\"" Sep 14 11:11:45.923102 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:45.923007 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 14 11:11:45.923102 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:45.923062 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 14 11:11:45.923572 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:45.923412 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 14 11:11:45.935221 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:45.935203 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp"] Sep 14 11:11:45.984405 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:45.984383 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e728a6b5-f5f2-4c35-bbee-2faa09e4059f-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp\" (UID: \"e728a6b5-f5f2-4c35-bbee-2faa09e4059f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp" Sep 14 11:11:45.984535 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:45.984425 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e728a6b5-f5f2-4c35-bbee-2faa09e4059f-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp\" (UID: \"e728a6b5-f5f2-4c35-bbee-2faa09e4059f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp" Sep 14 11:11:45.984535 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:45.984507 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e728a6b5-f5f2-4c35-bbee-2faa09e4059f-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp\" (UID: \"e728a6b5-f5f2-4c35-bbee-2faa09e4059f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp" Sep 14 11:11:45.984652 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:45.984587 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e728a6b5-f5f2-4c35-bbee-2faa09e4059f-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp\" (UID: \"e728a6b5-f5f2-4c35-bbee-2faa09e4059f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp" Sep 14 11:11:45.984652 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:45.984619 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js7rg\" (UniqueName: \"kubernetes.io/projected/e728a6b5-f5f2-4c35-bbee-2faa09e4059f-kube-api-access-js7rg\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp\" (UID: \"e728a6b5-f5f2-4c35-bbee-2faa09e4059f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp" Sep 14 11:11:45.984751 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:45.984668 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e728a6b5-f5f2-4c35-bbee-2faa09e4059f-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp\" (UID: \"e728a6b5-f5f2-4c35-bbee-2faa09e4059f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp" Sep 14 11:11:46.085753 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:46.085725 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e728a6b5-f5f2-4c35-bbee-2faa09e4059f-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp\" (UID: \"e728a6b5-f5f2-4c35-bbee-2faa09e4059f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp" Sep 14 11:11:46.085867 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:46.085767 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e728a6b5-f5f2-4c35-bbee-2faa09e4059f-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp\" (UID: \"e728a6b5-f5f2-4c35-bbee-2faa09e4059f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp" Sep 14 11:11:46.085867 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:46.085784 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js7rg\" (UniqueName: \"kubernetes.io/projected/e728a6b5-f5f2-4c35-bbee-2faa09e4059f-kube-api-access-js7rg\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp\" (UID: \"e728a6b5-f5f2-4c35-bbee-2faa09e4059f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp" Sep 14 11:11:46.085983 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:46.085917 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e728a6b5-f5f2-4c35-bbee-2faa09e4059f-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp\" (UID: \"e728a6b5-f5f2-4c35-bbee-2faa09e4059f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp" Sep 14 11:11:46.086034 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:46.085990 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e728a6b5-f5f2-4c35-bbee-2faa09e4059f-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp\" (UID: \"e728a6b5-f5f2-4c35-bbee-2faa09e4059f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp" Sep 14 11:11:46.086086 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:46.086047 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e728a6b5-f5f2-4c35-bbee-2faa09e4059f-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp\" (UID: \"e728a6b5-f5f2-4c35-bbee-2faa09e4059f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp" Sep 14 11:11:46.086135 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:46.086077 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e728a6b5-f5f2-4c35-bbee-2faa09e4059f-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp\" (UID: \"e728a6b5-f5f2-4c35-bbee-2faa09e4059f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp" Sep 14 11:11:46.086256 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:46.086235 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e728a6b5-f5f2-4c35-bbee-2faa09e4059f-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp\" (UID: \"e728a6b5-f5f2-4c35-bbee-2faa09e4059f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp" Sep 14 11:11:46.086341 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:46.086324 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e728a6b5-f5f2-4c35-bbee-2faa09e4059f-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp\" (UID: \"e728a6b5-f5f2-4c35-bbee-2faa09e4059f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp" Sep 14 11:11:46.088136 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:46.088116 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e728a6b5-f5f2-4c35-bbee-2faa09e4059f-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp\" (UID: \"e728a6b5-f5f2-4c35-bbee-2faa09e4059f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp" Sep 14 11:11:46.088571 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:46.088550 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e728a6b5-f5f2-4c35-bbee-2faa09e4059f-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp\" (UID: \"e728a6b5-f5f2-4c35-bbee-2faa09e4059f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp" Sep 14 11:11:46.094537 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:46.094508 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-js7rg\" (UniqueName: \"kubernetes.io/projected/e728a6b5-f5f2-4c35-bbee-2faa09e4059f-kube-api-access-js7rg\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp\" (UID: \"e728a6b5-f5f2-4c35-bbee-2faa09e4059f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp" Sep 14 11:11:46.227955 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:46.227929 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp" Sep 14 11:11:46.510723 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:46.510694 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp"] Sep 14 11:11:46.512356 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:11:46.512323 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode728a6b5_f5f2_4c35_bbee_2faa09e4059f.slice/crio-a5304f131beb83b170a4446d91863c39d0cafb8be37a922dcd182c17d15ee686 WatchSource:0}: Error finding container a5304f131beb83b170a4446d91863c39d0cafb8be37a922dcd182c17d15ee686: Status 404 returned error can't find the container with id a5304f131beb83b170a4446d91863c39d0cafb8be37a922dcd182c17d15ee686 Sep 14 11:11:46.934136 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:46.934099 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp" event={"ID":"e728a6b5-f5f2-4c35-bbee-2faa09e4059f","Type":"ContainerStarted","Data":"a5304f131beb83b170a4446d91863c39d0cafb8be37a922dcd182c17d15ee686"} Sep 14 11:11:47.939815 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:47.939781 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp" event={"ID":"e728a6b5-f5f2-4c35-bbee-2faa09e4059f","Type":"ContainerStarted","Data":"dc6f230e4f78f71b812de0232863b8239f077fe404d7e44ea5cf0ebded8d5681"} Sep 14 11:11:47.961646 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:47.961595 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp" podStartSLOduration=1.970761543 podStartE2EDuration="2.961578339s" totalImagesPullingTime="990.816796ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:11:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:11:47.960006302 +0000 UTC m=+652.580218838" watchObservedRunningTime="2026-09-14 11:11:47.961578339 +0000 UTC m=+652.581790863" Sep 14 11:11:56.228198 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:56.228160 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp" Sep 14 11:11:56.228605 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:56.228224 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp" Sep 14 11:11:56.241059 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:56.241034 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp" Sep 14 11:11:56.995599 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:11:56.995563 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp" Sep 14 11:12:43.755349 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:12:43.755316 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-m49mq"] Sep 14 11:12:43.759675 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:12:43.759647 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-m49mq" Sep 14 11:12:43.762199 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:12:43.762175 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 14 11:12:43.771996 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:12:43.771974 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-m49mq"] Sep 14 11:12:43.876253 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:12:43.876218 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/518192ea-4063-4e8c-a600-6a85c83c2dcb-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-m49mq\" (UID: \"518192ea-4063-4e8c-a600-6a85c83c2dcb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-m49mq" Sep 14 11:12:43.876426 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:12:43.876257 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rml2k\" (UniqueName: \"kubernetes.io/projected/518192ea-4063-4e8c-a600-6a85c83c2dcb-kube-api-access-rml2k\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-m49mq\" (UID: \"518192ea-4063-4e8c-a600-6a85c83c2dcb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-m49mq" Sep 14 11:12:43.876426 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:12:43.876337 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/518192ea-4063-4e8c-a600-6a85c83c2dcb-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-m49mq\" (UID: \"518192ea-4063-4e8c-a600-6a85c83c2dcb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-m49mq" Sep 14 11:12:43.876426 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:12:43.876383 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/518192ea-4063-4e8c-a600-6a85c83c2dcb-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-m49mq\" (UID: \"518192ea-4063-4e8c-a600-6a85c83c2dcb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-m49mq" Sep 14 11:12:43.876544 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:12:43.876499 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/518192ea-4063-4e8c-a600-6a85c83c2dcb-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-m49mq\" (UID: \"518192ea-4063-4e8c-a600-6a85c83c2dcb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-m49mq" Sep 14 11:12:43.876544 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:12:43.876528 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/518192ea-4063-4e8c-a600-6a85c83c2dcb-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-m49mq\" (UID: \"518192ea-4063-4e8c-a600-6a85c83c2dcb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-m49mq" Sep 14 11:12:43.977717 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:12:43.977671 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/518192ea-4063-4e8c-a600-6a85c83c2dcb-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-m49mq\" (UID: \"518192ea-4063-4e8c-a600-6a85c83c2dcb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-m49mq" Sep 14 11:12:43.977889 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:12:43.977712 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/518192ea-4063-4e8c-a600-6a85c83c2dcb-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-m49mq\" (UID: \"518192ea-4063-4e8c-a600-6a85c83c2dcb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-m49mq" Sep 14 11:12:43.977889 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:12:43.977827 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/518192ea-4063-4e8c-a600-6a85c83c2dcb-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-m49mq\" (UID: \"518192ea-4063-4e8c-a600-6a85c83c2dcb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-m49mq" Sep 14 11:12:43.977889 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:12:43.977857 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rml2k\" (UniqueName: \"kubernetes.io/projected/518192ea-4063-4e8c-a600-6a85c83c2dcb-kube-api-access-rml2k\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-m49mq\" (UID: \"518192ea-4063-4e8c-a600-6a85c83c2dcb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-m49mq" Sep 14 11:12:43.978027 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:12:43.977899 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/518192ea-4063-4e8c-a600-6a85c83c2dcb-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-m49mq\" (UID: \"518192ea-4063-4e8c-a600-6a85c83c2dcb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-m49mq" Sep 14 11:12:43.978027 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:12:43.977956 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/518192ea-4063-4e8c-a600-6a85c83c2dcb-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-m49mq\" (UID: \"518192ea-4063-4e8c-a600-6a85c83c2dcb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-m49mq" Sep 14 11:12:43.978134 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:12:43.978098 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/518192ea-4063-4e8c-a600-6a85c83c2dcb-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-m49mq\" (UID: \"518192ea-4063-4e8c-a600-6a85c83c2dcb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-m49mq" Sep 14 11:12:43.978212 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:12:43.978130 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/518192ea-4063-4e8c-a600-6a85c83c2dcb-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-m49mq\" (UID: \"518192ea-4063-4e8c-a600-6a85c83c2dcb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-m49mq" Sep 14 11:12:43.978407 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:12:43.978380 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/518192ea-4063-4e8c-a600-6a85c83c2dcb-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-m49mq\" (UID: \"518192ea-4063-4e8c-a600-6a85c83c2dcb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-m49mq" Sep 14 11:12:43.980329 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:12:43.980307 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/518192ea-4063-4e8c-a600-6a85c83c2dcb-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-m49mq\" (UID: \"518192ea-4063-4e8c-a600-6a85c83c2dcb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-m49mq" Sep 14 11:12:43.980726 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:12:43.980709 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/518192ea-4063-4e8c-a600-6a85c83c2dcb-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-m49mq\" (UID: \"518192ea-4063-4e8c-a600-6a85c83c2dcb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-m49mq" Sep 14 11:12:43.992598 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:12:43.992578 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rml2k\" (UniqueName: \"kubernetes.io/projected/518192ea-4063-4e8c-a600-6a85c83c2dcb-kube-api-access-rml2k\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-m49mq\" (UID: \"518192ea-4063-4e8c-a600-6a85c83c2dcb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-m49mq" Sep 14 11:12:44.072898 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:12:44.072816 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-m49mq" Sep 14 11:12:44.349508 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:12:44.349482 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-m49mq"] Sep 14 11:12:44.350907 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:12:44.350882 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod518192ea_4063_4e8c_a600_6a85c83c2dcb.slice/crio-41d7aac1128fa6e9fa5e816f9d7aca04f68458cfb6c331f441c2d455d1ff02e5 WatchSource:0}: Error finding container 41d7aac1128fa6e9fa5e816f9d7aca04f68458cfb6c331f441c2d455d1ff02e5: Status 404 returned error can't find the container with id 41d7aac1128fa6e9fa5e816f9d7aca04f68458cfb6c331f441c2d455d1ff02e5 Sep 14 11:12:45.233245 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:12:45.233206 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-m49mq" event={"ID":"518192ea-4063-4e8c-a600-6a85c83c2dcb","Type":"ContainerStarted","Data":"1efe15cfe6e48aa56061b69817f0e79ad3f0d97cb921bd3a06d0df88557dbfb5"} Sep 14 11:12:45.233616 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:12:45.233292 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-m49mq" event={"ID":"518192ea-4063-4e8c-a600-6a85c83c2dcb","Type":"ContainerStarted","Data":"41d7aac1128fa6e9fa5e816f9d7aca04f68458cfb6c331f441c2d455d1ff02e5"} Sep 14 11:12:45.286006 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:12:45.285952 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-m49mq" podStartSLOduration=2.097688573 podStartE2EDuration="2.285933581s" totalImagesPullingTime="188.245008ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:12:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:12:45.284299609 +0000 UTC m=+709.904512132" watchObservedRunningTime="2026-09-14 11:12:45.285933581 +0000 UTC m=+709.906146103" Sep 14 11:12:54.073377 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:12:54.073298 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-m49mq" Sep 14 11:12:54.073377 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:12:54.073367 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-m49mq" Sep 14 11:12:54.085761 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:12:54.085738 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-m49mq" Sep 14 11:12:54.288801 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:12:54.288770 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-m49mq" Sep 14 11:15:41.740952 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:41.740912 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-de0f7a-6fc97c6fdd-mnhwk"] Sep 14 11:15:41.744840 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:41.744823 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-de0f7a-6fc97c6fdd-mnhwk" Sep 14 11:15:41.754102 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:41.754069 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-de0f7a-serving-cert\"" Sep 14 11:15:41.761798 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:41.761763 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-de0f7a-6fc97c6fdd-mnhwk"] Sep 14 11:15:41.855664 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:41.855628 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/454c8e0e-edcf-4c29-b697-a6cc79092849-maas-api-tls\") pod \"maas-api-e2e-worker-w4-de0f7a-6fc97c6fdd-mnhwk\" (UID: \"454c8e0e-edcf-4c29-b697-a6cc79092849\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-de0f7a-6fc97c6fdd-mnhwk" Sep 14 11:15:41.855791 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:41.855686 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx5xc\" (UniqueName: \"kubernetes.io/projected/454c8e0e-edcf-4c29-b697-a6cc79092849-kube-api-access-jx5xc\") pod \"maas-api-e2e-worker-w4-de0f7a-6fc97c6fdd-mnhwk\" (UID: \"454c8e0e-edcf-4c29-b697-a6cc79092849\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-de0f7a-6fc97c6fdd-mnhwk" Sep 14 11:15:41.855791 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:41.855779 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/454c8e0e-edcf-4c29-b697-a6cc79092849-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-de0f7a-6fc97c6fdd-mnhwk\" (UID: \"454c8e0e-edcf-4c29-b697-a6cc79092849\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-de0f7a-6fc97c6fdd-mnhwk" Sep 14 11:15:41.855880 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:41.855805 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/454c8e0e-edcf-4c29-b697-a6cc79092849-metrics-certs\") pod \"maas-api-e2e-worker-w4-de0f7a-6fc97c6fdd-mnhwk\" (UID: \"454c8e0e-edcf-4c29-b697-a6cc79092849\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-de0f7a-6fc97c6fdd-mnhwk" Sep 14 11:15:41.956675 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:41.956641 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/454c8e0e-edcf-4c29-b697-a6cc79092849-maas-api-tls\") pod \"maas-api-e2e-worker-w4-de0f7a-6fc97c6fdd-mnhwk\" (UID: \"454c8e0e-edcf-4c29-b697-a6cc79092849\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-de0f7a-6fc97c6fdd-mnhwk" Sep 14 11:15:41.956854 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:41.956726 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx5xc\" (UniqueName: \"kubernetes.io/projected/454c8e0e-edcf-4c29-b697-a6cc79092849-kube-api-access-jx5xc\") pod \"maas-api-e2e-worker-w4-de0f7a-6fc97c6fdd-mnhwk\" (UID: \"454c8e0e-edcf-4c29-b697-a6cc79092849\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-de0f7a-6fc97c6fdd-mnhwk" Sep 14 11:15:41.956854 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:41.956768 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/454c8e0e-edcf-4c29-b697-a6cc79092849-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-de0f7a-6fc97c6fdd-mnhwk\" (UID: \"454c8e0e-edcf-4c29-b697-a6cc79092849\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-de0f7a-6fc97c6fdd-mnhwk" Sep 14 11:15:41.956854 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:41.956791 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/454c8e0e-edcf-4c29-b697-a6cc79092849-metrics-certs\") pod \"maas-api-e2e-worker-w4-de0f7a-6fc97c6fdd-mnhwk\" (UID: \"454c8e0e-edcf-4c29-b697-a6cc79092849\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-de0f7a-6fc97c6fdd-mnhwk" Sep 14 11:15:41.957530 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:41.957453 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/454c8e0e-edcf-4c29-b697-a6cc79092849-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-de0f7a-6fc97c6fdd-mnhwk\" (UID: \"454c8e0e-edcf-4c29-b697-a6cc79092849\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-de0f7a-6fc97c6fdd-mnhwk" Sep 14 11:15:41.959259 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:41.959237 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/454c8e0e-edcf-4c29-b697-a6cc79092849-maas-api-tls\") pod \"maas-api-e2e-worker-w4-de0f7a-6fc97c6fdd-mnhwk\" (UID: \"454c8e0e-edcf-4c29-b697-a6cc79092849\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-de0f7a-6fc97c6fdd-mnhwk" Sep 14 11:15:41.959380 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:41.959324 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/454c8e0e-edcf-4c29-b697-a6cc79092849-metrics-certs\") pod \"maas-api-e2e-worker-w4-de0f7a-6fc97c6fdd-mnhwk\" (UID: \"454c8e0e-edcf-4c29-b697-a6cc79092849\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-de0f7a-6fc97c6fdd-mnhwk" Sep 14 11:15:41.970002 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:41.969975 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx5xc\" (UniqueName: \"kubernetes.io/projected/454c8e0e-edcf-4c29-b697-a6cc79092849-kube-api-access-jx5xc\") pod \"maas-api-e2e-worker-w4-de0f7a-6fc97c6fdd-mnhwk\" (UID: \"454c8e0e-edcf-4c29-b697-a6cc79092849\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-de0f7a-6fc97c6fdd-mnhwk" Sep 14 11:15:42.078453 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:42.078384 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-de0f7a-6fc97c6fdd-mnhwk" Sep 14 11:15:42.371486 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:42.371459 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-de0f7a-6fc97c6fdd-mnhwk"] Sep 14 11:15:42.373458 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:15:42.373431 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod454c8e0e_edcf_4c29_b697_a6cc79092849.slice/crio-3ef9d85f712ecdc388fc4d32f899efa68e15974714e1baa6fd91cff46b9803df WatchSource:0}: Error finding container 3ef9d85f712ecdc388fc4d32f899efa68e15974714e1baa6fd91cff46b9803df: Status 404 returned error can't find the container with id 3ef9d85f712ecdc388fc4d32f899efa68e15974714e1baa6fd91cff46b9803df Sep 14 11:15:43.017988 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:43.017954 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-de0f7a-687bdd887d-b86hw"] Sep 14 11:15:43.021665 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:43.021645 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-de0f7a-687bdd887d-b86hw" Sep 14 11:15:43.024238 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:43.024219 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-de0f7a\"" Sep 14 11:15:43.033235 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:43.033213 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-de0f7a-687bdd887d-b86hw"] Sep 14 11:15:43.114647 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:43.114607 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-de0f7a-6fc97c6fdd-mnhwk" event={"ID":"454c8e0e-edcf-4c29-b697-a6cc79092849","Type":"ContainerStarted","Data":"780e35f8095c46f8fb844a05599285e186e9a342ee5244f06461673f3ba7f7aa"} Sep 14 11:15:43.114647 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:43.114654 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-de0f7a-6fc97c6fdd-mnhwk" event={"ID":"454c8e0e-edcf-4c29-b697-a6cc79092849","Type":"ContainerStarted","Data":"3ef9d85f712ecdc388fc4d32f899efa68e15974714e1baa6fd91cff46b9803df"} Sep 14 11:15:43.114873 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:43.114751 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-de0f7a-6fc97c6fdd-mnhwk" Sep 14 11:15:43.143348 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:43.143308 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-de0f7a-6ccfc74d69-9xjkc"] Sep 14 11:15:43.144614 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:43.144572 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-de0f7a-6fc97c6fdd-mnhwk" podStartSLOduration=1.59920405 podStartE2EDuration="2.144557345s" totalImagesPullingTime="545.353295ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:15:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:15:43.142373612 +0000 UTC m=+887.762586159" watchObservedRunningTime="2026-09-14 11:15:43.144557345 +0000 UTC m=+887.764769859" Sep 14 11:15:43.147060 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:43.147041 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-de0f7a-6ccfc74d69-9xjkc" Sep 14 11:15:43.152482 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:43.152450 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-de0f7a-dockercfg-d4vb8\"" Sep 14 11:15:43.162321 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:43.162292 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-de0f7a-6ccfc74d69-9xjkc"] Sep 14 11:15:43.168835 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:43.168808 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47j57\" (UniqueName: \"kubernetes.io/projected/56ede25d-658f-4465-94ec-b64625cf6f6f-kube-api-access-47j57\") pod \"payload-pre-processing-e2e-worker-w4-de0f7a-687bdd887d-b86hw\" (UID: \"56ede25d-658f-4465-94ec-b64625cf6f6f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-de0f7a-687bdd887d-b86hw" Sep 14 11:15:43.168986 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:43.168861 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56ede25d-658f-4465-94ec-b64625cf6f6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-de0f7a-687bdd887d-b86hw\" (UID: \"56ede25d-658f-4465-94ec-b64625cf6f6f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-de0f7a-687bdd887d-b86hw" Sep 14 11:15:43.269818 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:43.269784 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47j57\" (UniqueName: \"kubernetes.io/projected/56ede25d-658f-4465-94ec-b64625cf6f6f-kube-api-access-47j57\") pod \"payload-pre-processing-e2e-worker-w4-de0f7a-687bdd887d-b86hw\" (UID: \"56ede25d-658f-4465-94ec-b64625cf6f6f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-de0f7a-687bdd887d-b86hw" Sep 14 11:15:43.269999 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:43.269845 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jmk2\" (UniqueName: \"kubernetes.io/projected/af78aa4d-5236-4484-867d-0bf6e594072d-kube-api-access-4jmk2\") pod \"payload-processing-e2e-worker-w4-de0f7a-6ccfc74d69-9xjkc\" (UID: \"af78aa4d-5236-4484-867d-0bf6e594072d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-de0f7a-6ccfc74d69-9xjkc" Sep 14 11:15:43.269999 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:43.269881 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56ede25d-658f-4465-94ec-b64625cf6f6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-de0f7a-687bdd887d-b86hw\" (UID: \"56ede25d-658f-4465-94ec-b64625cf6f6f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-de0f7a-687bdd887d-b86hw" Sep 14 11:15:43.269999 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:43.269974 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af78aa4d-5236-4484-867d-0bf6e594072d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-de0f7a-6ccfc74d69-9xjkc\" (UID: \"af78aa4d-5236-4484-867d-0bf6e594072d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-de0f7a-6ccfc74d69-9xjkc" Sep 14 11:15:43.270639 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:43.270615 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56ede25d-658f-4465-94ec-b64625cf6f6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-de0f7a-687bdd887d-b86hw\" (UID: \"56ede25d-658f-4465-94ec-b64625cf6f6f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-de0f7a-687bdd887d-b86hw" Sep 14 11:15:43.280833 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:43.280805 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47j57\" (UniqueName: \"kubernetes.io/projected/56ede25d-658f-4465-94ec-b64625cf6f6f-kube-api-access-47j57\") pod \"payload-pre-processing-e2e-worker-w4-de0f7a-687bdd887d-b86hw\" (UID: \"56ede25d-658f-4465-94ec-b64625cf6f6f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-de0f7a-687bdd887d-b86hw" Sep 14 11:15:43.332208 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:43.332175 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-de0f7a-687bdd887d-b86hw" Sep 14 11:15:43.371418 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:43.371354 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jmk2\" (UniqueName: \"kubernetes.io/projected/af78aa4d-5236-4484-867d-0bf6e594072d-kube-api-access-4jmk2\") pod \"payload-processing-e2e-worker-w4-de0f7a-6ccfc74d69-9xjkc\" (UID: \"af78aa4d-5236-4484-867d-0bf6e594072d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-de0f7a-6ccfc74d69-9xjkc" Sep 14 11:15:43.371641 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:43.371461 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af78aa4d-5236-4484-867d-0bf6e594072d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-de0f7a-6ccfc74d69-9xjkc\" (UID: \"af78aa4d-5236-4484-867d-0bf6e594072d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-de0f7a-6ccfc74d69-9xjkc" Sep 14 11:15:43.372384 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:43.372361 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af78aa4d-5236-4484-867d-0bf6e594072d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-de0f7a-6ccfc74d69-9xjkc\" (UID: \"af78aa4d-5236-4484-867d-0bf6e594072d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-de0f7a-6ccfc74d69-9xjkc" Sep 14 11:15:43.383875 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:43.383771 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jmk2\" (UniqueName: \"kubernetes.io/projected/af78aa4d-5236-4484-867d-0bf6e594072d-kube-api-access-4jmk2\") pod \"payload-processing-e2e-worker-w4-de0f7a-6ccfc74d69-9xjkc\" (UID: \"af78aa4d-5236-4484-867d-0bf6e594072d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-de0f7a-6ccfc74d69-9xjkc" Sep 14 11:15:43.460135 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:43.460100 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-de0f7a-6ccfc74d69-9xjkc" Sep 14 11:15:43.629616 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:15:43.629582 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56ede25d_658f_4465_94ec_b64625cf6f6f.slice/crio-6b7d2b261d75e0b69ec5af2e972c446be9424ce85fba78132f9352399228934a WatchSource:0}: Error finding container 6b7d2b261d75e0b69ec5af2e972c446be9424ce85fba78132f9352399228934a: Status 404 returned error can't find the container with id 6b7d2b261d75e0b69ec5af2e972c446be9424ce85fba78132f9352399228934a Sep 14 11:15:43.630052 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:43.630027 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-de0f7a-687bdd887d-b86hw"] Sep 14 11:15:43.771124 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:43.771097 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-de0f7a-6ccfc74d69-9xjkc"] Sep 14 11:15:43.772424 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:15:43.772396 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf78aa4d_5236_4484_867d_0bf6e594072d.slice/crio-0c5a846ee7f9471077ff5389f3a9b338475c8acf932dd3c60801f3e330a55181 WatchSource:0}: Error finding container 0c5a846ee7f9471077ff5389f3a9b338475c8acf932dd3c60801f3e330a55181: Status 404 returned error can't find the container with id 0c5a846ee7f9471077ff5389f3a9b338475c8acf932dd3c60801f3e330a55181 Sep 14 11:15:44.125953 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:44.125903 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-de0f7a-6ccfc74d69-9xjkc" event={"ID":"af78aa4d-5236-4484-867d-0bf6e594072d","Type":"ContainerStarted","Data":"0fa844f05adef13fc10f9a4c25866e8efba541de26e3118af0232413ac7da6ef"} Sep 14 11:15:44.126443 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:44.125966 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-de0f7a-6ccfc74d69-9xjkc" event={"ID":"af78aa4d-5236-4484-867d-0bf6e594072d","Type":"ContainerStarted","Data":"0c5a846ee7f9471077ff5389f3a9b338475c8acf932dd3c60801f3e330a55181"} Sep 14 11:15:44.126443 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:44.126019 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-de0f7a-6ccfc74d69-9xjkc" Sep 14 11:15:44.127602 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:44.127578 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-de0f7a-687bdd887d-b86hw" event={"ID":"56ede25d-658f-4465-94ec-b64625cf6f6f","Type":"ContainerStarted","Data":"98ec95515f8cbe58ce36794f94eee0079e223be93047e513a0fe81af39af6fdc"} Sep 14 11:15:44.127742 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:44.127615 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-de0f7a-687bdd887d-b86hw" event={"ID":"56ede25d-658f-4465-94ec-b64625cf6f6f","Type":"ContainerStarted","Data":"6b7d2b261d75e0b69ec5af2e972c446be9424ce85fba78132f9352399228934a"} Sep 14 11:15:44.152679 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:44.152638 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-de0f7a-6ccfc74d69-9xjkc" podStartSLOduration=1.15262453 podStartE2EDuration="1.15262453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:15:44.151758884 +0000 UTC m=+888.771971425" watchObservedRunningTime="2026-09-14 11:15:44.15262453 +0000 UTC m=+888.772837053" Sep 14 11:15:44.179130 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:44.179088 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-de0f7a-687bdd887d-b86hw" podStartSLOduration=2.17907038 podStartE2EDuration="2.17907038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:15:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:15:44.177382163 +0000 UTC m=+888.797594700" watchObservedRunningTime="2026-09-14 11:15:44.17907038 +0000 UTC m=+888.799282967" Sep 14 11:15:44.598364 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:44.598331 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2"] Sep 14 11:15:44.602545 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:44.602523 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2" Sep 14 11:15:44.607911 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:44.607884 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-6f3d0f-serving-cert\"" Sep 14 11:15:44.623298 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:44.623259 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2"] Sep 14 11:15:44.682704 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:44.682671 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9b2415c-3a23-4717-ba02-71543cd63a88-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2\" (UID: \"f9b2415c-3a23-4717-ba02-71543cd63a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2" Sep 14 11:15:44.682844 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:44.682724 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9b2415c-3a23-4717-ba02-71543cd63a88-maas-api-tls\") pod \"maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2\" (UID: \"f9b2415c-3a23-4717-ba02-71543cd63a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2" Sep 14 11:15:44.682844 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:44.682798 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzdrf\" (UniqueName: \"kubernetes.io/projected/f9b2415c-3a23-4717-ba02-71543cd63a88-kube-api-access-lzdrf\") pod \"maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2\" (UID: \"f9b2415c-3a23-4717-ba02-71543cd63a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2" Sep 14 11:15:44.682922 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:44.682862 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9b2415c-3a23-4717-ba02-71543cd63a88-metrics-certs\") pod \"maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2\" (UID: \"f9b2415c-3a23-4717-ba02-71543cd63a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2" Sep 14 11:15:44.783786 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:44.783754 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9b2415c-3a23-4717-ba02-71543cd63a88-maas-api-tls\") pod \"maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2\" (UID: \"f9b2415c-3a23-4717-ba02-71543cd63a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2" Sep 14 11:15:44.783956 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:44.783797 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzdrf\" (UniqueName: \"kubernetes.io/projected/f9b2415c-3a23-4717-ba02-71543cd63a88-kube-api-access-lzdrf\") pod \"maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2\" (UID: \"f9b2415c-3a23-4717-ba02-71543cd63a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2" Sep 14 11:15:44.783956 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:44.783841 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9b2415c-3a23-4717-ba02-71543cd63a88-metrics-certs\") pod \"maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2\" (UID: \"f9b2415c-3a23-4717-ba02-71543cd63a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2" Sep 14 11:15:44.783956 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:44.783885 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9b2415c-3a23-4717-ba02-71543cd63a88-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2\" (UID: \"f9b2415c-3a23-4717-ba02-71543cd63a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2" Sep 14 11:15:44.783956 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:15:44.783918 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-6f3d0f-serving-cert: secret "maas-api-e2e-worker-w0-6f3d0f-serving-cert" not found Sep 14 11:15:44.784162 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:15:44.783990 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9b2415c-3a23-4717-ba02-71543cd63a88-maas-api-tls podName:f9b2415c-3a23-4717-ba02-71543cd63a88 nodeName:}" failed. No retries permitted until 2026-09-14 11:15:45.28396878 +0000 UTC m=+889.904181295 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f9b2415c-3a23-4717-ba02-71543cd63a88-maas-api-tls") pod "maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2" (UID: "f9b2415c-3a23-4717-ba02-71543cd63a88") : secret "maas-api-e2e-worker-w0-6f3d0f-serving-cert" not found Sep 14 11:15:44.784614 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:44.784582 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9b2415c-3a23-4717-ba02-71543cd63a88-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2\" (UID: \"f9b2415c-3a23-4717-ba02-71543cd63a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2" Sep 14 11:15:44.786397 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:44.786375 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9b2415c-3a23-4717-ba02-71543cd63a88-metrics-certs\") pod \"maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2\" (UID: \"f9b2415c-3a23-4717-ba02-71543cd63a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2" Sep 14 11:15:44.792194 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:44.792172 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzdrf\" (UniqueName: \"kubernetes.io/projected/f9b2415c-3a23-4717-ba02-71543cd63a88-kube-api-access-lzdrf\") pod \"maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2\" (UID: \"f9b2415c-3a23-4717-ba02-71543cd63a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2" Sep 14 11:15:45.134016 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:45.133981 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-de0f7a-687bdd887d-b86hw" Sep 14 11:15:45.290876 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:45.290839 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9b2415c-3a23-4717-ba02-71543cd63a88-maas-api-tls\") pod \"maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2\" (UID: \"f9b2415c-3a23-4717-ba02-71543cd63a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2" Sep 14 11:15:45.293588 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:45.293564 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9b2415c-3a23-4717-ba02-71543cd63a88-maas-api-tls\") pod \"maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2\" (UID: \"f9b2415c-3a23-4717-ba02-71543cd63a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2" Sep 14 11:15:45.514899 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:45.514869 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2" Sep 14 11:15:45.800059 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:45.800035 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2"] Sep 14 11:15:45.802057 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:15:45.802001 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9b2415c_3a23_4717_ba02_71543cd63a88.slice/crio-46dfef6cf78e76881f4fb2f333ccddb54cec823c4ac06b1a4c49c36dc97a1b04 WatchSource:0}: Error finding container 46dfef6cf78e76881f4fb2f333ccddb54cec823c4ac06b1a4c49c36dc97a1b04: Status 404 returned error can't find the container with id 46dfef6cf78e76881f4fb2f333ccddb54cec823c4ac06b1a4c49c36dc97a1b04 Sep 14 11:15:46.017292 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:46.017242 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf"] Sep 14 11:15:46.022807 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:46.022777 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf" Sep 14 11:15:46.025718 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:46.025696 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-6d272c-serving-cert\"" Sep 14 11:15:46.039839 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:46.039813 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf"] Sep 14 11:15:46.097540 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:46.097464 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c305a6f4-bb07-435f-9a79-c6580a82346c-maas-api-tls\") pod \"maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf\" (UID: \"c305a6f4-bb07-435f-9a79-c6580a82346c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf" Sep 14 11:15:46.097671 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:46.097540 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldgm5\" (UniqueName: \"kubernetes.io/projected/c305a6f4-bb07-435f-9a79-c6580a82346c-kube-api-access-ldgm5\") pod \"maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf\" (UID: \"c305a6f4-bb07-435f-9a79-c6580a82346c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf" Sep 14 11:15:46.097725 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:46.097677 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c305a6f4-bb07-435f-9a79-c6580a82346c-metrics-certs\") pod \"maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf\" (UID: \"c305a6f4-bb07-435f-9a79-c6580a82346c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf" Sep 14 11:15:46.097725 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:46.097703 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c305a6f4-bb07-435f-9a79-c6580a82346c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf\" (UID: \"c305a6f4-bb07-435f-9a79-c6580a82346c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf" Sep 14 11:15:46.146007 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:46.144199 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2" event={"ID":"f9b2415c-3a23-4717-ba02-71543cd63a88","Type":"ContainerStarted","Data":"46dfef6cf78e76881f4fb2f333ccddb54cec823c4ac06b1a4c49c36dc97a1b04"} Sep 14 11:15:46.198614 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:46.198531 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c305a6f4-bb07-435f-9a79-c6580a82346c-metrics-certs\") pod \"maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf\" (UID: \"c305a6f4-bb07-435f-9a79-c6580a82346c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf" Sep 14 11:15:46.198614 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:46.198569 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c305a6f4-bb07-435f-9a79-c6580a82346c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf\" (UID: \"c305a6f4-bb07-435f-9a79-c6580a82346c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf" Sep 14 11:15:46.198614 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:46.198599 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c305a6f4-bb07-435f-9a79-c6580a82346c-maas-api-tls\") pod \"maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf\" (UID: \"c305a6f4-bb07-435f-9a79-c6580a82346c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf" Sep 14 11:15:46.198796 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:46.198637 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldgm5\" (UniqueName: \"kubernetes.io/projected/c305a6f4-bb07-435f-9a79-c6580a82346c-kube-api-access-ldgm5\") pod \"maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf\" (UID: \"c305a6f4-bb07-435f-9a79-c6580a82346c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf" Sep 14 11:15:46.198796 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:15:46.198769 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-6d272c-serving-cert: secret "maas-api-e2e-inf-a-6d272c-serving-cert" not found Sep 14 11:15:46.198891 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:15:46.198845 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c305a6f4-bb07-435f-9a79-c6580a82346c-maas-api-tls podName:c305a6f4-bb07-435f-9a79-c6580a82346c nodeName:}" failed. No retries permitted until 2026-09-14 11:15:46.69882399 +0000 UTC m=+891.319036506 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c305a6f4-bb07-435f-9a79-c6580a82346c-maas-api-tls") pod "maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf" (UID: "c305a6f4-bb07-435f-9a79-c6580a82346c") : secret "maas-api-e2e-inf-a-6d272c-serving-cert" not found Sep 14 11:15:46.199411 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:46.199247 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c305a6f4-bb07-435f-9a79-c6580a82346c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf\" (UID: \"c305a6f4-bb07-435f-9a79-c6580a82346c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf" Sep 14 11:15:46.201060 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:46.201039 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c305a6f4-bb07-435f-9a79-c6580a82346c-metrics-certs\") pod \"maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf\" (UID: \"c305a6f4-bb07-435f-9a79-c6580a82346c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf" Sep 14 11:15:46.217583 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:46.217561 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldgm5\" (UniqueName: \"kubernetes.io/projected/c305a6f4-bb07-435f-9a79-c6580a82346c-kube-api-access-ldgm5\") pod \"maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf\" (UID: \"c305a6f4-bb07-435f-9a79-c6580a82346c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf" Sep 14 11:15:46.703988 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:46.703907 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c305a6f4-bb07-435f-9a79-c6580a82346c-maas-api-tls\") pod \"maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf\" (UID: \"c305a6f4-bb07-435f-9a79-c6580a82346c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf" Sep 14 11:15:46.706452 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:46.706424 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c305a6f4-bb07-435f-9a79-c6580a82346c-maas-api-tls\") pod \"maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf\" (UID: \"c305a6f4-bb07-435f-9a79-c6580a82346c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf" Sep 14 11:15:46.937518 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:46.937481 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf" Sep 14 11:15:47.150821 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:47.150784 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2" event={"ID":"f9b2415c-3a23-4717-ba02-71543cd63a88","Type":"ContainerStarted","Data":"e7c68eeb8ead3209c734e2d53248c8f5bbc7c1e5d32e21655bbfb15965196d12"} Sep 14 11:15:47.151187 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:47.150940 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2" Sep 14 11:15:47.194559 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:47.194505 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2" podStartSLOduration=2.867718884 podStartE2EDuration="3.19448266s" totalImagesPullingTime="326.763776ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:15:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:15:47.1931985 +0000 UTC m=+891.813411024" watchObservedRunningTime="2026-09-14 11:15:47.19448266 +0000 UTC m=+891.814695187" Sep 14 11:15:47.247917 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:47.247888 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf"] Sep 14 11:15:47.249069 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:15:47.249042 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc305a6f4_bb07_435f_9a79_c6580a82346c.slice/crio-395850fd7fff30f902e9ba5d49c22bb81ffb98b636f8902faa7b5e9e64f381e7 WatchSource:0}: Error finding container 395850fd7fff30f902e9ba5d49c22bb81ffb98b636f8902faa7b5e9e64f381e7: Status 404 returned error can't find the container with id 395850fd7fff30f902e9ba5d49c22bb81ffb98b636f8902faa7b5e9e64f381e7 Sep 14 11:15:47.882874 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:47.882790 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6d272c-54ccb485-dzrnb"] Sep 14 11:15:47.886612 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:47.886589 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6d272c-54ccb485-dzrnb" Sep 14 11:15:47.889677 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:47.889656 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-6d272c\"" Sep 14 11:15:47.889843 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:47.889824 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-6d272c-dockercfg-4m45b\"" Sep 14 11:15:47.924087 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:47.924056 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6d272c-54ccb485-dzrnb"] Sep 14 11:15:48.017922 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:48.017892 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z9kf\" (UniqueName: \"kubernetes.io/projected/a9563868-8c3b-4100-a7db-28c76e66fe40-kube-api-access-4z9kf\") pod \"payload-processing-e2e-inf-a-6d272c-54ccb485-dzrnb\" (UID: \"a9563868-8c3b-4100-a7db-28c76e66fe40\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6d272c-54ccb485-dzrnb" Sep 14 11:15:48.018081 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:48.017928 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9563868-8c3b-4100-a7db-28c76e66fe40-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6d272c-54ccb485-dzrnb\" (UID: \"a9563868-8c3b-4100-a7db-28c76e66fe40\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6d272c-54ccb485-dzrnb" Sep 14 11:15:48.118820 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:48.118786 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z9kf\" (UniqueName: \"kubernetes.io/projected/a9563868-8c3b-4100-a7db-28c76e66fe40-kube-api-access-4z9kf\") pod \"payload-processing-e2e-inf-a-6d272c-54ccb485-dzrnb\" (UID: \"a9563868-8c3b-4100-a7db-28c76e66fe40\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6d272c-54ccb485-dzrnb" Sep 14 11:15:48.118820 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:48.118824 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9563868-8c3b-4100-a7db-28c76e66fe40-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6d272c-54ccb485-dzrnb\" (UID: \"a9563868-8c3b-4100-a7db-28c76e66fe40\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6d272c-54ccb485-dzrnb" Sep 14 11:15:48.119830 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:48.119803 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9563868-8c3b-4100-a7db-28c76e66fe40-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6d272c-54ccb485-dzrnb\" (UID: \"a9563868-8c3b-4100-a7db-28c76e66fe40\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6d272c-54ccb485-dzrnb" Sep 14 11:15:48.140346 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:48.140253 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z9kf\" (UniqueName: \"kubernetes.io/projected/a9563868-8c3b-4100-a7db-28c76e66fe40-kube-api-access-4z9kf\") pod \"payload-processing-e2e-inf-a-6d272c-54ccb485-dzrnb\" (UID: \"a9563868-8c3b-4100-a7db-28c76e66fe40\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6d272c-54ccb485-dzrnb" Sep 14 11:15:48.159734 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:48.159697 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf" event={"ID":"c305a6f4-bb07-435f-9a79-c6580a82346c","Type":"ContainerStarted","Data":"76d6afe5f168ab5c498887d66bcb77537ce8a58a910521821d0c70d24f086c6f"} Sep 14 11:15:48.160115 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:48.159760 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf" event={"ID":"c305a6f4-bb07-435f-9a79-c6580a82346c","Type":"ContainerStarted","Data":"395850fd7fff30f902e9ba5d49c22bb81ffb98b636f8902faa7b5e9e64f381e7"} Sep 14 11:15:48.160115 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:48.159807 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf" Sep 14 11:15:48.199751 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:48.199721 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6d272c-54ccb485-dzrnb" Sep 14 11:15:48.488574 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:48.488517 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf" podStartSLOduration=3.142000674 podStartE2EDuration="3.488499006s" totalImagesPullingTime="346.498332ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:15:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:15:48.188502314 +0000 UTC m=+892.808714836" watchObservedRunningTime="2026-09-14 11:15:48.488499006 +0000 UTC m=+893.108711528" Sep 14 11:15:48.490286 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:48.490249 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6d272c-54ccb485-dzrnb"] Sep 14 11:15:48.492650 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:15:48.492621 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9563868_8c3b_4100_a7db_28c76e66fe40.slice/crio-d91167491a359dbec5e95c862cb09d8db8ad8e019b4b85e1858b684de81663db WatchSource:0}: Error finding container d91167491a359dbec5e95c862cb09d8db8ad8e019b4b85e1858b684de81663db: Status 404 returned error can't find the container with id d91167491a359dbec5e95c862cb09d8db8ad8e019b4b85e1858b684de81663db Sep 14 11:15:49.134119 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:49.134085 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-de0f7a-6fc97c6fdd-mnhwk" Sep 14 11:15:49.168044 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:49.168012 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6d272c-54ccb485-dzrnb" event={"ID":"a9563868-8c3b-4100-a7db-28c76e66fe40","Type":"ContainerStarted","Data":"4f89c3265bd6fadea498c04fbee51b3a4d9835ae2d3662676c22c055d2bdb0a4"} Sep 14 11:15:49.168497 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:49.168168 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6d272c-54ccb485-dzrnb" Sep 14 11:15:49.168497 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:49.168211 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6d272c-54ccb485-dzrnb" event={"ID":"a9563868-8c3b-4100-a7db-28c76e66fe40","Type":"ContainerStarted","Data":"d91167491a359dbec5e95c862cb09d8db8ad8e019b4b85e1858b684de81663db"} Sep 14 11:15:49.205737 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:49.205587 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-6d272c-54ccb485-dzrnb" podStartSLOduration=2.205570328 podStartE2EDuration="2.205570328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:15:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:15:49.202648557 +0000 UTC m=+893.822861096" watchObservedRunningTime="2026-09-14 11:15:49.205570328 +0000 UTC m=+893.825782850" Sep 14 11:15:52.460487 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:52.460456 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-27a1a2-5dcd48c789-z6n8r"] Sep 14 11:15:52.464423 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:52.464391 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-27a1a2-5dcd48c789-z6n8r" Sep 14 11:15:52.468860 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:52.468833 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-27a1a2-dockercfg-r4cfj\"" Sep 14 11:15:52.469171 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:52.469153 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-27a1a2\"" Sep 14 11:15:52.476385 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:52.476366 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-27a1a2-5dcd48c789-z6n8r"] Sep 14 11:15:52.564623 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:52.564592 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d120622-8756-4487-9798-b99f4d4279f4-plugins-config-volume\") pod \"payload-processing-e2e-autores-27a1a2-5dcd48c789-z6n8r\" (UID: \"8d120622-8756-4487-9798-b99f4d4279f4\") " pod="openshift-ingress/payload-processing-e2e-autores-27a1a2-5dcd48c789-z6n8r" Sep 14 11:15:52.564759 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:52.564686 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgpwv\" (UniqueName: \"kubernetes.io/projected/8d120622-8756-4487-9798-b99f4d4279f4-kube-api-access-fgpwv\") pod \"payload-processing-e2e-autores-27a1a2-5dcd48c789-z6n8r\" (UID: \"8d120622-8756-4487-9798-b99f4d4279f4\") " pod="openshift-ingress/payload-processing-e2e-autores-27a1a2-5dcd48c789-z6n8r" Sep 14 11:15:52.666101 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:52.666063 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgpwv\" (UniqueName: \"kubernetes.io/projected/8d120622-8756-4487-9798-b99f4d4279f4-kube-api-access-fgpwv\") pod \"payload-processing-e2e-autores-27a1a2-5dcd48c789-z6n8r\" (UID: \"8d120622-8756-4487-9798-b99f4d4279f4\") " pod="openshift-ingress/payload-processing-e2e-autores-27a1a2-5dcd48c789-z6n8r" Sep 14 11:15:52.666295 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:52.666142 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d120622-8756-4487-9798-b99f4d4279f4-plugins-config-volume\") pod \"payload-processing-e2e-autores-27a1a2-5dcd48c789-z6n8r\" (UID: \"8d120622-8756-4487-9798-b99f4d4279f4\") " pod="openshift-ingress/payload-processing-e2e-autores-27a1a2-5dcd48c789-z6n8r" Sep 14 11:15:52.666816 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:52.666797 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d120622-8756-4487-9798-b99f4d4279f4-plugins-config-volume\") pod \"payload-processing-e2e-autores-27a1a2-5dcd48c789-z6n8r\" (UID: \"8d120622-8756-4487-9798-b99f4d4279f4\") " pod="openshift-ingress/payload-processing-e2e-autores-27a1a2-5dcd48c789-z6n8r" Sep 14 11:15:52.678954 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:52.678930 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgpwv\" (UniqueName: \"kubernetes.io/projected/8d120622-8756-4487-9798-b99f4d4279f4-kube-api-access-fgpwv\") pod \"payload-processing-e2e-autores-27a1a2-5dcd48c789-z6n8r\" (UID: \"8d120622-8756-4487-9798-b99f4d4279f4\") " pod="openshift-ingress/payload-processing-e2e-autores-27a1a2-5dcd48c789-z6n8r" Sep 14 11:15:52.779837 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:52.779760 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-27a1a2-5dcd48c789-z6n8r" Sep 14 11:15:53.062522 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:53.062494 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-27a1a2-5dcd48c789-z6n8r"] Sep 14 11:15:53.063393 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:15:53.063370 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d120622_8756_4487_9798_b99f4d4279f4.slice/crio-654adc6662d13cc781b96e449ae38ceb720bf7e41810126b073fe11fd57e0830 WatchSource:0}: Error finding container 654adc6662d13cc781b96e449ae38ceb720bf7e41810126b073fe11fd57e0830: Status 404 returned error can't find the container with id 654adc6662d13cc781b96e449ae38ceb720bf7e41810126b073fe11fd57e0830 Sep 14 11:15:53.165073 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:53.165049 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2" Sep 14 11:15:53.194457 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:53.194427 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-27a1a2-5dcd48c789-z6n8r" event={"ID":"8d120622-8756-4487-9798-b99f4d4279f4","Type":"ContainerStarted","Data":"a2e1b947e0614ef58063d9f67fe5e3d4004b3d0cd363426bdd2c2fc2cc9edfc5"} Sep 14 11:15:53.194686 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:53.194484 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-27a1a2-5dcd48c789-z6n8r" event={"ID":"8d120622-8756-4487-9798-b99f4d4279f4","Type":"ContainerStarted","Data":"654adc6662d13cc781b96e449ae38ceb720bf7e41810126b073fe11fd57e0830"} Sep 14 11:15:53.194686 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:53.194527 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-27a1a2-5dcd48c789-z6n8r" Sep 14 11:15:53.216778 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:53.216705 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-27a1a2-5dcd48c789-z6n8r" podStartSLOduration=1.216684274 podStartE2EDuration="1.216684274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:15:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:15:53.214248052 +0000 UTC m=+897.834460576" watchObservedRunningTime="2026-09-14 11:15:53.216684274 +0000 UTC m=+897.836896798" Sep 14 11:15:54.173499 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:54.173470 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf" Sep 14 11:15:55.136044 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:55.136016 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-de0f7a-6ccfc74d69-9xjkc" Sep 14 11:15:56.147029 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:15:56.147002 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-de0f7a-687bdd887d-b86hw" Sep 14 11:16:00.178793 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:00.178761 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6d272c-54ccb485-dzrnb" Sep 14 11:16:04.201475 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:04.201440 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-27a1a2-5dcd48c789-z6n8r" Sep 14 11:16:13.264983 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:13.264948 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd"] Sep 14 11:16:13.269131 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:13.269108 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd" Sep 14 11:16:13.271880 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:13.271858 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-39ab34-serving-cert\"" Sep 14 11:16:13.284069 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:13.284048 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd"] Sep 14 11:16:13.377363 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:13.377309 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9136cf83-c206-4f67-b1c2-6b3f55eb5d24-maas-api-tls\") pod \"maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd\" (UID: \"9136cf83-c206-4f67-b1c2-6b3f55eb5d24\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd" Sep 14 11:16:13.377517 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:13.377394 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gcsd\" (UniqueName: \"kubernetes.io/projected/9136cf83-c206-4f67-b1c2-6b3f55eb5d24-kube-api-access-4gcsd\") pod \"maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd\" (UID: \"9136cf83-c206-4f67-b1c2-6b3f55eb5d24\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd" Sep 14 11:16:13.377517 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:13.377487 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9136cf83-c206-4f67-b1c2-6b3f55eb5d24-openshift-service-ca\") pod \"maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd\" (UID: \"9136cf83-c206-4f67-b1c2-6b3f55eb5d24\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd" Sep 14 11:16:13.377641 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:13.377622 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9136cf83-c206-4f67-b1c2-6b3f55eb5d24-metrics-certs\") pod \"maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd\" (UID: \"9136cf83-c206-4f67-b1c2-6b3f55eb5d24\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd" Sep 14 11:16:13.478525 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:13.478486 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9136cf83-c206-4f67-b1c2-6b3f55eb5d24-metrics-certs\") pod \"maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd\" (UID: \"9136cf83-c206-4f67-b1c2-6b3f55eb5d24\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd" Sep 14 11:16:13.478525 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:13.478529 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9136cf83-c206-4f67-b1c2-6b3f55eb5d24-maas-api-tls\") pod \"maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd\" (UID: \"9136cf83-c206-4f67-b1c2-6b3f55eb5d24\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd" Sep 14 11:16:13.478722 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:13.478557 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gcsd\" (UniqueName: \"kubernetes.io/projected/9136cf83-c206-4f67-b1c2-6b3f55eb5d24-kube-api-access-4gcsd\") pod \"maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd\" (UID: \"9136cf83-c206-4f67-b1c2-6b3f55eb5d24\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd" Sep 14 11:16:13.478722 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:13.478642 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9136cf83-c206-4f67-b1c2-6b3f55eb5d24-openshift-service-ca\") pod \"maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd\" (UID: \"9136cf83-c206-4f67-b1c2-6b3f55eb5d24\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd" Sep 14 11:16:13.478722 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:16:13.478648 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-39ab34-serving-cert: secret "maas-api-e2e-inf-b-39ab34-serving-cert" not found Sep 14 11:16:13.478836 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:16:13.478732 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9136cf83-c206-4f67-b1c2-6b3f55eb5d24-maas-api-tls podName:9136cf83-c206-4f67-b1c2-6b3f55eb5d24 nodeName:}" failed. No retries permitted until 2026-09-14 11:16:13.978710859 +0000 UTC m=+918.598923365 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9136cf83-c206-4f67-b1c2-6b3f55eb5d24-maas-api-tls") pod "maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd" (UID: "9136cf83-c206-4f67-b1c2-6b3f55eb5d24") : secret "maas-api-e2e-inf-b-39ab34-serving-cert" not found Sep 14 11:16:13.479383 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:13.479357 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9136cf83-c206-4f67-b1c2-6b3f55eb5d24-openshift-service-ca\") pod \"maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd\" (UID: \"9136cf83-c206-4f67-b1c2-6b3f55eb5d24\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd" Sep 14 11:16:13.481311 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:13.481293 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9136cf83-c206-4f67-b1c2-6b3f55eb5d24-metrics-certs\") pod \"maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd\" (UID: \"9136cf83-c206-4f67-b1c2-6b3f55eb5d24\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd" Sep 14 11:16:13.492394 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:13.492376 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gcsd\" (UniqueName: \"kubernetes.io/projected/9136cf83-c206-4f67-b1c2-6b3f55eb5d24-kube-api-access-4gcsd\") pod \"maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd\" (UID: \"9136cf83-c206-4f67-b1c2-6b3f55eb5d24\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd" Sep 14 11:16:13.983244 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:13.983210 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9136cf83-c206-4f67-b1c2-6b3f55eb5d24-maas-api-tls\") pod \"maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd\" (UID: \"9136cf83-c206-4f67-b1c2-6b3f55eb5d24\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd" Sep 14 11:16:13.985645 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:13.985617 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9136cf83-c206-4f67-b1c2-6b3f55eb5d24-maas-api-tls\") pod \"maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd\" (UID: \"9136cf83-c206-4f67-b1c2-6b3f55eb5d24\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd" Sep 14 11:16:14.180995 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:14.180966 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd" Sep 14 11:16:14.218780 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:14.218749 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-39ab34-5449674498-jzrkv"] Sep 14 11:16:14.222637 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:14.222615 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39ab34-5449674498-jzrkv" Sep 14 11:16:14.227951 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:14.227928 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-39ab34-dockercfg-s4r82\"" Sep 14 11:16:14.229578 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:14.229553 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-39ab34\"" Sep 14 11:16:14.244233 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:14.244092 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-39ab34-5449674498-jzrkv"] Sep 14 11:16:14.286646 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:14.286607 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89cjx\" (UniqueName: \"kubernetes.io/projected/84925d6e-d030-437c-a141-7168ad5f9f79-kube-api-access-89cjx\") pod \"payload-pre-processing-e2e-inf-b-39ab34-5449674498-jzrkv\" (UID: \"84925d6e-d030-437c-a141-7168ad5f9f79\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39ab34-5449674498-jzrkv" Sep 14 11:16:14.287064 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:14.286744 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84925d6e-d030-437c-a141-7168ad5f9f79-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-39ab34-5449674498-jzrkv\" (UID: \"84925d6e-d030-437c-a141-7168ad5f9f79\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39ab34-5449674498-jzrkv" Sep 14 11:16:14.387340 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:14.387305 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84925d6e-d030-437c-a141-7168ad5f9f79-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-39ab34-5449674498-jzrkv\" (UID: \"84925d6e-d030-437c-a141-7168ad5f9f79\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39ab34-5449674498-jzrkv" Sep 14 11:16:14.387579 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:14.387383 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89cjx\" (UniqueName: \"kubernetes.io/projected/84925d6e-d030-437c-a141-7168ad5f9f79-kube-api-access-89cjx\") pod \"payload-pre-processing-e2e-inf-b-39ab34-5449674498-jzrkv\" (UID: \"84925d6e-d030-437c-a141-7168ad5f9f79\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39ab34-5449674498-jzrkv" Sep 14 11:16:14.388067 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:14.388043 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84925d6e-d030-437c-a141-7168ad5f9f79-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-39ab34-5449674498-jzrkv\" (UID: \"84925d6e-d030-437c-a141-7168ad5f9f79\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39ab34-5449674498-jzrkv" Sep 14 11:16:14.400821 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:14.400794 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89cjx\" (UniqueName: \"kubernetes.io/projected/84925d6e-d030-437c-a141-7168ad5f9f79-kube-api-access-89cjx\") pod \"payload-pre-processing-e2e-inf-b-39ab34-5449674498-jzrkv\" (UID: \"84925d6e-d030-437c-a141-7168ad5f9f79\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39ab34-5449674498-jzrkv" Sep 14 11:16:14.475126 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:14.475101 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd"] Sep 14 11:16:14.476780 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:16:14.476759 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9136cf83_c206_4f67_b1c2_6b3f55eb5d24.slice/crio-71faf607a67c9ebd676f23968f7d5682f613eb1aa7a7a93f31fa7d61a980a9db WatchSource:0}: Error finding container 71faf607a67c9ebd676f23968f7d5682f613eb1aa7a7a93f31fa7d61a980a9db: Status 404 returned error can't find the container with id 71faf607a67c9ebd676f23968f7d5682f613eb1aa7a7a93f31fa7d61a980a9db Sep 14 11:16:14.478144 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:14.478127 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 11:16:14.535191 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:14.535128 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39ab34-5449674498-jzrkv" Sep 14 11:16:14.815611 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:14.815435 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-39ab34-5449674498-jzrkv"] Sep 14 11:16:14.817996 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:16:14.817956 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84925d6e_d030_437c_a141_7168ad5f9f79.slice/crio-ddebf45ca7283c5d06705a83d0c14fafcd45d376701120eaba24805093919596 WatchSource:0}: Error finding container ddebf45ca7283c5d06705a83d0c14fafcd45d376701120eaba24805093919596: Status 404 returned error can't find the container with id ddebf45ca7283c5d06705a83d0c14fafcd45d376701120eaba24805093919596 Sep 14 11:16:15.329106 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:15.329075 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39ab34-5449674498-jzrkv" event={"ID":"84925d6e-d030-437c-a141-7168ad5f9f79","Type":"ContainerStarted","Data":"a689cbdf31c5da326430d4016588521ccbe206dec64be7423d304e1ea5068a51"} Sep 14 11:16:15.329561 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:15.329143 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39ab34-5449674498-jzrkv" Sep 14 11:16:15.329561 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:15.329166 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39ab34-5449674498-jzrkv" event={"ID":"84925d6e-d030-437c-a141-7168ad5f9f79","Type":"ContainerStarted","Data":"ddebf45ca7283c5d06705a83d0c14fafcd45d376701120eaba24805093919596"} Sep 14 11:16:15.330685 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:15.330662 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd" event={"ID":"9136cf83-c206-4f67-b1c2-6b3f55eb5d24","Type":"ContainerStarted","Data":"f72d54efcae28c4d9e1b8deb0a44522df8f09ce6e32175d988aa0d2a3b4e37dc"} Sep 14 11:16:15.330787 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:15.330703 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd" event={"ID":"9136cf83-c206-4f67-b1c2-6b3f55eb5d24","Type":"ContainerStarted","Data":"71faf607a67c9ebd676f23968f7d5682f613eb1aa7a7a93f31fa7d61a980a9db"} Sep 14 11:16:15.330787 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:15.330747 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd" Sep 14 11:16:15.351789 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:15.351741 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39ab34-5449674498-jzrkv" podStartSLOduration=1.351722691 podStartE2EDuration="1.351722691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:16:15.348001336 +0000 UTC m=+919.968213858" watchObservedRunningTime="2026-09-14 11:16:15.351722691 +0000 UTC m=+919.971935216" Sep 14 11:16:15.371797 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:15.371755 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd" podStartSLOduration=2.031996765 podStartE2EDuration="2.371740087s" totalImagesPullingTime="339.743322ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:16:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:16:15.370490708 +0000 UTC m=+919.990703257" watchObservedRunningTime="2026-09-14 11:16:15.371740087 +0000 UTC m=+919.991952610" Sep 14 11:16:21.342403 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:21.342373 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd" Sep 14 11:16:26.338797 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:26.338765 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39ab34-5449674498-jzrkv" Sep 14 11:16:31.357830 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:31.357799 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1ac71380-59db949c9-2bjc2"] Sep 14 11:16:31.362817 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:31.362795 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1ac71380-59db949c9-2bjc2" Sep 14 11:16:31.366615 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:31.366594 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1ac71380\"" Sep 14 11:16:31.366877 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:31.366659 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1ac71380-dockercfg-k2gvr\"" Sep 14 11:16:31.376184 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:31.376156 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1ac71380-59db949c9-2bjc2"] Sep 14 11:16:31.458248 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:31.458212 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twnl6\" (UniqueName: \"kubernetes.io/projected/99b23390-0c7c-4213-9e5f-00643b8c78ca-kube-api-access-twnl6\") pod \"payload-processing-e2e-ait-1ac71380-59db949c9-2bjc2\" (UID: \"99b23390-0c7c-4213-9e5f-00643b8c78ca\") " pod="openshift-ingress/payload-processing-e2e-ait-1ac71380-59db949c9-2bjc2" Sep 14 11:16:31.458432 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:31.458410 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99b23390-0c7c-4213-9e5f-00643b8c78ca-plugins-config-volume\") pod \"payload-processing-e2e-ait-1ac71380-59db949c9-2bjc2\" (UID: \"99b23390-0c7c-4213-9e5f-00643b8c78ca\") " pod="openshift-ingress/payload-processing-e2e-ait-1ac71380-59db949c9-2bjc2" Sep 14 11:16:31.559711 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:31.559677 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99b23390-0c7c-4213-9e5f-00643b8c78ca-plugins-config-volume\") pod \"payload-processing-e2e-ait-1ac71380-59db949c9-2bjc2\" (UID: \"99b23390-0c7c-4213-9e5f-00643b8c78ca\") " pod="openshift-ingress/payload-processing-e2e-ait-1ac71380-59db949c9-2bjc2" Sep 14 11:16:31.559876 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:31.559721 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twnl6\" (UniqueName: \"kubernetes.io/projected/99b23390-0c7c-4213-9e5f-00643b8c78ca-kube-api-access-twnl6\") pod \"payload-processing-e2e-ait-1ac71380-59db949c9-2bjc2\" (UID: \"99b23390-0c7c-4213-9e5f-00643b8c78ca\") " pod="openshift-ingress/payload-processing-e2e-ait-1ac71380-59db949c9-2bjc2" Sep 14 11:16:31.560382 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:31.560362 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99b23390-0c7c-4213-9e5f-00643b8c78ca-plugins-config-volume\") pod \"payload-processing-e2e-ait-1ac71380-59db949c9-2bjc2\" (UID: \"99b23390-0c7c-4213-9e5f-00643b8c78ca\") " pod="openshift-ingress/payload-processing-e2e-ait-1ac71380-59db949c9-2bjc2" Sep 14 11:16:31.581552 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:31.581517 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twnl6\" (UniqueName: \"kubernetes.io/projected/99b23390-0c7c-4213-9e5f-00643b8c78ca-kube-api-access-twnl6\") pod \"payload-processing-e2e-ait-1ac71380-59db949c9-2bjc2\" (UID: \"99b23390-0c7c-4213-9e5f-00643b8c78ca\") " pod="openshift-ingress/payload-processing-e2e-ait-1ac71380-59db949c9-2bjc2" Sep 14 11:16:31.676312 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:31.676292 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1ac71380-59db949c9-2bjc2" Sep 14 11:16:31.968065 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:31.968041 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1ac71380-59db949c9-2bjc2"] Sep 14 11:16:31.970222 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:16:31.970194 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99b23390_0c7c_4213_9e5f_00643b8c78ca.slice/crio-fa45ff0286db233ea8b10188ef66514b8df6c712ebc351b84b2fffd3da54e28d WatchSource:0}: Error finding container fa45ff0286db233ea8b10188ef66514b8df6c712ebc351b84b2fffd3da54e28d: Status 404 returned error can't find the container with id fa45ff0286db233ea8b10188ef66514b8df6c712ebc351b84b2fffd3da54e28d Sep 14 11:16:32.438326 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:32.438293 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1ac71380-59db949c9-2bjc2" event={"ID":"99b23390-0c7c-4213-9e5f-00643b8c78ca","Type":"ContainerStarted","Data":"a6781911f7a50ed46526600eb0ebe90c278d2bc6243a73e9ac654f31422250d8"} Sep 14 11:16:32.438722 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:32.438358 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1ac71380-59db949c9-2bjc2" event={"ID":"99b23390-0c7c-4213-9e5f-00643b8c78ca","Type":"ContainerStarted","Data":"fa45ff0286db233ea8b10188ef66514b8df6c712ebc351b84b2fffd3da54e28d"} Sep 14 11:16:32.438722 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:32.438495 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-1ac71380-59db949c9-2bjc2" Sep 14 11:16:32.465502 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:32.465453 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-1ac71380-59db949c9-2bjc2" podStartSLOduration=1.465439714 podStartE2EDuration="1.465439714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:16:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:16:32.463185443 +0000 UTC m=+937.083397987" watchObservedRunningTime="2026-09-14 11:16:32.465439714 +0000 UTC m=+937.085652236" Sep 14 11:16:43.445774 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:16:43.445744 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-1ac71380-59db949c9-2bjc2" Sep 14 11:17:07.088875 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:07.088839 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1ac71380-59db949c9-2bjc2"] Sep 14 11:17:07.089486 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:07.089074 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-1ac71380-59db949c9-2bjc2" podUID="99b23390-0c7c-4213-9e5f-00643b8c78ca" containerName="payload-processing" containerID="cri-o://a6781911f7a50ed46526600eb0ebe90c278d2bc6243a73e9ac654f31422250d8" gracePeriod=30 Sep 14 11:17:07.360426 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:07.360397 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1ac71380-59db949c9-2bjc2" Sep 14 11:17:07.505940 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:07.505902 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99b23390-0c7c-4213-9e5f-00643b8c78ca-plugins-config-volume\") pod \"99b23390-0c7c-4213-9e5f-00643b8c78ca\" (UID: \"99b23390-0c7c-4213-9e5f-00643b8c78ca\") " Sep 14 11:17:07.506117 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:07.505994 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twnl6\" (UniqueName: \"kubernetes.io/projected/99b23390-0c7c-4213-9e5f-00643b8c78ca-kube-api-access-twnl6\") pod \"99b23390-0c7c-4213-9e5f-00643b8c78ca\" (UID: \"99b23390-0c7c-4213-9e5f-00643b8c78ca\") " Sep 14 11:17:07.506385 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:07.506351 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99b23390-0c7c-4213-9e5f-00643b8c78ca-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "99b23390-0c7c-4213-9e5f-00643b8c78ca" (UID: "99b23390-0c7c-4213-9e5f-00643b8c78ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:17:07.508798 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:07.508766 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99b23390-0c7c-4213-9e5f-00643b8c78ca-kube-api-access-twnl6" (OuterVolumeSpecName: "kube-api-access-twnl6") pod "99b23390-0c7c-4213-9e5f-00643b8c78ca" (UID: "99b23390-0c7c-4213-9e5f-00643b8c78ca"). InnerVolumeSpecName "kube-api-access-twnl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:17:07.607537 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:07.607448 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99b23390-0c7c-4213-9e5f-00643b8c78ca-plugins-config-volume\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:17:07.607537 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:07.607484 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twnl6\" (UniqueName: \"kubernetes.io/projected/99b23390-0c7c-4213-9e5f-00643b8c78ca-kube-api-access-twnl6\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:17:07.640719 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:07.640681 2574 generic.go:356] "Generic (PLEG): container finished" podID="99b23390-0c7c-4213-9e5f-00643b8c78ca" containerID="a6781911f7a50ed46526600eb0ebe90c278d2bc6243a73e9ac654f31422250d8" exitCode=0 Sep 14 11:17:07.640895 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:07.640746 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1ac71380-59db949c9-2bjc2" Sep 14 11:17:07.640895 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:07.640763 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1ac71380-59db949c9-2bjc2" event={"ID":"99b23390-0c7c-4213-9e5f-00643b8c78ca","Type":"ContainerDied","Data":"a6781911f7a50ed46526600eb0ebe90c278d2bc6243a73e9ac654f31422250d8"} Sep 14 11:17:07.640895 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:07.640847 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1ac71380-59db949c9-2bjc2" event={"ID":"99b23390-0c7c-4213-9e5f-00643b8c78ca","Type":"ContainerDied","Data":"fa45ff0286db233ea8b10188ef66514b8df6c712ebc351b84b2fffd3da54e28d"} Sep 14 11:17:07.640895 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:07.640872 2574 scope.go:117] "RemoveContainer" containerID="a6781911f7a50ed46526600eb0ebe90c278d2bc6243a73e9ac654f31422250d8" Sep 14 11:17:07.650936 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:07.650919 2574 scope.go:117] "RemoveContainer" containerID="a6781911f7a50ed46526600eb0ebe90c278d2bc6243a73e9ac654f31422250d8" Sep 14 11:17:07.651280 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:17:07.651240 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6781911f7a50ed46526600eb0ebe90c278d2bc6243a73e9ac654f31422250d8\": container with ID starting with a6781911f7a50ed46526600eb0ebe90c278d2bc6243a73e9ac654f31422250d8 not found: ID does not exist" containerID="a6781911f7a50ed46526600eb0ebe90c278d2bc6243a73e9ac654f31422250d8" Sep 14 11:17:07.651396 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:07.651298 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6781911f7a50ed46526600eb0ebe90c278d2bc6243a73e9ac654f31422250d8"} err="failed to get container status \"a6781911f7a50ed46526600eb0ebe90c278d2bc6243a73e9ac654f31422250d8\": rpc error: code = NotFound desc = could not find container \"a6781911f7a50ed46526600eb0ebe90c278d2bc6243a73e9ac654f31422250d8\": container with ID starting with a6781911f7a50ed46526600eb0ebe90c278d2bc6243a73e9ac654f31422250d8 not found: ID does not exist" Sep 14 11:17:07.671913 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:07.671887 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1ac71380-59db949c9-2bjc2"] Sep 14 11:17:07.678535 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:07.678514 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1ac71380-59db949c9-2bjc2"] Sep 14 11:17:07.900778 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:07.900706 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99b23390-0c7c-4213-9e5f-00643b8c78ca" path="/var/lib/kubelet/pods/99b23390-0c7c-4213-9e5f-00643b8c78ca/volumes" Sep 14 11:17:15.082546 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:15.082500 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c652e0cd-7d7f44bdcd-kcqmv"] Sep 14 11:17:15.083162 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:15.083143 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99b23390-0c7c-4213-9e5f-00643b8c78ca" containerName="payload-processing" Sep 14 11:17:15.083209 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:15.083165 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="99b23390-0c7c-4213-9e5f-00643b8c78ca" containerName="payload-processing" Sep 14 11:17:15.083247 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:15.083237 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="99b23390-0c7c-4213-9e5f-00643b8c78ca" containerName="payload-processing" Sep 14 11:17:15.087675 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:15.087656 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c652e0cd-7d7f44bdcd-kcqmv" Sep 14 11:17:15.090375 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:15.090350 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-c652e0cd-dockercfg-kc7cw\"" Sep 14 11:17:15.090375 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:15.090373 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-c652e0cd\"" Sep 14 11:17:15.104979 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:15.104956 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c652e0cd-7d7f44bdcd-kcqmv"] Sep 14 11:17:15.164997 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:15.164969 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c652e0cd-6b86796b66-sv67x"] Sep 14 11:17:15.168597 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:15.168581 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c652e0cd-6b86796b66-sv67x" Sep 14 11:17:15.181901 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:15.181877 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c652e0cd-6b86796b66-sv67x"] Sep 14 11:17:15.284303 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:15.284254 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2244g\" (UniqueName: \"kubernetes.io/projected/4b87a93f-78ae-44f6-bfd5-6b555f8da3b1-kube-api-access-2244g\") pod \"payload-pre-processing-e2e-migrate-c652e0cd-7d7f44bdcd-kcqmv\" (UID: \"4b87a93f-78ae-44f6-bfd5-6b555f8da3b1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c652e0cd-7d7f44bdcd-kcqmv" Sep 14 11:17:15.284575 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:15.284369 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a8742e8-21b6-41e3-a0eb-b6611f72c0a7-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c652e0cd-6b86796b66-sv67x\" (UID: \"1a8742e8-21b6-41e3-a0eb-b6611f72c0a7\") " pod="openshift-ingress/payload-processing-e2e-migrate-c652e0cd-6b86796b66-sv67x" Sep 14 11:17:15.284575 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:15.284416 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8ngg\" (UniqueName: \"kubernetes.io/projected/1a8742e8-21b6-41e3-a0eb-b6611f72c0a7-kube-api-access-q8ngg\") pod \"payload-processing-e2e-migrate-c652e0cd-6b86796b66-sv67x\" (UID: \"1a8742e8-21b6-41e3-a0eb-b6611f72c0a7\") " pod="openshift-ingress/payload-processing-e2e-migrate-c652e0cd-6b86796b66-sv67x" Sep 14 11:17:15.284575 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:15.284444 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b87a93f-78ae-44f6-bfd5-6b555f8da3b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c652e0cd-7d7f44bdcd-kcqmv\" (UID: \"4b87a93f-78ae-44f6-bfd5-6b555f8da3b1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c652e0cd-7d7f44bdcd-kcqmv" Sep 14 11:17:15.385620 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:15.385535 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a8742e8-21b6-41e3-a0eb-b6611f72c0a7-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c652e0cd-6b86796b66-sv67x\" (UID: \"1a8742e8-21b6-41e3-a0eb-b6611f72c0a7\") " pod="openshift-ingress/payload-processing-e2e-migrate-c652e0cd-6b86796b66-sv67x" Sep 14 11:17:15.385620 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:15.385592 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8ngg\" (UniqueName: \"kubernetes.io/projected/1a8742e8-21b6-41e3-a0eb-b6611f72c0a7-kube-api-access-q8ngg\") pod \"payload-processing-e2e-migrate-c652e0cd-6b86796b66-sv67x\" (UID: \"1a8742e8-21b6-41e3-a0eb-b6611f72c0a7\") " pod="openshift-ingress/payload-processing-e2e-migrate-c652e0cd-6b86796b66-sv67x" Sep 14 11:17:15.385620 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:15.385616 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b87a93f-78ae-44f6-bfd5-6b555f8da3b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c652e0cd-7d7f44bdcd-kcqmv\" (UID: \"4b87a93f-78ae-44f6-bfd5-6b555f8da3b1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c652e0cd-7d7f44bdcd-kcqmv" Sep 14 11:17:15.385841 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:15.385693 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2244g\" (UniqueName: \"kubernetes.io/projected/4b87a93f-78ae-44f6-bfd5-6b555f8da3b1-kube-api-access-2244g\") pod \"payload-pre-processing-e2e-migrate-c652e0cd-7d7f44bdcd-kcqmv\" (UID: \"4b87a93f-78ae-44f6-bfd5-6b555f8da3b1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c652e0cd-7d7f44bdcd-kcqmv" Sep 14 11:17:15.386289 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:15.386243 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a8742e8-21b6-41e3-a0eb-b6611f72c0a7-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c652e0cd-6b86796b66-sv67x\" (UID: \"1a8742e8-21b6-41e3-a0eb-b6611f72c0a7\") " pod="openshift-ingress/payload-processing-e2e-migrate-c652e0cd-6b86796b66-sv67x" Sep 14 11:17:15.386451 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:15.386432 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b87a93f-78ae-44f6-bfd5-6b555f8da3b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c652e0cd-7d7f44bdcd-kcqmv\" (UID: \"4b87a93f-78ae-44f6-bfd5-6b555f8da3b1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c652e0cd-7d7f44bdcd-kcqmv" Sep 14 11:17:15.394184 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:15.394160 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2244g\" (UniqueName: \"kubernetes.io/projected/4b87a93f-78ae-44f6-bfd5-6b555f8da3b1-kube-api-access-2244g\") pod \"payload-pre-processing-e2e-migrate-c652e0cd-7d7f44bdcd-kcqmv\" (UID: \"4b87a93f-78ae-44f6-bfd5-6b555f8da3b1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c652e0cd-7d7f44bdcd-kcqmv" Sep 14 11:17:15.394307 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:15.394287 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8ngg\" (UniqueName: \"kubernetes.io/projected/1a8742e8-21b6-41e3-a0eb-b6611f72c0a7-kube-api-access-q8ngg\") pod \"payload-processing-e2e-migrate-c652e0cd-6b86796b66-sv67x\" (UID: \"1a8742e8-21b6-41e3-a0eb-b6611f72c0a7\") " pod="openshift-ingress/payload-processing-e2e-migrate-c652e0cd-6b86796b66-sv67x" Sep 14 11:17:15.398536 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:15.398520 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c652e0cd-7d7f44bdcd-kcqmv" Sep 14 11:17:15.478891 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:15.478863 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c652e0cd-6b86796b66-sv67x" Sep 14 11:17:15.679096 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:15.679067 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c652e0cd-7d7f44bdcd-kcqmv"] Sep 14 11:17:15.680260 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:17:15.680233 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b87a93f_78ae_44f6_bfd5_6b555f8da3b1.slice/crio-2e14f8c6042b9ea6a2128a254b7247d89eeafd539c0411d4aefbbd8299d81f6d WatchSource:0}: Error finding container 2e14f8c6042b9ea6a2128a254b7247d89eeafd539c0411d4aefbbd8299d81f6d: Status 404 returned error can't find the container with id 2e14f8c6042b9ea6a2128a254b7247d89eeafd539c0411d4aefbbd8299d81f6d Sep 14 11:17:15.763196 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:17:15.763171 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8742e8_21b6_41e3_a0eb_b6611f72c0a7.slice/crio-c0b27ce1aa2921df612dbea1d7ac934177c4a83e2ce9bf904bdcb1868f734b62 WatchSource:0}: Error finding container c0b27ce1aa2921df612dbea1d7ac934177c4a83e2ce9bf904bdcb1868f734b62: Status 404 returned error can't find the container with id c0b27ce1aa2921df612dbea1d7ac934177c4a83e2ce9bf904bdcb1868f734b62 Sep 14 11:17:15.763328 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:15.763304 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c652e0cd-6b86796b66-sv67x"] Sep 14 11:17:16.693089 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:16.692991 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c652e0cd-7d7f44bdcd-kcqmv" event={"ID":"4b87a93f-78ae-44f6-bfd5-6b555f8da3b1","Type":"ContainerStarted","Data":"77a125dc9d004731f7cb129cdafc6300dae8affaa10bdb675b837eea57698b99"} Sep 14 11:17:16.693089 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:16.693066 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c652e0cd-7d7f44bdcd-kcqmv" Sep 14 11:17:16.693538 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:16.693092 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c652e0cd-7d7f44bdcd-kcqmv" event={"ID":"4b87a93f-78ae-44f6-bfd5-6b555f8da3b1","Type":"ContainerStarted","Data":"2e14f8c6042b9ea6a2128a254b7247d89eeafd539c0411d4aefbbd8299d81f6d"} Sep 14 11:17:16.694706 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:16.694681 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c652e0cd-6b86796b66-sv67x" event={"ID":"1a8742e8-21b6-41e3-a0eb-b6611f72c0a7","Type":"ContainerStarted","Data":"1b97cefd3794edc5b6d7a24fbc9b768f9d2f4d8a5691ed3068902b390f868142"} Sep 14 11:17:16.694824 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:16.694730 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c652e0cd-6b86796b66-sv67x" event={"ID":"1a8742e8-21b6-41e3-a0eb-b6611f72c0a7","Type":"ContainerStarted","Data":"c0b27ce1aa2921df612dbea1d7ac934177c4a83e2ce9bf904bdcb1868f734b62"} Sep 14 11:17:16.694824 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:16.694759 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-c652e0cd-6b86796b66-sv67x" Sep 14 11:17:16.714782 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:16.714726 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c652e0cd-7d7f44bdcd-kcqmv" podStartSLOduration=1.714705331 podStartE2EDuration="1.714705331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:17:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:17:16.714184769 +0000 UTC m=+981.334397316" watchObservedRunningTime="2026-09-14 11:17:16.714705331 +0000 UTC m=+981.334917862" Sep 14 11:17:16.735201 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:16.735151 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-c652e0cd-6b86796b66-sv67x" podStartSLOduration=1.735131006 podStartE2EDuration="1.735131006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:17:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:17:16.731668095 +0000 UTC m=+981.351880650" watchObservedRunningTime="2026-09-14 11:17:16.735131006 +0000 UTC m=+981.355343534" Sep 14 11:17:27.702484 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:27.702451 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c652e0cd-7d7f44bdcd-kcqmv" Sep 14 11:17:27.702869 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:27.702813 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-c652e0cd-6b86796b66-sv67x" Sep 14 11:17:55.674051 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:55.674015 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c652e0cd-6b86796b66-sv67x"] Sep 14 11:17:55.674670 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:55.674363 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-c652e0cd-6b86796b66-sv67x" podUID="1a8742e8-21b6-41e3-a0eb-b6611f72c0a7" containerName="payload-processing" containerID="cri-o://1b97cefd3794edc5b6d7a24fbc9b768f9d2f4d8a5691ed3068902b390f868142" gracePeriod=30 Sep 14 11:17:55.687037 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:55.687009 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c652e0cd-7d7f44bdcd-kcqmv"] Sep 14 11:17:55.687518 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:55.687454 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c652e0cd-7d7f44bdcd-kcqmv" podUID="4b87a93f-78ae-44f6-bfd5-6b555f8da3b1" containerName="payload-pre-processing" containerID="cri-o://77a125dc9d004731f7cb129cdafc6300dae8affaa10bdb675b837eea57698b99" gracePeriod=30 Sep 14 11:17:55.926748 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:55.926639 2574 generic.go:356] "Generic (PLEG): container finished" podID="1a8742e8-21b6-41e3-a0eb-b6611f72c0a7" containerID="1b97cefd3794edc5b6d7a24fbc9b768f9d2f4d8a5691ed3068902b390f868142" exitCode=0 Sep 14 11:17:55.927010 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:55.926736 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c652e0cd-6b86796b66-sv67x" event={"ID":"1a8742e8-21b6-41e3-a0eb-b6611f72c0a7","Type":"ContainerDied","Data":"1b97cefd3794edc5b6d7a24fbc9b768f9d2f4d8a5691ed3068902b390f868142"} Sep 14 11:17:55.931691 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:55.931658 2574 generic.go:356] "Generic (PLEG): container finished" podID="4b87a93f-78ae-44f6-bfd5-6b555f8da3b1" containerID="77a125dc9d004731f7cb129cdafc6300dae8affaa10bdb675b837eea57698b99" exitCode=0 Sep 14 11:17:55.931691 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:55.931691 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c652e0cd-7d7f44bdcd-kcqmv" event={"ID":"4b87a93f-78ae-44f6-bfd5-6b555f8da3b1","Type":"ContainerDied","Data":"77a125dc9d004731f7cb129cdafc6300dae8affaa10bdb675b837eea57698b99"} Sep 14 11:17:55.979780 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:55.979755 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c652e0cd-6b86796b66-sv67x" Sep 14 11:17:55.983097 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:55.983079 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c652e0cd-7d7f44bdcd-kcqmv" Sep 14 11:17:56.078741 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:56.078709 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a8742e8-21b6-41e3-a0eb-b6611f72c0a7-plugins-config-volume\") pod \"1a8742e8-21b6-41e3-a0eb-b6611f72c0a7\" (UID: \"1a8742e8-21b6-41e3-a0eb-b6611f72c0a7\") " Sep 14 11:17:56.078904 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:56.078748 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2244g\" (UniqueName: \"kubernetes.io/projected/4b87a93f-78ae-44f6-bfd5-6b555f8da3b1-kube-api-access-2244g\") pod \"4b87a93f-78ae-44f6-bfd5-6b555f8da3b1\" (UID: \"4b87a93f-78ae-44f6-bfd5-6b555f8da3b1\") " Sep 14 11:17:56.078904 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:56.078826 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8ngg\" (UniqueName: \"kubernetes.io/projected/1a8742e8-21b6-41e3-a0eb-b6611f72c0a7-kube-api-access-q8ngg\") pod \"1a8742e8-21b6-41e3-a0eb-b6611f72c0a7\" (UID: \"1a8742e8-21b6-41e3-a0eb-b6611f72c0a7\") " Sep 14 11:17:56.078904 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:56.078900 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b87a93f-78ae-44f6-bfd5-6b555f8da3b1-plugins-config-volume\") pod \"4b87a93f-78ae-44f6-bfd5-6b555f8da3b1\" (UID: \"4b87a93f-78ae-44f6-bfd5-6b555f8da3b1\") " Sep 14 11:17:56.079142 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:56.079106 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a8742e8-21b6-41e3-a0eb-b6611f72c0a7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1a8742e8-21b6-41e3-a0eb-b6611f72c0a7" (UID: "1a8742e8-21b6-41e3-a0eb-b6611f72c0a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:17:56.079331 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:56.079308 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b87a93f-78ae-44f6-bfd5-6b555f8da3b1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4b87a93f-78ae-44f6-bfd5-6b555f8da3b1" (UID: "4b87a93f-78ae-44f6-bfd5-6b555f8da3b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:17:56.080984 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:56.080956 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b87a93f-78ae-44f6-bfd5-6b555f8da3b1-kube-api-access-2244g" (OuterVolumeSpecName: "kube-api-access-2244g") pod "4b87a93f-78ae-44f6-bfd5-6b555f8da3b1" (UID: "4b87a93f-78ae-44f6-bfd5-6b555f8da3b1"). InnerVolumeSpecName "kube-api-access-2244g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:17:56.081073 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:56.081037 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a8742e8-21b6-41e3-a0eb-b6611f72c0a7-kube-api-access-q8ngg" (OuterVolumeSpecName: "kube-api-access-q8ngg") pod "1a8742e8-21b6-41e3-a0eb-b6611f72c0a7" (UID: "1a8742e8-21b6-41e3-a0eb-b6611f72c0a7"). InnerVolumeSpecName "kube-api-access-q8ngg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:17:56.180111 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:56.180033 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8ngg\" (UniqueName: \"kubernetes.io/projected/1a8742e8-21b6-41e3-a0eb-b6611f72c0a7-kube-api-access-q8ngg\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:17:56.180111 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:56.180059 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b87a93f-78ae-44f6-bfd5-6b555f8da3b1-plugins-config-volume\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:17:56.180111 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:56.180068 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a8742e8-21b6-41e3-a0eb-b6611f72c0a7-plugins-config-volume\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:17:56.180111 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:56.180079 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2244g\" (UniqueName: \"kubernetes.io/projected/4b87a93f-78ae-44f6-bfd5-6b555f8da3b1-kube-api-access-2244g\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:17:56.942516 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:56.942487 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c652e0cd-7d7f44bdcd-kcqmv" Sep 14 11:17:56.943076 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:56.942502 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c652e0cd-7d7f44bdcd-kcqmv" event={"ID":"4b87a93f-78ae-44f6-bfd5-6b555f8da3b1","Type":"ContainerDied","Data":"2e14f8c6042b9ea6a2128a254b7247d89eeafd539c0411d4aefbbd8299d81f6d"} Sep 14 11:17:56.943231 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:56.943124 2574 scope.go:117] "RemoveContainer" containerID="77a125dc9d004731f7cb129cdafc6300dae8affaa10bdb675b837eea57698b99" Sep 14 11:17:56.944161 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:56.944140 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c652e0cd-6b86796b66-sv67x" event={"ID":"1a8742e8-21b6-41e3-a0eb-b6611f72c0a7","Type":"ContainerDied","Data":"c0b27ce1aa2921df612dbea1d7ac934177c4a83e2ce9bf904bdcb1868f734b62"} Sep 14 11:17:56.944307 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:56.944166 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c652e0cd-6b86796b66-sv67x" Sep 14 11:17:56.954522 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:56.954504 2574 scope.go:117] "RemoveContainer" containerID="1b97cefd3794edc5b6d7a24fbc9b768f9d2f4d8a5691ed3068902b390f868142" Sep 14 11:17:56.970818 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:56.970798 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c652e0cd-6b86796b66-sv67x"] Sep 14 11:17:56.973143 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:56.973124 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c652e0cd-6b86796b66-sv67x"] Sep 14 11:17:56.984431 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:56.984410 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c652e0cd-7d7f44bdcd-kcqmv"] Sep 14 11:17:56.988129 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:56.988109 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c652e0cd-7d7f44bdcd-kcqmv"] Sep 14 11:17:57.901592 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:57.901556 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a8742e8-21b6-41e3-a0eb-b6611f72c0a7" path="/var/lib/kubelet/pods/1a8742e8-21b6-41e3-a0eb-b6611f72c0a7/volumes" Sep 14 11:17:57.901980 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:17:57.901960 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b87a93f-78ae-44f6-bfd5-6b555f8da3b1" path="/var/lib/kubelet/pods/4b87a93f-78ae-44f6-bfd5-6b555f8da3b1/volumes" Sep 14 11:18:00.907017 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:00.906984 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-27a1a2-5dcd48c789-z6n8r"] Sep 14 11:18:00.907533 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:00.907175 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-27a1a2-5dcd48c789-z6n8r" podUID="8d120622-8756-4487-9798-b99f4d4279f4" containerName="payload-processing" containerID="cri-o://a2e1b947e0614ef58063d9f67fe5e3d4004b3d0cd363426bdd2c2fc2cc9edfc5" gracePeriod=30 Sep 14 11:18:01.193125 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:01.193097 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-27a1a2-5dcd48c789-z6n8r" Sep 14 11:18:01.233878 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:01.233827 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgpwv\" (UniqueName: \"kubernetes.io/projected/8d120622-8756-4487-9798-b99f4d4279f4-kube-api-access-fgpwv\") pod \"8d120622-8756-4487-9798-b99f4d4279f4\" (UID: \"8d120622-8756-4487-9798-b99f4d4279f4\") " Sep 14 11:18:01.234079 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:01.233996 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d120622-8756-4487-9798-b99f4d4279f4-plugins-config-volume\") pod \"8d120622-8756-4487-9798-b99f4d4279f4\" (UID: \"8d120622-8756-4487-9798-b99f4d4279f4\") " Sep 14 11:18:01.234890 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:01.234710 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d120622-8756-4487-9798-b99f4d4279f4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8d120622-8756-4487-9798-b99f4d4279f4" (UID: "8d120622-8756-4487-9798-b99f4d4279f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:18:01.237135 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:01.237092 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d120622-8756-4487-9798-b99f4d4279f4-kube-api-access-fgpwv" (OuterVolumeSpecName: "kube-api-access-fgpwv") pod "8d120622-8756-4487-9798-b99f4d4279f4" (UID: "8d120622-8756-4487-9798-b99f4d4279f4"). InnerVolumeSpecName "kube-api-access-fgpwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:18:01.335019 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:01.334983 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d120622-8756-4487-9798-b99f4d4279f4-plugins-config-volume\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:18:01.335019 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:01.335012 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgpwv\" (UniqueName: \"kubernetes.io/projected/8d120622-8756-4487-9798-b99f4d4279f4-kube-api-access-fgpwv\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:18:01.976113 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:01.976076 2574 generic.go:356] "Generic (PLEG): container finished" podID="8d120622-8756-4487-9798-b99f4d4279f4" containerID="a2e1b947e0614ef58063d9f67fe5e3d4004b3d0cd363426bdd2c2fc2cc9edfc5" exitCode=0 Sep 14 11:18:01.976638 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:01.976144 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-27a1a2-5dcd48c789-z6n8r" Sep 14 11:18:01.976638 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:01.976164 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-27a1a2-5dcd48c789-z6n8r" event={"ID":"8d120622-8756-4487-9798-b99f4d4279f4","Type":"ContainerDied","Data":"a2e1b947e0614ef58063d9f67fe5e3d4004b3d0cd363426bdd2c2fc2cc9edfc5"} Sep 14 11:18:01.976638 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:01.976246 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-27a1a2-5dcd48c789-z6n8r" event={"ID":"8d120622-8756-4487-9798-b99f4d4279f4","Type":"ContainerDied","Data":"654adc6662d13cc781b96e449ae38ceb720bf7e41810126b073fe11fd57e0830"} Sep 14 11:18:01.976638 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:01.976307 2574 scope.go:117] "RemoveContainer" containerID="a2e1b947e0614ef58063d9f67fe5e3d4004b3d0cd363426bdd2c2fc2cc9edfc5" Sep 14 11:18:01.985995 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:01.985978 2574 scope.go:117] "RemoveContainer" containerID="a2e1b947e0614ef58063d9f67fe5e3d4004b3d0cd363426bdd2c2fc2cc9edfc5" Sep 14 11:18:01.986321 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:18:01.986294 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2e1b947e0614ef58063d9f67fe5e3d4004b3d0cd363426bdd2c2fc2cc9edfc5\": container with ID starting with a2e1b947e0614ef58063d9f67fe5e3d4004b3d0cd363426bdd2c2fc2cc9edfc5 not found: ID does not exist" containerID="a2e1b947e0614ef58063d9f67fe5e3d4004b3d0cd363426bdd2c2fc2cc9edfc5" Sep 14 11:18:01.986397 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:01.986334 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2e1b947e0614ef58063d9f67fe5e3d4004b3d0cd363426bdd2c2fc2cc9edfc5"} err="failed to get container status \"a2e1b947e0614ef58063d9f67fe5e3d4004b3d0cd363426bdd2c2fc2cc9edfc5\": rpc error: code = NotFound desc = could not find container \"a2e1b947e0614ef58063d9f67fe5e3d4004b3d0cd363426bdd2c2fc2cc9edfc5\": container with ID starting with a2e1b947e0614ef58063d9f67fe5e3d4004b3d0cd363426bdd2c2fc2cc9edfc5 not found: ID does not exist" Sep 14 11:18:02.003029 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:02.002995 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-27a1a2-5dcd48c789-z6n8r"] Sep 14 11:18:02.010418 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:02.010391 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-27a1a2-5dcd48c789-z6n8r"] Sep 14 11:18:03.898960 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:03.898926 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d120622-8756-4487-9798-b99f4d4279f4" path="/var/lib/kubelet/pods/8d120622-8756-4487-9798-b99f4d4279f4/volumes" Sep 14 11:18:03.935388 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:03.935362 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-88bdd8bf-697785c6f8-gtm6q"] Sep 14 11:18:03.935823 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:03.935811 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b87a93f-78ae-44f6-bfd5-6b555f8da3b1" containerName="payload-pre-processing" Sep 14 11:18:03.935873 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:03.935826 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b87a93f-78ae-44f6-bfd5-6b555f8da3b1" containerName="payload-pre-processing" Sep 14 11:18:03.935873 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:03.935840 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d120622-8756-4487-9798-b99f4d4279f4" containerName="payload-processing" Sep 14 11:18:03.935873 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:03.935846 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d120622-8756-4487-9798-b99f4d4279f4" containerName="payload-processing" Sep 14 11:18:03.935873 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:03.935853 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a8742e8-21b6-41e3-a0eb-b6611f72c0a7" containerName="payload-processing" Sep 14 11:18:03.935873 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:03.935859 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a8742e8-21b6-41e3-a0eb-b6611f72c0a7" containerName="payload-processing" Sep 14 11:18:03.936031 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:03.935943 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b87a93f-78ae-44f6-bfd5-6b555f8da3b1" containerName="payload-pre-processing" Sep 14 11:18:03.936031 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:03.935950 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a8742e8-21b6-41e3-a0eb-b6611f72c0a7" containerName="payload-processing" Sep 14 11:18:03.936031 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:03.935959 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d120622-8756-4487-9798-b99f4d4279f4" containerName="payload-processing" Sep 14 11:18:03.940639 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:03.940621 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-88bdd8bf-697785c6f8-gtm6q" Sep 14 11:18:03.944475 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:03.944455 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-88bdd8bf-dockercfg-x8qpz\"" Sep 14 11:18:03.944774 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:03.944756 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-88bdd8bf\"" Sep 14 11:18:03.961936 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:03.961916 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt75n\" (UniqueName: \"kubernetes.io/projected/3de628bf-a52d-4d22-9b34-2bc5742e88f4-kube-api-access-bt75n\") pod \"payload-pre-processing-e2e-ait-88bdd8bf-697785c6f8-gtm6q\" (UID: \"3de628bf-a52d-4d22-9b34-2bc5742e88f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-88bdd8bf-697785c6f8-gtm6q" Sep 14 11:18:03.962040 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:03.961955 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3de628bf-a52d-4d22-9b34-2bc5742e88f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-88bdd8bf-697785c6f8-gtm6q\" (UID: \"3de628bf-a52d-4d22-9b34-2bc5742e88f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-88bdd8bf-697785c6f8-gtm6q" Sep 14 11:18:04.039139 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:04.039112 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-88bdd8bf-697785c6f8-gtm6q"] Sep 14 11:18:04.062561 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:04.062529 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt75n\" (UniqueName: \"kubernetes.io/projected/3de628bf-a52d-4d22-9b34-2bc5742e88f4-kube-api-access-bt75n\") pod \"payload-pre-processing-e2e-ait-88bdd8bf-697785c6f8-gtm6q\" (UID: \"3de628bf-a52d-4d22-9b34-2bc5742e88f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-88bdd8bf-697785c6f8-gtm6q" Sep 14 11:18:04.062730 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:04.062571 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3de628bf-a52d-4d22-9b34-2bc5742e88f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-88bdd8bf-697785c6f8-gtm6q\" (UID: \"3de628bf-a52d-4d22-9b34-2bc5742e88f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-88bdd8bf-697785c6f8-gtm6q" Sep 14 11:18:04.063358 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:04.063333 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3de628bf-a52d-4d22-9b34-2bc5742e88f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-88bdd8bf-697785c6f8-gtm6q\" (UID: \"3de628bf-a52d-4d22-9b34-2bc5742e88f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-88bdd8bf-697785c6f8-gtm6q" Sep 14 11:18:04.075487 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:04.075460 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt75n\" (UniqueName: \"kubernetes.io/projected/3de628bf-a52d-4d22-9b34-2bc5742e88f4-kube-api-access-bt75n\") pod \"payload-pre-processing-e2e-ait-88bdd8bf-697785c6f8-gtm6q\" (UID: \"3de628bf-a52d-4d22-9b34-2bc5742e88f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-88bdd8bf-697785c6f8-gtm6q" Sep 14 11:18:04.108474 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:04.108446 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-88bdd8bf-8649d85875-lm74z"] Sep 14 11:18:04.112323 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:04.112306 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-88bdd8bf-8649d85875-lm74z" Sep 14 11:18:04.134692 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:04.134665 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-88bdd8bf-8649d85875-lm74z"] Sep 14 11:18:04.163246 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:04.163178 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kx4j\" (UniqueName: \"kubernetes.io/projected/97c9e671-bc21-4197-9689-f06779a12978-kube-api-access-7kx4j\") pod \"payload-processing-e2e-ait-88bdd8bf-8649d85875-lm74z\" (UID: \"97c9e671-bc21-4197-9689-f06779a12978\") " pod="openshift-ingress/payload-processing-e2e-ait-88bdd8bf-8649d85875-lm74z" Sep 14 11:18:04.163409 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:04.163261 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97c9e671-bc21-4197-9689-f06779a12978-plugins-config-volume\") pod \"payload-processing-e2e-ait-88bdd8bf-8649d85875-lm74z\" (UID: \"97c9e671-bc21-4197-9689-f06779a12978\") " pod="openshift-ingress/payload-processing-e2e-ait-88bdd8bf-8649d85875-lm74z" Sep 14 11:18:04.250928 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:04.250900 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-88bdd8bf-697785c6f8-gtm6q" Sep 14 11:18:04.264301 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:04.264255 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97c9e671-bc21-4197-9689-f06779a12978-plugins-config-volume\") pod \"payload-processing-e2e-ait-88bdd8bf-8649d85875-lm74z\" (UID: \"97c9e671-bc21-4197-9689-f06779a12978\") " pod="openshift-ingress/payload-processing-e2e-ait-88bdd8bf-8649d85875-lm74z" Sep 14 11:18:04.264612 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:04.264568 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kx4j\" (UniqueName: \"kubernetes.io/projected/97c9e671-bc21-4197-9689-f06779a12978-kube-api-access-7kx4j\") pod \"payload-processing-e2e-ait-88bdd8bf-8649d85875-lm74z\" (UID: \"97c9e671-bc21-4197-9689-f06779a12978\") " pod="openshift-ingress/payload-processing-e2e-ait-88bdd8bf-8649d85875-lm74z" Sep 14 11:18:04.265243 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:04.265221 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97c9e671-bc21-4197-9689-f06779a12978-plugins-config-volume\") pod \"payload-processing-e2e-ait-88bdd8bf-8649d85875-lm74z\" (UID: \"97c9e671-bc21-4197-9689-f06779a12978\") " pod="openshift-ingress/payload-processing-e2e-ait-88bdd8bf-8649d85875-lm74z" Sep 14 11:18:04.280551 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:04.280494 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kx4j\" (UniqueName: \"kubernetes.io/projected/97c9e671-bc21-4197-9689-f06779a12978-kube-api-access-7kx4j\") pod \"payload-processing-e2e-ait-88bdd8bf-8649d85875-lm74z\" (UID: \"97c9e671-bc21-4197-9689-f06779a12978\") " pod="openshift-ingress/payload-processing-e2e-ait-88bdd8bf-8649d85875-lm74z" Sep 14 11:18:04.423692 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:04.423659 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-88bdd8bf-8649d85875-lm74z" Sep 14 11:18:04.536419 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:04.536396 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-88bdd8bf-697785c6f8-gtm6q"] Sep 14 11:18:04.538200 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:18:04.538178 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3de628bf_a52d_4d22_9b34_2bc5742e88f4.slice/crio-557f3cb44eaf07c80aaa64626d780f55ed2752b94d47fffa71f4c71e7c3ebfb5 WatchSource:0}: Error finding container 557f3cb44eaf07c80aaa64626d780f55ed2752b94d47fffa71f4c71e7c3ebfb5: Status 404 returned error can't find the container with id 557f3cb44eaf07c80aaa64626d780f55ed2752b94d47fffa71f4c71e7c3ebfb5 Sep 14 11:18:04.704253 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:04.704230 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-88bdd8bf-8649d85875-lm74z"] Sep 14 11:18:04.705690 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:18:04.705664 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97c9e671_bc21_4197_9689_f06779a12978.slice/crio-45ef296e0a1a5e8b99ae04ebdfc5291e684ce6cc000f4b86ea0615b19a1bf5a4 WatchSource:0}: Error finding container 45ef296e0a1a5e8b99ae04ebdfc5291e684ce6cc000f4b86ea0615b19a1bf5a4: Status 404 returned error can't find the container with id 45ef296e0a1a5e8b99ae04ebdfc5291e684ce6cc000f4b86ea0615b19a1bf5a4 Sep 14 11:18:05.004824 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:05.004721 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-88bdd8bf-8649d85875-lm74z" event={"ID":"97c9e671-bc21-4197-9689-f06779a12978","Type":"ContainerStarted","Data":"b28df878e39a70e630d47975f535f42746a4cedcfdf84b26d4596718439e4979"} Sep 14 11:18:05.004824 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:05.004788 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-88bdd8bf-8649d85875-lm74z" event={"ID":"97c9e671-bc21-4197-9689-f06779a12978","Type":"ContainerStarted","Data":"45ef296e0a1a5e8b99ae04ebdfc5291e684ce6cc000f4b86ea0615b19a1bf5a4"} Sep 14 11:18:05.005467 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:05.004875 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-88bdd8bf-8649d85875-lm74z" Sep 14 11:18:05.006465 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:05.006440 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-88bdd8bf-697785c6f8-gtm6q" event={"ID":"3de628bf-a52d-4d22-9b34-2bc5742e88f4","Type":"ContainerStarted","Data":"a602e15004eae5ed2812c48d0e1f537af56d1f7d4627df51e850bcfcc8bbf9f5"} Sep 14 11:18:05.006605 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:05.006493 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-88bdd8bf-697785c6f8-gtm6q" Sep 14 11:18:05.006605 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:05.006518 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-88bdd8bf-697785c6f8-gtm6q" event={"ID":"3de628bf-a52d-4d22-9b34-2bc5742e88f4","Type":"ContainerStarted","Data":"557f3cb44eaf07c80aaa64626d780f55ed2752b94d47fffa71f4c71e7c3ebfb5"} Sep 14 11:18:05.026912 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:05.026864 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-88bdd8bf-8649d85875-lm74z" podStartSLOduration=1.026849131 podStartE2EDuration="1.026849131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:18:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:18:05.026843702 +0000 UTC m=+1029.647056244" watchObservedRunningTime="2026-09-14 11:18:05.026849131 +0000 UTC m=+1029.647061655" Sep 14 11:18:05.052196 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:05.052149 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-88bdd8bf-697785c6f8-gtm6q" podStartSLOduration=2.052135232 podStartE2EDuration="2.052135232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:18:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:18:05.049985377 +0000 UTC m=+1029.670197918" watchObservedRunningTime="2026-09-14 11:18:05.052135232 +0000 UTC m=+1029.672347754" Sep 14 11:18:16.014512 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:16.014479 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-88bdd8bf-697785c6f8-gtm6q" Sep 14 11:18:16.014920 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:16.014758 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-88bdd8bf-8649d85875-lm74z" Sep 14 11:18:41.445832 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:41.445799 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-88bdd8bf-8649d85875-lm74z"] Sep 14 11:18:41.446392 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:41.446023 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-88bdd8bf-8649d85875-lm74z" podUID="97c9e671-bc21-4197-9689-f06779a12978" containerName="payload-processing" containerID="cri-o://b28df878e39a70e630d47975f535f42746a4cedcfdf84b26d4596718439e4979" gracePeriod=30 Sep 14 11:18:41.458788 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:41.458460 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-88bdd8bf-697785c6f8-gtm6q"] Sep 14 11:18:41.458788 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:41.458687 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-88bdd8bf-697785c6f8-gtm6q" podUID="3de628bf-a52d-4d22-9b34-2bc5742e88f4" containerName="payload-pre-processing" containerID="cri-o://a602e15004eae5ed2812c48d0e1f537af56d1f7d4627df51e850bcfcc8bbf9f5" gracePeriod=30 Sep 14 11:18:41.719371 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:41.719346 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-88bdd8bf-8649d85875-lm74z" Sep 14 11:18:41.756077 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:41.756055 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-88bdd8bf-697785c6f8-gtm6q" Sep 14 11:18:41.830759 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:41.830726 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kx4j\" (UniqueName: \"kubernetes.io/projected/97c9e671-bc21-4197-9689-f06779a12978-kube-api-access-7kx4j\") pod \"97c9e671-bc21-4197-9689-f06779a12978\" (UID: \"97c9e671-bc21-4197-9689-f06779a12978\") " Sep 14 11:18:41.830942 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:41.830796 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3de628bf-a52d-4d22-9b34-2bc5742e88f4-plugins-config-volume\") pod \"3de628bf-a52d-4d22-9b34-2bc5742e88f4\" (UID: \"3de628bf-a52d-4d22-9b34-2bc5742e88f4\") " Sep 14 11:18:41.830942 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:41.830913 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97c9e671-bc21-4197-9689-f06779a12978-plugins-config-volume\") pod \"97c9e671-bc21-4197-9689-f06779a12978\" (UID: \"97c9e671-bc21-4197-9689-f06779a12978\") " Sep 14 11:18:41.831060 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:41.830957 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bt75n\" (UniqueName: \"kubernetes.io/projected/3de628bf-a52d-4d22-9b34-2bc5742e88f4-kube-api-access-bt75n\") pod \"3de628bf-a52d-4d22-9b34-2bc5742e88f4\" (UID: \"3de628bf-a52d-4d22-9b34-2bc5742e88f4\") " Sep 14 11:18:41.831226 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:41.831197 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3de628bf-a52d-4d22-9b34-2bc5742e88f4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3de628bf-a52d-4d22-9b34-2bc5742e88f4" (UID: "3de628bf-a52d-4d22-9b34-2bc5742e88f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:18:41.831308 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:41.831250 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97c9e671-bc21-4197-9689-f06779a12978-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "97c9e671-bc21-4197-9689-f06779a12978" (UID: "97c9e671-bc21-4197-9689-f06779a12978"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:18:41.831413 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:41.831356 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97c9e671-bc21-4197-9689-f06779a12978-plugins-config-volume\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:18:41.831413 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:41.831374 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3de628bf-a52d-4d22-9b34-2bc5742e88f4-plugins-config-volume\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:18:41.833005 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:41.832977 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97c9e671-bc21-4197-9689-f06779a12978-kube-api-access-7kx4j" (OuterVolumeSpecName: "kube-api-access-7kx4j") pod "97c9e671-bc21-4197-9689-f06779a12978" (UID: "97c9e671-bc21-4197-9689-f06779a12978"). InnerVolumeSpecName "kube-api-access-7kx4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:18:41.833086 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:41.833042 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3de628bf-a52d-4d22-9b34-2bc5742e88f4-kube-api-access-bt75n" (OuterVolumeSpecName: "kube-api-access-bt75n") pod "3de628bf-a52d-4d22-9b34-2bc5742e88f4" (UID: "3de628bf-a52d-4d22-9b34-2bc5742e88f4"). InnerVolumeSpecName "kube-api-access-bt75n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:18:41.932086 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:41.932062 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt75n\" (UniqueName: \"kubernetes.io/projected/3de628bf-a52d-4d22-9b34-2bc5742e88f4-kube-api-access-bt75n\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:18:41.932086 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:41.932087 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kx4j\" (UniqueName: \"kubernetes.io/projected/97c9e671-bc21-4197-9689-f06779a12978-kube-api-access-7kx4j\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:18:42.208975 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:42.208936 2574 generic.go:356] "Generic (PLEG): container finished" podID="97c9e671-bc21-4197-9689-f06779a12978" containerID="b28df878e39a70e630d47975f535f42746a4cedcfdf84b26d4596718439e4979" exitCode=0 Sep 14 11:18:42.209170 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:42.209003 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-88bdd8bf-8649d85875-lm74z" Sep 14 11:18:42.209170 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:42.209023 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-88bdd8bf-8649d85875-lm74z" event={"ID":"97c9e671-bc21-4197-9689-f06779a12978","Type":"ContainerDied","Data":"b28df878e39a70e630d47975f535f42746a4cedcfdf84b26d4596718439e4979"} Sep 14 11:18:42.209170 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:42.209069 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-88bdd8bf-8649d85875-lm74z" event={"ID":"97c9e671-bc21-4197-9689-f06779a12978","Type":"ContainerDied","Data":"45ef296e0a1a5e8b99ae04ebdfc5291e684ce6cc000f4b86ea0615b19a1bf5a4"} Sep 14 11:18:42.209170 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:42.209091 2574 scope.go:117] "RemoveContainer" containerID="b28df878e39a70e630d47975f535f42746a4cedcfdf84b26d4596718439e4979" Sep 14 11:18:42.210506 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:42.210486 2574 generic.go:356] "Generic (PLEG): container finished" podID="3de628bf-a52d-4d22-9b34-2bc5742e88f4" containerID="a602e15004eae5ed2812c48d0e1f537af56d1f7d4627df51e850bcfcc8bbf9f5" exitCode=0 Sep 14 11:18:42.210636 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:42.210518 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-88bdd8bf-697785c6f8-gtm6q" event={"ID":"3de628bf-a52d-4d22-9b34-2bc5742e88f4","Type":"ContainerDied","Data":"a602e15004eae5ed2812c48d0e1f537af56d1f7d4627df51e850bcfcc8bbf9f5"} Sep 14 11:18:42.210636 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:42.210530 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-88bdd8bf-697785c6f8-gtm6q" Sep 14 11:18:42.210636 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:42.210557 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-88bdd8bf-697785c6f8-gtm6q" event={"ID":"3de628bf-a52d-4d22-9b34-2bc5742e88f4","Type":"ContainerDied","Data":"557f3cb44eaf07c80aaa64626d780f55ed2752b94d47fffa71f4c71e7c3ebfb5"} Sep 14 11:18:42.219227 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:42.219211 2574 scope.go:117] "RemoveContainer" containerID="b28df878e39a70e630d47975f535f42746a4cedcfdf84b26d4596718439e4979" Sep 14 11:18:42.219574 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:18:42.219555 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b28df878e39a70e630d47975f535f42746a4cedcfdf84b26d4596718439e4979\": container with ID starting with b28df878e39a70e630d47975f535f42746a4cedcfdf84b26d4596718439e4979 not found: ID does not exist" containerID="b28df878e39a70e630d47975f535f42746a4cedcfdf84b26d4596718439e4979" Sep 14 11:18:42.219666 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:42.219582 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b28df878e39a70e630d47975f535f42746a4cedcfdf84b26d4596718439e4979"} err="failed to get container status \"b28df878e39a70e630d47975f535f42746a4cedcfdf84b26d4596718439e4979\": rpc error: code = NotFound desc = could not find container \"b28df878e39a70e630d47975f535f42746a4cedcfdf84b26d4596718439e4979\": container with ID starting with b28df878e39a70e630d47975f535f42746a4cedcfdf84b26d4596718439e4979 not found: ID does not exist" Sep 14 11:18:42.219666 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:42.219598 2574 scope.go:117] "RemoveContainer" containerID="a602e15004eae5ed2812c48d0e1f537af56d1f7d4627df51e850bcfcc8bbf9f5" Sep 14 11:18:42.228413 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:42.228390 2574 scope.go:117] "RemoveContainer" containerID="a602e15004eae5ed2812c48d0e1f537af56d1f7d4627df51e850bcfcc8bbf9f5" Sep 14 11:18:42.228715 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:18:42.228697 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a602e15004eae5ed2812c48d0e1f537af56d1f7d4627df51e850bcfcc8bbf9f5\": container with ID starting with a602e15004eae5ed2812c48d0e1f537af56d1f7d4627df51e850bcfcc8bbf9f5 not found: ID does not exist" containerID="a602e15004eae5ed2812c48d0e1f537af56d1f7d4627df51e850bcfcc8bbf9f5" Sep 14 11:18:42.228766 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:42.228724 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a602e15004eae5ed2812c48d0e1f537af56d1f7d4627df51e850bcfcc8bbf9f5"} err="failed to get container status \"a602e15004eae5ed2812c48d0e1f537af56d1f7d4627df51e850bcfcc8bbf9f5\": rpc error: code = NotFound desc = could not find container \"a602e15004eae5ed2812c48d0e1f537af56d1f7d4627df51e850bcfcc8bbf9f5\": container with ID starting with a602e15004eae5ed2812c48d0e1f537af56d1f7d4627df51e850bcfcc8bbf9f5 not found: ID does not exist" Sep 14 11:18:42.229100 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:42.229082 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-88bdd8bf-697785c6f8-gtm6q"] Sep 14 11:18:42.235198 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:42.235177 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-88bdd8bf-697785c6f8-gtm6q"] Sep 14 11:18:42.247147 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:42.247123 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-88bdd8bf-8649d85875-lm74z"] Sep 14 11:18:42.249169 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:42.249151 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-88bdd8bf-8649d85875-lm74z"] Sep 14 11:18:43.900179 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:43.900141 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3de628bf-a52d-4d22-9b34-2bc5742e88f4" path="/var/lib/kubelet/pods/3de628bf-a52d-4d22-9b34-2bc5742e88f4/volumes" Sep 14 11:18:43.900596 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:43.900580 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97c9e671-bc21-4197-9689-f06779a12978" path="/var/lib/kubelet/pods/97c9e671-bc21-4197-9689-f06779a12978/volumes" Sep 14 11:18:52.239984 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:52.239904 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6fcec1be-69868f6b5d-fwqh6"] Sep 14 11:18:52.240390 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:52.240377 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3de628bf-a52d-4d22-9b34-2bc5742e88f4" containerName="payload-pre-processing" Sep 14 11:18:52.240438 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:52.240392 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3de628bf-a52d-4d22-9b34-2bc5742e88f4" containerName="payload-pre-processing" Sep 14 11:18:52.240438 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:52.240401 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97c9e671-bc21-4197-9689-f06779a12978" containerName="payload-processing" Sep 14 11:18:52.240438 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:52.240407 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="97c9e671-bc21-4197-9689-f06779a12978" containerName="payload-processing" Sep 14 11:18:52.240535 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:52.240490 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="97c9e671-bc21-4197-9689-f06779a12978" containerName="payload-processing" Sep 14 11:18:52.240535 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:52.240501 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3de628bf-a52d-4d22-9b34-2bc5742e88f4" containerName="payload-pre-processing" Sep 14 11:18:52.243758 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:52.243741 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-6fcec1be-69868f6b5d-fwqh6" Sep 14 11:18:52.247988 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:52.247958 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-6fcec1be\"" Sep 14 11:18:52.248110 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:52.248014 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-6fcec1be-dockercfg-554vj\"" Sep 14 11:18:52.259687 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:52.259666 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6fcec1be-69868f6b5d-fwqh6"] Sep 14 11:18:52.324868 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:52.324839 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6fcec1be-7b4d6d9857-6mdld"] Sep 14 11:18:52.328649 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:52.328626 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-6fcec1be-7b4d6d9857-6mdld" Sep 14 11:18:52.337213 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:52.337182 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bd7c1fd-9850-4475-83c7-09d570b39653-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-6fcec1be-69868f6b5d-fwqh6\" (UID: \"7bd7c1fd-9850-4475-83c7-09d570b39653\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6fcec1be-69868f6b5d-fwqh6" Sep 14 11:18:52.337528 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:52.337506 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d6qk\" (UniqueName: \"kubernetes.io/projected/7bd7c1fd-9850-4475-83c7-09d570b39653-kube-api-access-4d6qk\") pod \"payload-pre-processing-e2e-derive-6fcec1be-69868f6b5d-fwqh6\" (UID: \"7bd7c1fd-9850-4475-83c7-09d570b39653\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6fcec1be-69868f6b5d-fwqh6" Sep 14 11:18:52.346168 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:52.346143 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6fcec1be-7b4d6d9857-6mdld"] Sep 14 11:18:52.438979 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:52.438947 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bd7c1fd-9850-4475-83c7-09d570b39653-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-6fcec1be-69868f6b5d-fwqh6\" (UID: \"7bd7c1fd-9850-4475-83c7-09d570b39653\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6fcec1be-69868f6b5d-fwqh6" Sep 14 11:18:52.439110 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:52.439011 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d6qk\" (UniqueName: \"kubernetes.io/projected/7bd7c1fd-9850-4475-83c7-09d570b39653-kube-api-access-4d6qk\") pod \"payload-pre-processing-e2e-derive-6fcec1be-69868f6b5d-fwqh6\" (UID: \"7bd7c1fd-9850-4475-83c7-09d570b39653\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6fcec1be-69868f6b5d-fwqh6" Sep 14 11:18:52.439110 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:52.439037 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe737bee-a759-47c6-b9de-13478bde556e-plugins-config-volume\") pod \"payload-processing-e2e-derive-6fcec1be-7b4d6d9857-6mdld\" (UID: \"fe737bee-a759-47c6-b9de-13478bde556e\") " pod="openshift-ingress/payload-processing-e2e-derive-6fcec1be-7b4d6d9857-6mdld" Sep 14 11:18:52.439110 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:52.439068 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7wqm\" (UniqueName: \"kubernetes.io/projected/fe737bee-a759-47c6-b9de-13478bde556e-kube-api-access-m7wqm\") pod \"payload-processing-e2e-derive-6fcec1be-7b4d6d9857-6mdld\" (UID: \"fe737bee-a759-47c6-b9de-13478bde556e\") " pod="openshift-ingress/payload-processing-e2e-derive-6fcec1be-7b4d6d9857-6mdld" Sep 14 11:18:52.439847 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:52.439821 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bd7c1fd-9850-4475-83c7-09d570b39653-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-6fcec1be-69868f6b5d-fwqh6\" (UID: \"7bd7c1fd-9850-4475-83c7-09d570b39653\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6fcec1be-69868f6b5d-fwqh6" Sep 14 11:18:52.449194 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:52.449174 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d6qk\" (UniqueName: \"kubernetes.io/projected/7bd7c1fd-9850-4475-83c7-09d570b39653-kube-api-access-4d6qk\") pod \"payload-pre-processing-e2e-derive-6fcec1be-69868f6b5d-fwqh6\" (UID: \"7bd7c1fd-9850-4475-83c7-09d570b39653\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-6fcec1be-69868f6b5d-fwqh6" Sep 14 11:18:52.539914 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:52.539849 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7wqm\" (UniqueName: \"kubernetes.io/projected/fe737bee-a759-47c6-b9de-13478bde556e-kube-api-access-m7wqm\") pod \"payload-processing-e2e-derive-6fcec1be-7b4d6d9857-6mdld\" (UID: \"fe737bee-a759-47c6-b9de-13478bde556e\") " pod="openshift-ingress/payload-processing-e2e-derive-6fcec1be-7b4d6d9857-6mdld" Sep 14 11:18:52.540029 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:52.540012 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe737bee-a759-47c6-b9de-13478bde556e-plugins-config-volume\") pod \"payload-processing-e2e-derive-6fcec1be-7b4d6d9857-6mdld\" (UID: \"fe737bee-a759-47c6-b9de-13478bde556e\") " pod="openshift-ingress/payload-processing-e2e-derive-6fcec1be-7b4d6d9857-6mdld" Sep 14 11:18:52.540736 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:52.540717 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe737bee-a759-47c6-b9de-13478bde556e-plugins-config-volume\") pod \"payload-processing-e2e-derive-6fcec1be-7b4d6d9857-6mdld\" (UID: \"fe737bee-a759-47c6-b9de-13478bde556e\") " pod="openshift-ingress/payload-processing-e2e-derive-6fcec1be-7b4d6d9857-6mdld" Sep 14 11:18:52.549013 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:52.548993 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7wqm\" (UniqueName: \"kubernetes.io/projected/fe737bee-a759-47c6-b9de-13478bde556e-kube-api-access-m7wqm\") pod \"payload-processing-e2e-derive-6fcec1be-7b4d6d9857-6mdld\" (UID: \"fe737bee-a759-47c6-b9de-13478bde556e\") " pod="openshift-ingress/payload-processing-e2e-derive-6fcec1be-7b4d6d9857-6mdld" Sep 14 11:18:52.554763 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:52.554747 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-6fcec1be-69868f6b5d-fwqh6" Sep 14 11:18:52.644765 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:52.644737 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-6fcec1be-7b4d6d9857-6mdld" Sep 14 11:18:52.832711 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:52.832683 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6fcec1be-69868f6b5d-fwqh6"] Sep 14 11:18:52.834731 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:18:52.834705 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd7c1fd_9850_4475_83c7_09d570b39653.slice/crio-a51d79fbce79ca0f83b25689bba14974b0dd1a9f13752f829161f9ec97af07bd WatchSource:0}: Error finding container a51d79fbce79ca0f83b25689bba14974b0dd1a9f13752f829161f9ec97af07bd: Status 404 returned error can't find the container with id a51d79fbce79ca0f83b25689bba14974b0dd1a9f13752f829161f9ec97af07bd Sep 14 11:18:52.924050 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:52.924028 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6fcec1be-7b4d6d9857-6mdld"] Sep 14 11:18:52.924836 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:18:52.924816 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe737bee_a759_47c6_b9de_13478bde556e.slice/crio-a077060b7a47fdbdb1fd5ce06964f365ece5d6d60a114f65d185dc7267a5d399 WatchSource:0}: Error finding container a077060b7a47fdbdb1fd5ce06964f365ece5d6d60a114f65d185dc7267a5d399: Status 404 returned error can't find the container with id a077060b7a47fdbdb1fd5ce06964f365ece5d6d60a114f65d185dc7267a5d399 Sep 14 11:18:53.278012 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:53.277968 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-6fcec1be-7b4d6d9857-6mdld" event={"ID":"fe737bee-a759-47c6-b9de-13478bde556e","Type":"ContainerStarted","Data":"fc9b28b783cbcbfe4848d8a6e4af1664473ed54dab555ec2c150d31520a1f3a7"} Sep 14 11:18:53.278488 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:53.278039 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-6fcec1be-7b4d6d9857-6mdld" event={"ID":"fe737bee-a759-47c6-b9de-13478bde556e","Type":"ContainerStarted","Data":"a077060b7a47fdbdb1fd5ce06964f365ece5d6d60a114f65d185dc7267a5d399"} Sep 14 11:18:53.278488 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:53.278070 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-6fcec1be-7b4d6d9857-6mdld" Sep 14 11:18:53.279767 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:53.279737 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-6fcec1be-69868f6b5d-fwqh6" event={"ID":"7bd7c1fd-9850-4475-83c7-09d570b39653","Type":"ContainerStarted","Data":"3080affe66e40b5288f4533189d92e3376d5ed049e94ab23f1150572c4695cec"} Sep 14 11:18:53.279903 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:53.279829 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-6fcec1be-69868f6b5d-fwqh6" Sep 14 11:18:53.279903 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:53.279856 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-6fcec1be-69868f6b5d-fwqh6" event={"ID":"7bd7c1fd-9850-4475-83c7-09d570b39653","Type":"ContainerStarted","Data":"a51d79fbce79ca0f83b25689bba14974b0dd1a9f13752f829161f9ec97af07bd"} Sep 14 11:18:53.298558 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:53.298511 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-6fcec1be-7b4d6d9857-6mdld" podStartSLOduration=1.2984942130000001 podStartE2EDuration="1.298494213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:18:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:18:53.29681776 +0000 UTC m=+1077.917030304" watchObservedRunningTime="2026-09-14 11:18:53.298494213 +0000 UTC m=+1077.918706738" Sep 14 11:18:53.318819 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:18:53.318772 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-6fcec1be-69868f6b5d-fwqh6" podStartSLOduration=1.318760651 podStartE2EDuration="1.318760651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:18:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:18:53.31589802 +0000 UTC m=+1077.936110545" watchObservedRunningTime="2026-09-14 11:18:53.318760651 +0000 UTC m=+1077.938973173" Sep 14 11:19:01.146836 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:01.146797 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-41158e-5c545f4f6b-jbvwc"] Sep 14 11:19:01.151942 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:01.151920 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-41158e-5c545f4f6b-jbvwc" Sep 14 11:19:01.154643 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:01.154627 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-41158e-dockercfg-cmf6s\"" Sep 14 11:19:01.154845 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:01.154831 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-41158e\"" Sep 14 11:19:01.164588 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:01.164567 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-41158e-5c545f4f6b-jbvwc"] Sep 14 11:19:01.218746 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:01.218716 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-41158e-76d6556445-mwcks"] Sep 14 11:19:01.222935 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:01.222914 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-41158e-76d6556445-mwcks" Sep 14 11:19:01.229185 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:01.229154 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e5c7c21-ff24-472f-bf12-5f0f75e5fcd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-41158e-5c545f4f6b-jbvwc\" (UID: \"6e5c7c21-ff24-472f-bf12-5f0f75e5fcd7\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-41158e-5c545f4f6b-jbvwc" Sep 14 11:19:01.229331 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:01.229224 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b72hw\" (UniqueName: \"kubernetes.io/projected/6e5c7c21-ff24-472f-bf12-5f0f75e5fcd7-kube-api-access-b72hw\") pod \"payload-pre-processing-e2e-explicit-41158e-5c545f4f6b-jbvwc\" (UID: \"6e5c7c21-ff24-472f-bf12-5f0f75e5fcd7\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-41158e-5c545f4f6b-jbvwc" Sep 14 11:19:01.234698 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:01.234675 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-41158e-76d6556445-mwcks"] Sep 14 11:19:01.330327 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:01.330262 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c0aba7e-0dd8-4fdb-ae72-c79003e102f9-plugins-config-volume\") pod \"payload-processing-e2e-explicit-41158e-76d6556445-mwcks\" (UID: \"1c0aba7e-0dd8-4fdb-ae72-c79003e102f9\") " pod="openshift-ingress/payload-processing-e2e-explicit-41158e-76d6556445-mwcks" Sep 14 11:19:01.330327 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:01.330335 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e5c7c21-ff24-472f-bf12-5f0f75e5fcd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-41158e-5c545f4f6b-jbvwc\" (UID: \"6e5c7c21-ff24-472f-bf12-5f0f75e5fcd7\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-41158e-5c545f4f6b-jbvwc" Sep 14 11:19:01.330581 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:01.330380 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b72hw\" (UniqueName: \"kubernetes.io/projected/6e5c7c21-ff24-472f-bf12-5f0f75e5fcd7-kube-api-access-b72hw\") pod \"payload-pre-processing-e2e-explicit-41158e-5c545f4f6b-jbvwc\" (UID: \"6e5c7c21-ff24-472f-bf12-5f0f75e5fcd7\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-41158e-5c545f4f6b-jbvwc" Sep 14 11:19:01.330581 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:01.330429 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg9sx\" (UniqueName: \"kubernetes.io/projected/1c0aba7e-0dd8-4fdb-ae72-c79003e102f9-kube-api-access-vg9sx\") pod \"payload-processing-e2e-explicit-41158e-76d6556445-mwcks\" (UID: \"1c0aba7e-0dd8-4fdb-ae72-c79003e102f9\") " pod="openshift-ingress/payload-processing-e2e-explicit-41158e-76d6556445-mwcks" Sep 14 11:19:01.331017 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:01.330993 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e5c7c21-ff24-472f-bf12-5f0f75e5fcd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-41158e-5c545f4f6b-jbvwc\" (UID: \"6e5c7c21-ff24-472f-bf12-5f0f75e5fcd7\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-41158e-5c545f4f6b-jbvwc" Sep 14 11:19:01.340843 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:01.340813 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b72hw\" (UniqueName: \"kubernetes.io/projected/6e5c7c21-ff24-472f-bf12-5f0f75e5fcd7-kube-api-access-b72hw\") pod \"payload-pre-processing-e2e-explicit-41158e-5c545f4f6b-jbvwc\" (UID: \"6e5c7c21-ff24-472f-bf12-5f0f75e5fcd7\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-41158e-5c545f4f6b-jbvwc" Sep 14 11:19:01.431964 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:01.431922 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg9sx\" (UniqueName: \"kubernetes.io/projected/1c0aba7e-0dd8-4fdb-ae72-c79003e102f9-kube-api-access-vg9sx\") pod \"payload-processing-e2e-explicit-41158e-76d6556445-mwcks\" (UID: \"1c0aba7e-0dd8-4fdb-ae72-c79003e102f9\") " pod="openshift-ingress/payload-processing-e2e-explicit-41158e-76d6556445-mwcks" Sep 14 11:19:01.432166 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:01.432011 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c0aba7e-0dd8-4fdb-ae72-c79003e102f9-plugins-config-volume\") pod \"payload-processing-e2e-explicit-41158e-76d6556445-mwcks\" (UID: \"1c0aba7e-0dd8-4fdb-ae72-c79003e102f9\") " pod="openshift-ingress/payload-processing-e2e-explicit-41158e-76d6556445-mwcks" Sep 14 11:19:01.432717 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:01.432697 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c0aba7e-0dd8-4fdb-ae72-c79003e102f9-plugins-config-volume\") pod \"payload-processing-e2e-explicit-41158e-76d6556445-mwcks\" (UID: \"1c0aba7e-0dd8-4fdb-ae72-c79003e102f9\") " pod="openshift-ingress/payload-processing-e2e-explicit-41158e-76d6556445-mwcks" Sep 14 11:19:01.441789 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:01.441745 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg9sx\" (UniqueName: \"kubernetes.io/projected/1c0aba7e-0dd8-4fdb-ae72-c79003e102f9-kube-api-access-vg9sx\") pod \"payload-processing-e2e-explicit-41158e-76d6556445-mwcks\" (UID: \"1c0aba7e-0dd8-4fdb-ae72-c79003e102f9\") " pod="openshift-ingress/payload-processing-e2e-explicit-41158e-76d6556445-mwcks" Sep 14 11:19:01.464811 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:01.464780 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-41158e-5c545f4f6b-jbvwc" Sep 14 11:19:01.535896 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:01.535800 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-41158e-76d6556445-mwcks" Sep 14 11:19:01.752953 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:01.752921 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-41158e-5c545f4f6b-jbvwc"] Sep 14 11:19:01.753454 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:19:01.753428 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e5c7c21_ff24_472f_bf12_5f0f75e5fcd7.slice/crio-00ba9d3e41846841de1c4a0dbefdb24275b87b853aab8f172684a05d3be2dcf3 WatchSource:0}: Error finding container 00ba9d3e41846841de1c4a0dbefdb24275b87b853aab8f172684a05d3be2dcf3: Status 404 returned error can't find the container with id 00ba9d3e41846841de1c4a0dbefdb24275b87b853aab8f172684a05d3be2dcf3 Sep 14 11:19:01.826663 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:01.826637 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-41158e-76d6556445-mwcks"] Sep 14 11:19:01.827905 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:19:01.827877 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c0aba7e_0dd8_4fdb_ae72_c79003e102f9.slice/crio-16718c24fef7d8119e1cf6322a1803f26e7a149a20869ecdcb663e9f6a8afef3 WatchSource:0}: Error finding container 16718c24fef7d8119e1cf6322a1803f26e7a149a20869ecdcb663e9f6a8afef3: Status 404 returned error can't find the container with id 16718c24fef7d8119e1cf6322a1803f26e7a149a20869ecdcb663e9f6a8afef3 Sep 14 11:19:02.333157 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:02.333116 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-41158e-5c545f4f6b-jbvwc" event={"ID":"6e5c7c21-ff24-472f-bf12-5f0f75e5fcd7","Type":"ContainerStarted","Data":"9b7153691ef51de1fc6d5b016188ab39b48058e62c1090fe5bad89ed444fb4d7"} Sep 14 11:19:02.333662 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:02.333190 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-41158e-5c545f4f6b-jbvwc" event={"ID":"6e5c7c21-ff24-472f-bf12-5f0f75e5fcd7","Type":"ContainerStarted","Data":"00ba9d3e41846841de1c4a0dbefdb24275b87b853aab8f172684a05d3be2dcf3"} Sep 14 11:19:02.333662 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:02.333297 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-41158e-5c545f4f6b-jbvwc" Sep 14 11:19:02.334792 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:02.334760 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-41158e-76d6556445-mwcks" event={"ID":"1c0aba7e-0dd8-4fdb-ae72-c79003e102f9","Type":"ContainerStarted","Data":"863452a66be288d1ddce5738568ae1edcd792977f294160f866e13f27eab04ae"} Sep 14 11:19:02.334919 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:02.334814 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-41158e-76d6556445-mwcks" Sep 14 11:19:02.334919 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:02.334839 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-41158e-76d6556445-mwcks" event={"ID":"1c0aba7e-0dd8-4fdb-ae72-c79003e102f9","Type":"ContainerStarted","Data":"16718c24fef7d8119e1cf6322a1803f26e7a149a20869ecdcb663e9f6a8afef3"} Sep 14 11:19:02.360388 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:02.360338 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-41158e-5c545f4f6b-jbvwc" podStartSLOduration=1.360323723 podStartE2EDuration="1.360323723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:19:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:19:02.358136848 +0000 UTC m=+1086.978349395" watchObservedRunningTime="2026-09-14 11:19:02.360323723 +0000 UTC m=+1086.980536246" Sep 14 11:19:02.381757 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:02.381713 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-41158e-76d6556445-mwcks" podStartSLOduration=1.381697939 podStartE2EDuration="1.381697939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:19:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:19:02.380525287 +0000 UTC m=+1087.000737829" watchObservedRunningTime="2026-09-14 11:19:02.381697939 +0000 UTC m=+1087.001910462" Sep 14 11:19:04.288245 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:04.288210 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-6fcec1be-7b4d6d9857-6mdld" Sep 14 11:19:04.288665 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:04.288372 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-6fcec1be-69868f6b5d-fwqh6" Sep 14 11:19:11.984131 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:11.984098 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6fcec1be-7b4d6d9857-6mdld"] Sep 14 11:19:11.984725 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:11.984404 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-6fcec1be-7b4d6d9857-6mdld" podUID="fe737bee-a759-47c6-b9de-13478bde556e" containerName="payload-processing" containerID="cri-o://fc9b28b783cbcbfe4848d8a6e4af1664473ed54dab555ec2c150d31520a1f3a7" gracePeriod=30 Sep 14 11:19:11.992722 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:11.992528 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6fcec1be-69868f6b5d-fwqh6"] Sep 14 11:19:11.993138 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:11.992749 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-6fcec1be-69868f6b5d-fwqh6" podUID="7bd7c1fd-9850-4475-83c7-09d570b39653" containerName="payload-pre-processing" containerID="cri-o://3080affe66e40b5288f4533189d92e3376d5ed049e94ab23f1150572c4695cec" gracePeriod=30 Sep 14 11:19:12.271318 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:12.271293 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-6fcec1be-7b4d6d9857-6mdld" Sep 14 11:19:12.298520 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:12.298496 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-6fcec1be-69868f6b5d-fwqh6" Sep 14 11:19:12.340820 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:12.340786 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bd7c1fd-9850-4475-83c7-09d570b39653-plugins-config-volume\") pod \"7bd7c1fd-9850-4475-83c7-09d570b39653\" (UID: \"7bd7c1fd-9850-4475-83c7-09d570b39653\") " Sep 14 11:19:12.341006 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:12.340936 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7wqm\" (UniqueName: \"kubernetes.io/projected/fe737bee-a759-47c6-b9de-13478bde556e-kube-api-access-m7wqm\") pod \"fe737bee-a759-47c6-b9de-13478bde556e\" (UID: \"fe737bee-a759-47c6-b9de-13478bde556e\") " Sep 14 11:19:12.341086 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:12.341029 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4d6qk\" (UniqueName: \"kubernetes.io/projected/7bd7c1fd-9850-4475-83c7-09d570b39653-kube-api-access-4d6qk\") pod \"7bd7c1fd-9850-4475-83c7-09d570b39653\" (UID: \"7bd7c1fd-9850-4475-83c7-09d570b39653\") " Sep 14 11:19:12.341152 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:12.341129 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe737bee-a759-47c6-b9de-13478bde556e-plugins-config-volume\") pod \"fe737bee-a759-47c6-b9de-13478bde556e\" (UID: \"fe737bee-a759-47c6-b9de-13478bde556e\") " Sep 14 11:19:12.341303 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:12.341249 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd7c1fd-9850-4475-83c7-09d570b39653-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7bd7c1fd-9850-4475-83c7-09d570b39653" (UID: "7bd7c1fd-9850-4475-83c7-09d570b39653"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:19:12.341583 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:12.341543 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bd7c1fd-9850-4475-83c7-09d570b39653-plugins-config-volume\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:19:12.341704 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:12.341653 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe737bee-a759-47c6-b9de-13478bde556e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fe737bee-a759-47c6-b9de-13478bde556e" (UID: "fe737bee-a759-47c6-b9de-13478bde556e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:19:12.344160 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:12.344127 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe737bee-a759-47c6-b9de-13478bde556e-kube-api-access-m7wqm" (OuterVolumeSpecName: "kube-api-access-m7wqm") pod "fe737bee-a759-47c6-b9de-13478bde556e" (UID: "fe737bee-a759-47c6-b9de-13478bde556e"). InnerVolumeSpecName "kube-api-access-m7wqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:19:12.345429 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:12.345392 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd7c1fd-9850-4475-83c7-09d570b39653-kube-api-access-4d6qk" (OuterVolumeSpecName: "kube-api-access-4d6qk") pod "7bd7c1fd-9850-4475-83c7-09d570b39653" (UID: "7bd7c1fd-9850-4475-83c7-09d570b39653"). InnerVolumeSpecName "kube-api-access-4d6qk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:19:12.406121 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:12.406093 2574 generic.go:356] "Generic (PLEG): container finished" podID="7bd7c1fd-9850-4475-83c7-09d570b39653" containerID="3080affe66e40b5288f4533189d92e3376d5ed049e94ab23f1150572c4695cec" exitCode=0 Sep 14 11:19:12.406245 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:12.406151 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-6fcec1be-69868f6b5d-fwqh6" Sep 14 11:19:12.406245 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:12.406188 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-6fcec1be-69868f6b5d-fwqh6" event={"ID":"7bd7c1fd-9850-4475-83c7-09d570b39653","Type":"ContainerDied","Data":"3080affe66e40b5288f4533189d92e3376d5ed049e94ab23f1150572c4695cec"} Sep 14 11:19:12.406393 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:12.406249 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-6fcec1be-69868f6b5d-fwqh6" event={"ID":"7bd7c1fd-9850-4475-83c7-09d570b39653","Type":"ContainerDied","Data":"a51d79fbce79ca0f83b25689bba14974b0dd1a9f13752f829161f9ec97af07bd"} Sep 14 11:19:12.406393 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:12.406306 2574 scope.go:117] "RemoveContainer" containerID="3080affe66e40b5288f4533189d92e3376d5ed049e94ab23f1150572c4695cec" Sep 14 11:19:12.407749 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:12.407706 2574 generic.go:356] "Generic (PLEG): container finished" podID="fe737bee-a759-47c6-b9de-13478bde556e" containerID="fc9b28b783cbcbfe4848d8a6e4af1664473ed54dab555ec2c150d31520a1f3a7" exitCode=0 Sep 14 11:19:12.407749 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:12.407733 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-6fcec1be-7b4d6d9857-6mdld" event={"ID":"fe737bee-a759-47c6-b9de-13478bde556e","Type":"ContainerDied","Data":"fc9b28b783cbcbfe4848d8a6e4af1664473ed54dab555ec2c150d31520a1f3a7"} Sep 14 11:19:12.407749 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:12.407750 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-6fcec1be-7b4d6d9857-6mdld" Sep 14 11:19:12.407987 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:12.407782 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-6fcec1be-7b4d6d9857-6mdld" event={"ID":"fe737bee-a759-47c6-b9de-13478bde556e","Type":"ContainerDied","Data":"a077060b7a47fdbdb1fd5ce06964f365ece5d6d60a114f65d185dc7267a5d399"} Sep 14 11:19:12.420190 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:12.420170 2574 scope.go:117] "RemoveContainer" containerID="3080affe66e40b5288f4533189d92e3376d5ed049e94ab23f1150572c4695cec" Sep 14 11:19:12.421374 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:19:12.421078 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3080affe66e40b5288f4533189d92e3376d5ed049e94ab23f1150572c4695cec\": container with ID starting with 3080affe66e40b5288f4533189d92e3376d5ed049e94ab23f1150572c4695cec not found: ID does not exist" containerID="3080affe66e40b5288f4533189d92e3376d5ed049e94ab23f1150572c4695cec" Sep 14 11:19:12.421374 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:12.421125 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3080affe66e40b5288f4533189d92e3376d5ed049e94ab23f1150572c4695cec"} err="failed to get container status \"3080affe66e40b5288f4533189d92e3376d5ed049e94ab23f1150572c4695cec\": rpc error: code = NotFound desc = could not find container \"3080affe66e40b5288f4533189d92e3376d5ed049e94ab23f1150572c4695cec\": container with ID starting with 3080affe66e40b5288f4533189d92e3376d5ed049e94ab23f1150572c4695cec not found: ID does not exist" Sep 14 11:19:12.421374 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:12.421148 2574 scope.go:117] "RemoveContainer" containerID="fc9b28b783cbcbfe4848d8a6e4af1664473ed54dab555ec2c150d31520a1f3a7" Sep 14 11:19:12.430220 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:12.430200 2574 scope.go:117] "RemoveContainer" containerID="fc9b28b783cbcbfe4848d8a6e4af1664473ed54dab555ec2c150d31520a1f3a7" Sep 14 11:19:12.430572 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:19:12.430550 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc9b28b783cbcbfe4848d8a6e4af1664473ed54dab555ec2c150d31520a1f3a7\": container with ID starting with fc9b28b783cbcbfe4848d8a6e4af1664473ed54dab555ec2c150d31520a1f3a7 not found: ID does not exist" containerID="fc9b28b783cbcbfe4848d8a6e4af1664473ed54dab555ec2c150d31520a1f3a7" Sep 14 11:19:12.430656 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:12.430580 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc9b28b783cbcbfe4848d8a6e4af1664473ed54dab555ec2c150d31520a1f3a7"} err="failed to get container status \"fc9b28b783cbcbfe4848d8a6e4af1664473ed54dab555ec2c150d31520a1f3a7\": rpc error: code = NotFound desc = could not find container \"fc9b28b783cbcbfe4848d8a6e4af1664473ed54dab555ec2c150d31520a1f3a7\": container with ID starting with fc9b28b783cbcbfe4848d8a6e4af1664473ed54dab555ec2c150d31520a1f3a7 not found: ID does not exist" Sep 14 11:19:12.439984 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:12.439959 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6fcec1be-69868f6b5d-fwqh6"] Sep 14 11:19:12.445744 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:12.445718 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d6qk\" (UniqueName: \"kubernetes.io/projected/7bd7c1fd-9850-4475-83c7-09d570b39653-kube-api-access-4d6qk\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:19:12.445854 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:12.445777 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe737bee-a759-47c6-b9de-13478bde556e-plugins-config-volume\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:19:12.445854 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:12.445799 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7wqm\" (UniqueName: \"kubernetes.io/projected/fe737bee-a759-47c6-b9de-13478bde556e-kube-api-access-m7wqm\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:19:12.448951 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:12.448932 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-6fcec1be-69868f6b5d-fwqh6"] Sep 14 11:19:12.463113 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:12.463087 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6fcec1be-7b4d6d9857-6mdld"] Sep 14 11:19:12.466818 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:12.466799 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-6fcec1be-7b4d6d9857-6mdld"] Sep 14 11:19:13.343712 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:13.343633 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-41158e-76d6556445-mwcks" Sep 14 11:19:13.343712 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:13.343707 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-41158e-5c545f4f6b-jbvwc" Sep 14 11:19:13.899865 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:13.899833 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bd7c1fd-9850-4475-83c7-09d570b39653" path="/var/lib/kubelet/pods/7bd7c1fd-9850-4475-83c7-09d570b39653/volumes" Sep 14 11:19:13.900213 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:19:13.900199 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe737bee-a759-47c6-b9de-13478bde556e" path="/var/lib/kubelet/pods/fe737bee-a759-47c6-b9de-13478bde556e/volumes" Sep 14 11:20:47.717699 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:20:47.717662 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-41158e-76d6556445-mwcks"] Sep 14 11:20:47.718580 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:20:47.718543 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-41158e-76d6556445-mwcks" podUID="1c0aba7e-0dd8-4fdb-ae72-c79003e102f9" containerName="payload-processing" containerID="cri-o://863452a66be288d1ddce5738568ae1edcd792977f294160f866e13f27eab04ae" gracePeriod=30 Sep 14 11:20:47.728698 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:20:47.728446 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-41158e-5c545f4f6b-jbvwc"] Sep 14 11:20:47.728698 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:20:47.728644 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-41158e-5c545f4f6b-jbvwc" podUID="6e5c7c21-ff24-472f-bf12-5f0f75e5fcd7" containerName="payload-pre-processing" containerID="cri-o://9b7153691ef51de1fc6d5b016188ab39b48058e62c1090fe5bad89ed444fb4d7" gracePeriod=30 Sep 14 11:20:47.927432 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:20:47.927398 2574 generic.go:356] "Generic (PLEG): container finished" podID="6e5c7c21-ff24-472f-bf12-5f0f75e5fcd7" containerID="9b7153691ef51de1fc6d5b016188ab39b48058e62c1090fe5bad89ed444fb4d7" exitCode=0 Sep 14 11:20:47.927604 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:20:47.927473 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-41158e-5c545f4f6b-jbvwc" event={"ID":"6e5c7c21-ff24-472f-bf12-5f0f75e5fcd7","Type":"ContainerDied","Data":"9b7153691ef51de1fc6d5b016188ab39b48058e62c1090fe5bad89ed444fb4d7"} Sep 14 11:20:47.929100 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:20:47.929078 2574 generic.go:356] "Generic (PLEG): container finished" podID="1c0aba7e-0dd8-4fdb-ae72-c79003e102f9" containerID="863452a66be288d1ddce5738568ae1edcd792977f294160f866e13f27eab04ae" exitCode=0 Sep 14 11:20:47.929224 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:20:47.929110 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-41158e-76d6556445-mwcks" event={"ID":"1c0aba7e-0dd8-4fdb-ae72-c79003e102f9","Type":"ContainerDied","Data":"863452a66be288d1ddce5738568ae1edcd792977f294160f866e13f27eab04ae"} Sep 14 11:20:48.008083 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:20:48.008061 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-41158e-76d6556445-mwcks" Sep 14 11:20:48.040634 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:20:48.040606 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-41158e-5c545f4f6b-jbvwc" Sep 14 11:20:48.054842 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:20:48.054813 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c0aba7e-0dd8-4fdb-ae72-c79003e102f9-plugins-config-volume\") pod \"1c0aba7e-0dd8-4fdb-ae72-c79003e102f9\" (UID: \"1c0aba7e-0dd8-4fdb-ae72-c79003e102f9\") " Sep 14 11:20:48.054991 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:20:48.054918 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vg9sx\" (UniqueName: \"kubernetes.io/projected/1c0aba7e-0dd8-4fdb-ae72-c79003e102f9-kube-api-access-vg9sx\") pod \"1c0aba7e-0dd8-4fdb-ae72-c79003e102f9\" (UID: \"1c0aba7e-0dd8-4fdb-ae72-c79003e102f9\") " Sep 14 11:20:48.055231 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:20:48.055207 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c0aba7e-0dd8-4fdb-ae72-c79003e102f9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1c0aba7e-0dd8-4fdb-ae72-c79003e102f9" (UID: "1c0aba7e-0dd8-4fdb-ae72-c79003e102f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:20:48.057228 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:20:48.057203 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c0aba7e-0dd8-4fdb-ae72-c79003e102f9-kube-api-access-vg9sx" (OuterVolumeSpecName: "kube-api-access-vg9sx") pod "1c0aba7e-0dd8-4fdb-ae72-c79003e102f9" (UID: "1c0aba7e-0dd8-4fdb-ae72-c79003e102f9"). InnerVolumeSpecName "kube-api-access-vg9sx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:20:48.156206 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:20:48.156166 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e5c7c21-ff24-472f-bf12-5f0f75e5fcd7-plugins-config-volume\") pod \"6e5c7c21-ff24-472f-bf12-5f0f75e5fcd7\" (UID: \"6e5c7c21-ff24-472f-bf12-5f0f75e5fcd7\") " Sep 14 11:20:48.156424 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:20:48.156227 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b72hw\" (UniqueName: \"kubernetes.io/projected/6e5c7c21-ff24-472f-bf12-5f0f75e5fcd7-kube-api-access-b72hw\") pod \"6e5c7c21-ff24-472f-bf12-5f0f75e5fcd7\" (UID: \"6e5c7c21-ff24-472f-bf12-5f0f75e5fcd7\") " Sep 14 11:20:48.156558 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:20:48.156533 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e5c7c21-ff24-472f-bf12-5f0f75e5fcd7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6e5c7c21-ff24-472f-bf12-5f0f75e5fcd7" (UID: "6e5c7c21-ff24-472f-bf12-5f0f75e5fcd7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:20:48.156637 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:20:48.156616 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg9sx\" (UniqueName: \"kubernetes.io/projected/1c0aba7e-0dd8-4fdb-ae72-c79003e102f9-kube-api-access-vg9sx\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:20:48.156692 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:20:48.156644 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c0aba7e-0dd8-4fdb-ae72-c79003e102f9-plugins-config-volume\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:20:48.158519 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:20:48.158498 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e5c7c21-ff24-472f-bf12-5f0f75e5fcd7-kube-api-access-b72hw" (OuterVolumeSpecName: "kube-api-access-b72hw") pod "6e5c7c21-ff24-472f-bf12-5f0f75e5fcd7" (UID: "6e5c7c21-ff24-472f-bf12-5f0f75e5fcd7"). InnerVolumeSpecName "kube-api-access-b72hw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:20:48.257216 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:20:48.257139 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e5c7c21-ff24-472f-bf12-5f0f75e5fcd7-plugins-config-volume\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:20:48.257216 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:20:48.257167 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b72hw\" (UniqueName: \"kubernetes.io/projected/6e5c7c21-ff24-472f-bf12-5f0f75e5fcd7-kube-api-access-b72hw\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:20:48.935600 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:20:48.935571 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-41158e-5c545f4f6b-jbvwc" Sep 14 11:20:48.935600 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:20:48.935597 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-41158e-5c545f4f6b-jbvwc" event={"ID":"6e5c7c21-ff24-472f-bf12-5f0f75e5fcd7","Type":"ContainerDied","Data":"00ba9d3e41846841de1c4a0dbefdb24275b87b853aab8f172684a05d3be2dcf3"} Sep 14 11:20:48.936064 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:20:48.935647 2574 scope.go:117] "RemoveContainer" containerID="9b7153691ef51de1fc6d5b016188ab39b48058e62c1090fe5bad89ed444fb4d7" Sep 14 11:20:48.937116 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:20:48.937098 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-41158e-76d6556445-mwcks" Sep 14 11:20:48.937238 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:20:48.937170 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-41158e-76d6556445-mwcks" event={"ID":"1c0aba7e-0dd8-4fdb-ae72-c79003e102f9","Type":"ContainerDied","Data":"16718c24fef7d8119e1cf6322a1803f26e7a149a20869ecdcb663e9f6a8afef3"} Sep 14 11:20:48.945814 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:20:48.945798 2574 scope.go:117] "RemoveContainer" containerID="863452a66be288d1ddce5738568ae1edcd792977f294160f866e13f27eab04ae" Sep 14 11:20:48.960111 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:20:48.960090 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-41158e-76d6556445-mwcks"] Sep 14 11:20:48.962377 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:20:48.962355 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-41158e-76d6556445-mwcks"] Sep 14 11:20:48.971035 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:20:48.971016 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-41158e-5c545f4f6b-jbvwc"] Sep 14 11:20:48.974779 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:20:48.974753 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-41158e-5c545f4f6b-jbvwc"] Sep 14 11:20:49.898973 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:20:49.898940 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c0aba7e-0dd8-4fdb-ae72-c79003e102f9" path="/var/lib/kubelet/pods/1c0aba7e-0dd8-4fdb-ae72-c79003e102f9/volumes" Sep 14 11:20:49.899342 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:20:49.899328 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e5c7c21-ff24-472f-bf12-5f0f75e5fcd7" path="/var/lib/kubelet/pods/6e5c7c21-ff24-472f-bf12-5f0f75e5fcd7/volumes" Sep 14 11:21:01.660781 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:01.660746 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-508ed449-6769f67d55-b4d89"] Sep 14 11:21:01.661515 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:01.661494 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c0aba7e-0dd8-4fdb-ae72-c79003e102f9" containerName="payload-processing" Sep 14 11:21:01.661582 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:01.661521 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c0aba7e-0dd8-4fdb-ae72-c79003e102f9" containerName="payload-processing" Sep 14 11:21:01.661582 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:01.661545 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe737bee-a759-47c6-b9de-13478bde556e" containerName="payload-processing" Sep 14 11:21:01.661582 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:01.661554 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe737bee-a759-47c6-b9de-13478bde556e" containerName="payload-processing" Sep 14 11:21:01.661582 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:01.661572 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e5c7c21-ff24-472f-bf12-5f0f75e5fcd7" containerName="payload-pre-processing" Sep 14 11:21:01.661582 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:01.661581 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e5c7c21-ff24-472f-bf12-5f0f75e5fcd7" containerName="payload-pre-processing" Sep 14 11:21:01.661770 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:01.661604 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bd7c1fd-9850-4475-83c7-09d570b39653" containerName="payload-pre-processing" Sep 14 11:21:01.661770 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:01.661613 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bd7c1fd-9850-4475-83c7-09d570b39653" containerName="payload-pre-processing" Sep 14 11:21:01.661770 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:01.661732 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe737bee-a759-47c6-b9de-13478bde556e" containerName="payload-processing" Sep 14 11:21:01.661770 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:01.661746 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c0aba7e-0dd8-4fdb-ae72-c79003e102f9" containerName="payload-processing" Sep 14 11:21:01.661770 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:01.661757 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bd7c1fd-9850-4475-83c7-09d570b39653" containerName="payload-pre-processing" Sep 14 11:21:01.661935 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:01.661770 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e5c7c21-ff24-472f-bf12-5f0f75e5fcd7" containerName="payload-pre-processing" Sep 14 11:21:01.667411 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:01.667387 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-508ed449-6769f67d55-b4d89" Sep 14 11:21:01.669820 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:01.669750 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-508ed449-serving-cert\"" Sep 14 11:21:01.673229 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:01.673204 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-508ed449-6769f67d55-b4d89"] Sep 14 11:21:01.786210 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:01.785221 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb7hw\" (UniqueName: \"kubernetes.io/projected/cca8370e-bd73-4077-b848-8e4ab5f525f6-kube-api-access-kb7hw\") pod \"maas-api-e2e-mt-508ed449-6769f67d55-b4d89\" (UID: \"cca8370e-bd73-4077-b848-8e4ab5f525f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-508ed449-6769f67d55-b4d89" Sep 14 11:21:01.786210 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:01.785327 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cca8370e-bd73-4077-b848-8e4ab5f525f6-openshift-service-ca\") pod \"maas-api-e2e-mt-508ed449-6769f67d55-b4d89\" (UID: \"cca8370e-bd73-4077-b848-8e4ab5f525f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-508ed449-6769f67d55-b4d89" Sep 14 11:21:01.786210 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:01.785358 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cca8370e-bd73-4077-b848-8e4ab5f525f6-maas-api-tls\") pod \"maas-api-e2e-mt-508ed449-6769f67d55-b4d89\" (UID: \"cca8370e-bd73-4077-b848-8e4ab5f525f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-508ed449-6769f67d55-b4d89" Sep 14 11:21:01.786210 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:01.785477 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cca8370e-bd73-4077-b848-8e4ab5f525f6-metrics-certs\") pod \"maas-api-e2e-mt-508ed449-6769f67d55-b4d89\" (UID: \"cca8370e-bd73-4077-b848-8e4ab5f525f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-508ed449-6769f67d55-b4d89" Sep 14 11:21:01.886482 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:01.886437 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cca8370e-bd73-4077-b848-8e4ab5f525f6-metrics-certs\") pod \"maas-api-e2e-mt-508ed449-6769f67d55-b4d89\" (UID: \"cca8370e-bd73-4077-b848-8e4ab5f525f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-508ed449-6769f67d55-b4d89" Sep 14 11:21:01.886680 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:01.886510 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb7hw\" (UniqueName: \"kubernetes.io/projected/cca8370e-bd73-4077-b848-8e4ab5f525f6-kube-api-access-kb7hw\") pod \"maas-api-e2e-mt-508ed449-6769f67d55-b4d89\" (UID: \"cca8370e-bd73-4077-b848-8e4ab5f525f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-508ed449-6769f67d55-b4d89" Sep 14 11:21:01.886680 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:01.886562 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cca8370e-bd73-4077-b848-8e4ab5f525f6-openshift-service-ca\") pod \"maas-api-e2e-mt-508ed449-6769f67d55-b4d89\" (UID: \"cca8370e-bd73-4077-b848-8e4ab5f525f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-508ed449-6769f67d55-b4d89" Sep 14 11:21:01.886680 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:01.886653 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cca8370e-bd73-4077-b848-8e4ab5f525f6-maas-api-tls\") pod \"maas-api-e2e-mt-508ed449-6769f67d55-b4d89\" (UID: \"cca8370e-bd73-4077-b848-8e4ab5f525f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-508ed449-6769f67d55-b4d89" Sep 14 11:21:01.887238 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:01.887214 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cca8370e-bd73-4077-b848-8e4ab5f525f6-openshift-service-ca\") pod \"maas-api-e2e-mt-508ed449-6769f67d55-b4d89\" (UID: \"cca8370e-bd73-4077-b848-8e4ab5f525f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-508ed449-6769f67d55-b4d89" Sep 14 11:21:01.889055 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:01.889031 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cca8370e-bd73-4077-b848-8e4ab5f525f6-metrics-certs\") pod \"maas-api-e2e-mt-508ed449-6769f67d55-b4d89\" (UID: \"cca8370e-bd73-4077-b848-8e4ab5f525f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-508ed449-6769f67d55-b4d89" Sep 14 11:21:01.889260 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:01.889239 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cca8370e-bd73-4077-b848-8e4ab5f525f6-maas-api-tls\") pod \"maas-api-e2e-mt-508ed449-6769f67d55-b4d89\" (UID: \"cca8370e-bd73-4077-b848-8e4ab5f525f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-508ed449-6769f67d55-b4d89" Sep 14 11:21:01.894657 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:01.894635 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb7hw\" (UniqueName: \"kubernetes.io/projected/cca8370e-bd73-4077-b848-8e4ab5f525f6-kube-api-access-kb7hw\") pod \"maas-api-e2e-mt-508ed449-6769f67d55-b4d89\" (UID: \"cca8370e-bd73-4077-b848-8e4ab5f525f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-508ed449-6769f67d55-b4d89" Sep 14 11:21:01.981650 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:01.981610 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-508ed449-6769f67d55-b4d89" Sep 14 11:21:02.263101 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:02.263072 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-508ed449-6769f67d55-b4d89"] Sep 14 11:21:02.264982 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:21:02.264953 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcca8370e_bd73_4077_b848_8e4ab5f525f6.slice/crio-4681b590195db3b842e6a121a57e369af62a4e6575f21857663379dfb2adfd7d WatchSource:0}: Error finding container 4681b590195db3b842e6a121a57e369af62a4e6575f21857663379dfb2adfd7d: Status 404 returned error can't find the container with id 4681b590195db3b842e6a121a57e369af62a4e6575f21857663379dfb2adfd7d Sep 14 11:21:02.272246 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:02.272221 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-508ed449-6d76d9498-8pcnj"] Sep 14 11:21:02.277438 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:02.277419 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-508ed449-6d76d9498-8pcnj" Sep 14 11:21:02.279743 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:02.279724 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-508ed449\"" Sep 14 11:21:02.279854 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:02.279836 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-508ed449-dockercfg-9w8bc\"" Sep 14 11:21:02.286831 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:02.286808 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-508ed449-6d76d9498-8pcnj"] Sep 14 11:21:02.311335 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:02.311310 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-508ed449-6974bf9485-bkjsb"] Sep 14 11:21:02.315035 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:02.315019 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-508ed449-6974bf9485-bkjsb" Sep 14 11:21:02.327542 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:02.327502 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-508ed449-6974bf9485-bkjsb"] Sep 14 11:21:02.392902 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:02.392868 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sst9x\" (UniqueName: \"kubernetes.io/projected/98688e3e-c222-4390-8025-63c9f3653e86-kube-api-access-sst9x\") pod \"payload-pre-processing-e2e-mt-508ed449-6d76d9498-8pcnj\" (UID: \"98688e3e-c222-4390-8025-63c9f3653e86\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-508ed449-6d76d9498-8pcnj" Sep 14 11:21:02.393066 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:02.393047 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98688e3e-c222-4390-8025-63c9f3653e86-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-508ed449-6d76d9498-8pcnj\" (UID: \"98688e3e-c222-4390-8025-63c9f3653e86\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-508ed449-6d76d9498-8pcnj" Sep 14 11:21:02.494292 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:02.494232 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98688e3e-c222-4390-8025-63c9f3653e86-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-508ed449-6d76d9498-8pcnj\" (UID: \"98688e3e-c222-4390-8025-63c9f3653e86\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-508ed449-6d76d9498-8pcnj" Sep 14 11:21:02.494292 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:02.494296 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1176be80-d388-465d-836b-d16026901e62-plugins-config-volume\") pod \"payload-processing-e2e-mt-508ed449-6974bf9485-bkjsb\" (UID: \"1176be80-d388-465d-836b-d16026901e62\") " pod="openshift-ingress/payload-processing-e2e-mt-508ed449-6974bf9485-bkjsb" Sep 14 11:21:02.494501 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:02.494349 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sst9x\" (UniqueName: \"kubernetes.io/projected/98688e3e-c222-4390-8025-63c9f3653e86-kube-api-access-sst9x\") pod \"payload-pre-processing-e2e-mt-508ed449-6d76d9498-8pcnj\" (UID: \"98688e3e-c222-4390-8025-63c9f3653e86\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-508ed449-6d76d9498-8pcnj" Sep 14 11:21:02.494558 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:02.494494 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2pv9\" (UniqueName: \"kubernetes.io/projected/1176be80-d388-465d-836b-d16026901e62-kube-api-access-v2pv9\") pod \"payload-processing-e2e-mt-508ed449-6974bf9485-bkjsb\" (UID: \"1176be80-d388-465d-836b-d16026901e62\") " pod="openshift-ingress/payload-processing-e2e-mt-508ed449-6974bf9485-bkjsb" Sep 14 11:21:02.494994 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:02.494977 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98688e3e-c222-4390-8025-63c9f3653e86-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-508ed449-6d76d9498-8pcnj\" (UID: \"98688e3e-c222-4390-8025-63c9f3653e86\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-508ed449-6d76d9498-8pcnj" Sep 14 11:21:02.503394 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:02.503372 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sst9x\" (UniqueName: \"kubernetes.io/projected/98688e3e-c222-4390-8025-63c9f3653e86-kube-api-access-sst9x\") pod \"payload-pre-processing-e2e-mt-508ed449-6d76d9498-8pcnj\" (UID: \"98688e3e-c222-4390-8025-63c9f3653e86\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-508ed449-6d76d9498-8pcnj" Sep 14 11:21:02.590083 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:02.590000 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-508ed449-6d76d9498-8pcnj" Sep 14 11:21:02.595058 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:02.595032 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2pv9\" (UniqueName: \"kubernetes.io/projected/1176be80-d388-465d-836b-d16026901e62-kube-api-access-v2pv9\") pod \"payload-processing-e2e-mt-508ed449-6974bf9485-bkjsb\" (UID: \"1176be80-d388-465d-836b-d16026901e62\") " pod="openshift-ingress/payload-processing-e2e-mt-508ed449-6974bf9485-bkjsb" Sep 14 11:21:02.595173 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:02.595087 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1176be80-d388-465d-836b-d16026901e62-plugins-config-volume\") pod \"payload-processing-e2e-mt-508ed449-6974bf9485-bkjsb\" (UID: \"1176be80-d388-465d-836b-d16026901e62\") " pod="openshift-ingress/payload-processing-e2e-mt-508ed449-6974bf9485-bkjsb" Sep 14 11:21:02.595814 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:02.595791 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1176be80-d388-465d-836b-d16026901e62-plugins-config-volume\") pod \"payload-processing-e2e-mt-508ed449-6974bf9485-bkjsb\" (UID: \"1176be80-d388-465d-836b-d16026901e62\") " pod="openshift-ingress/payload-processing-e2e-mt-508ed449-6974bf9485-bkjsb" Sep 14 11:21:02.603714 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:02.603690 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2pv9\" (UniqueName: \"kubernetes.io/projected/1176be80-d388-465d-836b-d16026901e62-kube-api-access-v2pv9\") pod \"payload-processing-e2e-mt-508ed449-6974bf9485-bkjsb\" (UID: \"1176be80-d388-465d-836b-d16026901e62\") " pod="openshift-ingress/payload-processing-e2e-mt-508ed449-6974bf9485-bkjsb" Sep 14 11:21:02.626192 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:02.626161 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-508ed449-6974bf9485-bkjsb" Sep 14 11:21:02.890929 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:02.890896 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-508ed449-6d76d9498-8pcnj"] Sep 14 11:21:02.892715 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:21:02.892670 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98688e3e_c222_4390_8025_63c9f3653e86.slice/crio-9174c52a57be41de7b29ed36e4eaf5b642c48056dd2c7d92d883979e4730429c WatchSource:0}: Error finding container 9174c52a57be41de7b29ed36e4eaf5b642c48056dd2c7d92d883979e4730429c: Status 404 returned error can't find the container with id 9174c52a57be41de7b29ed36e4eaf5b642c48056dd2c7d92d883979e4730429c Sep 14 11:21:02.947984 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:21:02.947864 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1176be80_d388_465d_836b_d16026901e62.slice/crio-55ddac7e911b7ab295080b55cd1dd1ef5ea44bd154745d415d9fa0439554de11 WatchSource:0}: Error finding container 55ddac7e911b7ab295080b55cd1dd1ef5ea44bd154745d415d9fa0439554de11: Status 404 returned error can't find the container with id 55ddac7e911b7ab295080b55cd1dd1ef5ea44bd154745d415d9fa0439554de11 Sep 14 11:21:02.947984 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:02.947922 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-508ed449-6974bf9485-bkjsb"] Sep 14 11:21:03.020936 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:03.020878 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-508ed449-6d76d9498-8pcnj" event={"ID":"98688e3e-c222-4390-8025-63c9f3653e86","Type":"ContainerStarted","Data":"9a497ce11ff56e314da0258af5f7468fe0dd9c9610c57eeebcf20e958c9e3804"} Sep 14 11:21:03.021054 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:03.020945 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-508ed449-6d76d9498-8pcnj" event={"ID":"98688e3e-c222-4390-8025-63c9f3653e86","Type":"ContainerStarted","Data":"9174c52a57be41de7b29ed36e4eaf5b642c48056dd2c7d92d883979e4730429c"} Sep 14 11:21:03.021446 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:03.021076 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-508ed449-6d76d9498-8pcnj" Sep 14 11:21:03.022680 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:03.022659 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-508ed449-6974bf9485-bkjsb" event={"ID":"1176be80-d388-465d-836b-d16026901e62","Type":"ContainerStarted","Data":"55ddac7e911b7ab295080b55cd1dd1ef5ea44bd154745d415d9fa0439554de11"} Sep 14 11:21:03.024450 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:03.024423 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-508ed449-6769f67d55-b4d89" event={"ID":"cca8370e-bd73-4077-b848-8e4ab5f525f6","Type":"ContainerStarted","Data":"456485e1295fcd3704284028570f5eaeef8c3a1dff55d457ca7d93e7afc33a0e"} Sep 14 11:21:03.024552 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:03.024472 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-508ed449-6769f67d55-b4d89" event={"ID":"cca8370e-bd73-4077-b848-8e4ab5f525f6","Type":"ContainerStarted","Data":"4681b590195db3b842e6a121a57e369af62a4e6575f21857663379dfb2adfd7d"} Sep 14 11:21:03.024552 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:03.024502 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-508ed449-6769f67d55-b4d89" Sep 14 11:21:03.060415 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:03.060362 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-508ed449-6d76d9498-8pcnj" podStartSLOduration=1.060341307 podStartE2EDuration="1.060341307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:21:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:21:03.057338849 +0000 UTC m=+1207.677551391" watchObservedRunningTime="2026-09-14 11:21:03.060341307 +0000 UTC m=+1207.680553830" Sep 14 11:21:03.078213 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:03.078158 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-508ed449-6769f67d55-b4d89" podStartSLOduration=1.582201252 podStartE2EDuration="2.078141494s" totalImagesPullingTime="495.940242ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:21:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:21:03.078004599 +0000 UTC m=+1207.698217146" watchObservedRunningTime="2026-09-14 11:21:03.078141494 +0000 UTC m=+1207.698354016" Sep 14 11:21:04.032943 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:04.032904 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-508ed449-6974bf9485-bkjsb" event={"ID":"1176be80-d388-465d-836b-d16026901e62","Type":"ContainerStarted","Data":"c69fd7e3be40bbccc7efef6480c045b4841a309df1804deec7b4c690437eed46"} Sep 14 11:21:04.033416 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:04.033399 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-508ed449-6974bf9485-bkjsb" Sep 14 11:21:04.050461 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:04.050420 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-508ed449-6974bf9485-bkjsb" podStartSLOduration=2.050406685 podStartE2EDuration="2.050406685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:21:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:21:04.049304452 +0000 UTC m=+1208.669516996" watchObservedRunningTime="2026-09-14 11:21:04.050406685 +0000 UTC m=+1208.670619207" Sep 14 11:21:09.038815 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:09.038782 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-508ed449-6769f67d55-b4d89" Sep 14 11:21:14.034912 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:14.034872 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-508ed449-6d76d9498-8pcnj" Sep 14 11:21:15.041457 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:15.041429 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-508ed449-6974bf9485-bkjsb" Sep 14 11:21:46.861472 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:46.861395 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-508ed449-6769f67d55-b4d89"] Sep 14 11:21:46.862004 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:46.861624 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-508ed449-6769f67d55-b4d89" podUID="cca8370e-bd73-4077-b848-8e4ab5f525f6" containerName="maas-api" containerID="cri-o://456485e1295fcd3704284028570f5eaeef8c3a1dff55d457ca7d93e7afc33a0e" gracePeriod=30 Sep 14 11:21:46.913505 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:46.913470 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-508ed449-6974bf9485-bkjsb"] Sep 14 11:21:46.913969 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:46.913936 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-508ed449-6974bf9485-bkjsb" podUID="1176be80-d388-465d-836b-d16026901e62" containerName="payload-processing" containerID="cri-o://c69fd7e3be40bbccc7efef6480c045b4841a309df1804deec7b4c690437eed46" gracePeriod=30 Sep 14 11:21:46.922833 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:46.922805 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-508ed449-6d76d9498-8pcnj"] Sep 14 11:21:46.923014 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:46.922993 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-508ed449-6d76d9498-8pcnj" podUID="98688e3e-c222-4390-8025-63c9f3653e86" containerName="payload-pre-processing" containerID="cri-o://9a497ce11ff56e314da0258af5f7468fe0dd9c9610c57eeebcf20e958c9e3804" gracePeriod=30 Sep 14 11:21:47.198238 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.198212 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-508ed449-6769f67d55-b4d89" Sep 14 11:21:47.235250 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.234795 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-508ed449-6974bf9485-bkjsb" Sep 14 11:21:47.244394 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.244365 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-508ed449-6d76d9498-8pcnj" Sep 14 11:21:47.276892 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.276852 2574 generic.go:356] "Generic (PLEG): container finished" podID="1176be80-d388-465d-836b-d16026901e62" containerID="c69fd7e3be40bbccc7efef6480c045b4841a309df1804deec7b4c690437eed46" exitCode=0 Sep 14 11:21:47.277073 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.276996 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-508ed449-6974bf9485-bkjsb" Sep 14 11:21:47.277145 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.277103 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-508ed449-6974bf9485-bkjsb" event={"ID":"1176be80-d388-465d-836b-d16026901e62","Type":"ContainerDied","Data":"c69fd7e3be40bbccc7efef6480c045b4841a309df1804deec7b4c690437eed46"} Sep 14 11:21:47.277220 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.277203 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-508ed449-6974bf9485-bkjsb" event={"ID":"1176be80-d388-465d-836b-d16026901e62","Type":"ContainerDied","Data":"55ddac7e911b7ab295080b55cd1dd1ef5ea44bd154745d415d9fa0439554de11"} Sep 14 11:21:47.277308 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.277242 2574 scope.go:117] "RemoveContainer" containerID="c69fd7e3be40bbccc7efef6480c045b4841a309df1804deec7b4c690437eed46" Sep 14 11:21:47.279369 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.279346 2574 generic.go:356] "Generic (PLEG): container finished" podID="cca8370e-bd73-4077-b848-8e4ab5f525f6" containerID="456485e1295fcd3704284028570f5eaeef8c3a1dff55d457ca7d93e7afc33a0e" exitCode=0 Sep 14 11:21:47.279550 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.279422 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-508ed449-6769f67d55-b4d89" Sep 14 11:21:47.279550 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.279465 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-508ed449-6769f67d55-b4d89" event={"ID":"cca8370e-bd73-4077-b848-8e4ab5f525f6","Type":"ContainerDied","Data":"456485e1295fcd3704284028570f5eaeef8c3a1dff55d457ca7d93e7afc33a0e"} Sep 14 11:21:47.279704 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.279551 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-508ed449-6769f67d55-b4d89" event={"ID":"cca8370e-bd73-4077-b848-8e4ab5f525f6","Type":"ContainerDied","Data":"4681b590195db3b842e6a121a57e369af62a4e6575f21857663379dfb2adfd7d"} Sep 14 11:21:47.281630 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.281595 2574 generic.go:356] "Generic (PLEG): container finished" podID="98688e3e-c222-4390-8025-63c9f3653e86" containerID="9a497ce11ff56e314da0258af5f7468fe0dd9c9610c57eeebcf20e958c9e3804" exitCode=0 Sep 14 11:21:47.281737 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.281631 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-508ed449-6d76d9498-8pcnj" event={"ID":"98688e3e-c222-4390-8025-63c9f3653e86","Type":"ContainerDied","Data":"9a497ce11ff56e314da0258af5f7468fe0dd9c9610c57eeebcf20e958c9e3804"} Sep 14 11:21:47.281737 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.281635 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-508ed449-6d76d9498-8pcnj" Sep 14 11:21:47.281737 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.281670 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-508ed449-6d76d9498-8pcnj" event={"ID":"98688e3e-c222-4390-8025-63c9f3653e86","Type":"ContainerDied","Data":"9174c52a57be41de7b29ed36e4eaf5b642c48056dd2c7d92d883979e4730429c"} Sep 14 11:21:47.289813 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.289792 2574 scope.go:117] "RemoveContainer" containerID="c69fd7e3be40bbccc7efef6480c045b4841a309df1804deec7b4c690437eed46" Sep 14 11:21:47.290117 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:21:47.290091 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c69fd7e3be40bbccc7efef6480c045b4841a309df1804deec7b4c690437eed46\": container with ID starting with c69fd7e3be40bbccc7efef6480c045b4841a309df1804deec7b4c690437eed46 not found: ID does not exist" containerID="c69fd7e3be40bbccc7efef6480c045b4841a309df1804deec7b4c690437eed46" Sep 14 11:21:47.290199 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.290122 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c69fd7e3be40bbccc7efef6480c045b4841a309df1804deec7b4c690437eed46"} err="failed to get container status \"c69fd7e3be40bbccc7efef6480c045b4841a309df1804deec7b4c690437eed46\": rpc error: code = NotFound desc = could not find container \"c69fd7e3be40bbccc7efef6480c045b4841a309df1804deec7b4c690437eed46\": container with ID starting with c69fd7e3be40bbccc7efef6480c045b4841a309df1804deec7b4c690437eed46 not found: ID does not exist" Sep 14 11:21:47.290199 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.290140 2574 scope.go:117] "RemoveContainer" containerID="456485e1295fcd3704284028570f5eaeef8c3a1dff55d457ca7d93e7afc33a0e" Sep 14 11:21:47.299314 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.299260 2574 scope.go:117] "RemoveContainer" containerID="456485e1295fcd3704284028570f5eaeef8c3a1dff55d457ca7d93e7afc33a0e" Sep 14 11:21:47.299590 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:21:47.299568 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"456485e1295fcd3704284028570f5eaeef8c3a1dff55d457ca7d93e7afc33a0e\": container with ID starting with 456485e1295fcd3704284028570f5eaeef8c3a1dff55d457ca7d93e7afc33a0e not found: ID does not exist" containerID="456485e1295fcd3704284028570f5eaeef8c3a1dff55d457ca7d93e7afc33a0e" Sep 14 11:21:47.299683 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.299596 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"456485e1295fcd3704284028570f5eaeef8c3a1dff55d457ca7d93e7afc33a0e"} err="failed to get container status \"456485e1295fcd3704284028570f5eaeef8c3a1dff55d457ca7d93e7afc33a0e\": rpc error: code = NotFound desc = could not find container \"456485e1295fcd3704284028570f5eaeef8c3a1dff55d457ca7d93e7afc33a0e\": container with ID starting with 456485e1295fcd3704284028570f5eaeef8c3a1dff55d457ca7d93e7afc33a0e not found: ID does not exist" Sep 14 11:21:47.299683 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.299614 2574 scope.go:117] "RemoveContainer" containerID="9a497ce11ff56e314da0258af5f7468fe0dd9c9610c57eeebcf20e958c9e3804" Sep 14 11:21:47.310824 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.310770 2574 scope.go:117] "RemoveContainer" containerID="9a497ce11ff56e314da0258af5f7468fe0dd9c9610c57eeebcf20e958c9e3804" Sep 14 11:21:47.311253 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:21:47.311208 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a497ce11ff56e314da0258af5f7468fe0dd9c9610c57eeebcf20e958c9e3804\": container with ID starting with 9a497ce11ff56e314da0258af5f7468fe0dd9c9610c57eeebcf20e958c9e3804 not found: ID does not exist" containerID="9a497ce11ff56e314da0258af5f7468fe0dd9c9610c57eeebcf20e958c9e3804" Sep 14 11:21:47.311394 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.311247 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a497ce11ff56e314da0258af5f7468fe0dd9c9610c57eeebcf20e958c9e3804"} err="failed to get container status \"9a497ce11ff56e314da0258af5f7468fe0dd9c9610c57eeebcf20e958c9e3804\": rpc error: code = NotFound desc = could not find container \"9a497ce11ff56e314da0258af5f7468fe0dd9c9610c57eeebcf20e958c9e3804\": container with ID starting with 9a497ce11ff56e314da0258af5f7468fe0dd9c9610c57eeebcf20e958c9e3804 not found: ID does not exist" Sep 14 11:21:47.357638 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.357604 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kb7hw\" (UniqueName: \"kubernetes.io/projected/cca8370e-bd73-4077-b848-8e4ab5f525f6-kube-api-access-kb7hw\") pod \"cca8370e-bd73-4077-b848-8e4ab5f525f6\" (UID: \"cca8370e-bd73-4077-b848-8e4ab5f525f6\") " Sep 14 11:21:47.357823 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.357645 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sst9x\" (UniqueName: \"kubernetes.io/projected/98688e3e-c222-4390-8025-63c9f3653e86-kube-api-access-sst9x\") pod \"98688e3e-c222-4390-8025-63c9f3653e86\" (UID: \"98688e3e-c222-4390-8025-63c9f3653e86\") " Sep 14 11:21:47.357823 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.357666 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98688e3e-c222-4390-8025-63c9f3653e86-plugins-config-volume\") pod \"98688e3e-c222-4390-8025-63c9f3653e86\" (UID: \"98688e3e-c222-4390-8025-63c9f3653e86\") " Sep 14 11:21:47.357823 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.357687 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2pv9\" (UniqueName: \"kubernetes.io/projected/1176be80-d388-465d-836b-d16026901e62-kube-api-access-v2pv9\") pod \"1176be80-d388-465d-836b-d16026901e62\" (UID: \"1176be80-d388-465d-836b-d16026901e62\") " Sep 14 11:21:47.357823 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.357733 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cca8370e-bd73-4077-b848-8e4ab5f525f6-maas-api-tls\") pod \"cca8370e-bd73-4077-b848-8e4ab5f525f6\" (UID: \"cca8370e-bd73-4077-b848-8e4ab5f525f6\") " Sep 14 11:21:47.357823 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.357776 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1176be80-d388-465d-836b-d16026901e62-plugins-config-volume\") pod \"1176be80-d388-465d-836b-d16026901e62\" (UID: \"1176be80-d388-465d-836b-d16026901e62\") " Sep 14 11:21:47.357823 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.357815 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cca8370e-bd73-4077-b848-8e4ab5f525f6-openshift-service-ca\") pod \"cca8370e-bd73-4077-b848-8e4ab5f525f6\" (UID: \"cca8370e-bd73-4077-b848-8e4ab5f525f6\") " Sep 14 11:21:47.358229 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.357940 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cca8370e-bd73-4077-b848-8e4ab5f525f6-metrics-certs\") pod \"cca8370e-bd73-4077-b848-8e4ab5f525f6\" (UID: \"cca8370e-bd73-4077-b848-8e4ab5f525f6\") " Sep 14 11:21:47.359162 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.358750 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1176be80-d388-465d-836b-d16026901e62-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1176be80-d388-465d-836b-d16026901e62" (UID: "1176be80-d388-465d-836b-d16026901e62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:21:47.359162 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.359029 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cca8370e-bd73-4077-b848-8e4ab5f525f6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cca8370e-bd73-4077-b848-8e4ab5f525f6" (UID: "cca8370e-bd73-4077-b848-8e4ab5f525f6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:21:47.359956 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.359234 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98688e3e-c222-4390-8025-63c9f3653e86-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "98688e3e-c222-4390-8025-63c9f3653e86" (UID: "98688e3e-c222-4390-8025-63c9f3653e86"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:21:47.361841 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.361813 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cca8370e-bd73-4077-b848-8e4ab5f525f6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cca8370e-bd73-4077-b848-8e4ab5f525f6" (UID: "cca8370e-bd73-4077-b848-8e4ab5f525f6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:21:47.361960 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.361888 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cca8370e-bd73-4077-b848-8e4ab5f525f6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cca8370e-bd73-4077-b848-8e4ab5f525f6" (UID: "cca8370e-bd73-4077-b848-8e4ab5f525f6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:21:47.362461 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.362437 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cca8370e-bd73-4077-b848-8e4ab5f525f6-kube-api-access-kb7hw" (OuterVolumeSpecName: "kube-api-access-kb7hw") pod "cca8370e-bd73-4077-b848-8e4ab5f525f6" (UID: "cca8370e-bd73-4077-b848-8e4ab5f525f6"). InnerVolumeSpecName "kube-api-access-kb7hw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:21:47.362936 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.362920 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1176be80-d388-465d-836b-d16026901e62-kube-api-access-v2pv9" (OuterVolumeSpecName: "kube-api-access-v2pv9") pod "1176be80-d388-465d-836b-d16026901e62" (UID: "1176be80-d388-465d-836b-d16026901e62"). InnerVolumeSpecName "kube-api-access-v2pv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:21:47.363517 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.363467 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98688e3e-c222-4390-8025-63c9f3653e86-kube-api-access-sst9x" (OuterVolumeSpecName: "kube-api-access-sst9x") pod "98688e3e-c222-4390-8025-63c9f3653e86" (UID: "98688e3e-c222-4390-8025-63c9f3653e86"). InnerVolumeSpecName "kube-api-access-sst9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:21:47.459617 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.459587 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cca8370e-bd73-4077-b848-8e4ab5f525f6-metrics-certs\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:21:47.459617 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.459615 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb7hw\" (UniqueName: \"kubernetes.io/projected/cca8370e-bd73-4077-b848-8e4ab5f525f6-kube-api-access-kb7hw\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:21:47.459617 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.459626 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sst9x\" (UniqueName: \"kubernetes.io/projected/98688e3e-c222-4390-8025-63c9f3653e86-kube-api-access-sst9x\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:21:47.459842 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.459635 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98688e3e-c222-4390-8025-63c9f3653e86-plugins-config-volume\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:21:47.459842 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.459644 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2pv9\" (UniqueName: \"kubernetes.io/projected/1176be80-d388-465d-836b-d16026901e62-kube-api-access-v2pv9\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:21:47.459842 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.459654 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cca8370e-bd73-4077-b848-8e4ab5f525f6-maas-api-tls\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:21:47.459842 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.459662 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1176be80-d388-465d-836b-d16026901e62-plugins-config-volume\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:21:47.459842 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.459671 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cca8370e-bd73-4077-b848-8e4ab5f525f6-openshift-service-ca\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:21:47.609258 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.609224 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-508ed449-6974bf9485-bkjsb"] Sep 14 11:21:47.614807 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.614745 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-508ed449-6974bf9485-bkjsb"] Sep 14 11:21:47.651954 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.651923 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-508ed449-6d76d9498-8pcnj"] Sep 14 11:21:47.658948 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.658922 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-508ed449-6d76d9498-8pcnj"] Sep 14 11:21:47.682120 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.682091 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-508ed449-6769f67d55-b4d89"] Sep 14 11:21:47.702354 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.702327 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-508ed449-6769f67d55-b4d89"] Sep 14 11:21:47.900227 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.900194 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1176be80-d388-465d-836b-d16026901e62" path="/var/lib/kubelet/pods/1176be80-d388-465d-836b-d16026901e62/volumes" Sep 14 11:21:47.900628 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.900612 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98688e3e-c222-4390-8025-63c9f3653e86" path="/var/lib/kubelet/pods/98688e3e-c222-4390-8025-63c9f3653e86/volumes" Sep 14 11:21:47.900981 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:21:47.900968 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cca8370e-bd73-4077-b848-8e4ab5f525f6" path="/var/lib/kubelet/pods/cca8370e-bd73-4077-b848-8e4ab5f525f6/volumes" Sep 14 11:22:15.664242 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:15.664207 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh"] Sep 14 11:22:15.664820 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:15.664690 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1176be80-d388-465d-836b-d16026901e62" containerName="payload-processing" Sep 14 11:22:15.664820 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:15.664702 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1176be80-d388-465d-836b-d16026901e62" containerName="payload-processing" Sep 14 11:22:15.664820 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:15.664725 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cca8370e-bd73-4077-b848-8e4ab5f525f6" containerName="maas-api" Sep 14 11:22:15.664820 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:15.664730 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cca8370e-bd73-4077-b848-8e4ab5f525f6" containerName="maas-api" Sep 14 11:22:15.664820 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:15.664740 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98688e3e-c222-4390-8025-63c9f3653e86" containerName="payload-pre-processing" Sep 14 11:22:15.664820 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:15.664746 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="98688e3e-c222-4390-8025-63c9f3653e86" containerName="payload-pre-processing" Sep 14 11:22:15.664820 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:15.664814 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="cca8370e-bd73-4077-b848-8e4ab5f525f6" containerName="maas-api" Sep 14 11:22:15.664820 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:15.664822 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1176be80-d388-465d-836b-d16026901e62" containerName="payload-processing" Sep 14 11:22:15.665280 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:15.664831 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="98688e3e-c222-4390-8025-63c9f3653e86" containerName="payload-pre-processing" Sep 14 11:22:15.669546 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:15.669523 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh" Sep 14 11:22:15.672656 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:15.672635 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-e42fe6-serving-cert\"" Sep 14 11:22:15.690748 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:15.690721 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh"] Sep 14 11:22:15.852228 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:15.852190 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3221733-0209-46bb-a456-7bdcec80a1bc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh\" (UID: \"c3221733-0209-46bb-a456-7bdcec80a1bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh" Sep 14 11:22:15.852228 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:15.852227 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86bg8\" (UniqueName: \"kubernetes.io/projected/c3221733-0209-46bb-a456-7bdcec80a1bc-kube-api-access-86bg8\") pod \"maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh\" (UID: \"c3221733-0209-46bb-a456-7bdcec80a1bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh" Sep 14 11:22:15.852475 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:15.852365 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3221733-0209-46bb-a456-7bdcec80a1bc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh\" (UID: \"c3221733-0209-46bb-a456-7bdcec80a1bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh" Sep 14 11:22:15.852475 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:15.852429 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3221733-0209-46bb-a456-7bdcec80a1bc-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh\" (UID: \"c3221733-0209-46bb-a456-7bdcec80a1bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh" Sep 14 11:22:15.953717 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:15.953681 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3221733-0209-46bb-a456-7bdcec80a1bc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh\" (UID: \"c3221733-0209-46bb-a456-7bdcec80a1bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh" Sep 14 11:22:15.953934 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:15.953755 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3221733-0209-46bb-a456-7bdcec80a1bc-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh\" (UID: \"c3221733-0209-46bb-a456-7bdcec80a1bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh" Sep 14 11:22:15.953934 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:15.953869 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3221733-0209-46bb-a456-7bdcec80a1bc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh\" (UID: \"c3221733-0209-46bb-a456-7bdcec80a1bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh" Sep 14 11:22:15.953934 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:15.953893 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86bg8\" (UniqueName: \"kubernetes.io/projected/c3221733-0209-46bb-a456-7bdcec80a1bc-kube-api-access-86bg8\") pod \"maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh\" (UID: \"c3221733-0209-46bb-a456-7bdcec80a1bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh" Sep 14 11:22:15.954113 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:22:15.954024 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e42fe6-serving-cert: secret "maas-api-e2e-shared-a-w2-e42fe6-serving-cert" not found Sep 14 11:22:15.954113 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:22:15.954101 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3221733-0209-46bb-a456-7bdcec80a1bc-maas-api-tls podName:c3221733-0209-46bb-a456-7bdcec80a1bc nodeName:}" failed. No retries permitted until 2026-09-14 11:22:16.454082541 +0000 UTC m=+1281.074295060 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c3221733-0209-46bb-a456-7bdcec80a1bc-maas-api-tls") pod "maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh" (UID: "c3221733-0209-46bb-a456-7bdcec80a1bc") : secret "maas-api-e2e-shared-a-w2-e42fe6-serving-cert" not found Sep 14 11:22:15.954518 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:15.954498 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3221733-0209-46bb-a456-7bdcec80a1bc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh\" (UID: \"c3221733-0209-46bb-a456-7bdcec80a1bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh" Sep 14 11:22:15.956441 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:15.956419 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3221733-0209-46bb-a456-7bdcec80a1bc-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh\" (UID: \"c3221733-0209-46bb-a456-7bdcec80a1bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh" Sep 14 11:22:15.966127 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:15.966107 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86bg8\" (UniqueName: \"kubernetes.io/projected/c3221733-0209-46bb-a456-7bdcec80a1bc-kube-api-access-86bg8\") pod \"maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh\" (UID: \"c3221733-0209-46bb-a456-7bdcec80a1bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh" Sep 14 11:22:16.459564 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:16.459532 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3221733-0209-46bb-a456-7bdcec80a1bc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh\" (UID: \"c3221733-0209-46bb-a456-7bdcec80a1bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh" Sep 14 11:22:16.462231 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:16.462208 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3221733-0209-46bb-a456-7bdcec80a1bc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh\" (UID: \"c3221733-0209-46bb-a456-7bdcec80a1bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh" Sep 14 11:22:16.548009 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:16.547965 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e42fe6-75ddbdd474-f5klc"] Sep 14 11:22:16.552010 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:16.551987 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e42fe6-75ddbdd474-f5klc" Sep 14 11:22:16.555183 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:16.555153 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-e42fe6\"" Sep 14 11:22:16.555299 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:16.555202 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-e42fe6-dockercfg-ftqmg\"" Sep 14 11:22:16.579960 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:16.579932 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e42fe6-75ddbdd474-f5klc"] Sep 14 11:22:16.580529 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:16.580484 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh" Sep 14 11:22:16.661260 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:16.661217 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mjn2\" (UniqueName: \"kubernetes.io/projected/b7319163-0333-4d9f-ae8b-abe6beaaa690-kube-api-access-7mjn2\") pod \"payload-pre-processing-e2e-shared-a-w2-e42fe6-75ddbdd474-f5klc\" (UID: \"b7319163-0333-4d9f-ae8b-abe6beaaa690\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e42fe6-75ddbdd474-f5klc" Sep 14 11:22:16.661427 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:16.661372 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7319163-0333-4d9f-ae8b-abe6beaaa690-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-e42fe6-75ddbdd474-f5klc\" (UID: \"b7319163-0333-4d9f-ae8b-abe6beaaa690\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e42fe6-75ddbdd474-f5klc" Sep 14 11:22:16.670471 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:16.670442 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e42fe6-7f857b48b9-cp8r5"] Sep 14 11:22:16.675146 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:16.675130 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e42fe6-7f857b48b9-cp8r5" Sep 14 11:22:16.702258 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:16.702234 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e42fe6-7f857b48b9-cp8r5"] Sep 14 11:22:16.762539 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:16.762448 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7319163-0333-4d9f-ae8b-abe6beaaa690-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-e42fe6-75ddbdd474-f5klc\" (UID: \"b7319163-0333-4d9f-ae8b-abe6beaaa690\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e42fe6-75ddbdd474-f5klc" Sep 14 11:22:16.762717 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:16.762582 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mjn2\" (UniqueName: \"kubernetes.io/projected/b7319163-0333-4d9f-ae8b-abe6beaaa690-kube-api-access-7mjn2\") pod \"payload-pre-processing-e2e-shared-a-w2-e42fe6-75ddbdd474-f5klc\" (UID: \"b7319163-0333-4d9f-ae8b-abe6beaaa690\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e42fe6-75ddbdd474-f5klc" Sep 14 11:22:16.763123 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:16.763103 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7319163-0333-4d9f-ae8b-abe6beaaa690-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-e42fe6-75ddbdd474-f5klc\" (UID: \"b7319163-0333-4d9f-ae8b-abe6beaaa690\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e42fe6-75ddbdd474-f5klc" Sep 14 11:22:16.785758 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:16.785731 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mjn2\" (UniqueName: \"kubernetes.io/projected/b7319163-0333-4d9f-ae8b-abe6beaaa690-kube-api-access-7mjn2\") pod \"payload-pre-processing-e2e-shared-a-w2-e42fe6-75ddbdd474-f5klc\" (UID: \"b7319163-0333-4d9f-ae8b-abe6beaaa690\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e42fe6-75ddbdd474-f5klc" Sep 14 11:22:16.863426 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:16.863377 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e42fe6-75ddbdd474-f5klc" Sep 14 11:22:16.863752 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:16.863729 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dc5ea8f-4e68-4452-adb0-b20626023ea8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-e42fe6-7f857b48b9-cp8r5\" (UID: \"7dc5ea8f-4e68-4452-adb0-b20626023ea8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e42fe6-7f857b48b9-cp8r5" Sep 14 11:22:16.863877 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:16.863858 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-896wf\" (UniqueName: \"kubernetes.io/projected/7dc5ea8f-4e68-4452-adb0-b20626023ea8-kube-api-access-896wf\") pod \"payload-processing-e2e-shared-a-w2-e42fe6-7f857b48b9-cp8r5\" (UID: \"7dc5ea8f-4e68-4452-adb0-b20626023ea8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e42fe6-7f857b48b9-cp8r5" Sep 14 11:22:16.895741 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:16.895679 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh"] Sep 14 11:22:16.897573 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:22:16.896913 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3221733_0209_46bb_a456_7bdcec80a1bc.slice/crio-7070b4e4c9d9c36c11776edbbabb020c20661cff5bd9c4772fcb45ae96297865 WatchSource:0}: Error finding container 7070b4e4c9d9c36c11776edbbabb020c20661cff5bd9c4772fcb45ae96297865: Status 404 returned error can't find the container with id 7070b4e4c9d9c36c11776edbbabb020c20661cff5bd9c4772fcb45ae96297865 Sep 14 11:22:16.898604 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:16.898576 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 11:22:16.964816 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:16.964776 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-896wf\" (UniqueName: \"kubernetes.io/projected/7dc5ea8f-4e68-4452-adb0-b20626023ea8-kube-api-access-896wf\") pod \"payload-processing-e2e-shared-a-w2-e42fe6-7f857b48b9-cp8r5\" (UID: \"7dc5ea8f-4e68-4452-adb0-b20626023ea8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e42fe6-7f857b48b9-cp8r5" Sep 14 11:22:16.964974 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:16.964888 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dc5ea8f-4e68-4452-adb0-b20626023ea8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-e42fe6-7f857b48b9-cp8r5\" (UID: \"7dc5ea8f-4e68-4452-adb0-b20626023ea8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e42fe6-7f857b48b9-cp8r5" Sep 14 11:22:16.965843 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:16.965819 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dc5ea8f-4e68-4452-adb0-b20626023ea8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-e42fe6-7f857b48b9-cp8r5\" (UID: \"7dc5ea8f-4e68-4452-adb0-b20626023ea8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e42fe6-7f857b48b9-cp8r5" Sep 14 11:22:16.985469 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:16.985446 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-896wf\" (UniqueName: \"kubernetes.io/projected/7dc5ea8f-4e68-4452-adb0-b20626023ea8-kube-api-access-896wf\") pod \"payload-processing-e2e-shared-a-w2-e42fe6-7f857b48b9-cp8r5\" (UID: \"7dc5ea8f-4e68-4452-adb0-b20626023ea8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e42fe6-7f857b48b9-cp8r5" Sep 14 11:22:16.990180 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:16.990155 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e42fe6-7f857b48b9-cp8r5" Sep 14 11:22:17.155154 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:17.155126 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e42fe6-75ddbdd474-f5klc"] Sep 14 11:22:17.156764 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:22:17.156738 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7319163_0333_4d9f_ae8b_abe6beaaa690.slice/crio-9a2f95b68cfc9d9821ceb0466e85b93fed70b223b872168973b3f127682a1491 WatchSource:0}: Error finding container 9a2f95b68cfc9d9821ceb0466e85b93fed70b223b872168973b3f127682a1491: Status 404 returned error can't find the container with id 9a2f95b68cfc9d9821ceb0466e85b93fed70b223b872168973b3f127682a1491 Sep 14 11:22:17.283202 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:17.283179 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e42fe6-7f857b48b9-cp8r5"] Sep 14 11:22:17.285683 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:22:17.285650 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dc5ea8f_4e68_4452_adb0_b20626023ea8.slice/crio-9503c395ff56212adb27e14da42a4a3c3c3034a41541c36746e0cf98d9a1a04b WatchSource:0}: Error finding container 9503c395ff56212adb27e14da42a4a3c3c3034a41541c36746e0cf98d9a1a04b: Status 404 returned error can't find the container with id 9503c395ff56212adb27e14da42a4a3c3c3034a41541c36746e0cf98d9a1a04b Sep 14 11:22:17.456333 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:17.456290 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh" event={"ID":"c3221733-0209-46bb-a456-7bdcec80a1bc","Type":"ContainerStarted","Data":"7070b4e4c9d9c36c11776edbbabb020c20661cff5bd9c4772fcb45ae96297865"} Sep 14 11:22:17.458037 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:17.458011 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e42fe6-7f857b48b9-cp8r5" event={"ID":"7dc5ea8f-4e68-4452-adb0-b20626023ea8","Type":"ContainerStarted","Data":"0c18fc5bd6082a92958c4de2626438b7c2e44276cd77f93aa9708203c01f64c3"} Sep 14 11:22:17.458165 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:17.458062 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e42fe6-7f857b48b9-cp8r5" event={"ID":"7dc5ea8f-4e68-4452-adb0-b20626023ea8","Type":"ContainerStarted","Data":"9503c395ff56212adb27e14da42a4a3c3c3034a41541c36746e0cf98d9a1a04b"} Sep 14 11:22:17.458165 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:17.458110 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e42fe6-7f857b48b9-cp8r5" Sep 14 11:22:17.459715 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:17.459687 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e42fe6-75ddbdd474-f5klc" event={"ID":"b7319163-0333-4d9f-ae8b-abe6beaaa690","Type":"ContainerStarted","Data":"dc9f3e6c9702c6501547ae4cc48fd9a7fe24b5d9a402177eca22cab5b19cecc9"} Sep 14 11:22:17.459832 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:17.459741 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e42fe6-75ddbdd474-f5klc" Sep 14 11:22:17.459832 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:17.459758 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e42fe6-75ddbdd474-f5klc" event={"ID":"b7319163-0333-4d9f-ae8b-abe6beaaa690","Type":"ContainerStarted","Data":"9a2f95b68cfc9d9821ceb0466e85b93fed70b223b872168973b3f127682a1491"} Sep 14 11:22:17.499710 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:17.499667 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e42fe6-7f857b48b9-cp8r5" podStartSLOduration=1.499653483 podStartE2EDuration="1.499653483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:22:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:22:17.497793034 +0000 UTC m=+1282.118005577" watchObservedRunningTime="2026-09-14 11:22:17.499653483 +0000 UTC m=+1282.119865997" Sep 14 11:22:17.526726 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:17.526679 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e42fe6-75ddbdd474-f5klc" podStartSLOduration=1.526664706 podStartE2EDuration="1.526664706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:22:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:22:17.524400004 +0000 UTC m=+1282.144612541" watchObservedRunningTime="2026-09-14 11:22:17.526664706 +0000 UTC m=+1282.146877229" Sep 14 11:22:18.466731 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:18.466696 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh" event={"ID":"c3221733-0209-46bb-a456-7bdcec80a1bc","Type":"ContainerStarted","Data":"4ef22319893da69ae80ade8e501acdc71636444baee86111aa221b223b03b563"} Sep 14 11:22:18.467140 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:18.466758 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh" Sep 14 11:22:18.493019 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:18.492975 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh" podStartSLOduration=2.613185683 podStartE2EDuration="3.492958557s" totalImagesPullingTime="879.772874ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:22:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:22:18.491119608 +0000 UTC m=+1283.111332142" watchObservedRunningTime="2026-09-14 11:22:18.492958557 +0000 UTC m=+1283.113171112" Sep 14 11:22:24.478344 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:24.478314 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh" Sep 14 11:22:28.468693 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:28.468660 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e42fe6-75ddbdd474-f5klc" Sep 14 11:22:28.469055 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:28.469047 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e42fe6-7f857b48b9-cp8r5" Sep 14 11:22:52.554531 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:52.554495 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-41ae25-7c78f9c687-gqbtv"] Sep 14 11:22:52.559827 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:52.559811 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-41ae25-7c78f9c687-gqbtv" Sep 14 11:22:52.563606 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:52.563581 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-41ae25\"" Sep 14 11:22:52.572876 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:52.572855 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-41ae25-7c78f9c687-gqbtv"] Sep 14 11:22:52.707011 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:52.706966 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd813a8-2d06-4faf-99ed-23c6d83a2b82-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-41ae25-7c78f9c687-gqbtv\" (UID: \"dfd813a8-2d06-4faf-99ed-23c6d83a2b82\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-41ae25-7c78f9c687-gqbtv" Sep 14 11:22:52.707169 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:52.707058 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsxk2\" (UniqueName: \"kubernetes.io/projected/dfd813a8-2d06-4faf-99ed-23c6d83a2b82-kube-api-access-vsxk2\") pod \"payload-processing-e2e-shared-b-w2-41ae25-7c78f9c687-gqbtv\" (UID: \"dfd813a8-2d06-4faf-99ed-23c6d83a2b82\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-41ae25-7c78f9c687-gqbtv" Sep 14 11:22:52.807768 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:52.807676 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd813a8-2d06-4faf-99ed-23c6d83a2b82-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-41ae25-7c78f9c687-gqbtv\" (UID: \"dfd813a8-2d06-4faf-99ed-23c6d83a2b82\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-41ae25-7c78f9c687-gqbtv" Sep 14 11:22:52.807768 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:52.807728 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsxk2\" (UniqueName: \"kubernetes.io/projected/dfd813a8-2d06-4faf-99ed-23c6d83a2b82-kube-api-access-vsxk2\") pod \"payload-processing-e2e-shared-b-w2-41ae25-7c78f9c687-gqbtv\" (UID: \"dfd813a8-2d06-4faf-99ed-23c6d83a2b82\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-41ae25-7c78f9c687-gqbtv" Sep 14 11:22:52.808505 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:52.808482 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd813a8-2d06-4faf-99ed-23c6d83a2b82-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-41ae25-7c78f9c687-gqbtv\" (UID: \"dfd813a8-2d06-4faf-99ed-23c6d83a2b82\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-41ae25-7c78f9c687-gqbtv" Sep 14 11:22:52.817056 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:52.817035 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsxk2\" (UniqueName: \"kubernetes.io/projected/dfd813a8-2d06-4faf-99ed-23c6d83a2b82-kube-api-access-vsxk2\") pod \"payload-processing-e2e-shared-b-w2-41ae25-7c78f9c687-gqbtv\" (UID: \"dfd813a8-2d06-4faf-99ed-23c6d83a2b82\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-41ae25-7c78f9c687-gqbtv" Sep 14 11:22:52.872398 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:52.872359 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-41ae25-7c78f9c687-gqbtv" Sep 14 11:22:53.162181 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:53.162157 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-41ae25-7c78f9c687-gqbtv"] Sep 14 11:22:53.164945 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:22:53.164917 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd813a8_2d06_4faf_99ed_23c6d83a2b82.slice/crio-46c1fcbcf1c10bf9bcfb93e4536006a57d7626004e43365915d00d7cf99fa840 WatchSource:0}: Error finding container 46c1fcbcf1c10bf9bcfb93e4536006a57d7626004e43365915d00d7cf99fa840: Status 404 returned error can't find the container with id 46c1fcbcf1c10bf9bcfb93e4536006a57d7626004e43365915d00d7cf99fa840 Sep 14 11:22:53.659801 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:53.659766 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-41ae25-7c78f9c687-gqbtv" event={"ID":"dfd813a8-2d06-4faf-99ed-23c6d83a2b82","Type":"ContainerStarted","Data":"5b3bdc7cf447cbb2cd4c1c388c0a5ed6cfc956c36c36a10f69965fe9de8db87b"} Sep 14 11:22:53.660297 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:53.659822 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-41ae25-7c78f9c687-gqbtv" event={"ID":"dfd813a8-2d06-4faf-99ed-23c6d83a2b82","Type":"ContainerStarted","Data":"46c1fcbcf1c10bf9bcfb93e4536006a57d7626004e43365915d00d7cf99fa840"} Sep 14 11:22:53.660297 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:53.659864 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-41ae25-7c78f9c687-gqbtv" Sep 14 11:22:53.679761 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:22:53.679714 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-41ae25-7c78f9c687-gqbtv" podStartSLOduration=1.6796980559999999 podStartE2EDuration="1.679698056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:22:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:22:53.677706916 +0000 UTC m=+1318.297919464" watchObservedRunningTime="2026-09-14 11:22:53.679698056 +0000 UTC m=+1318.299910597" Sep 14 11:23:04.667465 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:04.667429 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-41ae25-7c78f9c687-gqbtv" Sep 14 11:23:27.967068 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:27.966995 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf"] Sep 14 11:23:27.967579 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:27.967235 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf" podUID="c305a6f4-bb07-435f-9a79-c6580a82346c" containerName="maas-api" containerID="cri-o://76d6afe5f168ab5c498887d66bcb77537ce8a58a910521821d0c70d24f086c6f" gracePeriod=30 Sep 14 11:23:28.042196 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:28.042166 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6d272c-54ccb485-dzrnb"] Sep 14 11:23:28.042478 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:28.042447 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-6d272c-54ccb485-dzrnb" podUID="a9563868-8c3b-4100-a7db-28c76e66fe40" containerName="payload-processing" containerID="cri-o://4f89c3265bd6fadea498c04fbee51b3a4d9835ae2d3662676c22c055d2bdb0a4" gracePeriod=30 Sep 14 11:23:28.307564 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:28.307538 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf" Sep 14 11:23:28.357340 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:28.357310 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6d272c-54ccb485-dzrnb" Sep 14 11:23:28.468251 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:28.468225 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4z9kf\" (UniqueName: \"kubernetes.io/projected/a9563868-8c3b-4100-a7db-28c76e66fe40-kube-api-access-4z9kf\") pod \"a9563868-8c3b-4100-a7db-28c76e66fe40\" (UID: \"a9563868-8c3b-4100-a7db-28c76e66fe40\") " Sep 14 11:23:28.468433 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:28.468314 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c305a6f4-bb07-435f-9a79-c6580a82346c-metrics-certs\") pod \"c305a6f4-bb07-435f-9a79-c6580a82346c\" (UID: \"c305a6f4-bb07-435f-9a79-c6580a82346c\") " Sep 14 11:23:28.468433 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:28.468339 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c305a6f4-bb07-435f-9a79-c6580a82346c-maas-api-tls\") pod \"c305a6f4-bb07-435f-9a79-c6580a82346c\" (UID: \"c305a6f4-bb07-435f-9a79-c6580a82346c\") " Sep 14 11:23:28.468433 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:28.468373 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldgm5\" (UniqueName: \"kubernetes.io/projected/c305a6f4-bb07-435f-9a79-c6580a82346c-kube-api-access-ldgm5\") pod \"c305a6f4-bb07-435f-9a79-c6580a82346c\" (UID: \"c305a6f4-bb07-435f-9a79-c6580a82346c\") " Sep 14 11:23:28.468433 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:28.468406 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9563868-8c3b-4100-a7db-28c76e66fe40-plugins-config-volume\") pod \"a9563868-8c3b-4100-a7db-28c76e66fe40\" (UID: \"a9563868-8c3b-4100-a7db-28c76e66fe40\") " Sep 14 11:23:28.468674 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:28.468514 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c305a6f4-bb07-435f-9a79-c6580a82346c-openshift-service-ca\") pod \"c305a6f4-bb07-435f-9a79-c6580a82346c\" (UID: \"c305a6f4-bb07-435f-9a79-c6580a82346c\") " Sep 14 11:23:28.468859 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:28.468821 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c305a6f4-bb07-435f-9a79-c6580a82346c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c305a6f4-bb07-435f-9a79-c6580a82346c" (UID: "c305a6f4-bb07-435f-9a79-c6580a82346c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:23:28.468968 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:28.468846 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9563868-8c3b-4100-a7db-28c76e66fe40-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a9563868-8c3b-4100-a7db-28c76e66fe40" (UID: "a9563868-8c3b-4100-a7db-28c76e66fe40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:23:28.468968 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:28.468955 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9563868-8c3b-4100-a7db-28c76e66fe40-plugins-config-volume\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:23:28.469088 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:28.468975 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c305a6f4-bb07-435f-9a79-c6580a82346c-openshift-service-ca\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:23:28.470809 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:28.470785 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c305a6f4-bb07-435f-9a79-c6580a82346c-kube-api-access-ldgm5" (OuterVolumeSpecName: "kube-api-access-ldgm5") pod "c305a6f4-bb07-435f-9a79-c6580a82346c" (UID: "c305a6f4-bb07-435f-9a79-c6580a82346c"). InnerVolumeSpecName "kube-api-access-ldgm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:23:28.470914 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:28.470784 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c305a6f4-bb07-435f-9a79-c6580a82346c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c305a6f4-bb07-435f-9a79-c6580a82346c" (UID: "c305a6f4-bb07-435f-9a79-c6580a82346c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:23:28.471079 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:28.471058 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c305a6f4-bb07-435f-9a79-c6580a82346c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c305a6f4-bb07-435f-9a79-c6580a82346c" (UID: "c305a6f4-bb07-435f-9a79-c6580a82346c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:23:28.471079 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:28.471061 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9563868-8c3b-4100-a7db-28c76e66fe40-kube-api-access-4z9kf" (OuterVolumeSpecName: "kube-api-access-4z9kf") pod "a9563868-8c3b-4100-a7db-28c76e66fe40" (UID: "a9563868-8c3b-4100-a7db-28c76e66fe40"). InnerVolumeSpecName "kube-api-access-4z9kf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:23:28.570509 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:28.570436 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z9kf\" (UniqueName: \"kubernetes.io/projected/a9563868-8c3b-4100-a7db-28c76e66fe40-kube-api-access-4z9kf\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:23:28.570509 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:28.570462 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c305a6f4-bb07-435f-9a79-c6580a82346c-metrics-certs\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:23:28.570509 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:28.570472 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c305a6f4-bb07-435f-9a79-c6580a82346c-maas-api-tls\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:23:28.570509 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:28.570481 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldgm5\" (UniqueName: \"kubernetes.io/projected/c305a6f4-bb07-435f-9a79-c6580a82346c-kube-api-access-ldgm5\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:23:28.861005 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:28.860915 2574 generic.go:356] "Generic (PLEG): container finished" podID="a9563868-8c3b-4100-a7db-28c76e66fe40" containerID="4f89c3265bd6fadea498c04fbee51b3a4d9835ae2d3662676c22c055d2bdb0a4" exitCode=0 Sep 14 11:23:28.861005 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:28.860965 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6d272c-54ccb485-dzrnb" Sep 14 11:23:28.861240 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:28.861009 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6d272c-54ccb485-dzrnb" event={"ID":"a9563868-8c3b-4100-a7db-28c76e66fe40","Type":"ContainerDied","Data":"4f89c3265bd6fadea498c04fbee51b3a4d9835ae2d3662676c22c055d2bdb0a4"} Sep 14 11:23:28.861240 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:28.861071 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6d272c-54ccb485-dzrnb" event={"ID":"a9563868-8c3b-4100-a7db-28c76e66fe40","Type":"ContainerDied","Data":"d91167491a359dbec5e95c862cb09d8db8ad8e019b4b85e1858b684de81663db"} Sep 14 11:23:28.861240 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:28.861103 2574 scope.go:117] "RemoveContainer" containerID="4f89c3265bd6fadea498c04fbee51b3a4d9835ae2d3662676c22c055d2bdb0a4" Sep 14 11:23:28.862757 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:28.862733 2574 generic.go:356] "Generic (PLEG): container finished" podID="c305a6f4-bb07-435f-9a79-c6580a82346c" containerID="76d6afe5f168ab5c498887d66bcb77537ce8a58a910521821d0c70d24f086c6f" exitCode=0 Sep 14 11:23:28.862915 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:28.862766 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf" event={"ID":"c305a6f4-bb07-435f-9a79-c6580a82346c","Type":"ContainerDied","Data":"76d6afe5f168ab5c498887d66bcb77537ce8a58a910521821d0c70d24f086c6f"} Sep 14 11:23:28.862915 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:28.862803 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf" Sep 14 11:23:28.862915 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:28.862832 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf" event={"ID":"c305a6f4-bb07-435f-9a79-c6580a82346c","Type":"ContainerDied","Data":"395850fd7fff30f902e9ba5d49c22bb81ffb98b636f8902faa7b5e9e64f381e7"} Sep 14 11:23:28.872559 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:28.872527 2574 scope.go:117] "RemoveContainer" containerID="4f89c3265bd6fadea498c04fbee51b3a4d9835ae2d3662676c22c055d2bdb0a4" Sep 14 11:23:28.872945 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:23:28.872907 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f89c3265bd6fadea498c04fbee51b3a4d9835ae2d3662676c22c055d2bdb0a4\": container with ID starting with 4f89c3265bd6fadea498c04fbee51b3a4d9835ae2d3662676c22c055d2bdb0a4 not found: ID does not exist" containerID="4f89c3265bd6fadea498c04fbee51b3a4d9835ae2d3662676c22c055d2bdb0a4" Sep 14 11:23:28.873033 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:28.872948 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f89c3265bd6fadea498c04fbee51b3a4d9835ae2d3662676c22c055d2bdb0a4"} err="failed to get container status \"4f89c3265bd6fadea498c04fbee51b3a4d9835ae2d3662676c22c055d2bdb0a4\": rpc error: code = NotFound desc = could not find container \"4f89c3265bd6fadea498c04fbee51b3a4d9835ae2d3662676c22c055d2bdb0a4\": container with ID starting with 4f89c3265bd6fadea498c04fbee51b3a4d9835ae2d3662676c22c055d2bdb0a4 not found: ID does not exist" Sep 14 11:23:28.873033 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:28.872970 2574 scope.go:117] "RemoveContainer" containerID="76d6afe5f168ab5c498887d66bcb77537ce8a58a910521821d0c70d24f086c6f" Sep 14 11:23:28.881751 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:28.881735 2574 scope.go:117] "RemoveContainer" containerID="76d6afe5f168ab5c498887d66bcb77537ce8a58a910521821d0c70d24f086c6f" Sep 14 11:23:28.881991 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:23:28.881974 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76d6afe5f168ab5c498887d66bcb77537ce8a58a910521821d0c70d24f086c6f\": container with ID starting with 76d6afe5f168ab5c498887d66bcb77537ce8a58a910521821d0c70d24f086c6f not found: ID does not exist" containerID="76d6afe5f168ab5c498887d66bcb77537ce8a58a910521821d0c70d24f086c6f" Sep 14 11:23:28.882038 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:28.881998 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76d6afe5f168ab5c498887d66bcb77537ce8a58a910521821d0c70d24f086c6f"} err="failed to get container status \"76d6afe5f168ab5c498887d66bcb77537ce8a58a910521821d0c70d24f086c6f\": rpc error: code = NotFound desc = could not find container \"76d6afe5f168ab5c498887d66bcb77537ce8a58a910521821d0c70d24f086c6f\": container with ID starting with 76d6afe5f168ab5c498887d66bcb77537ce8a58a910521821d0c70d24f086c6f not found: ID does not exist" Sep 14 11:23:28.917334 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:28.917311 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf"] Sep 14 11:23:28.925593 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:28.925572 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6d272c-568f45cd-rrmxf"] Sep 14 11:23:28.988206 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:28.988184 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6d272c-54ccb485-dzrnb"] Sep 14 11:23:29.009277 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:29.009241 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6d272c-54ccb485-dzrnb"] Sep 14 11:23:29.898708 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:29.898678 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9563868-8c3b-4100-a7db-28c76e66fe40" path="/var/lib/kubelet/pods/a9563868-8c3b-4100-a7db-28c76e66fe40/volumes" Sep 14 11:23:29.899073 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:29.899060 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c305a6f4-bb07-435f-9a79-c6580a82346c" path="/var/lib/kubelet/pods/c305a6f4-bb07-435f-9a79-c6580a82346c/volumes" Sep 14 11:23:35.310894 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:35.310861 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7cc544b6-577b49b478-855h5"] Sep 14 11:23:35.311355 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:35.311342 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9563868-8c3b-4100-a7db-28c76e66fe40" containerName="payload-processing" Sep 14 11:23:35.311355 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:35.311357 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9563868-8c3b-4100-a7db-28c76e66fe40" containerName="payload-processing" Sep 14 11:23:35.311451 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:35.311380 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c305a6f4-bb07-435f-9a79-c6580a82346c" containerName="maas-api" Sep 14 11:23:35.311451 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:35.311386 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c305a6f4-bb07-435f-9a79-c6580a82346c" containerName="maas-api" Sep 14 11:23:35.311521 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:35.311454 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9563868-8c3b-4100-a7db-28c76e66fe40" containerName="payload-processing" Sep 14 11:23:35.311521 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:35.311466 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c305a6f4-bb07-435f-9a79-c6580a82346c" containerName="maas-api" Sep 14 11:23:35.316141 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:35.316124 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7cc544b6-577b49b478-855h5" Sep 14 11:23:35.318562 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:35.318542 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-7cc544b6-serving-cert\"" Sep 14 11:23:35.327994 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:35.327971 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7cc544b6-577b49b478-855h5"] Sep 14 11:23:35.340516 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:35.340484 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/726c5799-69bd-47ab-9158-91066369e82e-maas-api-tls\") pod \"maas-api-e2e-mt-7cc544b6-577b49b478-855h5\" (UID: \"726c5799-69bd-47ab-9158-91066369e82e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7cc544b6-577b49b478-855h5" Sep 14 11:23:35.340655 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:35.340572 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/726c5799-69bd-47ab-9158-91066369e82e-openshift-service-ca\") pod \"maas-api-e2e-mt-7cc544b6-577b49b478-855h5\" (UID: \"726c5799-69bd-47ab-9158-91066369e82e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7cc544b6-577b49b478-855h5" Sep 14 11:23:35.340655 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:35.340604 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkqmd\" (UniqueName: \"kubernetes.io/projected/726c5799-69bd-47ab-9158-91066369e82e-kube-api-access-xkqmd\") pod \"maas-api-e2e-mt-7cc544b6-577b49b478-855h5\" (UID: \"726c5799-69bd-47ab-9158-91066369e82e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7cc544b6-577b49b478-855h5" Sep 14 11:23:35.340792 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:35.340681 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/726c5799-69bd-47ab-9158-91066369e82e-metrics-certs\") pod \"maas-api-e2e-mt-7cc544b6-577b49b478-855h5\" (UID: \"726c5799-69bd-47ab-9158-91066369e82e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7cc544b6-577b49b478-855h5" Sep 14 11:23:35.441972 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:35.441938 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/726c5799-69bd-47ab-9158-91066369e82e-maas-api-tls\") pod \"maas-api-e2e-mt-7cc544b6-577b49b478-855h5\" (UID: \"726c5799-69bd-47ab-9158-91066369e82e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7cc544b6-577b49b478-855h5" Sep 14 11:23:35.442141 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:35.442009 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/726c5799-69bd-47ab-9158-91066369e82e-openshift-service-ca\") pod \"maas-api-e2e-mt-7cc544b6-577b49b478-855h5\" (UID: \"726c5799-69bd-47ab-9158-91066369e82e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7cc544b6-577b49b478-855h5" Sep 14 11:23:35.442141 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:35.442050 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkqmd\" (UniqueName: \"kubernetes.io/projected/726c5799-69bd-47ab-9158-91066369e82e-kube-api-access-xkqmd\") pod \"maas-api-e2e-mt-7cc544b6-577b49b478-855h5\" (UID: \"726c5799-69bd-47ab-9158-91066369e82e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7cc544b6-577b49b478-855h5" Sep 14 11:23:35.442141 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:35.442115 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/726c5799-69bd-47ab-9158-91066369e82e-metrics-certs\") pod \"maas-api-e2e-mt-7cc544b6-577b49b478-855h5\" (UID: \"726c5799-69bd-47ab-9158-91066369e82e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7cc544b6-577b49b478-855h5" Sep 14 11:23:35.442756 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:35.442678 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/726c5799-69bd-47ab-9158-91066369e82e-openshift-service-ca\") pod \"maas-api-e2e-mt-7cc544b6-577b49b478-855h5\" (UID: \"726c5799-69bd-47ab-9158-91066369e82e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7cc544b6-577b49b478-855h5" Sep 14 11:23:35.444554 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:35.444532 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/726c5799-69bd-47ab-9158-91066369e82e-maas-api-tls\") pod \"maas-api-e2e-mt-7cc544b6-577b49b478-855h5\" (UID: \"726c5799-69bd-47ab-9158-91066369e82e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7cc544b6-577b49b478-855h5" Sep 14 11:23:35.444637 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:35.444558 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/726c5799-69bd-47ab-9158-91066369e82e-metrics-certs\") pod \"maas-api-e2e-mt-7cc544b6-577b49b478-855h5\" (UID: \"726c5799-69bd-47ab-9158-91066369e82e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7cc544b6-577b49b478-855h5" Sep 14 11:23:35.453791 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:35.453770 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkqmd\" (UniqueName: \"kubernetes.io/projected/726c5799-69bd-47ab-9158-91066369e82e-kube-api-access-xkqmd\") pod \"maas-api-e2e-mt-7cc544b6-577b49b478-855h5\" (UID: \"726c5799-69bd-47ab-9158-91066369e82e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7cc544b6-577b49b478-855h5" Sep 14 11:23:35.629022 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:35.628945 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7cc544b6-577b49b478-855h5" Sep 14 11:23:35.917058 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:35.917035 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7cc544b6-577b49b478-855h5"] Sep 14 11:23:35.919160 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:23:35.919134 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod726c5799_69bd_47ab_9158_91066369e82e.slice/crio-090ed6b29d1299de260b0a176ae1931817ef822003e945aadc5c6af54f6a7bdf WatchSource:0}: Error finding container 090ed6b29d1299de260b0a176ae1931817ef822003e945aadc5c6af54f6a7bdf: Status 404 returned error can't find the container with id 090ed6b29d1299de260b0a176ae1931817ef822003e945aadc5c6af54f6a7bdf Sep 14 11:23:36.159292 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:36.159239 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7cc544b6-6cc9546b66-jxdkn"] Sep 14 11:23:36.164469 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:36.164447 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7cc544b6-6cc9546b66-jxdkn" Sep 14 11:23:36.167067 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:36.166981 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-7cc544b6-dockercfg-2rv8v\"" Sep 14 11:23:36.167201 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:36.167152 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7cc544b6\"" Sep 14 11:23:36.178330 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:36.178305 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7cc544b6-6cc9546b66-jxdkn"] Sep 14 11:23:36.251922 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:36.251741 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c107ae62-f202-42dc-ab3c-054285a1673c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7cc544b6-6cc9546b66-jxdkn\" (UID: \"c107ae62-f202-42dc-ab3c-054285a1673c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7cc544b6-6cc9546b66-jxdkn" Sep 14 11:23:36.251922 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:36.251812 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b56w4\" (UniqueName: \"kubernetes.io/projected/c107ae62-f202-42dc-ab3c-054285a1673c-kube-api-access-b56w4\") pod \"payload-pre-processing-e2e-mt-7cc544b6-6cc9546b66-jxdkn\" (UID: \"c107ae62-f202-42dc-ab3c-054285a1673c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7cc544b6-6cc9546b66-jxdkn" Sep 14 11:23:36.352778 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:36.352747 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c107ae62-f202-42dc-ab3c-054285a1673c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7cc544b6-6cc9546b66-jxdkn\" (UID: \"c107ae62-f202-42dc-ab3c-054285a1673c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7cc544b6-6cc9546b66-jxdkn" Sep 14 11:23:36.353173 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:36.352793 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b56w4\" (UniqueName: \"kubernetes.io/projected/c107ae62-f202-42dc-ab3c-054285a1673c-kube-api-access-b56w4\") pod \"payload-pre-processing-e2e-mt-7cc544b6-6cc9546b66-jxdkn\" (UID: \"c107ae62-f202-42dc-ab3c-054285a1673c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7cc544b6-6cc9546b66-jxdkn" Sep 14 11:23:36.353451 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:36.353424 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c107ae62-f202-42dc-ab3c-054285a1673c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7cc544b6-6cc9546b66-jxdkn\" (UID: \"c107ae62-f202-42dc-ab3c-054285a1673c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7cc544b6-6cc9546b66-jxdkn" Sep 14 11:23:36.361631 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:36.361612 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b56w4\" (UniqueName: \"kubernetes.io/projected/c107ae62-f202-42dc-ab3c-054285a1673c-kube-api-access-b56w4\") pod \"payload-pre-processing-e2e-mt-7cc544b6-6cc9546b66-jxdkn\" (UID: \"c107ae62-f202-42dc-ab3c-054285a1673c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7cc544b6-6cc9546b66-jxdkn" Sep 14 11:23:36.480115 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:36.480084 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7cc544b6-6cc9546b66-jxdkn" Sep 14 11:23:36.762941 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:36.762917 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7cc544b6-6cc9546b66-jxdkn"] Sep 14 11:23:36.764972 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:23:36.764938 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc107ae62_f202_42dc_ab3c_054285a1673c.slice/crio-34376b45f27b865769bceb2768f51ddf68d597c8068735e2ba02c05124284138 WatchSource:0}: Error finding container 34376b45f27b865769bceb2768f51ddf68d597c8068735e2ba02c05124284138: Status 404 returned error can't find the container with id 34376b45f27b865769bceb2768f51ddf68d597c8068735e2ba02c05124284138 Sep 14 11:23:36.916393 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:36.916353 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7cc544b6-577b49b478-855h5" event={"ID":"726c5799-69bd-47ab-9158-91066369e82e","Type":"ContainerStarted","Data":"78e33bac56da92aef35839c7b0af6108441550d64c9d25f049a776deb866e7e1"} Sep 14 11:23:36.916628 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:36.916600 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7cc544b6-577b49b478-855h5" Sep 14 11:23:36.916764 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:36.916650 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7cc544b6-577b49b478-855h5" event={"ID":"726c5799-69bd-47ab-9158-91066369e82e","Type":"ContainerStarted","Data":"090ed6b29d1299de260b0a176ae1931817ef822003e945aadc5c6af54f6a7bdf"} Sep 14 11:23:36.918366 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:36.918341 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7cc544b6-6cc9546b66-jxdkn" event={"ID":"c107ae62-f202-42dc-ab3c-054285a1673c","Type":"ContainerStarted","Data":"979f890669c8b2d3228d60c372d4a8ee8099e39b07670d8a9a881f0d39b4a568"} Sep 14 11:23:36.918476 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:36.918390 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7cc544b6-6cc9546b66-jxdkn" event={"ID":"c107ae62-f202-42dc-ab3c-054285a1673c","Type":"ContainerStarted","Data":"34376b45f27b865769bceb2768f51ddf68d597c8068735e2ba02c05124284138"} Sep 14 11:23:36.918536 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:36.918478 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7cc544b6-6cc9546b66-jxdkn" Sep 14 11:23:36.942416 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:36.942354 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7cc544b6-577b49b478-855h5" podStartSLOduration=1.470492773 podStartE2EDuration="1.94233584s" totalImagesPullingTime="471.843067ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:23:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:23:36.939601106 +0000 UTC m=+1361.559813651" watchObservedRunningTime="2026-09-14 11:23:36.94233584 +0000 UTC m=+1361.562548364" Sep 14 11:23:36.960902 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:36.960848 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-7cc544b6-6cc9546b66-jxdkn" podStartSLOduration=0.960834194 podStartE2EDuration="960.834194ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:23:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:23:36.957946075 +0000 UTC m=+1361.578158620" watchObservedRunningTime="2026-09-14 11:23:36.960834194 +0000 UTC m=+1361.581046717" Sep 14 11:23:42.930616 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:42.930584 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7cc544b6-577b49b478-855h5" Sep 14 11:23:47.258487 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:47.258448 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd"] Sep 14 11:23:47.258920 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:47.258753 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd" podUID="9136cf83-c206-4f67-b1c2-6b3f55eb5d24" containerName="maas-api" containerID="cri-o://f72d54efcae28c4d9e1b8deb0a44522df8f09ce6e32175d988aa0d2a3b4e37dc" gracePeriod=30 Sep 14 11:23:47.328891 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:47.328854 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-39ab34-5449674498-jzrkv"] Sep 14 11:23:47.329096 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:47.329070 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39ab34-5449674498-jzrkv" podUID="84925d6e-d030-437c-a141-7168ad5f9f79" containerName="payload-pre-processing" containerID="cri-o://a689cbdf31c5da326430d4016588521ccbe206dec64be7423d304e1ea5068a51" gracePeriod=30 Sep 14 11:23:47.598016 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:47.597988 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd" Sep 14 11:23:47.641441 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:47.641413 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39ab34-5449674498-jzrkv" Sep 14 11:23:47.663587 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:47.663552 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gcsd\" (UniqueName: \"kubernetes.io/projected/9136cf83-c206-4f67-b1c2-6b3f55eb5d24-kube-api-access-4gcsd\") pod \"9136cf83-c206-4f67-b1c2-6b3f55eb5d24\" (UID: \"9136cf83-c206-4f67-b1c2-6b3f55eb5d24\") " Sep 14 11:23:47.663760 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:47.663701 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9136cf83-c206-4f67-b1c2-6b3f55eb5d24-maas-api-tls\") pod \"9136cf83-c206-4f67-b1c2-6b3f55eb5d24\" (UID: \"9136cf83-c206-4f67-b1c2-6b3f55eb5d24\") " Sep 14 11:23:47.663760 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:47.663744 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9136cf83-c206-4f67-b1c2-6b3f55eb5d24-metrics-certs\") pod \"9136cf83-c206-4f67-b1c2-6b3f55eb5d24\" (UID: \"9136cf83-c206-4f67-b1c2-6b3f55eb5d24\") " Sep 14 11:23:47.663873 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:47.663778 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9136cf83-c206-4f67-b1c2-6b3f55eb5d24-openshift-service-ca\") pod \"9136cf83-c206-4f67-b1c2-6b3f55eb5d24\" (UID: \"9136cf83-c206-4f67-b1c2-6b3f55eb5d24\") " Sep 14 11:23:47.664501 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:47.664467 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9136cf83-c206-4f67-b1c2-6b3f55eb5d24-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9136cf83-c206-4f67-b1c2-6b3f55eb5d24" (UID: "9136cf83-c206-4f67-b1c2-6b3f55eb5d24"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:23:47.666810 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:47.666768 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9136cf83-c206-4f67-b1c2-6b3f55eb5d24-kube-api-access-4gcsd" (OuterVolumeSpecName: "kube-api-access-4gcsd") pod "9136cf83-c206-4f67-b1c2-6b3f55eb5d24" (UID: "9136cf83-c206-4f67-b1c2-6b3f55eb5d24"). InnerVolumeSpecName "kube-api-access-4gcsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:23:47.666909 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:47.666836 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9136cf83-c206-4f67-b1c2-6b3f55eb5d24-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9136cf83-c206-4f67-b1c2-6b3f55eb5d24" (UID: "9136cf83-c206-4f67-b1c2-6b3f55eb5d24"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:23:47.667230 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:47.667159 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9136cf83-c206-4f67-b1c2-6b3f55eb5d24-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9136cf83-c206-4f67-b1c2-6b3f55eb5d24" (UID: "9136cf83-c206-4f67-b1c2-6b3f55eb5d24"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:23:47.765097 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:47.765060 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84925d6e-d030-437c-a141-7168ad5f9f79-plugins-config-volume\") pod \"84925d6e-d030-437c-a141-7168ad5f9f79\" (UID: \"84925d6e-d030-437c-a141-7168ad5f9f79\") " Sep 14 11:23:47.765097 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:47.765104 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89cjx\" (UniqueName: \"kubernetes.io/projected/84925d6e-d030-437c-a141-7168ad5f9f79-kube-api-access-89cjx\") pod \"84925d6e-d030-437c-a141-7168ad5f9f79\" (UID: \"84925d6e-d030-437c-a141-7168ad5f9f79\") " Sep 14 11:23:47.765515 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:47.765489 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84925d6e-d030-437c-a141-7168ad5f9f79-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "84925d6e-d030-437c-a141-7168ad5f9f79" (UID: "84925d6e-d030-437c-a141-7168ad5f9f79"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:23:47.765618 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:47.765604 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9136cf83-c206-4f67-b1c2-6b3f55eb5d24-maas-api-tls\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:23:47.765663 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:47.765621 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9136cf83-c206-4f67-b1c2-6b3f55eb5d24-metrics-certs\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:23:47.765663 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:47.765631 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84925d6e-d030-437c-a141-7168ad5f9f79-plugins-config-volume\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:23:47.765663 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:47.765640 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9136cf83-c206-4f67-b1c2-6b3f55eb5d24-openshift-service-ca\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:23:47.765663 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:47.765648 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gcsd\" (UniqueName: \"kubernetes.io/projected/9136cf83-c206-4f67-b1c2-6b3f55eb5d24-kube-api-access-4gcsd\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:23:47.767411 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:47.767389 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84925d6e-d030-437c-a141-7168ad5f9f79-kube-api-access-89cjx" (OuterVolumeSpecName: "kube-api-access-89cjx") pod "84925d6e-d030-437c-a141-7168ad5f9f79" (UID: "84925d6e-d030-437c-a141-7168ad5f9f79"). InnerVolumeSpecName "kube-api-access-89cjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:23:47.866232 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:47.866155 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89cjx\" (UniqueName: \"kubernetes.io/projected/84925d6e-d030-437c-a141-7168ad5f9f79-kube-api-access-89cjx\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:23:47.927242 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:47.927217 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7cc544b6-6cc9546b66-jxdkn" Sep 14 11:23:47.981406 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:47.981369 2574 generic.go:356] "Generic (PLEG): container finished" podID="84925d6e-d030-437c-a141-7168ad5f9f79" containerID="a689cbdf31c5da326430d4016588521ccbe206dec64be7423d304e1ea5068a51" exitCode=0 Sep 14 11:23:47.981577 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:47.981418 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39ab34-5449674498-jzrkv" Sep 14 11:23:47.981577 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:47.981455 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39ab34-5449674498-jzrkv" event={"ID":"84925d6e-d030-437c-a141-7168ad5f9f79","Type":"ContainerDied","Data":"a689cbdf31c5da326430d4016588521ccbe206dec64be7423d304e1ea5068a51"} Sep 14 11:23:47.981577 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:47.981511 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-39ab34-5449674498-jzrkv" event={"ID":"84925d6e-d030-437c-a141-7168ad5f9f79","Type":"ContainerDied","Data":"ddebf45ca7283c5d06705a83d0c14fafcd45d376701120eaba24805093919596"} Sep 14 11:23:47.981577 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:47.981541 2574 scope.go:117] "RemoveContainer" containerID="a689cbdf31c5da326430d4016588521ccbe206dec64be7423d304e1ea5068a51" Sep 14 11:23:47.982869 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:47.982846 2574 generic.go:356] "Generic (PLEG): container finished" podID="9136cf83-c206-4f67-b1c2-6b3f55eb5d24" containerID="f72d54efcae28c4d9e1b8deb0a44522df8f09ce6e32175d988aa0d2a3b4e37dc" exitCode=0 Sep 14 11:23:47.982987 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:47.982903 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd" Sep 14 11:23:47.982987 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:47.982951 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd" event={"ID":"9136cf83-c206-4f67-b1c2-6b3f55eb5d24","Type":"ContainerDied","Data":"f72d54efcae28c4d9e1b8deb0a44522df8f09ce6e32175d988aa0d2a3b4e37dc"} Sep 14 11:23:47.983086 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:47.983000 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd" event={"ID":"9136cf83-c206-4f67-b1c2-6b3f55eb5d24","Type":"ContainerDied","Data":"71faf607a67c9ebd676f23968f7d5682f613eb1aa7a7a93f31fa7d61a980a9db"} Sep 14 11:23:47.991403 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:47.991382 2574 scope.go:117] "RemoveContainer" containerID="a689cbdf31c5da326430d4016588521ccbe206dec64be7423d304e1ea5068a51" Sep 14 11:23:47.991696 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:23:47.991676 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a689cbdf31c5da326430d4016588521ccbe206dec64be7423d304e1ea5068a51\": container with ID starting with a689cbdf31c5da326430d4016588521ccbe206dec64be7423d304e1ea5068a51 not found: ID does not exist" containerID="a689cbdf31c5da326430d4016588521ccbe206dec64be7423d304e1ea5068a51" Sep 14 11:23:47.991809 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:47.991711 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a689cbdf31c5da326430d4016588521ccbe206dec64be7423d304e1ea5068a51"} err="failed to get container status \"a689cbdf31c5da326430d4016588521ccbe206dec64be7423d304e1ea5068a51\": rpc error: code = NotFound desc = could not find container \"a689cbdf31c5da326430d4016588521ccbe206dec64be7423d304e1ea5068a51\": container with ID starting with a689cbdf31c5da326430d4016588521ccbe206dec64be7423d304e1ea5068a51 not found: ID does not exist" Sep 14 11:23:47.991809 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:47.991735 2574 scope.go:117] "RemoveContainer" containerID="f72d54efcae28c4d9e1b8deb0a44522df8f09ce6e32175d988aa0d2a3b4e37dc" Sep 14 11:23:48.002755 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:48.002735 2574 scope.go:117] "RemoveContainer" containerID="f72d54efcae28c4d9e1b8deb0a44522df8f09ce6e32175d988aa0d2a3b4e37dc" Sep 14 11:23:48.003097 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:23:48.003077 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f72d54efcae28c4d9e1b8deb0a44522df8f09ce6e32175d988aa0d2a3b4e37dc\": container with ID starting with f72d54efcae28c4d9e1b8deb0a44522df8f09ce6e32175d988aa0d2a3b4e37dc not found: ID does not exist" containerID="f72d54efcae28c4d9e1b8deb0a44522df8f09ce6e32175d988aa0d2a3b4e37dc" Sep 14 11:23:48.003210 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:48.003101 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f72d54efcae28c4d9e1b8deb0a44522df8f09ce6e32175d988aa0d2a3b4e37dc"} err="failed to get container status \"f72d54efcae28c4d9e1b8deb0a44522df8f09ce6e32175d988aa0d2a3b4e37dc\": rpc error: code = NotFound desc = could not find container \"f72d54efcae28c4d9e1b8deb0a44522df8f09ce6e32175d988aa0d2a3b4e37dc\": container with ID starting with f72d54efcae28c4d9e1b8deb0a44522df8f09ce6e32175d988aa0d2a3b4e37dc not found: ID does not exist" Sep 14 11:23:48.036910 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:48.036866 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd"] Sep 14 11:23:48.054661 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:48.054632 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-39ab34-85bf75bd6f-nhbnd"] Sep 14 11:23:48.092514 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:48.092480 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-39ab34-5449674498-jzrkv"] Sep 14 11:23:48.099977 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:48.099948 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-39ab34-5449674498-jzrkv"] Sep 14 11:23:49.899583 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:49.899549 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84925d6e-d030-437c-a141-7168ad5f9f79" path="/var/lib/kubelet/pods/84925d6e-d030-437c-a141-7168ad5f9f79/volumes" Sep 14 11:23:49.899966 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:23:49.899953 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9136cf83-c206-4f67-b1c2-6b3f55eb5d24" path="/var/lib/kubelet/pods/9136cf83-c206-4f67-b1c2-6b3f55eb5d24/volumes" Sep 14 11:24:02.859955 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:02.859923 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-55c025-66d8d54859-x2fb8"] Sep 14 11:24:02.860650 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:02.860632 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84925d6e-d030-437c-a141-7168ad5f9f79" containerName="payload-pre-processing" Sep 14 11:24:02.860738 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:02.860654 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="84925d6e-d030-437c-a141-7168ad5f9f79" containerName="payload-pre-processing" Sep 14 11:24:02.860738 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:02.860693 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9136cf83-c206-4f67-b1c2-6b3f55eb5d24" containerName="maas-api" Sep 14 11:24:02.860738 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:02.860702 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9136cf83-c206-4f67-b1c2-6b3f55eb5d24" containerName="maas-api" Sep 14 11:24:02.860901 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:02.860830 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="84925d6e-d030-437c-a141-7168ad5f9f79" containerName="payload-pre-processing" Sep 14 11:24:02.860901 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:02.860845 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9136cf83-c206-4f67-b1c2-6b3f55eb5d24" containerName="maas-api" Sep 14 11:24:02.865854 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:02.865835 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-55c025-66d8d54859-x2fb8" Sep 14 11:24:02.868562 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:02.868540 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-55c025\"" Sep 14 11:24:02.868667 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:02.868584 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-55c025-dockercfg-kgnk5\"" Sep 14 11:24:02.878896 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:02.878871 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-55c025-66d8d54859-x2fb8"] Sep 14 11:24:02.922338 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:02.922306 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx2tk\" (UniqueName: \"kubernetes.io/projected/47f89cf4-0168-4cf1-a4d2-b001d69e966e-kube-api-access-hx2tk\") pod \"payload-processing-e2e-shared-a-w3-55c025-66d8d54859-x2fb8\" (UID: \"47f89cf4-0168-4cf1-a4d2-b001d69e966e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-55c025-66d8d54859-x2fb8" Sep 14 11:24:02.922598 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:02.922572 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47f89cf4-0168-4cf1-a4d2-b001d69e966e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-55c025-66d8d54859-x2fb8\" (UID: \"47f89cf4-0168-4cf1-a4d2-b001d69e966e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-55c025-66d8d54859-x2fb8" Sep 14 11:24:03.023494 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:03.023458 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47f89cf4-0168-4cf1-a4d2-b001d69e966e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-55c025-66d8d54859-x2fb8\" (UID: \"47f89cf4-0168-4cf1-a4d2-b001d69e966e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-55c025-66d8d54859-x2fb8" Sep 14 11:24:03.023656 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:03.023502 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx2tk\" (UniqueName: \"kubernetes.io/projected/47f89cf4-0168-4cf1-a4d2-b001d69e966e-kube-api-access-hx2tk\") pod \"payload-processing-e2e-shared-a-w3-55c025-66d8d54859-x2fb8\" (UID: \"47f89cf4-0168-4cf1-a4d2-b001d69e966e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-55c025-66d8d54859-x2fb8" Sep 14 11:24:03.024129 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:03.024100 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47f89cf4-0168-4cf1-a4d2-b001d69e966e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-55c025-66d8d54859-x2fb8\" (UID: \"47f89cf4-0168-4cf1-a4d2-b001d69e966e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-55c025-66d8d54859-x2fb8" Sep 14 11:24:03.032303 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:03.032244 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx2tk\" (UniqueName: \"kubernetes.io/projected/47f89cf4-0168-4cf1-a4d2-b001d69e966e-kube-api-access-hx2tk\") pod \"payload-processing-e2e-shared-a-w3-55c025-66d8d54859-x2fb8\" (UID: \"47f89cf4-0168-4cf1-a4d2-b001d69e966e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-55c025-66d8d54859-x2fb8" Sep 14 11:24:03.176871 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:03.176840 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-55c025-66d8d54859-x2fb8" Sep 14 11:24:03.459807 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:03.459782 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-55c025-66d8d54859-x2fb8"] Sep 14 11:24:03.461676 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:24:03.461649 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47f89cf4_0168_4cf1_a4d2_b001d69e966e.slice/crio-3ea8dbe7a8dbf059e4f43c2fc3424801a5035e02d0e444d5fe68194739535e24 WatchSource:0}: Error finding container 3ea8dbe7a8dbf059e4f43c2fc3424801a5035e02d0e444d5fe68194739535e24: Status 404 returned error can't find the container with id 3ea8dbe7a8dbf059e4f43c2fc3424801a5035e02d0e444d5fe68194739535e24 Sep 14 11:24:04.080167 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:04.080126 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-55c025-66d8d54859-x2fb8" event={"ID":"47f89cf4-0168-4cf1-a4d2-b001d69e966e","Type":"ContainerStarted","Data":"cc809165b66981ccd6fd901ea99ef6bac9d1e1bb030c2dcf1b1da99e95eb9de9"} Sep 14 11:24:04.080598 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:04.080191 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-55c025-66d8d54859-x2fb8" event={"ID":"47f89cf4-0168-4cf1-a4d2-b001d69e966e","Type":"ContainerStarted","Data":"3ea8dbe7a8dbf059e4f43c2fc3424801a5035e02d0e444d5fe68194739535e24"} Sep 14 11:24:04.080598 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:04.080229 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-55c025-66d8d54859-x2fb8" Sep 14 11:24:04.103257 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:04.103210 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-55c025-66d8d54859-x2fb8" podStartSLOduration=2.103193895 podStartE2EDuration="2.103193895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:24:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:24:04.100763808 +0000 UTC m=+1388.720976355" watchObservedRunningTime="2026-09-14 11:24:04.103193895 +0000 UTC m=+1388.723406417" Sep 14 11:24:15.088848 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:15.088819 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-55c025-66d8d54859-x2fb8" Sep 14 11:24:41.807798 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:41.807762 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3f02a5-857f4fc78-7pl5f"] Sep 14 11:24:41.812569 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:41.812553 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3f02a5-857f4fc78-7pl5f" Sep 14 11:24:41.815510 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:41.815493 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-3f02a5\"" Sep 14 11:24:41.829111 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:41.829085 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3f02a5-857f4fc78-7pl5f"] Sep 14 11:24:41.838303 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:41.838256 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3f02a5-5449d5f5f9-5l7hw"] Sep 14 11:24:41.842734 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:41.842717 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3f02a5-5449d5f5f9-5l7hw" Sep 14 11:24:41.845449 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:41.845423 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-3f02a5-dockercfg-kjqq7\"" Sep 14 11:24:41.855461 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:41.855440 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3f02a5-5449d5f5f9-5l7hw"] Sep 14 11:24:41.901283 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:41.901243 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebd4bc7e-3e14-4fd1-bc25-d16c664b9af4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-3f02a5-5449d5f5f9-5l7hw\" (UID: \"ebd4bc7e-3e14-4fd1-bc25-d16c664b9af4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3f02a5-5449d5f5f9-5l7hw" Sep 14 11:24:41.901432 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:41.901307 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57685ac4-6a88-4780-8fcc-40d91b2aa5fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-3f02a5-857f4fc78-7pl5f\" (UID: \"57685ac4-6a88-4780-8fcc-40d91b2aa5fd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3f02a5-857f4fc78-7pl5f" Sep 14 11:24:41.901432 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:41.901377 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lspn\" (UniqueName: \"kubernetes.io/projected/57685ac4-6a88-4780-8fcc-40d91b2aa5fd-kube-api-access-2lspn\") pod \"payload-pre-processing-e2e-shared-b-w3-3f02a5-857f4fc78-7pl5f\" (UID: \"57685ac4-6a88-4780-8fcc-40d91b2aa5fd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3f02a5-857f4fc78-7pl5f" Sep 14 11:24:41.901552 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:41.901449 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54rbh\" (UniqueName: \"kubernetes.io/projected/ebd4bc7e-3e14-4fd1-bc25-d16c664b9af4-kube-api-access-54rbh\") pod \"payload-processing-e2e-shared-b-w3-3f02a5-5449d5f5f9-5l7hw\" (UID: \"ebd4bc7e-3e14-4fd1-bc25-d16c664b9af4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3f02a5-5449d5f5f9-5l7hw" Sep 14 11:24:42.002087 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:42.002050 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54rbh\" (UniqueName: \"kubernetes.io/projected/ebd4bc7e-3e14-4fd1-bc25-d16c664b9af4-kube-api-access-54rbh\") pod \"payload-processing-e2e-shared-b-w3-3f02a5-5449d5f5f9-5l7hw\" (UID: \"ebd4bc7e-3e14-4fd1-bc25-d16c664b9af4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3f02a5-5449d5f5f9-5l7hw" Sep 14 11:24:42.002256 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:42.002178 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebd4bc7e-3e14-4fd1-bc25-d16c664b9af4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-3f02a5-5449d5f5f9-5l7hw\" (UID: \"ebd4bc7e-3e14-4fd1-bc25-d16c664b9af4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3f02a5-5449d5f5f9-5l7hw" Sep 14 11:24:42.002256 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:42.002202 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57685ac4-6a88-4780-8fcc-40d91b2aa5fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-3f02a5-857f4fc78-7pl5f\" (UID: \"57685ac4-6a88-4780-8fcc-40d91b2aa5fd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3f02a5-857f4fc78-7pl5f" Sep 14 11:24:42.002256 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:42.002230 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lspn\" (UniqueName: \"kubernetes.io/projected/57685ac4-6a88-4780-8fcc-40d91b2aa5fd-kube-api-access-2lspn\") pod \"payload-pre-processing-e2e-shared-b-w3-3f02a5-857f4fc78-7pl5f\" (UID: \"57685ac4-6a88-4780-8fcc-40d91b2aa5fd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3f02a5-857f4fc78-7pl5f" Sep 14 11:24:42.002978 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:42.002954 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebd4bc7e-3e14-4fd1-bc25-d16c664b9af4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-3f02a5-5449d5f5f9-5l7hw\" (UID: \"ebd4bc7e-3e14-4fd1-bc25-d16c664b9af4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3f02a5-5449d5f5f9-5l7hw" Sep 14 11:24:42.003086 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:42.003009 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57685ac4-6a88-4780-8fcc-40d91b2aa5fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-3f02a5-857f4fc78-7pl5f\" (UID: \"57685ac4-6a88-4780-8fcc-40d91b2aa5fd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3f02a5-857f4fc78-7pl5f" Sep 14 11:24:42.012076 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:42.012048 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54rbh\" (UniqueName: \"kubernetes.io/projected/ebd4bc7e-3e14-4fd1-bc25-d16c664b9af4-kube-api-access-54rbh\") pod \"payload-processing-e2e-shared-b-w3-3f02a5-5449d5f5f9-5l7hw\" (UID: \"ebd4bc7e-3e14-4fd1-bc25-d16c664b9af4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3f02a5-5449d5f5f9-5l7hw" Sep 14 11:24:42.012230 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:42.012207 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lspn\" (UniqueName: \"kubernetes.io/projected/57685ac4-6a88-4780-8fcc-40d91b2aa5fd-kube-api-access-2lspn\") pod \"payload-pre-processing-e2e-shared-b-w3-3f02a5-857f4fc78-7pl5f\" (UID: \"57685ac4-6a88-4780-8fcc-40d91b2aa5fd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3f02a5-857f4fc78-7pl5f" Sep 14 11:24:42.123076 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:42.122984 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3f02a5-857f4fc78-7pl5f" Sep 14 11:24:42.155463 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:42.155425 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3f02a5-5449d5f5f9-5l7hw" Sep 14 11:24:42.409452 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:42.409426 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3f02a5-857f4fc78-7pl5f"] Sep 14 11:24:42.409972 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:24:42.409951 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57685ac4_6a88_4780_8fcc_40d91b2aa5fd.slice/crio-f177ab2c3c123aa093db64762fb6bfe5d6c5264d5a1398e737dfe0dd81cd961c WatchSource:0}: Error finding container f177ab2c3c123aa093db64762fb6bfe5d6c5264d5a1398e737dfe0dd81cd961c: Status 404 returned error can't find the container with id f177ab2c3c123aa093db64762fb6bfe5d6c5264d5a1398e737dfe0dd81cd961c Sep 14 11:24:42.446518 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:42.446494 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3f02a5-5449d5f5f9-5l7hw"] Sep 14 11:24:42.448569 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:24:42.448544 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebd4bc7e_3e14_4fd1_bc25_d16c664b9af4.slice/crio-d67dbb8788d250e76cc21413ac04fca0619af3ec292778fdbced24106ab4f7f2 WatchSource:0}: Error finding container d67dbb8788d250e76cc21413ac04fca0619af3ec292778fdbced24106ab4f7f2: Status 404 returned error can't find the container with id d67dbb8788d250e76cc21413ac04fca0619af3ec292778fdbced24106ab4f7f2 Sep 14 11:24:43.301794 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:43.301756 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3f02a5-5449d5f5f9-5l7hw" event={"ID":"ebd4bc7e-3e14-4fd1-bc25-d16c664b9af4","Type":"ContainerStarted","Data":"51e03f2f0a09a85be876b1eee80bef19dd3100e9e4d1723e0aee56f10fd35615"} Sep 14 11:24:43.302255 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:43.301858 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3f02a5-5449d5f5f9-5l7hw" Sep 14 11:24:43.302255 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:43.301884 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3f02a5-5449d5f5f9-5l7hw" event={"ID":"ebd4bc7e-3e14-4fd1-bc25-d16c664b9af4","Type":"ContainerStarted","Data":"d67dbb8788d250e76cc21413ac04fca0619af3ec292778fdbced24106ab4f7f2"} Sep 14 11:24:43.303393 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:43.303369 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3f02a5-857f4fc78-7pl5f" event={"ID":"57685ac4-6a88-4780-8fcc-40d91b2aa5fd","Type":"ContainerStarted","Data":"5e7a6b71a1ca1f16e7e23061288c61dff2b50a24ba89e129ed51d7de81186103"} Sep 14 11:24:43.303478 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:43.303415 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3f02a5-857f4fc78-7pl5f" event={"ID":"57685ac4-6a88-4780-8fcc-40d91b2aa5fd","Type":"ContainerStarted","Data":"f177ab2c3c123aa093db64762fb6bfe5d6c5264d5a1398e737dfe0dd81cd961c"} Sep 14 11:24:43.303523 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:43.303494 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3f02a5-857f4fc78-7pl5f" Sep 14 11:24:43.326765 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:43.326714 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3f02a5-5449d5f5f9-5l7hw" podStartSLOduration=2.326700481 podStartE2EDuration="2.326700481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:24:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:24:43.32260522 +0000 UTC m=+1427.942817745" watchObservedRunningTime="2026-09-14 11:24:43.326700481 +0000 UTC m=+1427.946913003" Sep 14 11:24:43.345029 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:43.344984 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3f02a5-857f4fc78-7pl5f" podStartSLOduration=2.344969605 podStartE2EDuration="2.344969605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:24:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:24:43.342657205 +0000 UTC m=+1427.962869856" watchObservedRunningTime="2026-09-14 11:24:43.344969605 +0000 UTC m=+1427.965182128" Sep 14 11:24:54.311315 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:54.311219 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3f02a5-5449d5f5f9-5l7hw" Sep 14 11:24:54.311675 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:24:54.311349 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3f02a5-857f4fc78-7pl5f" Sep 14 11:25:49.329575 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:25:49.329542 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7cc544b6-577b49b478-855h5"] Sep 14 11:25:49.330119 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:25:49.329800 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7cc544b6-577b49b478-855h5" podUID="726c5799-69bd-47ab-9158-91066369e82e" containerName="maas-api" containerID="cri-o://78e33bac56da92aef35839c7b0af6108441550d64c9d25f049a776deb866e7e1" gracePeriod=30 Sep 14 11:25:49.400182 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:25:49.400150 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7cc544b6-6cc9546b66-jxdkn"] Sep 14 11:25:49.400593 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:25:49.400559 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-7cc544b6-6cc9546b66-jxdkn" podUID="c107ae62-f202-42dc-ab3c-054285a1673c" containerName="payload-pre-processing" containerID="cri-o://979f890669c8b2d3228d60c372d4a8ee8099e39b07670d8a9a881f0d39b4a568" gracePeriod=30 Sep 14 11:25:49.680737 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:25:49.679800 2574 generic.go:356] "Generic (PLEG): container finished" podID="c107ae62-f202-42dc-ab3c-054285a1673c" containerID="979f890669c8b2d3228d60c372d4a8ee8099e39b07670d8a9a881f0d39b4a568" exitCode=0 Sep 14 11:25:49.680737 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:25:49.679936 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7cc544b6-6cc9546b66-jxdkn" event={"ID":"c107ae62-f202-42dc-ab3c-054285a1673c","Type":"ContainerDied","Data":"979f890669c8b2d3228d60c372d4a8ee8099e39b07670d8a9a881f0d39b4a568"} Sep 14 11:25:49.681951 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:25:49.681741 2574 generic.go:356] "Generic (PLEG): container finished" podID="726c5799-69bd-47ab-9158-91066369e82e" containerID="78e33bac56da92aef35839c7b0af6108441550d64c9d25f049a776deb866e7e1" exitCode=0 Sep 14 11:25:49.681951 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:25:49.681852 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7cc544b6-577b49b478-855h5" event={"ID":"726c5799-69bd-47ab-9158-91066369e82e","Type":"ContainerDied","Data":"78e33bac56da92aef35839c7b0af6108441550d64c9d25f049a776deb866e7e1"} Sep 14 11:25:49.681951 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:25:49.681886 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7cc544b6-577b49b478-855h5" Sep 14 11:25:49.681951 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:25:49.681897 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7cc544b6-577b49b478-855h5" event={"ID":"726c5799-69bd-47ab-9158-91066369e82e","Type":"ContainerDied","Data":"090ed6b29d1299de260b0a176ae1931817ef822003e945aadc5c6af54f6a7bdf"} Sep 14 11:25:49.681951 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:25:49.681920 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="090ed6b29d1299de260b0a176ae1931817ef822003e945aadc5c6af54f6a7bdf" Sep 14 11:25:49.745502 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:25:49.745472 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7cc544b6-6cc9546b66-jxdkn" Sep 14 11:25:49.775354 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:25:49.775324 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/726c5799-69bd-47ab-9158-91066369e82e-metrics-certs\") pod \"726c5799-69bd-47ab-9158-91066369e82e\" (UID: \"726c5799-69bd-47ab-9158-91066369e82e\") " Sep 14 11:25:49.775495 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:25:49.775435 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/726c5799-69bd-47ab-9158-91066369e82e-maas-api-tls\") pod \"726c5799-69bd-47ab-9158-91066369e82e\" (UID: \"726c5799-69bd-47ab-9158-91066369e82e\") " Sep 14 11:25:49.775495 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:25:49.775489 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/726c5799-69bd-47ab-9158-91066369e82e-openshift-service-ca\") pod \"726c5799-69bd-47ab-9158-91066369e82e\" (UID: \"726c5799-69bd-47ab-9158-91066369e82e\") " Sep 14 11:25:49.775584 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:25:49.775555 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkqmd\" (UniqueName: \"kubernetes.io/projected/726c5799-69bd-47ab-9158-91066369e82e-kube-api-access-xkqmd\") pod \"726c5799-69bd-47ab-9158-91066369e82e\" (UID: \"726c5799-69bd-47ab-9158-91066369e82e\") " Sep 14 11:25:49.775936 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:25:49.775903 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/726c5799-69bd-47ab-9158-91066369e82e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "726c5799-69bd-47ab-9158-91066369e82e" (UID: "726c5799-69bd-47ab-9158-91066369e82e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:25:49.777726 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:25:49.777697 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/726c5799-69bd-47ab-9158-91066369e82e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "726c5799-69bd-47ab-9158-91066369e82e" (UID: "726c5799-69bd-47ab-9158-91066369e82e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:25:49.777832 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:25:49.777720 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/726c5799-69bd-47ab-9158-91066369e82e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "726c5799-69bd-47ab-9158-91066369e82e" (UID: "726c5799-69bd-47ab-9158-91066369e82e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:25:49.778118 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:25:49.778103 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/726c5799-69bd-47ab-9158-91066369e82e-kube-api-access-xkqmd" (OuterVolumeSpecName: "kube-api-access-xkqmd") pod "726c5799-69bd-47ab-9158-91066369e82e" (UID: "726c5799-69bd-47ab-9158-91066369e82e"). InnerVolumeSpecName "kube-api-access-xkqmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:25:49.876591 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:25:49.876542 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b56w4\" (UniqueName: \"kubernetes.io/projected/c107ae62-f202-42dc-ab3c-054285a1673c-kube-api-access-b56w4\") pod \"c107ae62-f202-42dc-ab3c-054285a1673c\" (UID: \"c107ae62-f202-42dc-ab3c-054285a1673c\") " Sep 14 11:25:49.876762 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:25:49.876659 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c107ae62-f202-42dc-ab3c-054285a1673c-plugins-config-volume\") pod \"c107ae62-f202-42dc-ab3c-054285a1673c\" (UID: \"c107ae62-f202-42dc-ab3c-054285a1673c\") " Sep 14 11:25:49.877038 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:25:49.877018 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/726c5799-69bd-47ab-9158-91066369e82e-openshift-service-ca\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:25:49.877112 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:25:49.877045 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkqmd\" (UniqueName: \"kubernetes.io/projected/726c5799-69bd-47ab-9158-91066369e82e-kube-api-access-xkqmd\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:25:49.877112 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:25:49.877061 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/726c5799-69bd-47ab-9158-91066369e82e-metrics-certs\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:25:49.877112 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:25:49.877074 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/726c5799-69bd-47ab-9158-91066369e82e-maas-api-tls\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:25:49.877224 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:25:49.877144 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c107ae62-f202-42dc-ab3c-054285a1673c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c107ae62-f202-42dc-ab3c-054285a1673c" (UID: "c107ae62-f202-42dc-ab3c-054285a1673c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:25:49.878833 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:25:49.878806 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c107ae62-f202-42dc-ab3c-054285a1673c-kube-api-access-b56w4" (OuterVolumeSpecName: "kube-api-access-b56w4") pod "c107ae62-f202-42dc-ab3c-054285a1673c" (UID: "c107ae62-f202-42dc-ab3c-054285a1673c"). InnerVolumeSpecName "kube-api-access-b56w4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:25:49.978482 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:25:49.978449 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b56w4\" (UniqueName: \"kubernetes.io/projected/c107ae62-f202-42dc-ab3c-054285a1673c-kube-api-access-b56w4\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:25:49.978482 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:25:49.978478 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c107ae62-f202-42dc-ab3c-054285a1673c-plugins-config-volume\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:25:50.688340 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:25:50.688308 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7cc544b6-6cc9546b66-jxdkn" Sep 14 11:25:50.688739 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:25:50.688338 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7cc544b6-6cc9546b66-jxdkn" event={"ID":"c107ae62-f202-42dc-ab3c-054285a1673c","Type":"ContainerDied","Data":"34376b45f27b865769bceb2768f51ddf68d597c8068735e2ba02c05124284138"} Sep 14 11:25:50.688739 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:25:50.688401 2574 scope.go:117] "RemoveContainer" containerID="979f890669c8b2d3228d60c372d4a8ee8099e39b07670d8a9a881f0d39b4a568" Sep 14 11:25:50.688739 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:25:50.688349 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7cc544b6-577b49b478-855h5" Sep 14 11:25:50.712788 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:25:50.712761 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7cc544b6-6cc9546b66-jxdkn"] Sep 14 11:25:50.715685 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:25:50.715663 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7cc544b6-6cc9546b66-jxdkn"] Sep 14 11:25:50.727847 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:25:50.727811 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7cc544b6-577b49b478-855h5"] Sep 14 11:25:50.729901 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:25:50.729879 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7cc544b6-577b49b478-855h5"] Sep 14 11:25:51.901614 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:25:51.901581 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="726c5799-69bd-47ab-9158-91066369e82e" path="/var/lib/kubelet/pods/726c5799-69bd-47ab-9158-91066369e82e/volumes" Sep 14 11:25:51.901989 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:25:51.901929 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c107ae62-f202-42dc-ab3c-054285a1673c" path="/var/lib/kubelet/pods/c107ae62-f202-42dc-ab3c-054285a1673c/volumes" Sep 14 11:28:00.490320 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:00.490287 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e4fc87-d957544d6-pczbn"] Sep 14 11:28:00.490838 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:00.490765 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c107ae62-f202-42dc-ab3c-054285a1673c" containerName="payload-pre-processing" Sep 14 11:28:00.490838 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:00.490776 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c107ae62-f202-42dc-ab3c-054285a1673c" containerName="payload-pre-processing" Sep 14 11:28:00.490838 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:00.490798 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="726c5799-69bd-47ab-9158-91066369e82e" containerName="maas-api" Sep 14 11:28:00.490838 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:00.490804 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="726c5799-69bd-47ab-9158-91066369e82e" containerName="maas-api" Sep 14 11:28:00.491066 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:00.490869 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="726c5799-69bd-47ab-9158-91066369e82e" containerName="maas-api" Sep 14 11:28:00.491066 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:00.490878 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c107ae62-f202-42dc-ab3c-054285a1673c" containerName="payload-pre-processing" Sep 14 11:28:00.494148 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:00.494126 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4fc87-d957544d6-pczbn" Sep 14 11:28:00.497355 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:00.497334 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-e4fc87-serving-cert\"" Sep 14 11:28:00.504759 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:00.504737 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e4fc87-d957544d6-pczbn"] Sep 14 11:28:00.601982 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:00.601952 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6480efe3-8062-43b1-947a-b60bfdb029d5-maas-api-tls\") pod \"maas-api-e2e-api-a-e4fc87-d957544d6-pczbn\" (UID: \"6480efe3-8062-43b1-947a-b60bfdb029d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4fc87-d957544d6-pczbn" Sep 14 11:28:00.602148 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:00.601989 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtrq8\" (UniqueName: \"kubernetes.io/projected/6480efe3-8062-43b1-947a-b60bfdb029d5-kube-api-access-wtrq8\") pod \"maas-api-e2e-api-a-e4fc87-d957544d6-pczbn\" (UID: \"6480efe3-8062-43b1-947a-b60bfdb029d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4fc87-d957544d6-pczbn" Sep 14 11:28:00.602148 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:00.602015 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6480efe3-8062-43b1-947a-b60bfdb029d5-metrics-certs\") pod \"maas-api-e2e-api-a-e4fc87-d957544d6-pczbn\" (UID: \"6480efe3-8062-43b1-947a-b60bfdb029d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4fc87-d957544d6-pczbn" Sep 14 11:28:00.602148 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:00.602107 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6480efe3-8062-43b1-947a-b60bfdb029d5-openshift-service-ca\") pod \"maas-api-e2e-api-a-e4fc87-d957544d6-pczbn\" (UID: \"6480efe3-8062-43b1-947a-b60bfdb029d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4fc87-d957544d6-pczbn" Sep 14 11:28:00.702999 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:00.702967 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6480efe3-8062-43b1-947a-b60bfdb029d5-maas-api-tls\") pod \"maas-api-e2e-api-a-e4fc87-d957544d6-pczbn\" (UID: \"6480efe3-8062-43b1-947a-b60bfdb029d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4fc87-d957544d6-pczbn" Sep 14 11:28:00.703166 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:00.703016 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtrq8\" (UniqueName: \"kubernetes.io/projected/6480efe3-8062-43b1-947a-b60bfdb029d5-kube-api-access-wtrq8\") pod \"maas-api-e2e-api-a-e4fc87-d957544d6-pczbn\" (UID: \"6480efe3-8062-43b1-947a-b60bfdb029d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4fc87-d957544d6-pczbn" Sep 14 11:28:00.703166 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:00.703044 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6480efe3-8062-43b1-947a-b60bfdb029d5-metrics-certs\") pod \"maas-api-e2e-api-a-e4fc87-d957544d6-pczbn\" (UID: \"6480efe3-8062-43b1-947a-b60bfdb029d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4fc87-d957544d6-pczbn" Sep 14 11:28:00.703166 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:00.703087 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6480efe3-8062-43b1-947a-b60bfdb029d5-openshift-service-ca\") pod \"maas-api-e2e-api-a-e4fc87-d957544d6-pczbn\" (UID: \"6480efe3-8062-43b1-947a-b60bfdb029d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4fc87-d957544d6-pczbn" Sep 14 11:28:00.703166 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:28:00.703123 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-e4fc87-serving-cert: secret "maas-api-e2e-api-a-e4fc87-serving-cert" not found Sep 14 11:28:00.703435 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:28:00.703194 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6480efe3-8062-43b1-947a-b60bfdb029d5-maas-api-tls podName:6480efe3-8062-43b1-947a-b60bfdb029d5 nodeName:}" failed. No retries permitted until 2026-09-14 11:28:01.203172506 +0000 UTC m=+1625.823385023 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6480efe3-8062-43b1-947a-b60bfdb029d5-maas-api-tls") pod "maas-api-e2e-api-a-e4fc87-d957544d6-pczbn" (UID: "6480efe3-8062-43b1-947a-b60bfdb029d5") : secret "maas-api-e2e-api-a-e4fc87-serving-cert" not found Sep 14 11:28:00.703768 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:00.703749 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6480efe3-8062-43b1-947a-b60bfdb029d5-openshift-service-ca\") pod \"maas-api-e2e-api-a-e4fc87-d957544d6-pczbn\" (UID: \"6480efe3-8062-43b1-947a-b60bfdb029d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4fc87-d957544d6-pczbn" Sep 14 11:28:00.705668 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:00.705649 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6480efe3-8062-43b1-947a-b60bfdb029d5-metrics-certs\") pod \"maas-api-e2e-api-a-e4fc87-d957544d6-pczbn\" (UID: \"6480efe3-8062-43b1-947a-b60bfdb029d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4fc87-d957544d6-pczbn" Sep 14 11:28:00.711797 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:00.711774 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtrq8\" (UniqueName: \"kubernetes.io/projected/6480efe3-8062-43b1-947a-b60bfdb029d5-kube-api-access-wtrq8\") pod \"maas-api-e2e-api-a-e4fc87-d957544d6-pczbn\" (UID: \"6480efe3-8062-43b1-947a-b60bfdb029d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4fc87-d957544d6-pczbn" Sep 14 11:28:01.207318 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:01.207254 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6480efe3-8062-43b1-947a-b60bfdb029d5-maas-api-tls\") pod \"maas-api-e2e-api-a-e4fc87-d957544d6-pczbn\" (UID: \"6480efe3-8062-43b1-947a-b60bfdb029d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4fc87-d957544d6-pczbn" Sep 14 11:28:01.209759 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:01.209738 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6480efe3-8062-43b1-947a-b60bfdb029d5-maas-api-tls\") pod \"maas-api-e2e-api-a-e4fc87-d957544d6-pczbn\" (UID: \"6480efe3-8062-43b1-947a-b60bfdb029d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4fc87-d957544d6-pczbn" Sep 14 11:28:01.266871 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:01.266846 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e4fc87-54f47fc48f-hb5ls"] Sep 14 11:28:01.270710 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:01.270694 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e4fc87-54f47fc48f-hb5ls" Sep 14 11:28:01.273037 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:01.273012 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-e4fc87-dockercfg-rlkbf\"" Sep 14 11:28:01.273171 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:01.273133 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-e4fc87\"" Sep 14 11:28:01.282485 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:01.282374 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e4fc87-54f47fc48f-hb5ls"] Sep 14 11:28:01.405935 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:01.405901 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4fc87-d957544d6-pczbn" Sep 14 11:28:01.409528 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:01.409332 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xl5k\" (UniqueName: \"kubernetes.io/projected/f2e0fb04-8585-4c7c-82bd-4655cd00042a-kube-api-access-5xl5k\") pod \"payload-processing-e2e-api-a-e4fc87-54f47fc48f-hb5ls\" (UID: \"f2e0fb04-8585-4c7c-82bd-4655cd00042a\") " pod="openshift-ingress/payload-processing-e2e-api-a-e4fc87-54f47fc48f-hb5ls" Sep 14 11:28:01.409629 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:01.409599 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2e0fb04-8585-4c7c-82bd-4655cd00042a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e4fc87-54f47fc48f-hb5ls\" (UID: \"f2e0fb04-8585-4c7c-82bd-4655cd00042a\") " pod="openshift-ingress/payload-processing-e2e-api-a-e4fc87-54f47fc48f-hb5ls" Sep 14 11:28:01.511022 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:01.510987 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2e0fb04-8585-4c7c-82bd-4655cd00042a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e4fc87-54f47fc48f-hb5ls\" (UID: \"f2e0fb04-8585-4c7c-82bd-4655cd00042a\") " pod="openshift-ingress/payload-processing-e2e-api-a-e4fc87-54f47fc48f-hb5ls" Sep 14 11:28:01.511443 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:01.511041 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xl5k\" (UniqueName: \"kubernetes.io/projected/f2e0fb04-8585-4c7c-82bd-4655cd00042a-kube-api-access-5xl5k\") pod \"payload-processing-e2e-api-a-e4fc87-54f47fc48f-hb5ls\" (UID: \"f2e0fb04-8585-4c7c-82bd-4655cd00042a\") " pod="openshift-ingress/payload-processing-e2e-api-a-e4fc87-54f47fc48f-hb5ls" Sep 14 11:28:01.511814 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:01.511795 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2e0fb04-8585-4c7c-82bd-4655cd00042a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e4fc87-54f47fc48f-hb5ls\" (UID: \"f2e0fb04-8585-4c7c-82bd-4655cd00042a\") " pod="openshift-ingress/payload-processing-e2e-api-a-e4fc87-54f47fc48f-hb5ls" Sep 14 11:28:01.519966 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:01.519936 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xl5k\" (UniqueName: \"kubernetes.io/projected/f2e0fb04-8585-4c7c-82bd-4655cd00042a-kube-api-access-5xl5k\") pod \"payload-processing-e2e-api-a-e4fc87-54f47fc48f-hb5ls\" (UID: \"f2e0fb04-8585-4c7c-82bd-4655cd00042a\") " pod="openshift-ingress/payload-processing-e2e-api-a-e4fc87-54f47fc48f-hb5ls" Sep 14 11:28:01.587849 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:01.587815 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e4fc87-54f47fc48f-hb5ls" Sep 14 11:28:01.689953 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:28:01.689924 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6480efe3_8062_43b1_947a_b60bfdb029d5.slice/crio-5c2e488a6e33608ea80c3ae91190d7cc7270e1a646e51a3f70b2efc0c487303f WatchSource:0}: Error finding container 5c2e488a6e33608ea80c3ae91190d7cc7270e1a646e51a3f70b2efc0c487303f: Status 404 returned error can't find the container with id 5c2e488a6e33608ea80c3ae91190d7cc7270e1a646e51a3f70b2efc0c487303f Sep 14 11:28:01.690094 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:01.690069 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e4fc87-d957544d6-pczbn"] Sep 14 11:28:01.691696 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:01.691680 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 11:28:01.863794 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:01.863766 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e4fc87-54f47fc48f-hb5ls"] Sep 14 11:28:01.866160 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:28:01.866133 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2e0fb04_8585_4c7c_82bd_4655cd00042a.slice/crio-5d269351d0dd47380b4941f13c936994dba1415e6c68baa6e827640d5158b3f3 WatchSource:0}: Error finding container 5d269351d0dd47380b4941f13c936994dba1415e6c68baa6e827640d5158b3f3: Status 404 returned error can't find the container with id 5d269351d0dd47380b4941f13c936994dba1415e6c68baa6e827640d5158b3f3 Sep 14 11:28:02.428329 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:02.428297 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e4fc87-54f47fc48f-hb5ls" event={"ID":"f2e0fb04-8585-4c7c-82bd-4655cd00042a","Type":"ContainerStarted","Data":"1253f06149565f74d6f45966f86fe4575089e6dddb9b9a091ad6a27b4b130b8a"} Sep 14 11:28:02.428529 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:02.428361 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e4fc87-54f47fc48f-hb5ls" event={"ID":"f2e0fb04-8585-4c7c-82bd-4655cd00042a","Type":"ContainerStarted","Data":"5d269351d0dd47380b4941f13c936994dba1415e6c68baa6e827640d5158b3f3"} Sep 14 11:28:02.428529 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:02.428490 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-e4fc87-54f47fc48f-hb5ls" Sep 14 11:28:02.429632 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:02.429602 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4fc87-d957544d6-pczbn" event={"ID":"6480efe3-8062-43b1-947a-b60bfdb029d5","Type":"ContainerStarted","Data":"5c2e488a6e33608ea80c3ae91190d7cc7270e1a646e51a3f70b2efc0c487303f"} Sep 14 11:28:02.448786 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:02.448743 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-e4fc87-54f47fc48f-hb5ls" podStartSLOduration=1.4487265169999999 podStartE2EDuration="1.448726517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:28:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:28:02.44592925 +0000 UTC m=+1627.066141778" watchObservedRunningTime="2026-09-14 11:28:02.448726517 +0000 UTC m=+1627.068939051" Sep 14 11:28:03.436977 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:03.436930 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4fc87-d957544d6-pczbn" event={"ID":"6480efe3-8062-43b1-947a-b60bfdb029d5","Type":"ContainerStarted","Data":"49781a7fa5cf0cf39a0145c747405a2e997b88d87b4271b1b99520787092f7e1"} Sep 14 11:28:03.437430 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:03.437208 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4fc87-d957544d6-pczbn" Sep 14 11:28:03.459391 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:03.459331 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4fc87-d957544d6-pczbn" podStartSLOduration=2.622044988 podStartE2EDuration="3.459312377s" totalImagesPullingTime="837.267389ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:28:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:28:03.456440594 +0000 UTC m=+1628.076653117" watchObservedRunningTime="2026-09-14 11:28:03.459312377 +0000 UTC m=+1628.079524900" Sep 14 11:28:09.448843 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:09.448815 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4fc87-d957544d6-pczbn" Sep 14 11:28:13.439071 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:13.439033 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-e4fc87-54f47fc48f-hb5ls" Sep 14 11:28:35.592335 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:35.592301 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-420d64-58d698886-sh4gs"] Sep 14 11:28:35.596406 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:35.596388 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-420d64-58d698886-sh4gs" Sep 14 11:28:35.598950 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:35.598924 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-420d64\"" Sep 14 11:28:35.605012 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:35.604770 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-420d64-58d698886-sh4gs"] Sep 14 11:28:35.658926 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:35.658892 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbbkr\" (UniqueName: \"kubernetes.io/projected/8a0970df-5b89-491f-9ef9-ac65af2f54d5-kube-api-access-fbbkr\") pod \"payload-pre-processing-e2e-api-b-420d64-58d698886-sh4gs\" (UID: \"8a0970df-5b89-491f-9ef9-ac65af2f54d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-420d64-58d698886-sh4gs" Sep 14 11:28:35.659078 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:35.658949 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a0970df-5b89-491f-9ef9-ac65af2f54d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-420d64-58d698886-sh4gs\" (UID: \"8a0970df-5b89-491f-9ef9-ac65af2f54d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-420d64-58d698886-sh4gs" Sep 14 11:28:35.760378 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:35.760345 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbbkr\" (UniqueName: \"kubernetes.io/projected/8a0970df-5b89-491f-9ef9-ac65af2f54d5-kube-api-access-fbbkr\") pod \"payload-pre-processing-e2e-api-b-420d64-58d698886-sh4gs\" (UID: \"8a0970df-5b89-491f-9ef9-ac65af2f54d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-420d64-58d698886-sh4gs" Sep 14 11:28:35.760688 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:35.760435 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a0970df-5b89-491f-9ef9-ac65af2f54d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-420d64-58d698886-sh4gs\" (UID: \"8a0970df-5b89-491f-9ef9-ac65af2f54d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-420d64-58d698886-sh4gs" Sep 14 11:28:35.761372 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:35.761349 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a0970df-5b89-491f-9ef9-ac65af2f54d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-420d64-58d698886-sh4gs\" (UID: \"8a0970df-5b89-491f-9ef9-ac65af2f54d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-420d64-58d698886-sh4gs" Sep 14 11:28:35.769779 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:35.769759 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbbkr\" (UniqueName: \"kubernetes.io/projected/8a0970df-5b89-491f-9ef9-ac65af2f54d5-kube-api-access-fbbkr\") pod \"payload-pre-processing-e2e-api-b-420d64-58d698886-sh4gs\" (UID: \"8a0970df-5b89-491f-9ef9-ac65af2f54d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-420d64-58d698886-sh4gs" Sep 14 11:28:35.908956 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:35.908879 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-420d64-58d698886-sh4gs" Sep 14 11:28:36.192469 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:36.192447 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-420d64-58d698886-sh4gs"] Sep 14 11:28:36.193725 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:28:36.193698 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a0970df_5b89_491f_9ef9_ac65af2f54d5.slice/crio-a5881defe5d366b85049db9f0ac22ffa6247bf10eb2ff62544d20ae99e02843a WatchSource:0}: Error finding container a5881defe5d366b85049db9f0ac22ffa6247bf10eb2ff62544d20ae99e02843a: Status 404 returned error can't find the container with id a5881defe5d366b85049db9f0ac22ffa6247bf10eb2ff62544d20ae99e02843a Sep 14 11:28:36.632461 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:36.632366 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-420d64-58d698886-sh4gs" event={"ID":"8a0970df-5b89-491f-9ef9-ac65af2f54d5","Type":"ContainerStarted","Data":"a96e750551a0e334424519e6d40cf7b7a11889aa13695ced0edfe25d256356b0"} Sep 14 11:28:36.632461 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:36.632438 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-420d64-58d698886-sh4gs" Sep 14 11:28:36.632461 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:36.632462 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-420d64-58d698886-sh4gs" event={"ID":"8a0970df-5b89-491f-9ef9-ac65af2f54d5","Type":"ContainerStarted","Data":"a5881defe5d366b85049db9f0ac22ffa6247bf10eb2ff62544d20ae99e02843a"} Sep 14 11:28:36.654405 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:36.654348 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-420d64-58d698886-sh4gs" podStartSLOduration=1.6543311969999999 podStartE2EDuration="1.654331197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:28:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:28:36.652195625 +0000 UTC m=+1661.272408161" watchObservedRunningTime="2026-09-14 11:28:36.654331197 +0000 UTC m=+1661.274543720" Sep 14 11:28:47.640118 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:28:47.640083 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-420d64-58d698886-sh4gs" Sep 14 11:29:05.338618 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:05.338580 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2"] Sep 14 11:29:05.339150 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:05.338848 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2" podUID="f9b2415c-3a23-4717-ba02-71543cd63a88" containerName="maas-api" containerID="cri-o://e7c68eeb8ead3209c734e2d53248c8f5bbc7c1e5d32e21655bbfb15965196d12" gracePeriod=30 Sep 14 11:29:05.597138 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:05.597077 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2" Sep 14 11:29:05.649189 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:05.649163 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9b2415c-3a23-4717-ba02-71543cd63a88-metrics-certs\") pod \"f9b2415c-3a23-4717-ba02-71543cd63a88\" (UID: \"f9b2415c-3a23-4717-ba02-71543cd63a88\") " Sep 14 11:29:05.649437 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:05.649241 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzdrf\" (UniqueName: \"kubernetes.io/projected/f9b2415c-3a23-4717-ba02-71543cd63a88-kube-api-access-lzdrf\") pod \"f9b2415c-3a23-4717-ba02-71543cd63a88\" (UID: \"f9b2415c-3a23-4717-ba02-71543cd63a88\") " Sep 14 11:29:05.649437 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:05.649345 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9b2415c-3a23-4717-ba02-71543cd63a88-openshift-service-ca\") pod \"f9b2415c-3a23-4717-ba02-71543cd63a88\" (UID: \"f9b2415c-3a23-4717-ba02-71543cd63a88\") " Sep 14 11:29:05.649437 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:05.649400 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9b2415c-3a23-4717-ba02-71543cd63a88-maas-api-tls\") pod \"f9b2415c-3a23-4717-ba02-71543cd63a88\" (UID: \"f9b2415c-3a23-4717-ba02-71543cd63a88\") " Sep 14 11:29:05.649785 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:05.649757 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9b2415c-3a23-4717-ba02-71543cd63a88-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f9b2415c-3a23-4717-ba02-71543cd63a88" (UID: "f9b2415c-3a23-4717-ba02-71543cd63a88"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:29:05.651700 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:05.651665 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9b2415c-3a23-4717-ba02-71543cd63a88-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f9b2415c-3a23-4717-ba02-71543cd63a88" (UID: "f9b2415c-3a23-4717-ba02-71543cd63a88"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:29:05.652402 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:05.652375 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9b2415c-3a23-4717-ba02-71543cd63a88-kube-api-access-lzdrf" (OuterVolumeSpecName: "kube-api-access-lzdrf") pod "f9b2415c-3a23-4717-ba02-71543cd63a88" (UID: "f9b2415c-3a23-4717-ba02-71543cd63a88"). InnerVolumeSpecName "kube-api-access-lzdrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:29:05.652642 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:05.652621 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9b2415c-3a23-4717-ba02-71543cd63a88-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f9b2415c-3a23-4717-ba02-71543cd63a88" (UID: "f9b2415c-3a23-4717-ba02-71543cd63a88"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:29:05.750677 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:05.750641 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9b2415c-3a23-4717-ba02-71543cd63a88-openshift-service-ca\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:29:05.750677 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:05.750672 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f9b2415c-3a23-4717-ba02-71543cd63a88-maas-api-tls\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:29:05.750677 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:05.750682 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9b2415c-3a23-4717-ba02-71543cd63a88-metrics-certs\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:29:05.750905 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:05.750689 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzdrf\" (UniqueName: \"kubernetes.io/projected/f9b2415c-3a23-4717-ba02-71543cd63a88-kube-api-access-lzdrf\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:29:05.805898 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:05.805868 2574 generic.go:356] "Generic (PLEG): container finished" podID="f9b2415c-3a23-4717-ba02-71543cd63a88" containerID="e7c68eeb8ead3209c734e2d53248c8f5bbc7c1e5d32e21655bbfb15965196d12" exitCode=0 Sep 14 11:29:05.806073 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:05.805922 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2" Sep 14 11:29:05.806073 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:05.805950 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2" event={"ID":"f9b2415c-3a23-4717-ba02-71543cd63a88","Type":"ContainerDied","Data":"e7c68eeb8ead3209c734e2d53248c8f5bbc7c1e5d32e21655bbfb15965196d12"} Sep 14 11:29:05.806073 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:05.806004 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2" event={"ID":"f9b2415c-3a23-4717-ba02-71543cd63a88","Type":"ContainerDied","Data":"46dfef6cf78e76881f4fb2f333ccddb54cec823c4ac06b1a4c49c36dc97a1b04"} Sep 14 11:29:05.806073 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:05.806030 2574 scope.go:117] "RemoveContainer" containerID="e7c68eeb8ead3209c734e2d53248c8f5bbc7c1e5d32e21655bbfb15965196d12" Sep 14 11:29:05.816823 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:05.816801 2574 scope.go:117] "RemoveContainer" containerID="e7c68eeb8ead3209c734e2d53248c8f5bbc7c1e5d32e21655bbfb15965196d12" Sep 14 11:29:05.817129 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:29:05.817110 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7c68eeb8ead3209c734e2d53248c8f5bbc7c1e5d32e21655bbfb15965196d12\": container with ID starting with e7c68eeb8ead3209c734e2d53248c8f5bbc7c1e5d32e21655bbfb15965196d12 not found: ID does not exist" containerID="e7c68eeb8ead3209c734e2d53248c8f5bbc7c1e5d32e21655bbfb15965196d12" Sep 14 11:29:05.817204 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:05.817140 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7c68eeb8ead3209c734e2d53248c8f5bbc7c1e5d32e21655bbfb15965196d12"} err="failed to get container status \"e7c68eeb8ead3209c734e2d53248c8f5bbc7c1e5d32e21655bbfb15965196d12\": rpc error: code = NotFound desc = could not find container \"e7c68eeb8ead3209c734e2d53248c8f5bbc7c1e5d32e21655bbfb15965196d12\": container with ID starting with e7c68eeb8ead3209c734e2d53248c8f5bbc7c1e5d32e21655bbfb15965196d12 not found: ID does not exist" Sep 14 11:29:05.833465 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:05.833429 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2"] Sep 14 11:29:05.837190 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:05.837166 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-6f3d0f-5fbf86d6cb-z96r2"] Sep 14 11:29:05.902436 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:05.902363 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9b2415c-3a23-4717-ba02-71543cd63a88" path="/var/lib/kubelet/pods/f9b2415c-3a23-4717-ba02-71543cd63a88/volumes" Sep 14 11:29:12.567539 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:12.567505 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e4fc87-d957544d6-pczbn"] Sep 14 11:29:12.568008 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:12.567719 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4fc87-d957544d6-pczbn" podUID="6480efe3-8062-43b1-947a-b60bfdb029d5" containerName="maas-api" containerID="cri-o://49781a7fa5cf0cf39a0145c747405a2e997b88d87b4271b1b99520787092f7e1" gracePeriod=30 Sep 14 11:29:12.649317 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:12.649286 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e4fc87-54f47fc48f-hb5ls"] Sep 14 11:29:12.649535 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:12.649508 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-e4fc87-54f47fc48f-hb5ls" podUID="f2e0fb04-8585-4c7c-82bd-4655cd00042a" containerName="payload-processing" containerID="cri-o://1253f06149565f74d6f45966f86fe4575089e6dddb9b9a091ad6a27b4b130b8a" gracePeriod=30 Sep 14 11:29:12.850796 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:12.850764 2574 generic.go:356] "Generic (PLEG): container finished" podID="f2e0fb04-8585-4c7c-82bd-4655cd00042a" containerID="1253f06149565f74d6f45966f86fe4575089e6dddb9b9a091ad6a27b4b130b8a" exitCode=0 Sep 14 11:29:12.850977 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:12.850856 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e4fc87-54f47fc48f-hb5ls" event={"ID":"f2e0fb04-8585-4c7c-82bd-4655cd00042a","Type":"ContainerDied","Data":"1253f06149565f74d6f45966f86fe4575089e6dddb9b9a091ad6a27b4b130b8a"} Sep 14 11:29:12.853184 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:12.853161 2574 generic.go:356] "Generic (PLEG): container finished" podID="6480efe3-8062-43b1-947a-b60bfdb029d5" containerID="49781a7fa5cf0cf39a0145c747405a2e997b88d87b4271b1b99520787092f7e1" exitCode=0 Sep 14 11:29:12.853333 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:12.853315 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4fc87-d957544d6-pczbn" event={"ID":"6480efe3-8062-43b1-947a-b60bfdb029d5","Type":"ContainerDied","Data":"49781a7fa5cf0cf39a0145c747405a2e997b88d87b4271b1b99520787092f7e1"} Sep 14 11:29:12.918707 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:12.918681 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4fc87-d957544d6-pczbn" Sep 14 11:29:12.964849 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:12.964820 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e4fc87-54f47fc48f-hb5ls" Sep 14 11:29:13.026946 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:13.026913 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtrq8\" (UniqueName: \"kubernetes.io/projected/6480efe3-8062-43b1-947a-b60bfdb029d5-kube-api-access-wtrq8\") pod \"6480efe3-8062-43b1-947a-b60bfdb029d5\" (UID: \"6480efe3-8062-43b1-947a-b60bfdb029d5\") " Sep 14 11:29:13.027124 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:13.026995 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6480efe3-8062-43b1-947a-b60bfdb029d5-openshift-service-ca\") pod \"6480efe3-8062-43b1-947a-b60bfdb029d5\" (UID: \"6480efe3-8062-43b1-947a-b60bfdb029d5\") " Sep 14 11:29:13.027124 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:13.027041 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2e0fb04-8585-4c7c-82bd-4655cd00042a-plugins-config-volume\") pod \"f2e0fb04-8585-4c7c-82bd-4655cd00042a\" (UID: \"f2e0fb04-8585-4c7c-82bd-4655cd00042a\") " Sep 14 11:29:13.027124 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:13.027083 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6480efe3-8062-43b1-947a-b60bfdb029d5-maas-api-tls\") pod \"6480efe3-8062-43b1-947a-b60bfdb029d5\" (UID: \"6480efe3-8062-43b1-947a-b60bfdb029d5\") " Sep 14 11:29:13.027325 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:13.027132 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6480efe3-8062-43b1-947a-b60bfdb029d5-metrics-certs\") pod \"6480efe3-8062-43b1-947a-b60bfdb029d5\" (UID: \"6480efe3-8062-43b1-947a-b60bfdb029d5\") " Sep 14 11:29:13.027325 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:13.027156 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xl5k\" (UniqueName: \"kubernetes.io/projected/f2e0fb04-8585-4c7c-82bd-4655cd00042a-kube-api-access-5xl5k\") pod \"f2e0fb04-8585-4c7c-82bd-4655cd00042a\" (UID: \"f2e0fb04-8585-4c7c-82bd-4655cd00042a\") " Sep 14 11:29:13.027466 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:13.027432 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6480efe3-8062-43b1-947a-b60bfdb029d5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6480efe3-8062-43b1-947a-b60bfdb029d5" (UID: "6480efe3-8062-43b1-947a-b60bfdb029d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:29:13.027466 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:13.027456 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2e0fb04-8585-4c7c-82bd-4655cd00042a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f2e0fb04-8585-4c7c-82bd-4655cd00042a" (UID: "f2e0fb04-8585-4c7c-82bd-4655cd00042a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:29:13.027720 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:13.027695 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2e0fb04-8585-4c7c-82bd-4655cd00042a-plugins-config-volume\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:29:13.027720 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:13.027722 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6480efe3-8062-43b1-947a-b60bfdb029d5-openshift-service-ca\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:29:13.029576 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:13.029553 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6480efe3-8062-43b1-947a-b60bfdb029d5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6480efe3-8062-43b1-947a-b60bfdb029d5" (UID: "6480efe3-8062-43b1-947a-b60bfdb029d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:29:13.029704 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:13.029676 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2e0fb04-8585-4c7c-82bd-4655cd00042a-kube-api-access-5xl5k" (OuterVolumeSpecName: "kube-api-access-5xl5k") pod "f2e0fb04-8585-4c7c-82bd-4655cd00042a" (UID: "f2e0fb04-8585-4c7c-82bd-4655cd00042a"). InnerVolumeSpecName "kube-api-access-5xl5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:29:13.029853 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:13.029833 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6480efe3-8062-43b1-947a-b60bfdb029d5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6480efe3-8062-43b1-947a-b60bfdb029d5" (UID: "6480efe3-8062-43b1-947a-b60bfdb029d5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:29:13.029903 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:13.029845 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6480efe3-8062-43b1-947a-b60bfdb029d5-kube-api-access-wtrq8" (OuterVolumeSpecName: "kube-api-access-wtrq8") pod "6480efe3-8062-43b1-947a-b60bfdb029d5" (UID: "6480efe3-8062-43b1-947a-b60bfdb029d5"). InnerVolumeSpecName "kube-api-access-wtrq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:29:13.128960 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:13.128884 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtrq8\" (UniqueName: \"kubernetes.io/projected/6480efe3-8062-43b1-947a-b60bfdb029d5-kube-api-access-wtrq8\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:29:13.128960 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:13.128912 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6480efe3-8062-43b1-947a-b60bfdb029d5-maas-api-tls\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:29:13.128960 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:13.128922 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6480efe3-8062-43b1-947a-b60bfdb029d5-metrics-certs\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:29:13.128960 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:13.128930 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xl5k\" (UniqueName: \"kubernetes.io/projected/f2e0fb04-8585-4c7c-82bd-4655cd00042a-kube-api-access-5xl5k\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:29:13.859800 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:13.859730 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e4fc87-54f47fc48f-hb5ls" Sep 14 11:29:13.859800 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:13.859762 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e4fc87-54f47fc48f-hb5ls" event={"ID":"f2e0fb04-8585-4c7c-82bd-4655cd00042a","Type":"ContainerDied","Data":"5d269351d0dd47380b4941f13c936994dba1415e6c68baa6e827640d5158b3f3"} Sep 14 11:29:13.860233 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:13.859829 2574 scope.go:117] "RemoveContainer" containerID="1253f06149565f74d6f45966f86fe4575089e6dddb9b9a091ad6a27b4b130b8a" Sep 14 11:29:13.861353 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:13.861243 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4fc87-d957544d6-pczbn" event={"ID":"6480efe3-8062-43b1-947a-b60bfdb029d5","Type":"ContainerDied","Data":"5c2e488a6e33608ea80c3ae91190d7cc7270e1a646e51a3f70b2efc0c487303f"} Sep 14 11:29:13.861353 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:13.861287 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e4fc87-d957544d6-pczbn" Sep 14 11:29:13.869310 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:13.869291 2574 scope.go:117] "RemoveContainer" containerID="49781a7fa5cf0cf39a0145c747405a2e997b88d87b4271b1b99520787092f7e1" Sep 14 11:29:13.886161 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:13.886136 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e4fc87-d957544d6-pczbn"] Sep 14 11:29:13.890031 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:13.890009 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e4fc87-d957544d6-pczbn"] Sep 14 11:29:13.899403 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:13.899383 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6480efe3-8062-43b1-947a-b60bfdb029d5" path="/var/lib/kubelet/pods/6480efe3-8062-43b1-947a-b60bfdb029d5/volumes" Sep 14 11:29:13.900032 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:13.900015 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e4fc87-54f47fc48f-hb5ls"] Sep 14 11:29:13.903852 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:13.903832 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e4fc87-54f47fc48f-hb5ls"] Sep 14 11:29:15.899171 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:15.899140 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2e0fb04-8585-4c7c-82bd-4655cd00042a" path="/var/lib/kubelet/pods/f2e0fb04-8585-4c7c-82bd-4655cd00042a/volumes" Sep 14 11:29:36.062719 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:36.062679 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-420d64-58d698886-sh4gs"] Sep 14 11:29:36.063553 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:36.063513 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-420d64-58d698886-sh4gs" podUID="8a0970df-5b89-491f-9ef9-ac65af2f54d5" containerName="payload-pre-processing" containerID="cri-o://a96e750551a0e334424519e6d40cf7b7a11889aa13695ced0edfe25d256356b0" gracePeriod=30 Sep 14 11:29:36.339968 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:36.339943 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-420d64-58d698886-sh4gs" Sep 14 11:29:36.366226 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:36.366198 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a0970df-5b89-491f-9ef9-ac65af2f54d5-plugins-config-volume\") pod \"8a0970df-5b89-491f-9ef9-ac65af2f54d5\" (UID: \"8a0970df-5b89-491f-9ef9-ac65af2f54d5\") " Sep 14 11:29:36.366397 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:36.366344 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbbkr\" (UniqueName: \"kubernetes.io/projected/8a0970df-5b89-491f-9ef9-ac65af2f54d5-kube-api-access-fbbkr\") pod \"8a0970df-5b89-491f-9ef9-ac65af2f54d5\" (UID: \"8a0970df-5b89-491f-9ef9-ac65af2f54d5\") " Sep 14 11:29:36.366662 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:36.366633 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a0970df-5b89-491f-9ef9-ac65af2f54d5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8a0970df-5b89-491f-9ef9-ac65af2f54d5" (UID: "8a0970df-5b89-491f-9ef9-ac65af2f54d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:29:36.366794 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:36.366773 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a0970df-5b89-491f-9ef9-ac65af2f54d5-plugins-config-volume\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:29:36.369198 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:36.369169 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a0970df-5b89-491f-9ef9-ac65af2f54d5-kube-api-access-fbbkr" (OuterVolumeSpecName: "kube-api-access-fbbkr") pod "8a0970df-5b89-491f-9ef9-ac65af2f54d5" (UID: "8a0970df-5b89-491f-9ef9-ac65af2f54d5"). InnerVolumeSpecName "kube-api-access-fbbkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:29:36.467629 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:36.467595 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbbkr\" (UniqueName: \"kubernetes.io/projected/8a0970df-5b89-491f-9ef9-ac65af2f54d5-kube-api-access-fbbkr\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:29:36.996667 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:36.996631 2574 generic.go:356] "Generic (PLEG): container finished" podID="8a0970df-5b89-491f-9ef9-ac65af2f54d5" containerID="a96e750551a0e334424519e6d40cf7b7a11889aa13695ced0edfe25d256356b0" exitCode=0 Sep 14 11:29:36.996667 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:36.996668 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-420d64-58d698886-sh4gs" Sep 14 11:29:36.996881 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:36.996722 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-420d64-58d698886-sh4gs" event={"ID":"8a0970df-5b89-491f-9ef9-ac65af2f54d5","Type":"ContainerDied","Data":"a96e750551a0e334424519e6d40cf7b7a11889aa13695ced0edfe25d256356b0"} Sep 14 11:29:36.996881 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:36.996770 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-420d64-58d698886-sh4gs" event={"ID":"8a0970df-5b89-491f-9ef9-ac65af2f54d5","Type":"ContainerDied","Data":"a5881defe5d366b85049db9f0ac22ffa6247bf10eb2ff62544d20ae99e02843a"} Sep 14 11:29:36.996881 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:36.996795 2574 scope.go:117] "RemoveContainer" containerID="a96e750551a0e334424519e6d40cf7b7a11889aa13695ced0edfe25d256356b0" Sep 14 11:29:37.006025 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:37.006008 2574 scope.go:117] "RemoveContainer" containerID="a96e750551a0e334424519e6d40cf7b7a11889aa13695ced0edfe25d256356b0" Sep 14 11:29:37.006311 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:29:37.006290 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a96e750551a0e334424519e6d40cf7b7a11889aa13695ced0edfe25d256356b0\": container with ID starting with a96e750551a0e334424519e6d40cf7b7a11889aa13695ced0edfe25d256356b0 not found: ID does not exist" containerID="a96e750551a0e334424519e6d40cf7b7a11889aa13695ced0edfe25d256356b0" Sep 14 11:29:37.006394 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:37.006319 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a96e750551a0e334424519e6d40cf7b7a11889aa13695ced0edfe25d256356b0"} err="failed to get container status \"a96e750551a0e334424519e6d40cf7b7a11889aa13695ced0edfe25d256356b0\": rpc error: code = NotFound desc = could not find container \"a96e750551a0e334424519e6d40cf7b7a11889aa13695ced0edfe25d256356b0\": container with ID starting with a96e750551a0e334424519e6d40cf7b7a11889aa13695ced0edfe25d256356b0 not found: ID does not exist" Sep 14 11:29:37.017195 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:37.017173 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-420d64-58d698886-sh4gs"] Sep 14 11:29:37.020462 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:37.020444 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-420d64-58d698886-sh4gs"] Sep 14 11:29:37.904436 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:37.904398 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a0970df-5b89-491f-9ef9-ac65af2f54d5" path="/var/lib/kubelet/pods/8a0970df-5b89-491f-9ef9-ac65af2f54d5/volumes" Sep 14 11:29:54.123914 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:54.123880 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh"] Sep 14 11:29:54.124409 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:54.124182 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh" podUID="c3221733-0209-46bb-a456-7bdcec80a1bc" containerName="maas-api" containerID="cri-o://4ef22319893da69ae80ade8e501acdc71636444baee86111aa221b223b03b563" gracePeriod=30 Sep 14 11:29:54.187305 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:54.187252 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e42fe6-7f857b48b9-cp8r5"] Sep 14 11:29:54.187548 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:54.187520 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e42fe6-7f857b48b9-cp8r5" podUID="7dc5ea8f-4e68-4452-adb0-b20626023ea8" containerName="payload-processing" containerID="cri-o://0c18fc5bd6082a92958c4de2626438b7c2e44276cd77f93aa9708203c01f64c3" gracePeriod=30 Sep 14 11:29:54.195972 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:54.195942 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e42fe6-75ddbdd474-f5klc"] Sep 14 11:29:54.196195 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:54.196169 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e42fe6-75ddbdd474-f5klc" podUID="b7319163-0333-4d9f-ae8b-abe6beaaa690" containerName="payload-pre-processing" containerID="cri-o://dc9f3e6c9702c6501547ae4cc48fd9a7fe24b5d9a402177eca22cab5b19cecc9" gracePeriod=30 Sep 14 11:29:54.523299 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:54.523197 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e42fe6-75ddbdd474-f5klc" Sep 14 11:29:54.526808 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:54.526788 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh" Sep 14 11:29:54.530900 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:54.530879 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e42fe6-7f857b48b9-cp8r5" Sep 14 11:29:54.646410 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:54.646383 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dc5ea8f-4e68-4452-adb0-b20626023ea8-plugins-config-volume\") pod \"7dc5ea8f-4e68-4452-adb0-b20626023ea8\" (UID: \"7dc5ea8f-4e68-4452-adb0-b20626023ea8\") " Sep 14 11:29:54.646581 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:54.646434 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3221733-0209-46bb-a456-7bdcec80a1bc-openshift-service-ca\") pod \"c3221733-0209-46bb-a456-7bdcec80a1bc\" (UID: \"c3221733-0209-46bb-a456-7bdcec80a1bc\") " Sep 14 11:29:54.646581 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:54.646469 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-896wf\" (UniqueName: \"kubernetes.io/projected/7dc5ea8f-4e68-4452-adb0-b20626023ea8-kube-api-access-896wf\") pod \"7dc5ea8f-4e68-4452-adb0-b20626023ea8\" (UID: \"7dc5ea8f-4e68-4452-adb0-b20626023ea8\") " Sep 14 11:29:54.646581 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:54.646487 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3221733-0209-46bb-a456-7bdcec80a1bc-maas-api-tls\") pod \"c3221733-0209-46bb-a456-7bdcec80a1bc\" (UID: \"c3221733-0209-46bb-a456-7bdcec80a1bc\") " Sep 14 11:29:54.646581 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:54.646553 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86bg8\" (UniqueName: \"kubernetes.io/projected/c3221733-0209-46bb-a456-7bdcec80a1bc-kube-api-access-86bg8\") pod \"c3221733-0209-46bb-a456-7bdcec80a1bc\" (UID: \"c3221733-0209-46bb-a456-7bdcec80a1bc\") " Sep 14 11:29:54.646808 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:54.646596 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mjn2\" (UniqueName: \"kubernetes.io/projected/b7319163-0333-4d9f-ae8b-abe6beaaa690-kube-api-access-7mjn2\") pod \"b7319163-0333-4d9f-ae8b-abe6beaaa690\" (UID: \"b7319163-0333-4d9f-ae8b-abe6beaaa690\") " Sep 14 11:29:54.646808 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:54.646633 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7319163-0333-4d9f-ae8b-abe6beaaa690-plugins-config-volume\") pod \"b7319163-0333-4d9f-ae8b-abe6beaaa690\" (UID: \"b7319163-0333-4d9f-ae8b-abe6beaaa690\") " Sep 14 11:29:54.646808 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:54.646682 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3221733-0209-46bb-a456-7bdcec80a1bc-metrics-certs\") pod \"c3221733-0209-46bb-a456-7bdcec80a1bc\" (UID: \"c3221733-0209-46bb-a456-7bdcec80a1bc\") " Sep 14 11:29:54.646983 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:54.646950 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3221733-0209-46bb-a456-7bdcec80a1bc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c3221733-0209-46bb-a456-7bdcec80a1bc" (UID: "c3221733-0209-46bb-a456-7bdcec80a1bc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:29:54.647051 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:54.647023 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7319163-0333-4d9f-ae8b-abe6beaaa690-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b7319163-0333-4d9f-ae8b-abe6beaaa690" (UID: "b7319163-0333-4d9f-ae8b-abe6beaaa690"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:29:54.647051 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:54.647035 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dc5ea8f-4e68-4452-adb0-b20626023ea8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7dc5ea8f-4e68-4452-adb0-b20626023ea8" (UID: "7dc5ea8f-4e68-4452-adb0-b20626023ea8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:29:54.647166 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:54.647051 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3221733-0209-46bb-a456-7bdcec80a1bc-openshift-service-ca\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:29:54.648819 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:54.648789 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3221733-0209-46bb-a456-7bdcec80a1bc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c3221733-0209-46bb-a456-7bdcec80a1bc" (UID: "c3221733-0209-46bb-a456-7bdcec80a1bc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:29:54.648936 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:54.648846 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dc5ea8f-4e68-4452-adb0-b20626023ea8-kube-api-access-896wf" (OuterVolumeSpecName: "kube-api-access-896wf") pod "7dc5ea8f-4e68-4452-adb0-b20626023ea8" (UID: "7dc5ea8f-4e68-4452-adb0-b20626023ea8"). InnerVolumeSpecName "kube-api-access-896wf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:29:54.648936 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:54.648888 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7319163-0333-4d9f-ae8b-abe6beaaa690-kube-api-access-7mjn2" (OuterVolumeSpecName: "kube-api-access-7mjn2") pod "b7319163-0333-4d9f-ae8b-abe6beaaa690" (UID: "b7319163-0333-4d9f-ae8b-abe6beaaa690"). InnerVolumeSpecName "kube-api-access-7mjn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:29:54.649305 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:54.649256 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3221733-0209-46bb-a456-7bdcec80a1bc-kube-api-access-86bg8" (OuterVolumeSpecName: "kube-api-access-86bg8") pod "c3221733-0209-46bb-a456-7bdcec80a1bc" (UID: "c3221733-0209-46bb-a456-7bdcec80a1bc"). InnerVolumeSpecName "kube-api-access-86bg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:29:54.649404 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:54.649387 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3221733-0209-46bb-a456-7bdcec80a1bc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c3221733-0209-46bb-a456-7bdcec80a1bc" (UID: "c3221733-0209-46bb-a456-7bdcec80a1bc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:29:54.747928 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:54.747900 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86bg8\" (UniqueName: \"kubernetes.io/projected/c3221733-0209-46bb-a456-7bdcec80a1bc-kube-api-access-86bg8\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:29:54.747928 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:54.747927 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mjn2\" (UniqueName: \"kubernetes.io/projected/b7319163-0333-4d9f-ae8b-abe6beaaa690-kube-api-access-7mjn2\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:29:54.748127 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:54.747940 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7319163-0333-4d9f-ae8b-abe6beaaa690-plugins-config-volume\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:29:54.748127 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:54.747954 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3221733-0209-46bb-a456-7bdcec80a1bc-metrics-certs\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:29:54.748127 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:54.747966 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dc5ea8f-4e68-4452-adb0-b20626023ea8-plugins-config-volume\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:29:54.748127 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:54.747977 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-896wf\" (UniqueName: \"kubernetes.io/projected/7dc5ea8f-4e68-4452-adb0-b20626023ea8-kube-api-access-896wf\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:29:54.748127 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:54.747993 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3221733-0209-46bb-a456-7bdcec80a1bc-maas-api-tls\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:29:55.104207 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:55.104123 2574 generic.go:356] "Generic (PLEG): container finished" podID="7dc5ea8f-4e68-4452-adb0-b20626023ea8" containerID="0c18fc5bd6082a92958c4de2626438b7c2e44276cd77f93aa9708203c01f64c3" exitCode=0 Sep 14 11:29:55.104207 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:55.104162 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e42fe6-7f857b48b9-cp8r5" Sep 14 11:29:55.104207 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:55.104192 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e42fe6-7f857b48b9-cp8r5" event={"ID":"7dc5ea8f-4e68-4452-adb0-b20626023ea8","Type":"ContainerDied","Data":"0c18fc5bd6082a92958c4de2626438b7c2e44276cd77f93aa9708203c01f64c3"} Sep 14 11:29:55.104507 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:55.104238 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e42fe6-7f857b48b9-cp8r5" event={"ID":"7dc5ea8f-4e68-4452-adb0-b20626023ea8","Type":"ContainerDied","Data":"9503c395ff56212adb27e14da42a4a3c3c3034a41541c36746e0cf98d9a1a04b"} Sep 14 11:29:55.104507 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:55.104280 2574 scope.go:117] "RemoveContainer" containerID="0c18fc5bd6082a92958c4de2626438b7c2e44276cd77f93aa9708203c01f64c3" Sep 14 11:29:55.105817 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:55.105788 2574 generic.go:356] "Generic (PLEG): container finished" podID="b7319163-0333-4d9f-ae8b-abe6beaaa690" containerID="dc9f3e6c9702c6501547ae4cc48fd9a7fe24b5d9a402177eca22cab5b19cecc9" exitCode=0 Sep 14 11:29:55.105954 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:55.105828 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e42fe6-75ddbdd474-f5klc" Sep 14 11:29:55.105954 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:55.105828 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e42fe6-75ddbdd474-f5klc" event={"ID":"b7319163-0333-4d9f-ae8b-abe6beaaa690","Type":"ContainerDied","Data":"dc9f3e6c9702c6501547ae4cc48fd9a7fe24b5d9a402177eca22cab5b19cecc9"} Sep 14 11:29:55.105954 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:55.105875 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e42fe6-75ddbdd474-f5klc" event={"ID":"b7319163-0333-4d9f-ae8b-abe6beaaa690","Type":"ContainerDied","Data":"9a2f95b68cfc9d9821ceb0466e85b93fed70b223b872168973b3f127682a1491"} Sep 14 11:29:55.107588 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:55.107563 2574 generic.go:356] "Generic (PLEG): container finished" podID="c3221733-0209-46bb-a456-7bdcec80a1bc" containerID="4ef22319893da69ae80ade8e501acdc71636444baee86111aa221b223b03b563" exitCode=0 Sep 14 11:29:55.107708 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:55.107609 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh" event={"ID":"c3221733-0209-46bb-a456-7bdcec80a1bc","Type":"ContainerDied","Data":"4ef22319893da69ae80ade8e501acdc71636444baee86111aa221b223b03b563"} Sep 14 11:29:55.107708 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:55.107648 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh" event={"ID":"c3221733-0209-46bb-a456-7bdcec80a1bc","Type":"ContainerDied","Data":"7070b4e4c9d9c36c11776edbbabb020c20661cff5bd9c4772fcb45ae96297865"} Sep 14 11:29:55.107708 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:55.107612 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh" Sep 14 11:29:55.121028 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:55.119669 2574 scope.go:117] "RemoveContainer" containerID="0c18fc5bd6082a92958c4de2626438b7c2e44276cd77f93aa9708203c01f64c3" Sep 14 11:29:55.121920 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:29:55.121888 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c18fc5bd6082a92958c4de2626438b7c2e44276cd77f93aa9708203c01f64c3\": container with ID starting with 0c18fc5bd6082a92958c4de2626438b7c2e44276cd77f93aa9708203c01f64c3 not found: ID does not exist" containerID="0c18fc5bd6082a92958c4de2626438b7c2e44276cd77f93aa9708203c01f64c3" Sep 14 11:29:55.122048 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:55.121933 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c18fc5bd6082a92958c4de2626438b7c2e44276cd77f93aa9708203c01f64c3"} err="failed to get container status \"0c18fc5bd6082a92958c4de2626438b7c2e44276cd77f93aa9708203c01f64c3\": rpc error: code = NotFound desc = could not find container \"0c18fc5bd6082a92958c4de2626438b7c2e44276cd77f93aa9708203c01f64c3\": container with ID starting with 0c18fc5bd6082a92958c4de2626438b7c2e44276cd77f93aa9708203c01f64c3 not found: ID does not exist" Sep 14 11:29:55.122048 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:55.121961 2574 scope.go:117] "RemoveContainer" containerID="dc9f3e6c9702c6501547ae4cc48fd9a7fe24b5d9a402177eca22cab5b19cecc9" Sep 14 11:29:55.136130 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:55.136113 2574 scope.go:117] "RemoveContainer" containerID="dc9f3e6c9702c6501547ae4cc48fd9a7fe24b5d9a402177eca22cab5b19cecc9" Sep 14 11:29:55.136441 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:29:55.136422 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc9f3e6c9702c6501547ae4cc48fd9a7fe24b5d9a402177eca22cab5b19cecc9\": container with ID starting with dc9f3e6c9702c6501547ae4cc48fd9a7fe24b5d9a402177eca22cab5b19cecc9 not found: ID does not exist" containerID="dc9f3e6c9702c6501547ae4cc48fd9a7fe24b5d9a402177eca22cab5b19cecc9" Sep 14 11:29:55.136491 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:55.136449 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc9f3e6c9702c6501547ae4cc48fd9a7fe24b5d9a402177eca22cab5b19cecc9"} err="failed to get container status \"dc9f3e6c9702c6501547ae4cc48fd9a7fe24b5d9a402177eca22cab5b19cecc9\": rpc error: code = NotFound desc = could not find container \"dc9f3e6c9702c6501547ae4cc48fd9a7fe24b5d9a402177eca22cab5b19cecc9\": container with ID starting with dc9f3e6c9702c6501547ae4cc48fd9a7fe24b5d9a402177eca22cab5b19cecc9 not found: ID does not exist" Sep 14 11:29:55.136491 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:55.136466 2574 scope.go:117] "RemoveContainer" containerID="4ef22319893da69ae80ade8e501acdc71636444baee86111aa221b223b03b563" Sep 14 11:29:55.136662 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:55.136643 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e42fe6-7f857b48b9-cp8r5"] Sep 14 11:29:55.139715 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:55.139695 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e42fe6-7f857b48b9-cp8r5"] Sep 14 11:29:55.152732 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:55.152712 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e42fe6-75ddbdd474-f5klc"] Sep 14 11:29:55.153133 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:55.153109 2574 scope.go:117] "RemoveContainer" containerID="4ef22319893da69ae80ade8e501acdc71636444baee86111aa221b223b03b563" Sep 14 11:29:55.153497 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:29:55.153479 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ef22319893da69ae80ade8e501acdc71636444baee86111aa221b223b03b563\": container with ID starting with 4ef22319893da69ae80ade8e501acdc71636444baee86111aa221b223b03b563 not found: ID does not exist" containerID="4ef22319893da69ae80ade8e501acdc71636444baee86111aa221b223b03b563" Sep 14 11:29:55.153548 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:55.153505 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ef22319893da69ae80ade8e501acdc71636444baee86111aa221b223b03b563"} err="failed to get container status \"4ef22319893da69ae80ade8e501acdc71636444baee86111aa221b223b03b563\": rpc error: code = NotFound desc = could not find container \"4ef22319893da69ae80ade8e501acdc71636444baee86111aa221b223b03b563\": container with ID starting with 4ef22319893da69ae80ade8e501acdc71636444baee86111aa221b223b03b563 not found: ID does not exist" Sep 14 11:29:55.158456 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:55.158436 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e42fe6-75ddbdd474-f5klc"] Sep 14 11:29:55.168820 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:55.168801 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh"] Sep 14 11:29:55.173901 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:55.173883 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh"] Sep 14 11:29:55.899631 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:55.899602 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7dc5ea8f-4e68-4452-adb0-b20626023ea8" path="/var/lib/kubelet/pods/7dc5ea8f-4e68-4452-adb0-b20626023ea8/volumes" Sep 14 11:29:55.899946 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:55.899933 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7319163-0333-4d9f-ae8b-abe6beaaa690" path="/var/lib/kubelet/pods/b7319163-0333-4d9f-ae8b-abe6beaaa690/volumes" Sep 14 11:29:55.900289 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:55.900258 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3221733-0209-46bb-a456-7bdcec80a1bc" path="/var/lib/kubelet/pods/c3221733-0209-46bb-a456-7bdcec80a1bc/volumes" Sep 14 11:29:56.101702 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:56.101677 2574 scope.go:117] "RemoveContainer" containerID="78e33bac56da92aef35839c7b0af6108441550d64c9d25f049a776deb866e7e1" Sep 14 11:29:57.473760 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:29:57.473721 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e42fe6-76c4c45756-pdchh" podUID="c3221733-0209-46bb-a456-7bdcec80a1bc" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.62:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Sep 14 11:30:00.158537 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:00.158500 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-de0f7a-29823090-zgxps"] Sep 14 11:30:00.159192 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:00.159170 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2e0fb04-8585-4c7c-82bd-4655cd00042a" containerName="payload-processing" Sep 14 11:30:00.159355 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:00.159195 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2e0fb04-8585-4c7c-82bd-4655cd00042a" containerName="payload-processing" Sep 14 11:30:00.159355 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:00.159211 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6480efe3-8062-43b1-947a-b60bfdb029d5" containerName="maas-api" Sep 14 11:30:00.159355 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:00.159220 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6480efe3-8062-43b1-947a-b60bfdb029d5" containerName="maas-api" Sep 14 11:30:00.159355 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:00.159233 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a0970df-5b89-491f-9ef9-ac65af2f54d5" containerName="payload-pre-processing" Sep 14 11:30:00.159355 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:00.159241 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a0970df-5b89-491f-9ef9-ac65af2f54d5" containerName="payload-pre-processing" Sep 14 11:30:00.159355 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:00.159255 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dc5ea8f-4e68-4452-adb0-b20626023ea8" containerName="payload-processing" Sep 14 11:30:00.159355 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:00.159290 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dc5ea8f-4e68-4452-adb0-b20626023ea8" containerName="payload-processing" Sep 14 11:30:00.159355 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:00.159313 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3221733-0209-46bb-a456-7bdcec80a1bc" containerName="maas-api" Sep 14 11:30:00.159355 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:00.159322 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3221733-0209-46bb-a456-7bdcec80a1bc" containerName="maas-api" Sep 14 11:30:00.159355 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:00.159334 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7319163-0333-4d9f-ae8b-abe6beaaa690" containerName="payload-pre-processing" Sep 14 11:30:00.159355 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:00.159343 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7319163-0333-4d9f-ae8b-abe6beaaa690" containerName="payload-pre-processing" Sep 14 11:30:00.159771 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:00.159366 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9b2415c-3a23-4717-ba02-71543cd63a88" containerName="maas-api" Sep 14 11:30:00.159771 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:00.159375 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9b2415c-3a23-4717-ba02-71543cd63a88" containerName="maas-api" Sep 14 11:30:00.159771 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:00.159457 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dc5ea8f-4e68-4452-adb0-b20626023ea8" containerName="payload-processing" Sep 14 11:30:00.159771 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:00.159466 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7319163-0333-4d9f-ae8b-abe6beaaa690" containerName="payload-pre-processing" Sep 14 11:30:00.159771 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:00.159475 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2e0fb04-8585-4c7c-82bd-4655cd00042a" containerName="payload-processing" Sep 14 11:30:00.159771 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:00.159483 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3221733-0209-46bb-a456-7bdcec80a1bc" containerName="maas-api" Sep 14 11:30:00.159771 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:00.159490 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6480efe3-8062-43b1-947a-b60bfdb029d5" containerName="maas-api" Sep 14 11:30:00.159771 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:00.159500 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9b2415c-3a23-4717-ba02-71543cd63a88" containerName="maas-api" Sep 14 11:30:00.159771 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:00.159510 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a0970df-5b89-491f-9ef9-ac65af2f54d5" containerName="payload-pre-processing" Sep 14 11:30:00.162710 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:00.162687 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-41ae25-29823090-6pgs6"] Sep 14 11:30:00.162836 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:00.162799 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-de0f7a-29823090-zgxps" Sep 14 11:30:00.166474 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:00.166342 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f98vt\"" Sep 14 11:30:00.167784 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:00.167765 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-41ae25-29823090-6pgs6" Sep 14 11:30:00.173847 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:00.173406 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-de0f7a-29823090-zgxps"] Sep 14 11:30:00.177478 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:00.177449 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-41ae25-29823090-6pgs6"] Sep 14 11:30:00.178640 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:00.178622 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-de0f7a-29823090-zgxps" Sep 14 11:30:00.188237 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:00.188219 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-41ae25-29823090-6pgs6" Sep 14 11:30:00.464219 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:00.464190 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-de0f7a-29823090-zgxps"] Sep 14 11:30:00.465794 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:30:00.465766 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ad99492_2e73_45e3_a24d_6b7f3a6ad3ba.slice/crio-04b5eeffdb28229bd12f3ebf4114c2832a01e9d095bd1751f0345ec4538582ed WatchSource:0}: Error finding container 04b5eeffdb28229bd12f3ebf4114c2832a01e9d095bd1751f0345ec4538582ed: Status 404 returned error can't find the container with id 04b5eeffdb28229bd12f3ebf4114c2832a01e9d095bd1751f0345ec4538582ed Sep 14 11:30:00.485800 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:00.485780 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-41ae25-29823090-6pgs6"] Sep 14 11:30:00.487161 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:30:00.487136 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1444f46_198c_4e77_acd5_ca739282a903.slice/crio-7bc03f91909c5fe1c30f4d5f8c58ab24a8f76fc64bb4329e7cb7b1778ef2709a WatchSource:0}: Error finding container 7bc03f91909c5fe1c30f4d5f8c58ab24a8f76fc64bb4329e7cb7b1778ef2709a: Status 404 returned error can't find the container with id 7bc03f91909c5fe1c30f4d5f8c58ab24a8f76fc64bb4329e7cb7b1778ef2709a Sep 14 11:30:01.145636 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:01.145593 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-de0f7a-29823090-zgxps" event={"ID":"7ad99492-2e73-45e3-a24d-6b7f3a6ad3ba","Type":"ContainerStarted","Data":"04b5eeffdb28229bd12f3ebf4114c2832a01e9d095bd1751f0345ec4538582ed"} Sep 14 11:30:01.147165 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:01.147132 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-41ae25-29823090-6pgs6" event={"ID":"c1444f46-198c-4e77-acd5-ca739282a903","Type":"ContainerStarted","Data":"7bc03f91909c5fe1c30f4d5f8c58ab24a8f76fc64bb4329e7cb7b1778ef2709a"} Sep 14 11:30:04.168550 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:04.168454 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-de0f7a-29823090-zgxps" event={"ID":"7ad99492-2e73-45e3-a24d-6b7f3a6ad3ba","Type":"ContainerStarted","Data":"e45f52ddcf71532aab8e96b1feb7e6dd7681d4a3ed8648fdf5276ec4712ce7ea"} Sep 14 11:30:04.170110 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:04.170082 2574 generic.go:356] "Generic (PLEG): container finished" podID="c1444f46-198c-4e77-acd5-ca739282a903" containerID="4b057774c9b18d1548d86530f513fa1127945ec4a4abb9784aa6250abb495394" exitCode=0 Sep 14 11:30:04.170232 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:04.170150 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-41ae25-29823090-6pgs6" event={"ID":"c1444f46-198c-4e77-acd5-ca739282a903","Type":"ContainerDied","Data":"4b057774c9b18d1548d86530f513fa1127945ec4a4abb9784aa6250abb495394"} Sep 14 11:30:04.185516 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:04.185466 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-de0f7a-29823090-zgxps" podStartSLOduration=1.065484905 podStartE2EDuration="4.185448285s" totalImagesPullingTime="3.11996338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:30:04.185204093 +0000 UTC m=+1748.805416638" watchObservedRunningTime="2026-09-14 11:30:04.185448285 +0000 UTC m=+1748.805660809" Sep 14 11:30:05.177142 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:05.177107 2574 generic.go:356] "Generic (PLEG): container finished" podID="7ad99492-2e73-45e3-a24d-6b7f3a6ad3ba" containerID="e45f52ddcf71532aab8e96b1feb7e6dd7681d4a3ed8648fdf5276ec4712ce7ea" exitCode=0 Sep 14 11:30:05.177576 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:05.177189 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-de0f7a-29823090-zgxps" event={"ID":"7ad99492-2e73-45e3-a24d-6b7f3a6ad3ba","Type":"ContainerDied","Data":"e45f52ddcf71532aab8e96b1feb7e6dd7681d4a3ed8648fdf5276ec4712ce7ea"} Sep 14 11:30:05.307862 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:05.307832 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-41ae25-29823090-6pgs6" Sep 14 11:30:06.183629 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:06.183604 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-41ae25-29823090-6pgs6" Sep 14 11:30:06.184000 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:06.183638 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-41ae25-29823090-6pgs6" event={"ID":"c1444f46-198c-4e77-acd5-ca739282a903","Type":"ContainerDied","Data":"7bc03f91909c5fe1c30f4d5f8c58ab24a8f76fc64bb4329e7cb7b1778ef2709a"} Sep 14 11:30:06.184000 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:06.183681 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bc03f91909c5fe1c30f4d5f8c58ab24a8f76fc64bb4329e7cb7b1778ef2709a" Sep 14 11:30:06.334788 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:06.334764 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-de0f7a-29823090-zgxps" Sep 14 11:30:07.190494 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:07.190461 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-de0f7a-29823090-zgxps" Sep 14 11:30:07.190912 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:07.190490 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-de0f7a-29823090-zgxps" event={"ID":"7ad99492-2e73-45e3-a24d-6b7f3a6ad3ba","Type":"ContainerDied","Data":"04b5eeffdb28229bd12f3ebf4114c2832a01e9d095bd1751f0345ec4538582ed"} Sep 14 11:30:07.190912 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:07.190536 2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04b5eeffdb28229bd12f3ebf4114c2832a01e9d095bd1751f0345ec4538582ed" Sep 14 11:30:12.891948 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:12.891914 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-41ae25-29823090-6pgs6"] Sep 14 11:30:12.894263 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:12.894238 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-41ae25-29823090-6pgs6"] Sep 14 11:30:12.912934 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:12.912886 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-41ae25-7c78f9c687-gqbtv"] Sep 14 11:30:12.913157 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:12.913129 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-41ae25-7c78f9c687-gqbtv" podUID="dfd813a8-2d06-4faf-99ed-23c6d83a2b82" containerName="payload-processing" containerID="cri-o://5b3bdc7cf447cbb2cd4c1c388c0a5ed6cfc956c36c36a10f69965fe9de8db87b" gracePeriod=30 Sep 14 11:30:13.189179 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:13.189152 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-41ae25-7c78f9c687-gqbtv" Sep 14 11:30:13.226314 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:13.226238 2574 generic.go:356] "Generic (PLEG): container finished" podID="dfd813a8-2d06-4faf-99ed-23c6d83a2b82" containerID="5b3bdc7cf447cbb2cd4c1c388c0a5ed6cfc956c36c36a10f69965fe9de8db87b" exitCode=0 Sep 14 11:30:13.226619 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:13.226599 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-41ae25-7c78f9c687-gqbtv" Sep 14 11:30:13.226842 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:13.226663 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-41ae25-7c78f9c687-gqbtv" event={"ID":"dfd813a8-2d06-4faf-99ed-23c6d83a2b82","Type":"ContainerDied","Data":"5b3bdc7cf447cbb2cd4c1c388c0a5ed6cfc956c36c36a10f69965fe9de8db87b"} Sep 14 11:30:13.227067 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:13.226872 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-41ae25-7c78f9c687-gqbtv" event={"ID":"dfd813a8-2d06-4faf-99ed-23c6d83a2b82","Type":"ContainerDied","Data":"46c1fcbcf1c10bf9bcfb93e4536006a57d7626004e43365915d00d7cf99fa840"} Sep 14 11:30:13.227067 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:13.226908 2574 scope.go:117] "RemoveContainer" containerID="5b3bdc7cf447cbb2cd4c1c388c0a5ed6cfc956c36c36a10f69965fe9de8db87b" Sep 14 11:30:13.239490 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:13.239461 2574 scope.go:117] "RemoveContainer" containerID="5b3bdc7cf447cbb2cd4c1c388c0a5ed6cfc956c36c36a10f69965fe9de8db87b" Sep 14 11:30:13.239794 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:30:13.239772 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b3bdc7cf447cbb2cd4c1c388c0a5ed6cfc956c36c36a10f69965fe9de8db87b\": container with ID starting with 5b3bdc7cf447cbb2cd4c1c388c0a5ed6cfc956c36c36a10f69965fe9de8db87b not found: ID does not exist" containerID="5b3bdc7cf447cbb2cd4c1c388c0a5ed6cfc956c36c36a10f69965fe9de8db87b" Sep 14 11:30:13.239900 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:13.239801 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b3bdc7cf447cbb2cd4c1c388c0a5ed6cfc956c36c36a10f69965fe9de8db87b"} err="failed to get container status \"5b3bdc7cf447cbb2cd4c1c388c0a5ed6cfc956c36c36a10f69965fe9de8db87b\": rpc error: code = NotFound desc = could not find container \"5b3bdc7cf447cbb2cd4c1c388c0a5ed6cfc956c36c36a10f69965fe9de8db87b\": container with ID starting with 5b3bdc7cf447cbb2cd4c1c388c0a5ed6cfc956c36c36a10f69965fe9de8db87b not found: ID does not exist" Sep 14 11:30:13.336196 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:13.336169 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd813a8-2d06-4faf-99ed-23c6d83a2b82-plugins-config-volume\") pod \"dfd813a8-2d06-4faf-99ed-23c6d83a2b82\" (UID: \"dfd813a8-2d06-4faf-99ed-23c6d83a2b82\") " Sep 14 11:30:13.336352 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:13.336208 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vsxk2\" (UniqueName: \"kubernetes.io/projected/dfd813a8-2d06-4faf-99ed-23c6d83a2b82-kube-api-access-vsxk2\") pod \"dfd813a8-2d06-4faf-99ed-23c6d83a2b82\" (UID: \"dfd813a8-2d06-4faf-99ed-23c6d83a2b82\") " Sep 14 11:30:13.336534 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:13.336514 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfd813a8-2d06-4faf-99ed-23c6d83a2b82-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dfd813a8-2d06-4faf-99ed-23c6d83a2b82" (UID: "dfd813a8-2d06-4faf-99ed-23c6d83a2b82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:30:13.336651 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:13.336632 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd813a8-2d06-4faf-99ed-23c6d83a2b82-plugins-config-volume\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:30:13.338447 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:13.338418 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfd813a8-2d06-4faf-99ed-23c6d83a2b82-kube-api-access-vsxk2" (OuterVolumeSpecName: "kube-api-access-vsxk2") pod "dfd813a8-2d06-4faf-99ed-23c6d83a2b82" (UID: "dfd813a8-2d06-4faf-99ed-23c6d83a2b82"). InnerVolumeSpecName "kube-api-access-vsxk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:30:13.437930 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:13.437902 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsxk2\" (UniqueName: \"kubernetes.io/projected/dfd813a8-2d06-4faf-99ed-23c6d83a2b82-kube-api-access-vsxk2\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:30:13.553135 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:13.553107 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-41ae25-7c78f9c687-gqbtv"] Sep 14 11:30:13.559497 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:13.559472 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-41ae25-7c78f9c687-gqbtv"] Sep 14 11:30:13.899657 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:13.899587 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1444f46-198c-4e77-acd5-ca739282a903" path="/var/lib/kubelet/pods/c1444f46-198c-4e77-acd5-ca739282a903/volumes" Sep 14 11:30:13.900041 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:13.899906 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfd813a8-2d06-4faf-99ed-23c6d83a2b82" path="/var/lib/kubelet/pods/dfd813a8-2d06-4faf-99ed-23c6d83a2b82/volumes" Sep 14 11:30:20.964154 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:20.964118 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-de0f7a-6fc97c6fdd-mnhwk"] Sep 14 11:30:20.964593 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:20.964375 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-de0f7a-6fc97c6fdd-mnhwk" podUID="454c8e0e-edcf-4c29-b697-a6cc79092849" containerName="maas-api" containerID="cri-o://780e35f8095c46f8fb844a05599285e186e9a342ee5244f06461673f3ba7f7aa" gracePeriod=30 Sep 14 11:30:21.004244 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:21.004021 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-de0f7a-29823090-zgxps"] Sep 14 11:30:21.006644 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:21.006619 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-de0f7a-29823090-zgxps"] Sep 14 11:30:21.045206 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:21.045174 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-de0f7a-6ccfc74d69-9xjkc"] Sep 14 11:30:21.045472 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:21.045419 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-de0f7a-6ccfc74d69-9xjkc" podUID="af78aa4d-5236-4484-867d-0bf6e594072d" containerName="payload-processing" containerID="cri-o://0fa844f05adef13fc10f9a4c25866e8efba541de26e3118af0232413ac7da6ef" gracePeriod=30 Sep 14 11:30:21.050458 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:21.050429 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-de0f7a-687bdd887d-b86hw"] Sep 14 11:30:21.050638 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:21.050612 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-de0f7a-687bdd887d-b86hw" podUID="56ede25d-658f-4465-94ec-b64625cf6f6f" containerName="payload-pre-processing" containerID="cri-o://98ec95515f8cbe58ce36794f94eee0079e223be93047e513a0fe81af39af6fdc" gracePeriod=30 Sep 14 11:30:21.300708 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:21.300667 2574 generic.go:356] "Generic (PLEG): container finished" podID="454c8e0e-edcf-4c29-b697-a6cc79092849" containerID="780e35f8095c46f8fb844a05599285e186e9a342ee5244f06461673f3ba7f7aa" exitCode=0 Sep 14 11:30:21.301042 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:21.300782 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-de0f7a-6fc97c6fdd-mnhwk" event={"ID":"454c8e0e-edcf-4c29-b697-a6cc79092849","Type":"ContainerDied","Data":"780e35f8095c46f8fb844a05599285e186e9a342ee5244f06461673f3ba7f7aa"} Sep 14 11:30:21.307409 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:21.307333 2574 generic.go:356] "Generic (PLEG): container finished" podID="af78aa4d-5236-4484-867d-0bf6e594072d" containerID="0fa844f05adef13fc10f9a4c25866e8efba541de26e3118af0232413ac7da6ef" exitCode=0 Sep 14 11:30:21.307682 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:21.307658 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-de0f7a-6ccfc74d69-9xjkc" event={"ID":"af78aa4d-5236-4484-867d-0bf6e594072d","Type":"ContainerDied","Data":"0fa844f05adef13fc10f9a4c25866e8efba541de26e3118af0232413ac7da6ef"} Sep 14 11:30:21.310235 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:21.310214 2574 generic.go:356] "Generic (PLEG): container finished" podID="56ede25d-658f-4465-94ec-b64625cf6f6f" containerID="98ec95515f8cbe58ce36794f94eee0079e223be93047e513a0fe81af39af6fdc" exitCode=0 Sep 14 11:30:21.311652 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:21.310723 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-de0f7a-687bdd887d-b86hw" event={"ID":"56ede25d-658f-4465-94ec-b64625cf6f6f","Type":"ContainerDied","Data":"98ec95515f8cbe58ce36794f94eee0079e223be93047e513a0fe81af39af6fdc"} Sep 14 11:30:21.337914 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:21.337885 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-de0f7a-6fc97c6fdd-mnhwk" Sep 14 11:30:21.341988 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:21.341966 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-de0f7a-6ccfc74d69-9xjkc" Sep 14 11:30:21.386068 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:21.386041 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-de0f7a-687bdd887d-b86hw" Sep 14 11:30:21.414593 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:21.414566 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jx5xc\" (UniqueName: \"kubernetes.io/projected/454c8e0e-edcf-4c29-b697-a6cc79092849-kube-api-access-jx5xc\") pod \"454c8e0e-edcf-4c29-b697-a6cc79092849\" (UID: \"454c8e0e-edcf-4c29-b697-a6cc79092849\") " Sep 14 11:30:21.414732 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:21.414617 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/454c8e0e-edcf-4c29-b697-a6cc79092849-maas-api-tls\") pod \"454c8e0e-edcf-4c29-b697-a6cc79092849\" (UID: \"454c8e0e-edcf-4c29-b697-a6cc79092849\") " Sep 14 11:30:21.414732 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:21.414680 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47j57\" (UniqueName: \"kubernetes.io/projected/56ede25d-658f-4465-94ec-b64625cf6f6f-kube-api-access-47j57\") pod \"56ede25d-658f-4465-94ec-b64625cf6f6f\" (UID: \"56ede25d-658f-4465-94ec-b64625cf6f6f\") " Sep 14 11:30:21.414732 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:21.414708 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af78aa4d-5236-4484-867d-0bf6e594072d-plugins-config-volume\") pod \"af78aa4d-5236-4484-867d-0bf6e594072d\" (UID: \"af78aa4d-5236-4484-867d-0bf6e594072d\") " Sep 14 11:30:21.414936 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:21.414788 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/454c8e0e-edcf-4c29-b697-a6cc79092849-metrics-certs\") pod \"454c8e0e-edcf-4c29-b697-a6cc79092849\" (UID: \"454c8e0e-edcf-4c29-b697-a6cc79092849\") " Sep 14 11:30:21.414936 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:21.414853 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56ede25d-658f-4465-94ec-b64625cf6f6f-plugins-config-volume\") pod \"56ede25d-658f-4465-94ec-b64625cf6f6f\" (UID: \"56ede25d-658f-4465-94ec-b64625cf6f6f\") " Sep 14 11:30:21.414936 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:21.414893 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/454c8e0e-edcf-4c29-b697-a6cc79092849-openshift-service-ca\") pod \"454c8e0e-edcf-4c29-b697-a6cc79092849\" (UID: \"454c8e0e-edcf-4c29-b697-a6cc79092849\") " Sep 14 11:30:21.414936 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:21.414932 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jmk2\" (UniqueName: \"kubernetes.io/projected/af78aa4d-5236-4484-867d-0bf6e594072d-kube-api-access-4jmk2\") pod \"af78aa4d-5236-4484-867d-0bf6e594072d\" (UID: \"af78aa4d-5236-4484-867d-0bf6e594072d\") " Sep 14 11:30:21.415187 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:21.415030 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af78aa4d-5236-4484-867d-0bf6e594072d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "af78aa4d-5236-4484-867d-0bf6e594072d" (UID: "af78aa4d-5236-4484-867d-0bf6e594072d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:30:21.415533 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:21.415288 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/454c8e0e-edcf-4c29-b697-a6cc79092849-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "454c8e0e-edcf-4c29-b697-a6cc79092849" (UID: "454c8e0e-edcf-4c29-b697-a6cc79092849"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:30:21.415533 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:21.415452 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af78aa4d-5236-4484-867d-0bf6e594072d-plugins-config-volume\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:30:21.416198 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:21.416169 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56ede25d-658f-4465-94ec-b64625cf6f6f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "56ede25d-658f-4465-94ec-b64625cf6f6f" (UID: "56ede25d-658f-4465-94ec-b64625cf6f6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:30:21.418196 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:21.418173 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/454c8e0e-edcf-4c29-b697-a6cc79092849-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "454c8e0e-edcf-4c29-b697-a6cc79092849" (UID: "454c8e0e-edcf-4c29-b697-a6cc79092849"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:30:21.418357 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:21.418259 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/454c8e0e-edcf-4c29-b697-a6cc79092849-kube-api-access-jx5xc" (OuterVolumeSpecName: "kube-api-access-jx5xc") pod "454c8e0e-edcf-4c29-b697-a6cc79092849" (UID: "454c8e0e-edcf-4c29-b697-a6cc79092849"). InnerVolumeSpecName "kube-api-access-jx5xc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:30:21.419104 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:21.419070 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/454c8e0e-edcf-4c29-b697-a6cc79092849-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "454c8e0e-edcf-4c29-b697-a6cc79092849" (UID: "454c8e0e-edcf-4c29-b697-a6cc79092849"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:30:21.419321 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:21.419298 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56ede25d-658f-4465-94ec-b64625cf6f6f-kube-api-access-47j57" (OuterVolumeSpecName: "kube-api-access-47j57") pod "56ede25d-658f-4465-94ec-b64625cf6f6f" (UID: "56ede25d-658f-4465-94ec-b64625cf6f6f"). InnerVolumeSpecName "kube-api-access-47j57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:30:21.419594 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:21.419562 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af78aa4d-5236-4484-867d-0bf6e594072d-kube-api-access-4jmk2" (OuterVolumeSpecName: "kube-api-access-4jmk2") pod "af78aa4d-5236-4484-867d-0bf6e594072d" (UID: "af78aa4d-5236-4484-867d-0bf6e594072d"). InnerVolumeSpecName "kube-api-access-4jmk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:30:21.517164 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:21.517079 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jmk2\" (UniqueName: \"kubernetes.io/projected/af78aa4d-5236-4484-867d-0bf6e594072d-kube-api-access-4jmk2\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:30:21.517164 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:21.517122 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx5xc\" (UniqueName: \"kubernetes.io/projected/454c8e0e-edcf-4c29-b697-a6cc79092849-kube-api-access-jx5xc\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:30:21.517164 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:21.517138 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/454c8e0e-edcf-4c29-b697-a6cc79092849-maas-api-tls\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:30:21.517164 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:21.517154 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47j57\" (UniqueName: \"kubernetes.io/projected/56ede25d-658f-4465-94ec-b64625cf6f6f-kube-api-access-47j57\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:30:21.517417 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:21.517167 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/454c8e0e-edcf-4c29-b697-a6cc79092849-metrics-certs\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:30:21.517417 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:21.517181 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56ede25d-658f-4465-94ec-b64625cf6f6f-plugins-config-volume\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:30:21.517417 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:21.517193 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/454c8e0e-edcf-4c29-b697-a6cc79092849-openshift-service-ca\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:30:21.898896 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:21.898814 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ad99492-2e73-45e3-a24d-6b7f3a6ad3ba" path="/var/lib/kubelet/pods/7ad99492-2e73-45e3-a24d-6b7f3a6ad3ba/volumes" Sep 14 11:30:22.318328 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:22.318298 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-de0f7a-6fc97c6fdd-mnhwk" Sep 14 11:30:22.318328 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:22.318317 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-de0f7a-6fc97c6fdd-mnhwk" event={"ID":"454c8e0e-edcf-4c29-b697-a6cc79092849","Type":"ContainerDied","Data":"3ef9d85f712ecdc388fc4d32f899efa68e15974714e1baa6fd91cff46b9803df"} Sep 14 11:30:22.318887 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:22.318392 2574 scope.go:117] "RemoveContainer" containerID="780e35f8095c46f8fb844a05599285e186e9a342ee5244f06461673f3ba7f7aa" Sep 14 11:30:22.319997 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:22.319846 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-de0f7a-6ccfc74d69-9xjkc" event={"ID":"af78aa4d-5236-4484-867d-0bf6e594072d","Type":"ContainerDied","Data":"0c5a846ee7f9471077ff5389f3a9b338475c8acf932dd3c60801f3e330a55181"} Sep 14 11:30:22.319997 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:22.319911 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-de0f7a-6ccfc74d69-9xjkc" Sep 14 11:30:22.321947 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:22.321918 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-de0f7a-687bdd887d-b86hw" Sep 14 11:30:22.322083 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:22.321941 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-de0f7a-687bdd887d-b86hw" event={"ID":"56ede25d-658f-4465-94ec-b64625cf6f6f","Type":"ContainerDied","Data":"6b7d2b261d75e0b69ec5af2e972c446be9424ce85fba78132f9352399228934a"} Sep 14 11:30:22.330112 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:22.330095 2574 scope.go:117] "RemoveContainer" containerID="0fa844f05adef13fc10f9a4c25866e8efba541de26e3118af0232413ac7da6ef" Sep 14 11:30:22.338053 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:22.338036 2574 scope.go:117] "RemoveContainer" containerID="98ec95515f8cbe58ce36794f94eee0079e223be93047e513a0fe81af39af6fdc" Sep 14 11:30:22.347782 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:22.347762 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-de0f7a-6fc97c6fdd-mnhwk"] Sep 14 11:30:22.351710 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:22.351687 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-de0f7a-6fc97c6fdd-mnhwk"] Sep 14 11:30:22.363795 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:22.363773 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-de0f7a-687bdd887d-b86hw"] Sep 14 11:30:22.371628 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:22.371608 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-de0f7a-687bdd887d-b86hw"] Sep 14 11:30:22.383329 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:22.383304 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-de0f7a-6ccfc74d69-9xjkc"] Sep 14 11:30:22.388331 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:22.388310 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-de0f7a-6ccfc74d69-9xjkc"] Sep 14 11:30:23.899799 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:23.899765 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="454c8e0e-edcf-4c29-b697-a6cc79092849" path="/var/lib/kubelet/pods/454c8e0e-edcf-4c29-b697-a6cc79092849/volumes" Sep 14 11:30:23.900179 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:23.900134 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56ede25d-658f-4465-94ec-b64625cf6f6f" path="/var/lib/kubelet/pods/56ede25d-658f-4465-94ec-b64625cf6f6f/volumes" Sep 14 11:30:23.900486 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:30:23.900473 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af78aa4d-5236-4484-867d-0bf6e594072d" path="/var/lib/kubelet/pods/af78aa4d-5236-4484-867d-0bf6e594072d/volumes" Sep 14 11:33:26.050367 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:26.050336 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-df3763-6b8f57db8f-k45xv"] Sep 14 11:33:26.051010 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:26.050985 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="454c8e0e-edcf-4c29-b697-a6cc79092849" containerName="maas-api" Sep 14 11:33:26.051120 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:26.051014 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="454c8e0e-edcf-4c29-b697-a6cc79092849" containerName="maas-api" Sep 14 11:33:26.051120 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:26.051031 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af78aa4d-5236-4484-867d-0bf6e594072d" containerName="payload-processing" Sep 14 11:33:26.051120 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:26.051042 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="af78aa4d-5236-4484-867d-0bf6e594072d" containerName="payload-processing" Sep 14 11:33:26.051120 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:26.051058 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfd813a8-2d06-4faf-99ed-23c6d83a2b82" containerName="payload-processing" Sep 14 11:33:26.051120 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:26.051063 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfd813a8-2d06-4faf-99ed-23c6d83a2b82" containerName="payload-processing" Sep 14 11:33:26.051120 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:26.051077 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ad99492-2e73-45e3-a24d-6b7f3a6ad3ba" containerName="cleanup" Sep 14 11:33:26.051120 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:26.051082 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ad99492-2e73-45e3-a24d-6b7f3a6ad3ba" containerName="cleanup" Sep 14 11:33:26.051120 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:26.051099 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56ede25d-658f-4465-94ec-b64625cf6f6f" containerName="payload-pre-processing" Sep 14 11:33:26.051120 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:26.051107 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="56ede25d-658f-4465-94ec-b64625cf6f6f" containerName="payload-pre-processing" Sep 14 11:33:26.051120 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:26.051113 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1444f46-198c-4e77-acd5-ca739282a903" containerName="cleanup" Sep 14 11:33:26.051120 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:26.051118 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1444f46-198c-4e77-acd5-ca739282a903" containerName="cleanup" Sep 14 11:33:26.051531 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:26.051191 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="56ede25d-658f-4465-94ec-b64625cf6f6f" containerName="payload-pre-processing" Sep 14 11:33:26.051531 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:26.051201 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1444f46-198c-4e77-acd5-ca739282a903" containerName="cleanup" Sep 14 11:33:26.051531 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:26.051209 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="af78aa4d-5236-4484-867d-0bf6e594072d" containerName="payload-processing" Sep 14 11:33:26.051531 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:26.051218 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="454c8e0e-edcf-4c29-b697-a6cc79092849" containerName="maas-api" Sep 14 11:33:26.051531 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:26.051223 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ad99492-2e73-45e3-a24d-6b7f3a6ad3ba" containerName="cleanup" Sep 14 11:33:26.051531 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:26.051229 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfd813a8-2d06-4faf-99ed-23c6d83a2b82" containerName="payload-processing" Sep 14 11:33:26.054279 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:26.054246 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df3763-6b8f57db8f-k45xv" Sep 14 11:33:26.057151 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:26.057130 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-df3763-serving-cert\"" Sep 14 11:33:26.064603 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:26.064578 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-df3763-6b8f57db8f-k45xv"] Sep 14 11:33:26.091859 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:26.091837 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41e4b6e5-a721-45a7-8540-31f75937c7f6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-df3763-6b8f57db8f-k45xv\" (UID: \"41e4b6e5-a721-45a7-8540-31f75937c7f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df3763-6b8f57db8f-k45xv" Sep 14 11:33:26.091967 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:26.091879 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85m2d\" (UniqueName: \"kubernetes.io/projected/41e4b6e5-a721-45a7-8540-31f75937c7f6-kube-api-access-85m2d\") pod \"maas-api-e2e-ipp-a-df3763-6b8f57db8f-k45xv\" (UID: \"41e4b6e5-a721-45a7-8540-31f75937c7f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df3763-6b8f57db8f-k45xv" Sep 14 11:33:26.091967 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:26.091918 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41e4b6e5-a721-45a7-8540-31f75937c7f6-metrics-certs\") pod \"maas-api-e2e-ipp-a-df3763-6b8f57db8f-k45xv\" (UID: \"41e4b6e5-a721-45a7-8540-31f75937c7f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df3763-6b8f57db8f-k45xv" Sep 14 11:33:26.092068 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:26.091975 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e4b6e5-a721-45a7-8540-31f75937c7f6-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-df3763-6b8f57db8f-k45xv\" (UID: \"41e4b6e5-a721-45a7-8540-31f75937c7f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df3763-6b8f57db8f-k45xv" Sep 14 11:33:26.192603 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:26.192576 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e4b6e5-a721-45a7-8540-31f75937c7f6-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-df3763-6b8f57db8f-k45xv\" (UID: \"41e4b6e5-a721-45a7-8540-31f75937c7f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df3763-6b8f57db8f-k45xv" Sep 14 11:33:26.192763 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:26.192649 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41e4b6e5-a721-45a7-8540-31f75937c7f6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-df3763-6b8f57db8f-k45xv\" (UID: \"41e4b6e5-a721-45a7-8540-31f75937c7f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df3763-6b8f57db8f-k45xv" Sep 14 11:33:26.192763 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:26.192684 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85m2d\" (UniqueName: \"kubernetes.io/projected/41e4b6e5-a721-45a7-8540-31f75937c7f6-kube-api-access-85m2d\") pod \"maas-api-e2e-ipp-a-df3763-6b8f57db8f-k45xv\" (UID: \"41e4b6e5-a721-45a7-8540-31f75937c7f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df3763-6b8f57db8f-k45xv" Sep 14 11:33:26.192763 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:26.192712 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41e4b6e5-a721-45a7-8540-31f75937c7f6-metrics-certs\") pod \"maas-api-e2e-ipp-a-df3763-6b8f57db8f-k45xv\" (UID: \"41e4b6e5-a721-45a7-8540-31f75937c7f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df3763-6b8f57db8f-k45xv" Sep 14 11:33:26.193343 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:26.193319 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e4b6e5-a721-45a7-8540-31f75937c7f6-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-df3763-6b8f57db8f-k45xv\" (UID: \"41e4b6e5-a721-45a7-8540-31f75937c7f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df3763-6b8f57db8f-k45xv" Sep 14 11:33:26.195323 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:26.195301 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41e4b6e5-a721-45a7-8540-31f75937c7f6-metrics-certs\") pod \"maas-api-e2e-ipp-a-df3763-6b8f57db8f-k45xv\" (UID: \"41e4b6e5-a721-45a7-8540-31f75937c7f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df3763-6b8f57db8f-k45xv" Sep 14 11:33:26.195419 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:26.195405 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41e4b6e5-a721-45a7-8540-31f75937c7f6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-df3763-6b8f57db8f-k45xv\" (UID: \"41e4b6e5-a721-45a7-8540-31f75937c7f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df3763-6b8f57db8f-k45xv" Sep 14 11:33:26.200579 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:26.200559 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85m2d\" (UniqueName: \"kubernetes.io/projected/41e4b6e5-a721-45a7-8540-31f75937c7f6-kube-api-access-85m2d\") pod \"maas-api-e2e-ipp-a-df3763-6b8f57db8f-k45xv\" (UID: \"41e4b6e5-a721-45a7-8540-31f75937c7f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df3763-6b8f57db8f-k45xv" Sep 14 11:33:26.367934 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:26.367859 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df3763-6b8f57db8f-k45xv" Sep 14 11:33:26.649334 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:26.649309 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-df3763-6b8f57db8f-k45xv"] Sep 14 11:33:26.651607 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:33:26.651562 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41e4b6e5_a721_45a7_8540_31f75937c7f6.slice/crio-2b4bd280b75684ebd8ad7c31599e1028e5393afc39eda94540b994ea54319936 WatchSource:0}: Error finding container 2b4bd280b75684ebd8ad7c31599e1028e5393afc39eda94540b994ea54319936: Status 404 returned error can't find the container with id 2b4bd280b75684ebd8ad7c31599e1028e5393afc39eda94540b994ea54319936 Sep 14 11:33:26.653893 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:26.653436 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 11:33:27.301156 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:27.301126 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df3763-6b8f57db8f-k45xv" event={"ID":"41e4b6e5-a721-45a7-8540-31f75937c7f6","Type":"ContainerStarted","Data":"2b4bd280b75684ebd8ad7c31599e1028e5393afc39eda94540b994ea54319936"} Sep 14 11:33:28.308175 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:28.308134 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df3763-6b8f57db8f-k45xv" event={"ID":"41e4b6e5-a721-45a7-8540-31f75937c7f6","Type":"ContainerStarted","Data":"11b8c170fca00622b5ee82113c6384486b293151614062bdd9b6cbe5419fdd23"} Sep 14 11:33:28.308596 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:28.308303 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df3763-6b8f57db8f-k45xv" Sep 14 11:33:28.329124 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:28.329074 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df3763-6b8f57db8f-k45xv" podStartSLOduration=1.459688358 podStartE2EDuration="2.329060115s" totalImagesPullingTime="869.371757ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:33:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:33:28.326711353 +0000 UTC m=+1952.946923896" watchObservedRunningTime="2026-09-14 11:33:28.329060115 +0000 UTC m=+1952.949272638" Sep 14 11:33:34.319304 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:34.319259 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df3763-6b8f57db8f-k45xv" Sep 14 11:33:55.907291 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:55.907071 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv"] Sep 14 11:33:55.911227 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:55.911206 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv" Sep 14 11:33:55.913564 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:55.913544 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-d0f937-serving-cert\"" Sep 14 11:33:55.921448 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:55.921426 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv"] Sep 14 11:33:55.969641 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:55.969614 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkh9l\" (UniqueName: \"kubernetes.io/projected/db53c5cf-7e15-4711-990a-f3c2ede15e68-kube-api-access-fkh9l\") pod \"maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv\" (UID: \"db53c5cf-7e15-4711-990a-f3c2ede15e68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv" Sep 14 11:33:55.969770 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:55.969659 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db53c5cf-7e15-4711-990a-f3c2ede15e68-metrics-certs\") pod \"maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv\" (UID: \"db53c5cf-7e15-4711-990a-f3c2ede15e68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv" Sep 14 11:33:55.969770 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:55.969689 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db53c5cf-7e15-4711-990a-f3c2ede15e68-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv\" (UID: \"db53c5cf-7e15-4711-990a-f3c2ede15e68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv" Sep 14 11:33:55.969770 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:55.969704 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db53c5cf-7e15-4711-990a-f3c2ede15e68-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv\" (UID: \"db53c5cf-7e15-4711-990a-f3c2ede15e68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv" Sep 14 11:33:56.070442 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:56.070407 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkh9l\" (UniqueName: \"kubernetes.io/projected/db53c5cf-7e15-4711-990a-f3c2ede15e68-kube-api-access-fkh9l\") pod \"maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv\" (UID: \"db53c5cf-7e15-4711-990a-f3c2ede15e68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv" Sep 14 11:33:56.070645 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:56.070472 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db53c5cf-7e15-4711-990a-f3c2ede15e68-metrics-certs\") pod \"maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv\" (UID: \"db53c5cf-7e15-4711-990a-f3c2ede15e68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv" Sep 14 11:33:56.070645 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:56.070520 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db53c5cf-7e15-4711-990a-f3c2ede15e68-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv\" (UID: \"db53c5cf-7e15-4711-990a-f3c2ede15e68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv" Sep 14 11:33:56.070645 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:56.070541 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db53c5cf-7e15-4711-990a-f3c2ede15e68-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv\" (UID: \"db53c5cf-7e15-4711-990a-f3c2ede15e68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv" Sep 14 11:33:56.070645 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:33:56.070628 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-d0f937-serving-cert: secret "maas-api-e2e-ipp-b-d0f937-serving-cert" not found Sep 14 11:33:56.070813 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:33:56.070691 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db53c5cf-7e15-4711-990a-f3c2ede15e68-maas-api-tls podName:db53c5cf-7e15-4711-990a-f3c2ede15e68 nodeName:}" failed. No retries permitted until 2026-09-14 11:33:56.570675063 +0000 UTC m=+1981.190887564 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/db53c5cf-7e15-4711-990a-f3c2ede15e68-maas-api-tls") pod "maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv" (UID: "db53c5cf-7e15-4711-990a-f3c2ede15e68") : secret "maas-api-e2e-ipp-b-d0f937-serving-cert" not found Sep 14 11:33:56.071288 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:56.071242 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db53c5cf-7e15-4711-990a-f3c2ede15e68-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv\" (UID: \"db53c5cf-7e15-4711-990a-f3c2ede15e68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv" Sep 14 11:33:56.073142 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:56.073122 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db53c5cf-7e15-4711-990a-f3c2ede15e68-metrics-certs\") pod \"maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv\" (UID: \"db53c5cf-7e15-4711-990a-f3c2ede15e68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv" Sep 14 11:33:56.078671 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:56.078652 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkh9l\" (UniqueName: \"kubernetes.io/projected/db53c5cf-7e15-4711-990a-f3c2ede15e68-kube-api-access-fkh9l\") pod \"maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv\" (UID: \"db53c5cf-7e15-4711-990a-f3c2ede15e68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv" Sep 14 11:33:56.537422 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:56.537384 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d0f937-c8fcf6594-jj6z9"] Sep 14 11:33:56.540668 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:56.540653 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d0f937-c8fcf6594-jj6z9" Sep 14 11:33:56.543424 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:56.543403 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d0f937-dockercfg-2dktj\"" Sep 14 11:33:56.543540 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:56.543420 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d0f937\"" Sep 14 11:33:56.551221 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:56.551199 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d0f937-c8fcf6594-jj6z9"] Sep 14 11:33:56.576057 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:56.576029 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw7xj\" (UniqueName: \"kubernetes.io/projected/59c9f939-33c0-4467-9f6d-ec139b8f1800-kube-api-access-gw7xj\") pod \"payload-pre-processing-e2e-ipp-b-d0f937-c8fcf6594-jj6z9\" (UID: \"59c9f939-33c0-4467-9f6d-ec139b8f1800\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d0f937-c8fcf6594-jj6z9" Sep 14 11:33:56.576177 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:56.576062 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59c9f939-33c0-4467-9f6d-ec139b8f1800-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d0f937-c8fcf6594-jj6z9\" (UID: \"59c9f939-33c0-4467-9f6d-ec139b8f1800\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d0f937-c8fcf6594-jj6z9" Sep 14 11:33:56.576227 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:56.576209 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db53c5cf-7e15-4711-990a-f3c2ede15e68-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv\" (UID: \"db53c5cf-7e15-4711-990a-f3c2ede15e68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv" Sep 14 11:33:56.578829 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:56.578802 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db53c5cf-7e15-4711-990a-f3c2ede15e68-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv\" (UID: \"db53c5cf-7e15-4711-990a-f3c2ede15e68\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv" Sep 14 11:33:56.677326 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:56.677286 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw7xj\" (UniqueName: \"kubernetes.io/projected/59c9f939-33c0-4467-9f6d-ec139b8f1800-kube-api-access-gw7xj\") pod \"payload-pre-processing-e2e-ipp-b-d0f937-c8fcf6594-jj6z9\" (UID: \"59c9f939-33c0-4467-9f6d-ec139b8f1800\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d0f937-c8fcf6594-jj6z9" Sep 14 11:33:56.677486 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:56.677333 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59c9f939-33c0-4467-9f6d-ec139b8f1800-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d0f937-c8fcf6594-jj6z9\" (UID: \"59c9f939-33c0-4467-9f6d-ec139b8f1800\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d0f937-c8fcf6594-jj6z9" Sep 14 11:33:56.678043 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:56.678023 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59c9f939-33c0-4467-9f6d-ec139b8f1800-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d0f937-c8fcf6594-jj6z9\" (UID: \"59c9f939-33c0-4467-9f6d-ec139b8f1800\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d0f937-c8fcf6594-jj6z9" Sep 14 11:33:56.685696 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:56.685673 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw7xj\" (UniqueName: \"kubernetes.io/projected/59c9f939-33c0-4467-9f6d-ec139b8f1800-kube-api-access-gw7xj\") pod \"payload-pre-processing-e2e-ipp-b-d0f937-c8fcf6594-jj6z9\" (UID: \"59c9f939-33c0-4467-9f6d-ec139b8f1800\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d0f937-c8fcf6594-jj6z9" Sep 14 11:33:56.824761 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:56.824681 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv" Sep 14 11:33:56.853966 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:56.853941 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d0f937-c8fcf6594-jj6z9" Sep 14 11:33:57.100531 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:57.100500 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv"] Sep 14 11:33:57.102668 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:33:57.102642 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb53c5cf_7e15_4711_990a_f3c2ede15e68.slice/crio-492f7315bc1852ec12afd6ecff0273931adf9deac4f1e91a54c02027355209b0 WatchSource:0}: Error finding container 492f7315bc1852ec12afd6ecff0273931adf9deac4f1e91a54c02027355209b0: Status 404 returned error can't find the container with id 492f7315bc1852ec12afd6ecff0273931adf9deac4f1e91a54c02027355209b0 Sep 14 11:33:57.142346 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:57.142318 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d0f937-c8fcf6594-jj6z9"] Sep 14 11:33:57.143725 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:33:57.143705 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59c9f939_33c0_4467_9f6d_ec139b8f1800.slice/crio-d2c587e8836447639014be0d13c07e043e545c57e3dd1b99cc3c2d8f7f1a8a7f WatchSource:0}: Error finding container d2c587e8836447639014be0d13c07e043e545c57e3dd1b99cc3c2d8f7f1a8a7f: Status 404 returned error can't find the container with id d2c587e8836447639014be0d13c07e043e545c57e3dd1b99cc3c2d8f7f1a8a7f Sep 14 11:33:57.468867 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:57.468824 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv" event={"ID":"db53c5cf-7e15-4711-990a-f3c2ede15e68","Type":"ContainerStarted","Data":"492f7315bc1852ec12afd6ecff0273931adf9deac4f1e91a54c02027355209b0"} Sep 14 11:33:57.470502 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:57.470478 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d0f937-c8fcf6594-jj6z9" event={"ID":"59c9f939-33c0-4467-9f6d-ec139b8f1800","Type":"ContainerStarted","Data":"2959d93b1cfff7b70a23c1517c8e477bea9969e84343bbd1e99dcb2f57c00c41"} Sep 14 11:33:57.470633 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:57.470537 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d0f937-c8fcf6594-jj6z9" event={"ID":"59c9f939-33c0-4467-9f6d-ec139b8f1800","Type":"ContainerStarted","Data":"d2c587e8836447639014be0d13c07e043e545c57e3dd1b99cc3c2d8f7f1a8a7f"} Sep 14 11:33:57.470633 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:57.470568 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d0f937-c8fcf6594-jj6z9" Sep 14 11:33:57.488360 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:57.488314 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d0f937-c8fcf6594-jj6z9" podStartSLOduration=1.488301163 podStartE2EDuration="1.488301163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:33:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:33:57.486743865 +0000 UTC m=+1982.106956406" watchObservedRunningTime="2026-09-14 11:33:57.488301163 +0000 UTC m=+1982.108513677" Sep 14 11:33:58.477900 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:58.477833 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv" event={"ID":"db53c5cf-7e15-4711-990a-f3c2ede15e68","Type":"ContainerStarted","Data":"5d1b8f8fa2f45f2623c5e1bef5d4348c7671c47abf074f1dccce1a2916effa25"} Sep 14 11:33:58.478400 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:58.478157 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv" Sep 14 11:33:58.497624 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:33:58.497583 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv" podStartSLOduration=2.983471005 podStartE2EDuration="3.497569012s" totalImagesPullingTime="514.098007ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:33:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:33:58.496190735 +0000 UTC m=+1983.116403275" watchObservedRunningTime="2026-09-14 11:33:58.497569012 +0000 UTC m=+1983.117781536" Sep 14 11:34:04.489765 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:34:04.489732 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv" Sep 14 11:34:08.480050 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:34:08.480016 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d0f937-c8fcf6594-jj6z9" Sep 14 11:34:23.472292 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:34:23.472238 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-mvnvh_d4eea439-1943-4f74-9067-d1bb67a438f1/payload-processing/0.log" Sep 14 11:34:49.344435 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:34:49.344409 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-mvnvh_d4eea439-1943-4f74-9067-d1bb67a438f1/payload-processing/0.log" Sep 14 11:34:57.082779 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:34:57.082740 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ec353-8f6487bd6-6xxld"] Sep 14 11:34:57.086641 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:34:57.086624 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ec353-8f6487bd6-6xxld" Sep 14 11:34:57.088960 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:34:57.088937 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-1ec353-dockercfg-h47r6\"" Sep 14 11:34:57.089397 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:34:57.089376 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-1ec353\"" Sep 14 11:34:57.096714 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:34:57.096689 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ec353-8f6487bd6-6xxld"] Sep 14 11:34:57.119478 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:34:57.119447 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1ec353-69d5f9c955-859vb"] Sep 14 11:34:57.123200 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:34:57.123184 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ec353-69d5f9c955-859vb" Sep 14 11:34:57.131960 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:34:57.131929 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1ec353-69d5f9c955-859vb"] Sep 14 11:34:57.134833 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:34:57.134805 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bccbbc83-79a4-4da6-839c-4c392b80ffc6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-1ec353-8f6487bd6-6xxld\" (UID: \"bccbbc83-79a4-4da6-839c-4c392b80ffc6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ec353-8f6487bd6-6xxld" Sep 14 11:34:57.134922 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:34:57.134885 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29czl\" (UniqueName: \"kubernetes.io/projected/bccbbc83-79a4-4da6-839c-4c392b80ffc6-kube-api-access-29czl\") pod \"payload-pre-processing-e2e-ipp-cleanup-1ec353-8f6487bd6-6xxld\" (UID: \"bccbbc83-79a4-4da6-839c-4c392b80ffc6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ec353-8f6487bd6-6xxld" Sep 14 11:34:57.236667 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:34:57.236629 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5655d6e-9a1d-479d-a4cf-65c2414fcad4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-1ec353-69d5f9c955-859vb\" (UID: \"a5655d6e-9a1d-479d-a4cf-65c2414fcad4\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ec353-69d5f9c955-859vb" Sep 14 11:34:57.236833 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:34:57.236682 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfchd\" (UniqueName: \"kubernetes.io/projected/a5655d6e-9a1d-479d-a4cf-65c2414fcad4-kube-api-access-jfchd\") pod \"payload-processing-e2e-ipp-cleanup-1ec353-69d5f9c955-859vb\" (UID: \"a5655d6e-9a1d-479d-a4cf-65c2414fcad4\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ec353-69d5f9c955-859vb" Sep 14 11:34:57.236833 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:34:57.236753 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bccbbc83-79a4-4da6-839c-4c392b80ffc6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-1ec353-8f6487bd6-6xxld\" (UID: \"bccbbc83-79a4-4da6-839c-4c392b80ffc6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ec353-8f6487bd6-6xxld" Sep 14 11:34:57.236833 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:34:57.236810 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29czl\" (UniqueName: \"kubernetes.io/projected/bccbbc83-79a4-4da6-839c-4c392b80ffc6-kube-api-access-29czl\") pod \"payload-pre-processing-e2e-ipp-cleanup-1ec353-8f6487bd6-6xxld\" (UID: \"bccbbc83-79a4-4da6-839c-4c392b80ffc6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ec353-8f6487bd6-6xxld" Sep 14 11:34:57.237501 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:34:57.237480 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bccbbc83-79a4-4da6-839c-4c392b80ffc6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-1ec353-8f6487bd6-6xxld\" (UID: \"bccbbc83-79a4-4da6-839c-4c392b80ffc6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ec353-8f6487bd6-6xxld" Sep 14 11:34:57.244686 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:34:57.244661 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29czl\" (UniqueName: \"kubernetes.io/projected/bccbbc83-79a4-4da6-839c-4c392b80ffc6-kube-api-access-29czl\") pod \"payload-pre-processing-e2e-ipp-cleanup-1ec353-8f6487bd6-6xxld\" (UID: \"bccbbc83-79a4-4da6-839c-4c392b80ffc6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ec353-8f6487bd6-6xxld" Sep 14 11:34:57.337637 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:34:57.337553 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5655d6e-9a1d-479d-a4cf-65c2414fcad4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-1ec353-69d5f9c955-859vb\" (UID: \"a5655d6e-9a1d-479d-a4cf-65c2414fcad4\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ec353-69d5f9c955-859vb" Sep 14 11:34:57.337637 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:34:57.337594 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfchd\" (UniqueName: \"kubernetes.io/projected/a5655d6e-9a1d-479d-a4cf-65c2414fcad4-kube-api-access-jfchd\") pod \"payload-processing-e2e-ipp-cleanup-1ec353-69d5f9c955-859vb\" (UID: \"a5655d6e-9a1d-479d-a4cf-65c2414fcad4\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ec353-69d5f9c955-859vb" Sep 14 11:34:57.338214 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:34:57.338194 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5655d6e-9a1d-479d-a4cf-65c2414fcad4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-1ec353-69d5f9c955-859vb\" (UID: \"a5655d6e-9a1d-479d-a4cf-65c2414fcad4\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ec353-69d5f9c955-859vb" Sep 14 11:34:57.345098 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:34:57.345067 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfchd\" (UniqueName: \"kubernetes.io/projected/a5655d6e-9a1d-479d-a4cf-65c2414fcad4-kube-api-access-jfchd\") pod \"payload-processing-e2e-ipp-cleanup-1ec353-69d5f9c955-859vb\" (UID: \"a5655d6e-9a1d-479d-a4cf-65c2414fcad4\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ec353-69d5f9c955-859vb" Sep 14 11:34:57.397787 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:34:57.397759 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ec353-8f6487bd6-6xxld" Sep 14 11:34:57.435656 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:34:57.435391 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ec353-69d5f9c955-859vb" Sep 14 11:34:57.672367 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:34:57.672342 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ec353-8f6487bd6-6xxld"] Sep 14 11:34:57.673968 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:34:57.673936 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbccbbc83_79a4_4da6_839c_4c392b80ffc6.slice/crio-fad3b658de29c41ebced288e707b9b1dcb8188181ab04f581097582292d16df3 WatchSource:0}: Error finding container fad3b658de29c41ebced288e707b9b1dcb8188181ab04f581097582292d16df3: Status 404 returned error can't find the container with id fad3b658de29c41ebced288e707b9b1dcb8188181ab04f581097582292d16df3 Sep 14 11:34:57.722676 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:34:57.722653 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1ec353-69d5f9c955-859vb"] Sep 14 11:34:57.725064 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:34:57.725031 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5655d6e_9a1d_479d_a4cf_65c2414fcad4.slice/crio-46c85796c73156b2224a5d5885c54031a2798f6c184c2bfc05f5236463ca4b76 WatchSource:0}: Error finding container 46c85796c73156b2224a5d5885c54031a2798f6c184c2bfc05f5236463ca4b76: Status 404 returned error can't find the container with id 46c85796c73156b2224a5d5885c54031a2798f6c184c2bfc05f5236463ca4b76 Sep 14 11:34:57.809008 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:34:57.808972 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ec353-69d5f9c955-859vb" event={"ID":"a5655d6e-9a1d-479d-a4cf-65c2414fcad4","Type":"ContainerStarted","Data":"07ce9d7b4ff972a0d6c5189767a2d2ea37de2dd66d5770dd04b156f974d6b095"} Sep 14 11:34:57.809156 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:34:57.809040 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ec353-69d5f9c955-859vb" event={"ID":"a5655d6e-9a1d-479d-a4cf-65c2414fcad4","Type":"ContainerStarted","Data":"46c85796c73156b2224a5d5885c54031a2798f6c184c2bfc05f5236463ca4b76"} Sep 14 11:34:57.809212 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:34:57.809192 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ec353-69d5f9c955-859vb" Sep 14 11:34:57.810816 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:34:57.810794 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ec353-8f6487bd6-6xxld" event={"ID":"bccbbc83-79a4-4da6-839c-4c392b80ffc6","Type":"ContainerStarted","Data":"1518bc3cdb5cb5903ba11ff560aeddd9878fd2c5806ed4a52191c83eaa054131"} Sep 14 11:34:57.810929 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:34:57.810833 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ec353-8f6487bd6-6xxld" Sep 14 11:34:57.810929 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:34:57.810849 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ec353-8f6487bd6-6xxld" event={"ID":"bccbbc83-79a4-4da6-839c-4c392b80ffc6","Type":"ContainerStarted","Data":"fad3b658de29c41ebced288e707b9b1dcb8188181ab04f581097582292d16df3"} Sep 14 11:34:57.827635 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:34:57.827587 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ec353-69d5f9c955-859vb" podStartSLOduration=0.827572435 podStartE2EDuration="827.572435ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:34:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:34:57.826205568 +0000 UTC m=+2042.446418108" watchObservedRunningTime="2026-09-14 11:34:57.827572435 +0000 UTC m=+2042.447784978" Sep 14 11:34:57.844859 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:34:57.844812 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ec353-8f6487bd6-6xxld" podStartSLOduration=0.844797389 podStartE2EDuration="844.797389ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:34:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:34:57.842552404 +0000 UTC m=+2042.462764952" watchObservedRunningTime="2026-09-14 11:34:57.844797389 +0000 UTC m=+2042.465009913" Sep 14 11:35:08.819645 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:35:08.819611 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ec353-8f6487bd6-6xxld" Sep 14 11:35:08.820118 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:35:08.819710 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ec353-69d5f9c955-859vb" Sep 14 11:35:34.972415 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:35:34.972380 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1ec353-69d5f9c955-859vb"] Sep 14 11:35:34.973180 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:35:34.973144 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ec353-69d5f9c955-859vb" podUID="a5655d6e-9a1d-479d-a4cf-65c2414fcad4" containerName="payload-processing" containerID="cri-o://07ce9d7b4ff972a0d6c5189767a2d2ea37de2dd66d5770dd04b156f974d6b095" gracePeriod=30 Sep 14 11:35:34.978457 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:35:34.978426 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ec353-8f6487bd6-6xxld"] Sep 14 11:35:34.978705 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:35:34.978663 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ec353-8f6487bd6-6xxld" podUID="bccbbc83-79a4-4da6-839c-4c392b80ffc6" containerName="payload-pre-processing" containerID="cri-o://1518bc3cdb5cb5903ba11ff560aeddd9878fd2c5806ed4a52191c83eaa054131" gracePeriod=30 Sep 14 11:35:35.236175 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:35:35.236104 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ec353-69d5f9c955-859vb" Sep 14 11:35:35.270653 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:35:35.270627 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ec353-8f6487bd6-6xxld" Sep 14 11:35:35.309179 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:35:35.309130 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bccbbc83-79a4-4da6-839c-4c392b80ffc6-plugins-config-volume\") pod \"bccbbc83-79a4-4da6-839c-4c392b80ffc6\" (UID: \"bccbbc83-79a4-4da6-839c-4c392b80ffc6\") " Sep 14 11:35:35.309389 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:35:35.309333 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29czl\" (UniqueName: \"kubernetes.io/projected/bccbbc83-79a4-4da6-839c-4c392b80ffc6-kube-api-access-29czl\") pod \"bccbbc83-79a4-4da6-839c-4c392b80ffc6\" (UID: \"bccbbc83-79a4-4da6-839c-4c392b80ffc6\") " Sep 14 11:35:35.309389 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:35:35.309377 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfchd\" (UniqueName: \"kubernetes.io/projected/a5655d6e-9a1d-479d-a4cf-65c2414fcad4-kube-api-access-jfchd\") pod \"a5655d6e-9a1d-479d-a4cf-65c2414fcad4\" (UID: \"a5655d6e-9a1d-479d-a4cf-65c2414fcad4\") " Sep 14 11:35:35.309512 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:35:35.309412 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5655d6e-9a1d-479d-a4cf-65c2414fcad4-plugins-config-volume\") pod \"a5655d6e-9a1d-479d-a4cf-65c2414fcad4\" (UID: \"a5655d6e-9a1d-479d-a4cf-65c2414fcad4\") " Sep 14 11:35:35.309589 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:35:35.309555 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bccbbc83-79a4-4da6-839c-4c392b80ffc6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bccbbc83-79a4-4da6-839c-4c392b80ffc6" (UID: "bccbbc83-79a4-4da6-839c-4c392b80ffc6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:35:35.309961 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:35:35.309936 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5655d6e-9a1d-479d-a4cf-65c2414fcad4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a5655d6e-9a1d-479d-a4cf-65c2414fcad4" (UID: "a5655d6e-9a1d-479d-a4cf-65c2414fcad4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:35:35.310059 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:35:35.309955 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bccbbc83-79a4-4da6-839c-4c392b80ffc6-plugins-config-volume\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:35:35.311951 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:35:35.311926 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bccbbc83-79a4-4da6-839c-4c392b80ffc6-kube-api-access-29czl" (OuterVolumeSpecName: "kube-api-access-29czl") pod "bccbbc83-79a4-4da6-839c-4c392b80ffc6" (UID: "bccbbc83-79a4-4da6-839c-4c392b80ffc6"). InnerVolumeSpecName "kube-api-access-29czl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:35:35.312310 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:35:35.312285 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5655d6e-9a1d-479d-a4cf-65c2414fcad4-kube-api-access-jfchd" (OuterVolumeSpecName: "kube-api-access-jfchd") pod "a5655d6e-9a1d-479d-a4cf-65c2414fcad4" (UID: "a5655d6e-9a1d-479d-a4cf-65c2414fcad4"). InnerVolumeSpecName "kube-api-access-jfchd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:35:35.410756 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:35:35.410723 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfchd\" (UniqueName: \"kubernetes.io/projected/a5655d6e-9a1d-479d-a4cf-65c2414fcad4-kube-api-access-jfchd\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:35:35.410756 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:35:35.410749 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5655d6e-9a1d-479d-a4cf-65c2414fcad4-plugins-config-volume\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:35:35.410756 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:35:35.410761 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29czl\" (UniqueName: \"kubernetes.io/projected/bccbbc83-79a4-4da6-839c-4c392b80ffc6-kube-api-access-29czl\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:35:36.029663 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:35:36.029630 2574 generic.go:356] "Generic (PLEG): container finished" podID="a5655d6e-9a1d-479d-a4cf-65c2414fcad4" containerID="07ce9d7b4ff972a0d6c5189767a2d2ea37de2dd66d5770dd04b156f974d6b095" exitCode=0 Sep 14 11:35:36.030087 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:35:36.029673 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ec353-69d5f9c955-859vb" Sep 14 11:35:36.030087 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:35:36.029714 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ec353-69d5f9c955-859vb" event={"ID":"a5655d6e-9a1d-479d-a4cf-65c2414fcad4","Type":"ContainerDied","Data":"07ce9d7b4ff972a0d6c5189767a2d2ea37de2dd66d5770dd04b156f974d6b095"} Sep 14 11:35:36.030087 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:35:36.029771 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-1ec353-69d5f9c955-859vb" event={"ID":"a5655d6e-9a1d-479d-a4cf-65c2414fcad4","Type":"ContainerDied","Data":"46c85796c73156b2224a5d5885c54031a2798f6c184c2bfc05f5236463ca4b76"} Sep 14 11:35:36.030087 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:35:36.029802 2574 scope.go:117] "RemoveContainer" containerID="07ce9d7b4ff972a0d6c5189767a2d2ea37de2dd66d5770dd04b156f974d6b095" Sep 14 11:35:36.031598 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:35:36.031552 2574 generic.go:356] "Generic (PLEG): container finished" podID="bccbbc83-79a4-4da6-839c-4c392b80ffc6" containerID="1518bc3cdb5cb5903ba11ff560aeddd9878fd2c5806ed4a52191c83eaa054131" exitCode=0 Sep 14 11:35:36.031672 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:35:36.031617 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ec353-8f6487bd6-6xxld" Sep 14 11:35:36.031672 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:35:36.031660 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ec353-8f6487bd6-6xxld" event={"ID":"bccbbc83-79a4-4da6-839c-4c392b80ffc6","Type":"ContainerDied","Data":"1518bc3cdb5cb5903ba11ff560aeddd9878fd2c5806ed4a52191c83eaa054131"} Sep 14 11:35:36.031788 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:35:36.031707 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ec353-8f6487bd6-6xxld" event={"ID":"bccbbc83-79a4-4da6-839c-4c392b80ffc6","Type":"ContainerDied","Data":"fad3b658de29c41ebced288e707b9b1dcb8188181ab04f581097582292d16df3"} Sep 14 11:35:36.039941 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:35:36.039909 2574 scope.go:117] "RemoveContainer" containerID="07ce9d7b4ff972a0d6c5189767a2d2ea37de2dd66d5770dd04b156f974d6b095" Sep 14 11:35:36.040154 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:35:36.040135 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07ce9d7b4ff972a0d6c5189767a2d2ea37de2dd66d5770dd04b156f974d6b095\": container with ID starting with 07ce9d7b4ff972a0d6c5189767a2d2ea37de2dd66d5770dd04b156f974d6b095 not found: ID does not exist" containerID="07ce9d7b4ff972a0d6c5189767a2d2ea37de2dd66d5770dd04b156f974d6b095" Sep 14 11:35:36.040238 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:35:36.040167 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07ce9d7b4ff972a0d6c5189767a2d2ea37de2dd66d5770dd04b156f974d6b095"} err="failed to get container status \"07ce9d7b4ff972a0d6c5189767a2d2ea37de2dd66d5770dd04b156f974d6b095\": rpc error: code = NotFound desc = could not find container \"07ce9d7b4ff972a0d6c5189767a2d2ea37de2dd66d5770dd04b156f974d6b095\": container with ID starting with 07ce9d7b4ff972a0d6c5189767a2d2ea37de2dd66d5770dd04b156f974d6b095 not found: ID does not exist" Sep 14 11:35:36.040238 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:35:36.040190 2574 scope.go:117] "RemoveContainer" containerID="1518bc3cdb5cb5903ba11ff560aeddd9878fd2c5806ed4a52191c83eaa054131" Sep 14 11:35:36.049577 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:35:36.049561 2574 scope.go:117] "RemoveContainer" containerID="1518bc3cdb5cb5903ba11ff560aeddd9878fd2c5806ed4a52191c83eaa054131" Sep 14 11:35:36.049831 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:35:36.049813 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1518bc3cdb5cb5903ba11ff560aeddd9878fd2c5806ed4a52191c83eaa054131\": container with ID starting with 1518bc3cdb5cb5903ba11ff560aeddd9878fd2c5806ed4a52191c83eaa054131 not found: ID does not exist" containerID="1518bc3cdb5cb5903ba11ff560aeddd9878fd2c5806ed4a52191c83eaa054131" Sep 14 11:35:36.049900 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:35:36.049836 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1518bc3cdb5cb5903ba11ff560aeddd9878fd2c5806ed4a52191c83eaa054131"} err="failed to get container status \"1518bc3cdb5cb5903ba11ff560aeddd9878fd2c5806ed4a52191c83eaa054131\": rpc error: code = NotFound desc = could not find container \"1518bc3cdb5cb5903ba11ff560aeddd9878fd2c5806ed4a52191c83eaa054131\": container with ID starting with 1518bc3cdb5cb5903ba11ff560aeddd9878fd2c5806ed4a52191c83eaa054131 not found: ID does not exist" Sep 14 11:35:36.050522 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:35:36.050504 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1ec353-69d5f9c955-859vb"] Sep 14 11:35:36.058245 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:35:36.058224 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-1ec353-69d5f9c955-859vb"] Sep 14 11:35:36.067137 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:35:36.067116 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ec353-8f6487bd6-6xxld"] Sep 14 11:35:36.071655 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:35:36.071634 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-1ec353-8f6487bd6-6xxld"] Sep 14 11:35:37.899200 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:35:37.899168 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5655d6e-9a1d-479d-a4cf-65c2414fcad4" path="/var/lib/kubelet/pods/a5655d6e-9a1d-479d-a4cf-65c2414fcad4/volumes" Sep 14 11:35:37.899602 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:35:37.899594 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bccbbc83-79a4-4da6-839c-4c392b80ffc6" path="/var/lib/kubelet/pods/bccbbc83-79a4-4da6-839c-4c392b80ffc6/volumes" Sep 14 11:36:01.026934 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:36:01.026896 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-df3763-6b8f57db8f-k45xv"] Sep 14 11:36:01.027491 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:36:01.027127 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df3763-6b8f57db8f-k45xv" podUID="41e4b6e5-a721-45a7-8540-31f75937c7f6" containerName="maas-api" containerID="cri-o://11b8c170fca00622b5ee82113c6384486b293151614062bdd9b6cbe5419fdd23" gracePeriod=30 Sep 14 11:36:01.162971 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:36:01.162936 2574 generic.go:356] "Generic (PLEG): container finished" podID="41e4b6e5-a721-45a7-8540-31f75937c7f6" containerID="11b8c170fca00622b5ee82113c6384486b293151614062bdd9b6cbe5419fdd23" exitCode=0 Sep 14 11:36:01.163128 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:36:01.163015 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df3763-6b8f57db8f-k45xv" event={"ID":"41e4b6e5-a721-45a7-8540-31f75937c7f6","Type":"ContainerDied","Data":"11b8c170fca00622b5ee82113c6384486b293151614062bdd9b6cbe5419fdd23"} Sep 14 11:36:01.294255 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:36:01.294227 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df3763-6b8f57db8f-k45xv" Sep 14 11:36:01.368246 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:36:01.368214 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41e4b6e5-a721-45a7-8540-31f75937c7f6-metrics-certs\") pod \"41e4b6e5-a721-45a7-8540-31f75937c7f6\" (UID: \"41e4b6e5-a721-45a7-8540-31f75937c7f6\") " Sep 14 11:36:01.368480 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:36:01.368290 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e4b6e5-a721-45a7-8540-31f75937c7f6-openshift-service-ca\") pod \"41e4b6e5-a721-45a7-8540-31f75937c7f6\" (UID: \"41e4b6e5-a721-45a7-8540-31f75937c7f6\") " Sep 14 11:36:01.368480 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:36:01.368359 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85m2d\" (UniqueName: \"kubernetes.io/projected/41e4b6e5-a721-45a7-8540-31f75937c7f6-kube-api-access-85m2d\") pod \"41e4b6e5-a721-45a7-8540-31f75937c7f6\" (UID: \"41e4b6e5-a721-45a7-8540-31f75937c7f6\") " Sep 14 11:36:01.368480 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:36:01.368457 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41e4b6e5-a721-45a7-8540-31f75937c7f6-maas-api-tls\") pod \"41e4b6e5-a721-45a7-8540-31f75937c7f6\" (UID: \"41e4b6e5-a721-45a7-8540-31f75937c7f6\") " Sep 14 11:36:01.368696 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:36:01.368669 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41e4b6e5-a721-45a7-8540-31f75937c7f6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "41e4b6e5-a721-45a7-8540-31f75937c7f6" (UID: "41e4b6e5-a721-45a7-8540-31f75937c7f6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:36:01.368795 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:36:01.368772 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e4b6e5-a721-45a7-8540-31f75937c7f6-openshift-service-ca\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:36:01.370885 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:36:01.370861 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41e4b6e5-a721-45a7-8540-31f75937c7f6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "41e4b6e5-a721-45a7-8540-31f75937c7f6" (UID: "41e4b6e5-a721-45a7-8540-31f75937c7f6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:36:01.371024 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:36:01.370960 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41e4b6e5-a721-45a7-8540-31f75937c7f6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "41e4b6e5-a721-45a7-8540-31f75937c7f6" (UID: "41e4b6e5-a721-45a7-8540-31f75937c7f6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:36:01.371024 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:36:01.371003 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41e4b6e5-a721-45a7-8540-31f75937c7f6-kube-api-access-85m2d" (OuterVolumeSpecName: "kube-api-access-85m2d") pod "41e4b6e5-a721-45a7-8540-31f75937c7f6" (UID: "41e4b6e5-a721-45a7-8540-31f75937c7f6"). InnerVolumeSpecName "kube-api-access-85m2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:36:01.469664 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:36:01.469626 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85m2d\" (UniqueName: \"kubernetes.io/projected/41e4b6e5-a721-45a7-8540-31f75937c7f6-kube-api-access-85m2d\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:36:01.469664 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:36:01.469661 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41e4b6e5-a721-45a7-8540-31f75937c7f6-maas-api-tls\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:36:01.469664 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:36:01.469671 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41e4b6e5-a721-45a7-8540-31f75937c7f6-metrics-certs\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:36:02.169585 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:36:02.169489 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df3763-6b8f57db8f-k45xv" event={"ID":"41e4b6e5-a721-45a7-8540-31f75937c7f6","Type":"ContainerDied","Data":"2b4bd280b75684ebd8ad7c31599e1028e5393afc39eda94540b994ea54319936"} Sep 14 11:36:02.169585 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:36:02.169532 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-df3763-6b8f57db8f-k45xv" Sep 14 11:36:02.169585 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:36:02.169561 2574 scope.go:117] "RemoveContainer" containerID="11b8c170fca00622b5ee82113c6384486b293151614062bdd9b6cbe5419fdd23" Sep 14 11:36:02.185665 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:36:02.185635 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-df3763-6b8f57db8f-k45xv"] Sep 14 11:36:02.189658 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:36:02.189631 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-df3763-6b8f57db8f-k45xv"] Sep 14 11:36:03.899168 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:36:03.899130 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41e4b6e5-a721-45a7-8540-31f75937c7f6" path="/var/lib/kubelet/pods/41e4b6e5-a721-45a7-8540-31f75937c7f6/volumes" Sep 14 11:36:56.223221 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:36:56.223138 2574 scope.go:117] "RemoveContainer" containerID="e45f52ddcf71532aab8e96b1feb7e6dd7681d4a3ed8648fdf5276ec4712ce7ea" Sep 14 11:36:56.232278 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:36:56.232246 2574 scope.go:117] "RemoveContainer" containerID="4b057774c9b18d1548d86530f513fa1127945ec4a4abb9784aa6250abb495394" Sep 14 11:37:00.562036 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:00.561998 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv"] Sep 14 11:37:00.562564 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:00.562286 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv" podUID="db53c5cf-7e15-4711-990a-f3c2ede15e68" containerName="maas-api" containerID="cri-o://5d1b8f8fa2f45f2623c5e1bef5d4348c7671c47abf074f1dccce1a2916effa25" gracePeriod=30 Sep 14 11:37:00.628975 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:00.628944 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d0f937-c8fcf6594-jj6z9"] Sep 14 11:37:00.629188 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:00.629160 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d0f937-c8fcf6594-jj6z9" podUID="59c9f939-33c0-4467-9f6d-ec139b8f1800" containerName="payload-pre-processing" containerID="cri-o://2959d93b1cfff7b70a23c1517c8e477bea9969e84343bbd1e99dcb2f57c00c41" gracePeriod=30 Sep 14 11:37:00.896790 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:00.896762 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv" Sep 14 11:37:00.922232 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:00.922205 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d0f937-c8fcf6594-jj6z9" Sep 14 11:37:01.057399 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:01.057359 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db53c5cf-7e15-4711-990a-f3c2ede15e68-metrics-certs\") pod \"db53c5cf-7e15-4711-990a-f3c2ede15e68\" (UID: \"db53c5cf-7e15-4711-990a-f3c2ede15e68\") " Sep 14 11:37:01.057399 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:01.057403 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db53c5cf-7e15-4711-990a-f3c2ede15e68-maas-api-tls\") pod \"db53c5cf-7e15-4711-990a-f3c2ede15e68\" (UID: \"db53c5cf-7e15-4711-990a-f3c2ede15e68\") " Sep 14 11:37:01.057628 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:01.057435 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gw7xj\" (UniqueName: \"kubernetes.io/projected/59c9f939-33c0-4467-9f6d-ec139b8f1800-kube-api-access-gw7xj\") pod \"59c9f939-33c0-4467-9f6d-ec139b8f1800\" (UID: \"59c9f939-33c0-4467-9f6d-ec139b8f1800\") " Sep 14 11:37:01.057628 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:01.057463 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkh9l\" (UniqueName: \"kubernetes.io/projected/db53c5cf-7e15-4711-990a-f3c2ede15e68-kube-api-access-fkh9l\") pod \"db53c5cf-7e15-4711-990a-f3c2ede15e68\" (UID: \"db53c5cf-7e15-4711-990a-f3c2ede15e68\") " Sep 14 11:37:01.057628 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:01.057500 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db53c5cf-7e15-4711-990a-f3c2ede15e68-openshift-service-ca\") pod \"db53c5cf-7e15-4711-990a-f3c2ede15e68\" (UID: \"db53c5cf-7e15-4711-990a-f3c2ede15e68\") " Sep 14 11:37:01.057628 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:01.057548 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59c9f939-33c0-4467-9f6d-ec139b8f1800-plugins-config-volume\") pod \"59c9f939-33c0-4467-9f6d-ec139b8f1800\" (UID: \"59c9f939-33c0-4467-9f6d-ec139b8f1800\") " Sep 14 11:37:01.057952 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:01.057914 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db53c5cf-7e15-4711-990a-f3c2ede15e68-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "db53c5cf-7e15-4711-990a-f3c2ede15e68" (UID: "db53c5cf-7e15-4711-990a-f3c2ede15e68"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:37:01.058071 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:01.057965 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59c9f939-33c0-4467-9f6d-ec139b8f1800-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "59c9f939-33c0-4467-9f6d-ec139b8f1800" (UID: "59c9f939-33c0-4467-9f6d-ec139b8f1800"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:37:01.059744 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:01.059720 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db53c5cf-7e15-4711-990a-f3c2ede15e68-kube-api-access-fkh9l" (OuterVolumeSpecName: "kube-api-access-fkh9l") pod "db53c5cf-7e15-4711-990a-f3c2ede15e68" (UID: "db53c5cf-7e15-4711-990a-f3c2ede15e68"). InnerVolumeSpecName "kube-api-access-fkh9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:37:01.059744 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:01.059730 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db53c5cf-7e15-4711-990a-f3c2ede15e68-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "db53c5cf-7e15-4711-990a-f3c2ede15e68" (UID: "db53c5cf-7e15-4711-990a-f3c2ede15e68"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:37:01.060179 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:01.060158 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db53c5cf-7e15-4711-990a-f3c2ede15e68-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "db53c5cf-7e15-4711-990a-f3c2ede15e68" (UID: "db53c5cf-7e15-4711-990a-f3c2ede15e68"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:37:01.060179 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:01.060170 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59c9f939-33c0-4467-9f6d-ec139b8f1800-kube-api-access-gw7xj" (OuterVolumeSpecName: "kube-api-access-gw7xj") pod "59c9f939-33c0-4467-9f6d-ec139b8f1800" (UID: "59c9f939-33c0-4467-9f6d-ec139b8f1800"). InnerVolumeSpecName "kube-api-access-gw7xj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:37:01.158801 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:01.158718 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db53c5cf-7e15-4711-990a-f3c2ede15e68-metrics-certs\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:37:01.158801 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:01.158745 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db53c5cf-7e15-4711-990a-f3c2ede15e68-maas-api-tls\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:37:01.158801 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:01.158754 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw7xj\" (UniqueName: \"kubernetes.io/projected/59c9f939-33c0-4467-9f6d-ec139b8f1800-kube-api-access-gw7xj\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:37:01.158801 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:01.158765 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkh9l\" (UniqueName: \"kubernetes.io/projected/db53c5cf-7e15-4711-990a-f3c2ede15e68-kube-api-access-fkh9l\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:37:01.158801 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:01.158773 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db53c5cf-7e15-4711-990a-f3c2ede15e68-openshift-service-ca\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:37:01.158801 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:01.158783 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59c9f939-33c0-4467-9f6d-ec139b8f1800-plugins-config-volume\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:37:01.500950 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:01.500916 2574 generic.go:356] "Generic (PLEG): container finished" podID="db53c5cf-7e15-4711-990a-f3c2ede15e68" containerID="5d1b8f8fa2f45f2623c5e1bef5d4348c7671c47abf074f1dccce1a2916effa25" exitCode=0 Sep 14 11:37:01.501176 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:01.500980 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv" Sep 14 11:37:01.501176 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:01.501020 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv" event={"ID":"db53c5cf-7e15-4711-990a-f3c2ede15e68","Type":"ContainerDied","Data":"5d1b8f8fa2f45f2623c5e1bef5d4348c7671c47abf074f1dccce1a2916effa25"} Sep 14 11:37:01.501176 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:01.501087 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv" event={"ID":"db53c5cf-7e15-4711-990a-f3c2ede15e68","Type":"ContainerDied","Data":"492f7315bc1852ec12afd6ecff0273931adf9deac4f1e91a54c02027355209b0"} Sep 14 11:37:01.501176 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:01.501121 2574 scope.go:117] "RemoveContainer" containerID="5d1b8f8fa2f45f2623c5e1bef5d4348c7671c47abf074f1dccce1a2916effa25" Sep 14 11:37:01.510157 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:01.510129 2574 generic.go:356] "Generic (PLEG): container finished" podID="59c9f939-33c0-4467-9f6d-ec139b8f1800" containerID="2959d93b1cfff7b70a23c1517c8e477bea9969e84343bbd1e99dcb2f57c00c41" exitCode=0 Sep 14 11:37:01.510311 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:01.510241 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d0f937-c8fcf6594-jj6z9" event={"ID":"59c9f939-33c0-4467-9f6d-ec139b8f1800","Type":"ContainerDied","Data":"2959d93b1cfff7b70a23c1517c8e477bea9969e84343bbd1e99dcb2f57c00c41"} Sep 14 11:37:01.510383 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:01.510340 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d0f937-c8fcf6594-jj6z9" event={"ID":"59c9f939-33c0-4467-9f6d-ec139b8f1800","Type":"ContainerDied","Data":"d2c587e8836447639014be0d13c07e043e545c57e3dd1b99cc3c2d8f7f1a8a7f"} Sep 14 11:37:01.510511 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:01.510497 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d0f937-c8fcf6594-jj6z9" Sep 14 11:37:01.522358 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:01.522340 2574 scope.go:117] "RemoveContainer" containerID="5d1b8f8fa2f45f2623c5e1bef5d4348c7671c47abf074f1dccce1a2916effa25" Sep 14 11:37:01.522697 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:37:01.522677 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d1b8f8fa2f45f2623c5e1bef5d4348c7671c47abf074f1dccce1a2916effa25\": container with ID starting with 5d1b8f8fa2f45f2623c5e1bef5d4348c7671c47abf074f1dccce1a2916effa25 not found: ID does not exist" containerID="5d1b8f8fa2f45f2623c5e1bef5d4348c7671c47abf074f1dccce1a2916effa25" Sep 14 11:37:01.522752 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:01.522706 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d1b8f8fa2f45f2623c5e1bef5d4348c7671c47abf074f1dccce1a2916effa25"} err="failed to get container status \"5d1b8f8fa2f45f2623c5e1bef5d4348c7671c47abf074f1dccce1a2916effa25\": rpc error: code = NotFound desc = could not find container \"5d1b8f8fa2f45f2623c5e1bef5d4348c7671c47abf074f1dccce1a2916effa25\": container with ID starting with 5d1b8f8fa2f45f2623c5e1bef5d4348c7671c47abf074f1dccce1a2916effa25 not found: ID does not exist" Sep 14 11:37:01.522752 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:01.522724 2574 scope.go:117] "RemoveContainer" containerID="2959d93b1cfff7b70a23c1517c8e477bea9969e84343bbd1e99dcb2f57c00c41" Sep 14 11:37:01.529846 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:01.529815 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv"] Sep 14 11:37:01.532465 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:01.532439 2574 scope.go:117] "RemoveContainer" containerID="2959d93b1cfff7b70a23c1517c8e477bea9969e84343bbd1e99dcb2f57c00c41" Sep 14 11:37:01.532782 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:37:01.532763 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2959d93b1cfff7b70a23c1517c8e477bea9969e84343bbd1e99dcb2f57c00c41\": container with ID starting with 2959d93b1cfff7b70a23c1517c8e477bea9969e84343bbd1e99dcb2f57c00c41 not found: ID does not exist" containerID="2959d93b1cfff7b70a23c1517c8e477bea9969e84343bbd1e99dcb2f57c00c41" Sep 14 11:37:01.532847 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:01.532790 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2959d93b1cfff7b70a23c1517c8e477bea9969e84343bbd1e99dcb2f57c00c41"} err="failed to get container status \"2959d93b1cfff7b70a23c1517c8e477bea9969e84343bbd1e99dcb2f57c00c41\": rpc error: code = NotFound desc = could not find container \"2959d93b1cfff7b70a23c1517c8e477bea9969e84343bbd1e99dcb2f57c00c41\": container with ID starting with 2959d93b1cfff7b70a23c1517c8e477bea9969e84343bbd1e99dcb2f57c00c41 not found: ID does not exist" Sep 14 11:37:01.533426 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:01.533403 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d0f937-fbb47dcc6-2tdmv"] Sep 14 11:37:01.542639 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:01.542616 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d0f937-c8fcf6594-jj6z9"] Sep 14 11:37:01.546034 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:01.546015 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d0f937-c8fcf6594-jj6z9"] Sep 14 11:37:01.900452 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:01.900376 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59c9f939-33c0-4467-9f6d-ec139b8f1800" path="/var/lib/kubelet/pods/59c9f939-33c0-4467-9f6d-ec139b8f1800/volumes" Sep 14 11:37:01.900798 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:01.900773 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db53c5cf-7e15-4711-990a-f3c2ede15e68" path="/var/lib/kubelet/pods/db53c5cf-7e15-4711-990a-f3c2ede15e68/volumes" Sep 14 11:37:18.156772 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:18.156738 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-55c025-66d8d54859-x2fb8"] Sep 14 11:37:18.157523 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:18.156968 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-55c025-66d8d54859-x2fb8" podUID="47f89cf4-0168-4cf1-a4d2-b001d69e966e" containerName="payload-processing" containerID="cri-o://cc809165b66981ccd6fd901ea99ef6bac9d1e1bb030c2dcf1b1da99e95eb9de9" gracePeriod=30 Sep 14 11:37:18.425154 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:18.425123 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-55c025-66d8d54859-x2fb8" Sep 14 11:37:18.531107 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:18.531066 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47f89cf4-0168-4cf1-a4d2-b001d69e966e-plugins-config-volume\") pod \"47f89cf4-0168-4cf1-a4d2-b001d69e966e\" (UID: \"47f89cf4-0168-4cf1-a4d2-b001d69e966e\") " Sep 14 11:37:18.531255 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:18.531122 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hx2tk\" (UniqueName: \"kubernetes.io/projected/47f89cf4-0168-4cf1-a4d2-b001d69e966e-kube-api-access-hx2tk\") pod \"47f89cf4-0168-4cf1-a4d2-b001d69e966e\" (UID: \"47f89cf4-0168-4cf1-a4d2-b001d69e966e\") " Sep 14 11:37:18.531569 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:18.531541 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47f89cf4-0168-4cf1-a4d2-b001d69e966e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "47f89cf4-0168-4cf1-a4d2-b001d69e966e" (UID: "47f89cf4-0168-4cf1-a4d2-b001d69e966e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:37:18.533494 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:18.533471 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47f89cf4-0168-4cf1-a4d2-b001d69e966e-kube-api-access-hx2tk" (OuterVolumeSpecName: "kube-api-access-hx2tk") pod "47f89cf4-0168-4cf1-a4d2-b001d69e966e" (UID: "47f89cf4-0168-4cf1-a4d2-b001d69e966e"). InnerVolumeSpecName "kube-api-access-hx2tk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:37:18.603478 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:18.603446 2574 generic.go:356] "Generic (PLEG): container finished" podID="47f89cf4-0168-4cf1-a4d2-b001d69e966e" containerID="cc809165b66981ccd6fd901ea99ef6bac9d1e1bb030c2dcf1b1da99e95eb9de9" exitCode=0 Sep 14 11:37:18.603655 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:18.603507 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-55c025-66d8d54859-x2fb8" Sep 14 11:37:18.603655 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:18.603547 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-55c025-66d8d54859-x2fb8" event={"ID":"47f89cf4-0168-4cf1-a4d2-b001d69e966e","Type":"ContainerDied","Data":"cc809165b66981ccd6fd901ea99ef6bac9d1e1bb030c2dcf1b1da99e95eb9de9"} Sep 14 11:37:18.603655 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:18.603607 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-55c025-66d8d54859-x2fb8" event={"ID":"47f89cf4-0168-4cf1-a4d2-b001d69e966e","Type":"ContainerDied","Data":"3ea8dbe7a8dbf059e4f43c2fc3424801a5035e02d0e444d5fe68194739535e24"} Sep 14 11:37:18.603655 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:18.603631 2574 scope.go:117] "RemoveContainer" containerID="cc809165b66981ccd6fd901ea99ef6bac9d1e1bb030c2dcf1b1da99e95eb9de9" Sep 14 11:37:18.613012 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:18.612985 2574 scope.go:117] "RemoveContainer" containerID="cc809165b66981ccd6fd901ea99ef6bac9d1e1bb030c2dcf1b1da99e95eb9de9" Sep 14 11:37:18.613474 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:37:18.613449 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc809165b66981ccd6fd901ea99ef6bac9d1e1bb030c2dcf1b1da99e95eb9de9\": container with ID starting with cc809165b66981ccd6fd901ea99ef6bac9d1e1bb030c2dcf1b1da99e95eb9de9 not found: ID does not exist" containerID="cc809165b66981ccd6fd901ea99ef6bac9d1e1bb030c2dcf1b1da99e95eb9de9" Sep 14 11:37:18.613576 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:18.613484 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc809165b66981ccd6fd901ea99ef6bac9d1e1bb030c2dcf1b1da99e95eb9de9"} err="failed to get container status \"cc809165b66981ccd6fd901ea99ef6bac9d1e1bb030c2dcf1b1da99e95eb9de9\": rpc error: code = NotFound desc = could not find container \"cc809165b66981ccd6fd901ea99ef6bac9d1e1bb030c2dcf1b1da99e95eb9de9\": container with ID starting with cc809165b66981ccd6fd901ea99ef6bac9d1e1bb030c2dcf1b1da99e95eb9de9 not found: ID does not exist" Sep 14 11:37:18.628218 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:18.628194 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-55c025-66d8d54859-x2fb8"] Sep 14 11:37:18.632072 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:18.632047 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-55c025-66d8d54859-x2fb8"] Sep 14 11:37:18.632506 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:18.632485 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47f89cf4-0168-4cf1-a4d2-b001d69e966e-plugins-config-volume\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:37:18.632580 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:18.632510 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx2tk\" (UniqueName: \"kubernetes.io/projected/47f89cf4-0168-4cf1-a4d2-b001d69e966e-kube-api-access-hx2tk\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:37:19.899348 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:19.899306 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47f89cf4-0168-4cf1-a4d2-b001d69e966e" path="/var/lib/kubelet/pods/47f89cf4-0168-4cf1-a4d2-b001d69e966e/volumes" Sep 14 11:37:36.017681 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:36.017642 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3f02a5-5449d5f5f9-5l7hw"] Sep 14 11:37:36.018139 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:36.017920 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3f02a5-5449d5f5f9-5l7hw" podUID="ebd4bc7e-3e14-4fd1-bc25-d16c664b9af4" containerName="payload-processing" containerID="cri-o://51e03f2f0a09a85be876b1eee80bef19dd3100e9e4d1723e0aee56f10fd35615" gracePeriod=30 Sep 14 11:37:36.027435 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:36.027403 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3f02a5-857f4fc78-7pl5f"] Sep 14 11:37:36.027996 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:36.027651 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3f02a5-857f4fc78-7pl5f" podUID="57685ac4-6a88-4780-8fcc-40d91b2aa5fd" containerName="payload-pre-processing" containerID="cri-o://5e7a6b71a1ca1f16e7e23061288c61dff2b50a24ba89e129ed51d7de81186103" gracePeriod=30 Sep 14 11:37:36.290494 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:36.290428 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3f02a5-5449d5f5f9-5l7hw" Sep 14 11:37:36.332611 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:36.332572 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3f02a5-857f4fc78-7pl5f" Sep 14 11:37:36.410188 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:36.410147 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54rbh\" (UniqueName: \"kubernetes.io/projected/ebd4bc7e-3e14-4fd1-bc25-d16c664b9af4-kube-api-access-54rbh\") pod \"ebd4bc7e-3e14-4fd1-bc25-d16c664b9af4\" (UID: \"ebd4bc7e-3e14-4fd1-bc25-d16c664b9af4\") " Sep 14 11:37:36.410387 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:36.410202 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57685ac4-6a88-4780-8fcc-40d91b2aa5fd-plugins-config-volume\") pod \"57685ac4-6a88-4780-8fcc-40d91b2aa5fd\" (UID: \"57685ac4-6a88-4780-8fcc-40d91b2aa5fd\") " Sep 14 11:37:36.410387 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:36.410253 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2lspn\" (UniqueName: \"kubernetes.io/projected/57685ac4-6a88-4780-8fcc-40d91b2aa5fd-kube-api-access-2lspn\") pod \"57685ac4-6a88-4780-8fcc-40d91b2aa5fd\" (UID: \"57685ac4-6a88-4780-8fcc-40d91b2aa5fd\") " Sep 14 11:37:36.410515 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:36.410414 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebd4bc7e-3e14-4fd1-bc25-d16c664b9af4-plugins-config-volume\") pod \"ebd4bc7e-3e14-4fd1-bc25-d16c664b9af4\" (UID: \"ebd4bc7e-3e14-4fd1-bc25-d16c664b9af4\") " Sep 14 11:37:36.410709 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:36.410676 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57685ac4-6a88-4780-8fcc-40d91b2aa5fd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "57685ac4-6a88-4780-8fcc-40d91b2aa5fd" (UID: "57685ac4-6a88-4780-8fcc-40d91b2aa5fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:37:36.410838 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:36.410773 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebd4bc7e-3e14-4fd1-bc25-d16c664b9af4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ebd4bc7e-3e14-4fd1-bc25-d16c664b9af4" (UID: "ebd4bc7e-3e14-4fd1-bc25-d16c664b9af4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:37:36.410901 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:36.410840 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57685ac4-6a88-4780-8fcc-40d91b2aa5fd-plugins-config-volume\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:37:36.412457 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:36.412437 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebd4bc7e-3e14-4fd1-bc25-d16c664b9af4-kube-api-access-54rbh" (OuterVolumeSpecName: "kube-api-access-54rbh") pod "ebd4bc7e-3e14-4fd1-bc25-d16c664b9af4" (UID: "ebd4bc7e-3e14-4fd1-bc25-d16c664b9af4"). InnerVolumeSpecName "kube-api-access-54rbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:37:36.412851 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:36.412837 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57685ac4-6a88-4780-8fcc-40d91b2aa5fd-kube-api-access-2lspn" (OuterVolumeSpecName: "kube-api-access-2lspn") pod "57685ac4-6a88-4780-8fcc-40d91b2aa5fd" (UID: "57685ac4-6a88-4780-8fcc-40d91b2aa5fd"). InnerVolumeSpecName "kube-api-access-2lspn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:37:36.511496 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:36.511459 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54rbh\" (UniqueName: \"kubernetes.io/projected/ebd4bc7e-3e14-4fd1-bc25-d16c664b9af4-kube-api-access-54rbh\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:37:36.511496 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:36.511487 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lspn\" (UniqueName: \"kubernetes.io/projected/57685ac4-6a88-4780-8fcc-40d91b2aa5fd-kube-api-access-2lspn\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:37:36.511496 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:36.511503 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebd4bc7e-3e14-4fd1-bc25-d16c664b9af4-plugins-config-volume\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:37:36.699408 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:36.699369 2574 generic.go:356] "Generic (PLEG): container finished" podID="ebd4bc7e-3e14-4fd1-bc25-d16c664b9af4" containerID="51e03f2f0a09a85be876b1eee80bef19dd3100e9e4d1723e0aee56f10fd35615" exitCode=0 Sep 14 11:37:36.699408 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:36.699414 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3f02a5-5449d5f5f9-5l7hw" Sep 14 11:37:36.699687 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:36.699457 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3f02a5-5449d5f5f9-5l7hw" event={"ID":"ebd4bc7e-3e14-4fd1-bc25-d16c664b9af4","Type":"ContainerDied","Data":"51e03f2f0a09a85be876b1eee80bef19dd3100e9e4d1723e0aee56f10fd35615"} Sep 14 11:37:36.699687 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:36.699518 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-3f02a5-5449d5f5f9-5l7hw" event={"ID":"ebd4bc7e-3e14-4fd1-bc25-d16c664b9af4","Type":"ContainerDied","Data":"d67dbb8788d250e76cc21413ac04fca0619af3ec292778fdbced24106ab4f7f2"} Sep 14 11:37:36.699687 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:36.699549 2574 scope.go:117] "RemoveContainer" containerID="51e03f2f0a09a85be876b1eee80bef19dd3100e9e4d1723e0aee56f10fd35615" Sep 14 11:37:36.700948 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:36.700928 2574 generic.go:356] "Generic (PLEG): container finished" podID="57685ac4-6a88-4780-8fcc-40d91b2aa5fd" containerID="5e7a6b71a1ca1f16e7e23061288c61dff2b50a24ba89e129ed51d7de81186103" exitCode=0 Sep 14 11:37:36.701049 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:36.700953 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3f02a5-857f4fc78-7pl5f" event={"ID":"57685ac4-6a88-4780-8fcc-40d91b2aa5fd","Type":"ContainerDied","Data":"5e7a6b71a1ca1f16e7e23061288c61dff2b50a24ba89e129ed51d7de81186103"} Sep 14 11:37:36.701049 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:36.700979 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3f02a5-857f4fc78-7pl5f" Sep 14 11:37:36.701049 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:36.701016 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3f02a5-857f4fc78-7pl5f" event={"ID":"57685ac4-6a88-4780-8fcc-40d91b2aa5fd","Type":"ContainerDied","Data":"f177ab2c3c123aa093db64762fb6bfe5d6c5264d5a1398e737dfe0dd81cd961c"} Sep 14 11:37:36.710761 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:36.710729 2574 scope.go:117] "RemoveContainer" containerID="51e03f2f0a09a85be876b1eee80bef19dd3100e9e4d1723e0aee56f10fd35615" Sep 14 11:37:36.711069 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:37:36.711049 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51e03f2f0a09a85be876b1eee80bef19dd3100e9e4d1723e0aee56f10fd35615\": container with ID starting with 51e03f2f0a09a85be876b1eee80bef19dd3100e9e4d1723e0aee56f10fd35615 not found: ID does not exist" containerID="51e03f2f0a09a85be876b1eee80bef19dd3100e9e4d1723e0aee56f10fd35615" Sep 14 11:37:36.711212 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:36.711080 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51e03f2f0a09a85be876b1eee80bef19dd3100e9e4d1723e0aee56f10fd35615"} err="failed to get container status \"51e03f2f0a09a85be876b1eee80bef19dd3100e9e4d1723e0aee56f10fd35615\": rpc error: code = NotFound desc = could not find container \"51e03f2f0a09a85be876b1eee80bef19dd3100e9e4d1723e0aee56f10fd35615\": container with ID starting with 51e03f2f0a09a85be876b1eee80bef19dd3100e9e4d1723e0aee56f10fd35615 not found: ID does not exist" Sep 14 11:37:36.711212 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:36.711102 2574 scope.go:117] "RemoveContainer" containerID="5e7a6b71a1ca1f16e7e23061288c61dff2b50a24ba89e129ed51d7de81186103" Sep 14 11:37:36.719200 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:36.719178 2574 scope.go:117] "RemoveContainer" containerID="5e7a6b71a1ca1f16e7e23061288c61dff2b50a24ba89e129ed51d7de81186103" Sep 14 11:37:36.719646 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:37:36.719621 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e7a6b71a1ca1f16e7e23061288c61dff2b50a24ba89e129ed51d7de81186103\": container with ID starting with 5e7a6b71a1ca1f16e7e23061288c61dff2b50a24ba89e129ed51d7de81186103 not found: ID does not exist" containerID="5e7a6b71a1ca1f16e7e23061288c61dff2b50a24ba89e129ed51d7de81186103" Sep 14 11:37:36.719799 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:36.719653 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e7a6b71a1ca1f16e7e23061288c61dff2b50a24ba89e129ed51d7de81186103"} err="failed to get container status \"5e7a6b71a1ca1f16e7e23061288c61dff2b50a24ba89e129ed51d7de81186103\": rpc error: code = NotFound desc = could not find container \"5e7a6b71a1ca1f16e7e23061288c61dff2b50a24ba89e129ed51d7de81186103\": container with ID starting with 5e7a6b71a1ca1f16e7e23061288c61dff2b50a24ba89e129ed51d7de81186103 not found: ID does not exist" Sep 14 11:37:36.722105 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:36.722085 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3f02a5-5449d5f5f9-5l7hw"] Sep 14 11:37:36.725878 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:36.725859 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-3f02a5-5449d5f5f9-5l7hw"] Sep 14 11:37:36.738378 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:36.738303 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3f02a5-857f4fc78-7pl5f"] Sep 14 11:37:36.740235 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:36.740216 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-3f02a5-857f4fc78-7pl5f"] Sep 14 11:37:37.899442 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:37.899410 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57685ac4-6a88-4780-8fcc-40d91b2aa5fd" path="/var/lib/kubelet/pods/57685ac4-6a88-4780-8fcc-40d91b2aa5fd/volumes" Sep 14 11:37:37.899830 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:37.899773 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebd4bc7e-3e14-4fd1-bc25-d16c664b9af4" path="/var/lib/kubelet/pods/ebd4bc7e-3e14-4fd1-bc25-d16c664b9af4/volumes" Sep 14 11:37:45.739332 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:45.739288 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-796b4cc5fc-c4gnz"] Sep 14 11:37:45.739781 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:45.739496 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-796b4cc5fc-c4gnz" podUID="4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc" containerName="manager" containerID="cri-o://91b2bbe1a0d85e1ddb20b2dc3b9f6f5ec9077b5848ae581ee2f024e8098f9dcb" gracePeriod=30 Sep 14 11:37:45.995412 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:45.995351 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-796b4cc5fc-c4gnz" Sep 14 11:37:46.109060 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:46.109028 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc-webhook-cert\") pod \"4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc\" (UID: \"4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc\") " Sep 14 11:37:46.109217 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:46.109074 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc-metrics-certs\") pod \"4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc\" (UID: \"4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc\") " Sep 14 11:37:46.109217 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:46.109095 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6v2mn\" (UniqueName: \"kubernetes.io/projected/4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc-kube-api-access-6v2mn\") pod \"4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc\" (UID: \"4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc\") " Sep 14 11:37:46.111357 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:46.111320 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc" (UID: "4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:37:46.111496 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:46.111379 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc" (UID: "4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:37:46.111496 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:46.111394 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc-kube-api-access-6v2mn" (OuterVolumeSpecName: "kube-api-access-6v2mn") pod "4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc" (UID: "4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc"). InnerVolumeSpecName "kube-api-access-6v2mn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:37:46.210657 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:46.210624 2574 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc-webhook-cert\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:37:46.210657 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:46.210653 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc-metrics-certs\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:37:46.210657 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:46.210664 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v2mn\" (UniqueName: \"kubernetes.io/projected/4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc-kube-api-access-6v2mn\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:37:46.753467 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:46.753429 2574 generic.go:356] "Generic (PLEG): container finished" podID="4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc" containerID="91b2bbe1a0d85e1ddb20b2dc3b9f6f5ec9077b5848ae581ee2f024e8098f9dcb" exitCode=0 Sep 14 11:37:46.753848 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:46.753480 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-796b4cc5fc-c4gnz" Sep 14 11:37:46.753848 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:46.753510 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-796b4cc5fc-c4gnz" event={"ID":"4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc","Type":"ContainerDied","Data":"91b2bbe1a0d85e1ddb20b2dc3b9f6f5ec9077b5848ae581ee2f024e8098f9dcb"} Sep 14 11:37:46.753848 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:46.753565 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-796b4cc5fc-c4gnz" event={"ID":"4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc","Type":"ContainerDied","Data":"198a0fb93c42e3133295b360f32928919fc3197e6244b79708a9656cb2d16f96"} Sep 14 11:37:46.753848 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:46.753587 2574 scope.go:117] "RemoveContainer" containerID="91b2bbe1a0d85e1ddb20b2dc3b9f6f5ec9077b5848ae581ee2f024e8098f9dcb" Sep 14 11:37:46.763308 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:46.763290 2574 scope.go:117] "RemoveContainer" containerID="91b2bbe1a0d85e1ddb20b2dc3b9f6f5ec9077b5848ae581ee2f024e8098f9dcb" Sep 14 11:37:46.763604 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:37:46.763585 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91b2bbe1a0d85e1ddb20b2dc3b9f6f5ec9077b5848ae581ee2f024e8098f9dcb\": container with ID starting with 91b2bbe1a0d85e1ddb20b2dc3b9f6f5ec9077b5848ae581ee2f024e8098f9dcb not found: ID does not exist" containerID="91b2bbe1a0d85e1ddb20b2dc3b9f6f5ec9077b5848ae581ee2f024e8098f9dcb" Sep 14 11:37:46.763654 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:46.763612 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91b2bbe1a0d85e1ddb20b2dc3b9f6f5ec9077b5848ae581ee2f024e8098f9dcb"} err="failed to get container status \"91b2bbe1a0d85e1ddb20b2dc3b9f6f5ec9077b5848ae581ee2f024e8098f9dcb\": rpc error: code = NotFound desc = could not find container \"91b2bbe1a0d85e1ddb20b2dc3b9f6f5ec9077b5848ae581ee2f024e8098f9dcb\": container with ID starting with 91b2bbe1a0d85e1ddb20b2dc3b9f6f5ec9077b5848ae581ee2f024e8098f9dcb not found: ID does not exist" Sep 14 11:37:46.774599 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:46.774572 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-796b4cc5fc-c4gnz"] Sep 14 11:37:46.778806 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:46.778786 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-796b4cc5fc-c4gnz"] Sep 14 11:37:47.899172 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:37:47.899129 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc" path="/var/lib/kubelet/pods/4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc/volumes" Sep 14 11:38:47.978086 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:47.978049 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-796b4cc5fc-rmpbw"] Sep 14 11:38:47.978522 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:47.978447 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db53c5cf-7e15-4711-990a-f3c2ede15e68" containerName="maas-api" Sep 14 11:38:47.978522 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:47.978458 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="db53c5cf-7e15-4711-990a-f3c2ede15e68" containerName="maas-api" Sep 14 11:38:47.978522 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:47.978471 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47f89cf4-0168-4cf1-a4d2-b001d69e966e" containerName="payload-processing" Sep 14 11:38:47.978522 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:47.978477 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="47f89cf4-0168-4cf1-a4d2-b001d69e966e" containerName="payload-processing" Sep 14 11:38:47.978522 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:47.978483 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41e4b6e5-a721-45a7-8540-31f75937c7f6" containerName="maas-api" Sep 14 11:38:47.978522 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:47.978488 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="41e4b6e5-a721-45a7-8540-31f75937c7f6" containerName="maas-api" Sep 14 11:38:47.978522 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:47.978499 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57685ac4-6a88-4780-8fcc-40d91b2aa5fd" containerName="payload-pre-processing" Sep 14 11:38:47.978522 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:47.978504 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="57685ac4-6a88-4780-8fcc-40d91b2aa5fd" containerName="payload-pre-processing" Sep 14 11:38:47.978522 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:47.978511 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59c9f939-33c0-4467-9f6d-ec139b8f1800" containerName="payload-pre-processing" Sep 14 11:38:47.978522 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:47.978517 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="59c9f939-33c0-4467-9f6d-ec139b8f1800" containerName="payload-pre-processing" Sep 14 11:38:47.978522 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:47.978524 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bccbbc83-79a4-4da6-839c-4c392b80ffc6" containerName="payload-pre-processing" Sep 14 11:38:47.978522 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:47.978529 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bccbbc83-79a4-4da6-839c-4c392b80ffc6" containerName="payload-pre-processing" Sep 14 11:38:47.978894 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:47.978538 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5655d6e-9a1d-479d-a4cf-65c2414fcad4" containerName="payload-processing" Sep 14 11:38:47.978894 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:47.978543 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5655d6e-9a1d-479d-a4cf-65c2414fcad4" containerName="payload-processing" Sep 14 11:38:47.978894 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:47.978548 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc" containerName="manager" Sep 14 11:38:47.978894 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:47.978555 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc" containerName="manager" Sep 14 11:38:47.978894 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:47.978560 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebd4bc7e-3e14-4fd1-bc25-d16c664b9af4" containerName="payload-processing" Sep 14 11:38:47.978894 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:47.978565 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebd4bc7e-3e14-4fd1-bc25-d16c664b9af4" containerName="payload-processing" Sep 14 11:38:47.978894 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:47.978627 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="41e4b6e5-a721-45a7-8540-31f75937c7f6" containerName="maas-api" Sep 14 11:38:47.978894 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:47.978635 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="bccbbc83-79a4-4da6-839c-4c392b80ffc6" containerName="payload-pre-processing" Sep 14 11:38:47.978894 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:47.978642 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="47f89cf4-0168-4cf1-a4d2-b001d69e966e" containerName="payload-processing" Sep 14 11:38:47.978894 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:47.978647 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="59c9f939-33c0-4467-9f6d-ec139b8f1800" containerName="payload-pre-processing" Sep 14 11:38:47.978894 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:47.978652 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5655d6e-9a1d-479d-a4cf-65c2414fcad4" containerName="payload-processing" Sep 14 11:38:47.978894 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:47.978658 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4099bb96-fd23-4dc5-a9b7-fb37cd82c5dc" containerName="manager" Sep 14 11:38:47.978894 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:47.978663 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="db53c5cf-7e15-4711-990a-f3c2ede15e68" containerName="maas-api" Sep 14 11:38:47.978894 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:47.978670 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="57685ac4-6a88-4780-8fcc-40d91b2aa5fd" containerName="payload-pre-processing" Sep 14 11:38:47.978894 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:47.978676 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebd4bc7e-3e14-4fd1-bc25-d16c664b9af4" containerName="payload-processing" Sep 14 11:38:47.981822 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:47.981806 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-796b4cc5fc-rmpbw" Sep 14 11:38:47.985150 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:47.985122 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-txcjg\"" Sep 14 11:38:47.985301 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:47.985125 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 14 11:38:47.985301 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:47.985155 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 14 11:38:47.985403 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:47.985305 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 14 11:38:47.991834 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:47.991804 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-796b4cc5fc-rmpbw"] Sep 14 11:38:48.076997 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:48.076964 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96ce5789-014d-4795-87d1-35ad506c973e-metrics-certs\") pod \"maas-controller-796b4cc5fc-rmpbw\" (UID: \"96ce5789-014d-4795-87d1-35ad506c973e\") " pod="opendatahub/maas-controller-796b4cc5fc-rmpbw" Sep 14 11:38:48.077169 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:48.077015 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87smd\" (UniqueName: \"kubernetes.io/projected/96ce5789-014d-4795-87d1-35ad506c973e-kube-api-access-87smd\") pod \"maas-controller-796b4cc5fc-rmpbw\" (UID: \"96ce5789-014d-4795-87d1-35ad506c973e\") " pod="opendatahub/maas-controller-796b4cc5fc-rmpbw" Sep 14 11:38:48.077169 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:48.077071 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/96ce5789-014d-4795-87d1-35ad506c973e-webhook-cert\") pod \"maas-controller-796b4cc5fc-rmpbw\" (UID: \"96ce5789-014d-4795-87d1-35ad506c973e\") " pod="opendatahub/maas-controller-796b4cc5fc-rmpbw" Sep 14 11:38:48.177931 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:48.177896 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87smd\" (UniqueName: \"kubernetes.io/projected/96ce5789-014d-4795-87d1-35ad506c973e-kube-api-access-87smd\") pod \"maas-controller-796b4cc5fc-rmpbw\" (UID: \"96ce5789-014d-4795-87d1-35ad506c973e\") " pod="opendatahub/maas-controller-796b4cc5fc-rmpbw" Sep 14 11:38:48.178107 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:48.177948 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/96ce5789-014d-4795-87d1-35ad506c973e-webhook-cert\") pod \"maas-controller-796b4cc5fc-rmpbw\" (UID: \"96ce5789-014d-4795-87d1-35ad506c973e\") " pod="opendatahub/maas-controller-796b4cc5fc-rmpbw" Sep 14 11:38:48.178107 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:48.178000 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96ce5789-014d-4795-87d1-35ad506c973e-metrics-certs\") pod \"maas-controller-796b4cc5fc-rmpbw\" (UID: \"96ce5789-014d-4795-87d1-35ad506c973e\") " pod="opendatahub/maas-controller-796b4cc5fc-rmpbw" Sep 14 11:38:48.180574 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:48.180551 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/96ce5789-014d-4795-87d1-35ad506c973e-webhook-cert\") pod \"maas-controller-796b4cc5fc-rmpbw\" (UID: \"96ce5789-014d-4795-87d1-35ad506c973e\") " pod="opendatahub/maas-controller-796b4cc5fc-rmpbw" Sep 14 11:38:48.180574 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:48.180572 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96ce5789-014d-4795-87d1-35ad506c973e-metrics-certs\") pod \"maas-controller-796b4cc5fc-rmpbw\" (UID: \"96ce5789-014d-4795-87d1-35ad506c973e\") " pod="opendatahub/maas-controller-796b4cc5fc-rmpbw" Sep 14 11:38:48.185525 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:48.185507 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87smd\" (UniqueName: \"kubernetes.io/projected/96ce5789-014d-4795-87d1-35ad506c973e-kube-api-access-87smd\") pod \"maas-controller-796b4cc5fc-rmpbw\" (UID: \"96ce5789-014d-4795-87d1-35ad506c973e\") " pod="opendatahub/maas-controller-796b4cc5fc-rmpbw" Sep 14 11:38:48.293021 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:48.292942 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-796b4cc5fc-rmpbw" Sep 14 11:38:48.564183 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:48.564159 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-796b4cc5fc-rmpbw"] Sep 14 11:38:48.566682 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:38:48.566657 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96ce5789_014d_4795_87d1_35ad506c973e.slice/crio-fcd1aaa715d4b861b387d08e244154ed74a905bca8044c8d6e49408b752ff75b WatchSource:0}: Error finding container fcd1aaa715d4b861b387d08e244154ed74a905bca8044c8d6e49408b752ff75b: Status 404 returned error can't find the container with id fcd1aaa715d4b861b387d08e244154ed74a905bca8044c8d6e49408b752ff75b Sep 14 11:38:48.568382 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:48.568365 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 11:38:49.067544 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:49.067517 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-796b4cc5fc-rmpbw" event={"ID":"96ce5789-014d-4795-87d1-35ad506c973e","Type":"ContainerStarted","Data":"fcd1aaa715d4b861b387d08e244154ed74a905bca8044c8d6e49408b752ff75b"} Sep 14 11:38:50.073501 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:50.073458 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-796b4cc5fc-rmpbw" event={"ID":"96ce5789-014d-4795-87d1-35ad506c973e","Type":"ContainerStarted","Data":"3831ece71b88b18a73840934c53bd2070aad23264ac47d5fb38962fec698322d"} Sep 14 11:38:50.073891 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:50.073533 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-796b4cc5fc-rmpbw" Sep 14 11:38:50.096322 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:38:50.096256 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-796b4cc5fc-rmpbw" podStartSLOduration=2.605941344 podStartE2EDuration="3.096240452s" totalImagesPullingTime="490.299108ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:38:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:38:50.094633179 +0000 UTC m=+2274.714845723" watchObservedRunningTime="2026-09-14 11:38:50.096240452 +0000 UTC m=+2274.716452974" Sep 14 11:39:01.080374 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:01.080340 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-796b4cc5fc-rmpbw" Sep 14 11:39:10.547881 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:10.547844 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-796b4cc5fc-rmpbw"] Sep 14 11:39:10.548518 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:10.548151 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-796b4cc5fc-rmpbw" podUID="96ce5789-014d-4795-87d1-35ad506c973e" containerName="manager" containerID="cri-o://3831ece71b88b18a73840934c53bd2070aad23264ac47d5fb38962fec698322d" gracePeriod=30 Sep 14 11:39:10.793834 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:10.793808 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-796b4cc5fc-rmpbw" Sep 14 11:39:10.896901 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:10.896818 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/96ce5789-014d-4795-87d1-35ad506c973e-webhook-cert\") pod \"96ce5789-014d-4795-87d1-35ad506c973e\" (UID: \"96ce5789-014d-4795-87d1-35ad506c973e\") " Sep 14 11:39:10.896901 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:10.896870 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87smd\" (UniqueName: \"kubernetes.io/projected/96ce5789-014d-4795-87d1-35ad506c973e-kube-api-access-87smd\") pod \"96ce5789-014d-4795-87d1-35ad506c973e\" (UID: \"96ce5789-014d-4795-87d1-35ad506c973e\") " Sep 14 11:39:10.897095 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:10.896944 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96ce5789-014d-4795-87d1-35ad506c973e-metrics-certs\") pod \"96ce5789-014d-4795-87d1-35ad506c973e\" (UID: \"96ce5789-014d-4795-87d1-35ad506c973e\") " Sep 14 11:39:10.899111 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:10.899075 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96ce5789-014d-4795-87d1-35ad506c973e-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "96ce5789-014d-4795-87d1-35ad506c973e" (UID: "96ce5789-014d-4795-87d1-35ad506c973e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:39:10.899111 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:10.899086 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96ce5789-014d-4795-87d1-35ad506c973e-kube-api-access-87smd" (OuterVolumeSpecName: "kube-api-access-87smd") pod "96ce5789-014d-4795-87d1-35ad506c973e" (UID: "96ce5789-014d-4795-87d1-35ad506c973e"). InnerVolumeSpecName "kube-api-access-87smd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:39:10.899306 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:10.899092 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96ce5789-014d-4795-87d1-35ad506c973e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "96ce5789-014d-4795-87d1-35ad506c973e" (UID: "96ce5789-014d-4795-87d1-35ad506c973e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:39:10.997844 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:10.997803 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/96ce5789-014d-4795-87d1-35ad506c973e-metrics-certs\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:39:10.997844 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:10.997834 2574 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/96ce5789-014d-4795-87d1-35ad506c973e-webhook-cert\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:39:10.997844 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:10.997851 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87smd\" (UniqueName: \"kubernetes.io/projected/96ce5789-014d-4795-87d1-35ad506c973e-kube-api-access-87smd\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:39:11.187475 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:11.187439 2574 generic.go:356] "Generic (PLEG): container finished" podID="96ce5789-014d-4795-87d1-35ad506c973e" containerID="3831ece71b88b18a73840934c53bd2070aad23264ac47d5fb38962fec698322d" exitCode=0 Sep 14 11:39:11.187679 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:11.187486 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-796b4cc5fc-rmpbw" Sep 14 11:39:11.187679 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:11.187491 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-796b4cc5fc-rmpbw" event={"ID":"96ce5789-014d-4795-87d1-35ad506c973e","Type":"ContainerDied","Data":"3831ece71b88b18a73840934c53bd2070aad23264ac47d5fb38962fec698322d"} Sep 14 11:39:11.187679 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:11.187616 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-796b4cc5fc-rmpbw" event={"ID":"96ce5789-014d-4795-87d1-35ad506c973e","Type":"ContainerDied","Data":"fcd1aaa715d4b861b387d08e244154ed74a905bca8044c8d6e49408b752ff75b"} Sep 14 11:39:11.187679 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:11.187645 2574 scope.go:117] "RemoveContainer" containerID="3831ece71b88b18a73840934c53bd2070aad23264ac47d5fb38962fec698322d" Sep 14 11:39:11.197038 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:11.197021 2574 scope.go:117] "RemoveContainer" containerID="3831ece71b88b18a73840934c53bd2070aad23264ac47d5fb38962fec698322d" Sep 14 11:39:11.197356 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:39:11.197338 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3831ece71b88b18a73840934c53bd2070aad23264ac47d5fb38962fec698322d\": container with ID starting with 3831ece71b88b18a73840934c53bd2070aad23264ac47d5fb38962fec698322d not found: ID does not exist" containerID="3831ece71b88b18a73840934c53bd2070aad23264ac47d5fb38962fec698322d" Sep 14 11:39:11.197416 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:11.197366 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3831ece71b88b18a73840934c53bd2070aad23264ac47d5fb38962fec698322d"} err="failed to get container status \"3831ece71b88b18a73840934c53bd2070aad23264ac47d5fb38962fec698322d\": rpc error: code = NotFound desc = could not find container \"3831ece71b88b18a73840934c53bd2070aad23264ac47d5fb38962fec698322d\": container with ID starting with 3831ece71b88b18a73840934c53bd2070aad23264ac47d5fb38962fec698322d not found: ID does not exist" Sep 14 11:39:11.207694 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:11.207672 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-796b4cc5fc-rmpbw"] Sep 14 11:39:11.211242 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:11.211219 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-796b4cc5fc-rmpbw"] Sep 14 11:39:11.898935 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:11.898897 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96ce5789-014d-4795-87d1-35ad506c973e" path="/var/lib/kubelet/pods/96ce5789-014d-4795-87d1-35ad506c973e/volumes" Sep 14 11:39:11.942741 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:11.942706 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-796b4cc5fc-9pl9v"] Sep 14 11:39:11.943169 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:11.943156 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96ce5789-014d-4795-87d1-35ad506c973e" containerName="manager" Sep 14 11:39:11.943230 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:11.943173 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="96ce5789-014d-4795-87d1-35ad506c973e" containerName="manager" Sep 14 11:39:11.943283 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:11.943246 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="96ce5789-014d-4795-87d1-35ad506c973e" containerName="manager" Sep 14 11:39:11.947924 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:11.947900 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-796b4cc5fc-9pl9v" Sep 14 11:39:11.951411 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:11.951390 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 14 11:39:11.951558 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:11.951422 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-txcjg\"" Sep 14 11:39:11.951558 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:11.951395 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 14 11:39:11.951558 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:11.951390 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 14 11:39:11.956695 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:11.956674 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-796b4cc5fc-9pl9v"] Sep 14 11:39:12.110382 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:12.110334 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1eb2709a-547b-49ef-8ac6-0f05bece321b-webhook-cert\") pod \"maas-controller-796b4cc5fc-9pl9v\" (UID: \"1eb2709a-547b-49ef-8ac6-0f05bece321b\") " pod="opendatahub/maas-controller-796b4cc5fc-9pl9v" Sep 14 11:39:12.110570 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:12.110444 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1eb2709a-547b-49ef-8ac6-0f05bece321b-metrics-certs\") pod \"maas-controller-796b4cc5fc-9pl9v\" (UID: \"1eb2709a-547b-49ef-8ac6-0f05bece321b\") " pod="opendatahub/maas-controller-796b4cc5fc-9pl9v" Sep 14 11:39:12.110570 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:12.110524 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqntv\" (UniqueName: \"kubernetes.io/projected/1eb2709a-547b-49ef-8ac6-0f05bece321b-kube-api-access-tqntv\") pod \"maas-controller-796b4cc5fc-9pl9v\" (UID: \"1eb2709a-547b-49ef-8ac6-0f05bece321b\") " pod="opendatahub/maas-controller-796b4cc5fc-9pl9v" Sep 14 11:39:12.212042 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:12.212012 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1eb2709a-547b-49ef-8ac6-0f05bece321b-webhook-cert\") pod \"maas-controller-796b4cc5fc-9pl9v\" (UID: \"1eb2709a-547b-49ef-8ac6-0f05bece321b\") " pod="opendatahub/maas-controller-796b4cc5fc-9pl9v" Sep 14 11:39:12.212232 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:12.212098 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1eb2709a-547b-49ef-8ac6-0f05bece321b-metrics-certs\") pod \"maas-controller-796b4cc5fc-9pl9v\" (UID: \"1eb2709a-547b-49ef-8ac6-0f05bece321b\") " pod="opendatahub/maas-controller-796b4cc5fc-9pl9v" Sep 14 11:39:12.212232 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:12.212152 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqntv\" (UniqueName: \"kubernetes.io/projected/1eb2709a-547b-49ef-8ac6-0f05bece321b-kube-api-access-tqntv\") pod \"maas-controller-796b4cc5fc-9pl9v\" (UID: \"1eb2709a-547b-49ef-8ac6-0f05bece321b\") " pod="opendatahub/maas-controller-796b4cc5fc-9pl9v" Sep 14 11:39:12.214794 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:12.214765 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1eb2709a-547b-49ef-8ac6-0f05bece321b-metrics-certs\") pod \"maas-controller-796b4cc5fc-9pl9v\" (UID: \"1eb2709a-547b-49ef-8ac6-0f05bece321b\") " pod="opendatahub/maas-controller-796b4cc5fc-9pl9v" Sep 14 11:39:12.214891 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:12.214811 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1eb2709a-547b-49ef-8ac6-0f05bece321b-webhook-cert\") pod \"maas-controller-796b4cc5fc-9pl9v\" (UID: \"1eb2709a-547b-49ef-8ac6-0f05bece321b\") " pod="opendatahub/maas-controller-796b4cc5fc-9pl9v" Sep 14 11:39:12.219751 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:12.219728 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqntv\" (UniqueName: \"kubernetes.io/projected/1eb2709a-547b-49ef-8ac6-0f05bece321b-kube-api-access-tqntv\") pod \"maas-controller-796b4cc5fc-9pl9v\" (UID: \"1eb2709a-547b-49ef-8ac6-0f05bece321b\") " pod="opendatahub/maas-controller-796b4cc5fc-9pl9v" Sep 14 11:39:12.260371 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:12.260337 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-796b4cc5fc-9pl9v" Sep 14 11:39:12.532983 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:12.532955 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-796b4cc5fc-9pl9v"] Sep 14 11:39:12.534832 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:39:12.534797 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb2709a_547b_49ef_8ac6_0f05bece321b.slice/crio-5e1c90ef5a31f3fdd64f7eb8d76cb535c0f28974f08d34da85585da15da350b1 WatchSource:0}: Error finding container 5e1c90ef5a31f3fdd64f7eb8d76cb535c0f28974f08d34da85585da15da350b1: Status 404 returned error can't find the container with id 5e1c90ef5a31f3fdd64f7eb8d76cb535c0f28974f08d34da85585da15da350b1 Sep 14 11:39:13.201996 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:13.201957 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-796b4cc5fc-9pl9v" event={"ID":"1eb2709a-547b-49ef-8ac6-0f05bece321b","Type":"ContainerStarted","Data":"543f8636ad8d27c3987c4964e7375fafa1506799268724cd5fc5c8c598b6c55d"} Sep 14 11:39:13.202342 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:13.202058 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-796b4cc5fc-9pl9v" event={"ID":"1eb2709a-547b-49ef-8ac6-0f05bece321b","Type":"ContainerStarted","Data":"5e1c90ef5a31f3fdd64f7eb8d76cb535c0f28974f08d34da85585da15da350b1"} Sep 14 11:39:13.202342 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:13.202093 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-796b4cc5fc-9pl9v" Sep 14 11:39:13.220614 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:13.220566 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-796b4cc5fc-9pl9v" podStartSLOduration=1.621402225 podStartE2EDuration="2.220548064s" totalImagesPullingTime="599.145839ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:39:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:39:13.218728211 +0000 UTC m=+2297.838940757" watchObservedRunningTime="2026-09-14 11:39:13.220548064 +0000 UTC m=+2297.840760587" Sep 14 11:39:24.210635 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:39:24.210604 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-796b4cc5fc-9pl9v" Sep 14 11:43:04.814091 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:04.814052 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7749ddb494-qw7rz"] Sep 14 11:43:04.817877 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:04.817854 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7749ddb494-qw7rz" Sep 14 11:43:04.834811 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:04.834783 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7749ddb494-qw7rz"] Sep 14 11:43:04.924217 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:04.924178 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27hqr\" (UniqueName: \"kubernetes.io/projected/43d9df2d-9305-4a6e-b6ac-7d77f868b2b6-kube-api-access-27hqr\") pod \"maas-api-7749ddb494-qw7rz\" (UID: \"43d9df2d-9305-4a6e-b6ac-7d77f868b2b6\") " pod="odh-ai-gateway-infra/maas-api-7749ddb494-qw7rz" Sep 14 11:43:04.924453 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:04.924237 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43d9df2d-9305-4a6e-b6ac-7d77f868b2b6-openshift-service-ca\") pod \"maas-api-7749ddb494-qw7rz\" (UID: \"43d9df2d-9305-4a6e-b6ac-7d77f868b2b6\") " pod="odh-ai-gateway-infra/maas-api-7749ddb494-qw7rz" Sep 14 11:43:04.924453 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:04.924375 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43d9df2d-9305-4a6e-b6ac-7d77f868b2b6-metrics-certs\") pod \"maas-api-7749ddb494-qw7rz\" (UID: \"43d9df2d-9305-4a6e-b6ac-7d77f868b2b6\") " pod="odh-ai-gateway-infra/maas-api-7749ddb494-qw7rz" Sep 14 11:43:04.924592 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:04.924473 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43d9df2d-9305-4a6e-b6ac-7d77f868b2b6-maas-api-tls\") pod \"maas-api-7749ddb494-qw7rz\" (UID: \"43d9df2d-9305-4a6e-b6ac-7d77f868b2b6\") " pod="odh-ai-gateway-infra/maas-api-7749ddb494-qw7rz" Sep 14 11:43:05.025406 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:05.025301 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43d9df2d-9305-4a6e-b6ac-7d77f868b2b6-metrics-certs\") pod \"maas-api-7749ddb494-qw7rz\" (UID: \"43d9df2d-9305-4a6e-b6ac-7d77f868b2b6\") " pod="odh-ai-gateway-infra/maas-api-7749ddb494-qw7rz" Sep 14 11:43:05.025406 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:05.025379 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43d9df2d-9305-4a6e-b6ac-7d77f868b2b6-maas-api-tls\") pod \"maas-api-7749ddb494-qw7rz\" (UID: \"43d9df2d-9305-4a6e-b6ac-7d77f868b2b6\") " pod="odh-ai-gateway-infra/maas-api-7749ddb494-qw7rz" Sep 14 11:43:05.025717 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:05.025462 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27hqr\" (UniqueName: \"kubernetes.io/projected/43d9df2d-9305-4a6e-b6ac-7d77f868b2b6-kube-api-access-27hqr\") pod \"maas-api-7749ddb494-qw7rz\" (UID: \"43d9df2d-9305-4a6e-b6ac-7d77f868b2b6\") " pod="odh-ai-gateway-infra/maas-api-7749ddb494-qw7rz" Sep 14 11:43:05.025717 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:05.025511 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43d9df2d-9305-4a6e-b6ac-7d77f868b2b6-openshift-service-ca\") pod \"maas-api-7749ddb494-qw7rz\" (UID: \"43d9df2d-9305-4a6e-b6ac-7d77f868b2b6\") " pod="odh-ai-gateway-infra/maas-api-7749ddb494-qw7rz" Sep 14 11:43:05.026948 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:05.026920 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43d9df2d-9305-4a6e-b6ac-7d77f868b2b6-openshift-service-ca\") pod \"maas-api-7749ddb494-qw7rz\" (UID: \"43d9df2d-9305-4a6e-b6ac-7d77f868b2b6\") " pod="odh-ai-gateway-infra/maas-api-7749ddb494-qw7rz" Sep 14 11:43:05.028851 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:05.028793 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43d9df2d-9305-4a6e-b6ac-7d77f868b2b6-maas-api-tls\") pod \"maas-api-7749ddb494-qw7rz\" (UID: \"43d9df2d-9305-4a6e-b6ac-7d77f868b2b6\") " pod="odh-ai-gateway-infra/maas-api-7749ddb494-qw7rz" Sep 14 11:43:05.028982 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:05.028917 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43d9df2d-9305-4a6e-b6ac-7d77f868b2b6-metrics-certs\") pod \"maas-api-7749ddb494-qw7rz\" (UID: \"43d9df2d-9305-4a6e-b6ac-7d77f868b2b6\") " pod="odh-ai-gateway-infra/maas-api-7749ddb494-qw7rz" Sep 14 11:43:05.035360 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:05.035257 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27hqr\" (UniqueName: \"kubernetes.io/projected/43d9df2d-9305-4a6e-b6ac-7d77f868b2b6-kube-api-access-27hqr\") pod \"maas-api-7749ddb494-qw7rz\" (UID: \"43d9df2d-9305-4a6e-b6ac-7d77f868b2b6\") " pod="odh-ai-gateway-infra/maas-api-7749ddb494-qw7rz" Sep 14 11:43:05.128306 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:05.128198 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7749ddb494-qw7rz" Sep 14 11:43:05.430781 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:05.430750 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7749ddb494-qw7rz"] Sep 14 11:43:05.435414 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:43:05.435384 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43d9df2d_9305_4a6e_b6ac_7d77f868b2b6.slice/crio-70b8dc4748f89534de10335b0aec890aee6ec2e31547c2d39468c54e6a3c155f WatchSource:0}: Error finding container 70b8dc4748f89534de10335b0aec890aee6ec2e31547c2d39468c54e6a3c155f: Status 404 returned error can't find the container with id 70b8dc4748f89534de10335b0aec890aee6ec2e31547c2d39468c54e6a3c155f Sep 14 11:43:06.365342 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:06.365299 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7749ddb494-qw7rz" event={"ID":"43d9df2d-9305-4a6e-b6ac-7d77f868b2b6","Type":"ContainerStarted","Data":"d1a47c3dc57c425b403a1ad5a200fa27e0e82ddec4515b405d1adcba1015c90c"} Sep 14 11:43:06.365723 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:06.365365 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7749ddb494-qw7rz" event={"ID":"43d9df2d-9305-4a6e-b6ac-7d77f868b2b6","Type":"ContainerStarted","Data":"70b8dc4748f89534de10335b0aec890aee6ec2e31547c2d39468c54e6a3c155f"} Sep 14 11:43:06.365723 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:06.365478 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7749ddb494-qw7rz" Sep 14 11:43:06.423905 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:06.423848 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7749ddb494-qw7rz" podStartSLOduration=2.034152 podStartE2EDuration="2.423829513s" totalImagesPullingTime="389.677513ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:43:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:43:06.417183029 +0000 UTC m=+2531.037395552" watchObservedRunningTime="2026-09-14 11:43:06.423829513 +0000 UTC m=+2531.044042036" Sep 14 11:43:12.376758 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:12.376728 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7749ddb494-qw7rz" Sep 14 11:43:19.731593 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:19.731555 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-8f99c9866-4pd64"] Sep 14 11:43:19.732009 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:19.731784 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-8f99c9866-4pd64" podUID="be24615b-72c5-4376-a5b3-9f51d6df14d4" containerName="maas-api" containerID="cri-o://7052c65d76d01ce53900f65f7d0382a2306e1b6f6d332e21b85733ceef97d6a8" gracePeriod=30 Sep 14 11:43:19.750891 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:19.750853 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-8f99c9866-4pd64" podUID="be24615b-72c5-4376-a5b3-9f51d6df14d4" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.36:8443/health\": dial tcp 10.134.0.36:8443: connect: connection refused" Sep 14 11:43:20.984991 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:20.984965 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8f99c9866-4pd64" Sep 14 11:43:21.082121 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:21.082039 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be24615b-72c5-4376-a5b3-9f51d6df14d4-maas-api-tls\") pod \"be24615b-72c5-4376-a5b3-9f51d6df14d4\" (UID: \"be24615b-72c5-4376-a5b3-9f51d6df14d4\") " Sep 14 11:43:21.082121 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:21.082083 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be24615b-72c5-4376-a5b3-9f51d6df14d4-metrics-certs\") pod \"be24615b-72c5-4376-a5b3-9f51d6df14d4\" (UID: \"be24615b-72c5-4376-a5b3-9f51d6df14d4\") " Sep 14 11:43:21.082335 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:21.082160 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be24615b-72c5-4376-a5b3-9f51d6df14d4-openshift-service-ca\") pod \"be24615b-72c5-4376-a5b3-9f51d6df14d4\" (UID: \"be24615b-72c5-4376-a5b3-9f51d6df14d4\") " Sep 14 11:43:21.082335 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:21.082214 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnp6s\" (UniqueName: \"kubernetes.io/projected/be24615b-72c5-4376-a5b3-9f51d6df14d4-kube-api-access-cnp6s\") pod \"be24615b-72c5-4376-a5b3-9f51d6df14d4\" (UID: \"be24615b-72c5-4376-a5b3-9f51d6df14d4\") " Sep 14 11:43:21.082593 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:21.082560 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be24615b-72c5-4376-a5b3-9f51d6df14d4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "be24615b-72c5-4376-a5b3-9f51d6df14d4" (UID: "be24615b-72c5-4376-a5b3-9f51d6df14d4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:43:21.084406 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:21.084380 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be24615b-72c5-4376-a5b3-9f51d6df14d4-kube-api-access-cnp6s" (OuterVolumeSpecName: "kube-api-access-cnp6s") pod "be24615b-72c5-4376-a5b3-9f51d6df14d4" (UID: "be24615b-72c5-4376-a5b3-9f51d6df14d4"). InnerVolumeSpecName "kube-api-access-cnp6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:43:21.084767 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:21.084747 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be24615b-72c5-4376-a5b3-9f51d6df14d4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "be24615b-72c5-4376-a5b3-9f51d6df14d4" (UID: "be24615b-72c5-4376-a5b3-9f51d6df14d4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:43:21.084820 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:21.084759 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be24615b-72c5-4376-a5b3-9f51d6df14d4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "be24615b-72c5-4376-a5b3-9f51d6df14d4" (UID: "be24615b-72c5-4376-a5b3-9f51d6df14d4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:43:21.183480 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:21.183448 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be24615b-72c5-4376-a5b3-9f51d6df14d4-maas-api-tls\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:43:21.183480 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:21.183474 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be24615b-72c5-4376-a5b3-9f51d6df14d4-metrics-certs\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:43:21.183480 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:21.183485 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be24615b-72c5-4376-a5b3-9f51d6df14d4-openshift-service-ca\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:43:21.183725 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:21.183497 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnp6s\" (UniqueName: \"kubernetes.io/projected/be24615b-72c5-4376-a5b3-9f51d6df14d4-kube-api-access-cnp6s\") on node \"ip-10-0-137-230.ec2.internal\" DevicePath \"\"" Sep 14 11:43:21.443284 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:21.443232 2574 generic.go:356] "Generic (PLEG): container finished" podID="be24615b-72c5-4376-a5b3-9f51d6df14d4" containerID="7052c65d76d01ce53900f65f7d0382a2306e1b6f6d332e21b85733ceef97d6a8" exitCode=0 Sep 14 11:43:21.443460 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:21.443299 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8f99c9866-4pd64" Sep 14 11:43:21.443460 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:21.443319 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8f99c9866-4pd64" event={"ID":"be24615b-72c5-4376-a5b3-9f51d6df14d4","Type":"ContainerDied","Data":"7052c65d76d01ce53900f65f7d0382a2306e1b6f6d332e21b85733ceef97d6a8"} Sep 14 11:43:21.443460 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:21.443373 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8f99c9866-4pd64" event={"ID":"be24615b-72c5-4376-a5b3-9f51d6df14d4","Type":"ContainerDied","Data":"b255199da343040d0d676fbd8bc53197564ea0ec509c7ce8ee8e302c90c219a1"} Sep 14 11:43:21.443460 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:21.443394 2574 scope.go:117] "RemoveContainer" containerID="7052c65d76d01ce53900f65f7d0382a2306e1b6f6d332e21b85733ceef97d6a8" Sep 14 11:43:21.453806 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:21.453789 2574 scope.go:117] "RemoveContainer" containerID="7052c65d76d01ce53900f65f7d0382a2306e1b6f6d332e21b85733ceef97d6a8" Sep 14 11:43:21.454089 ip-10-0-137-230 kubenswrapper[2574]: E0914 11:43:21.454071 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7052c65d76d01ce53900f65f7d0382a2306e1b6f6d332e21b85733ceef97d6a8\": container with ID starting with 7052c65d76d01ce53900f65f7d0382a2306e1b6f6d332e21b85733ceef97d6a8 not found: ID does not exist" containerID="7052c65d76d01ce53900f65f7d0382a2306e1b6f6d332e21b85733ceef97d6a8" Sep 14 11:43:21.454143 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:21.454099 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7052c65d76d01ce53900f65f7d0382a2306e1b6f6d332e21b85733ceef97d6a8"} err="failed to get container status \"7052c65d76d01ce53900f65f7d0382a2306e1b6f6d332e21b85733ceef97d6a8\": rpc error: code = NotFound desc = could not find container \"7052c65d76d01ce53900f65f7d0382a2306e1b6f6d332e21b85733ceef97d6a8\": container with ID starting with 7052c65d76d01ce53900f65f7d0382a2306e1b6f6d332e21b85733ceef97d6a8 not found: ID does not exist" Sep 14 11:43:21.474392 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:21.474363 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-8f99c9866-4pd64"] Sep 14 11:43:21.486882 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:21.486857 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-8f99c9866-4pd64"] Sep 14 11:43:21.905160 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:21.905065 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be24615b-72c5-4376-a5b3-9f51d6df14d4" path="/var/lib/kubelet/pods/be24615b-72c5-4376-a5b3-9f51d6df14d4/volumes" Sep 14 11:43:40.674741 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:40.674709 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-796b4cc5fc-9pl9v_1eb2709a-547b-49ef-8ac6-0f05bece321b/manager/0.log" Sep 14 11:43:40.789232 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:40.789200 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-txc6g_2da68c9f-c790-4560-85bd-43dc1233bf8b/server/0.log" Sep 14 11:43:40.905625 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:40.905577 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-j8jfw_07d207b3-1e62-4a62-b090-3e56e9c10715/manager/0.log" Sep 14 11:43:41.287175 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:41.287151 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f85645c65-z2tnm_415f2f7b-4196-4d3d-b4ee-e237110a54bf/copy-manifests/0.log" Sep 14 11:43:41.302388 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:41.302365 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f85645c65-z2tnm_415f2f7b-4196-4d3d-b4ee-e237110a54bf/manager/0.log" Sep 14 11:43:41.772318 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:41.772291 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7749ddb494-qw7rz_43d9df2d-9305-4a6e-b6ac-7d77f868b2b6/maas-api/0.log" Sep 14 11:43:42.123920 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:42.123849 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-hfwtg_5677dd72-2c2c-4ef0-877c-fd3cde06194e/postgres/0.log" Sep 14 11:43:43.032423 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:43.032390 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-zvw7m_66f6a622-980a-4372-b50c-fc633812faa9/manager/0.log" Sep 14 11:43:43.645873 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:43.645828 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-m29tk_30723dd6-3820-45a5-b47a-3ba0942e29b5/manager/0.log" Sep 14 11:43:44.131635 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:44.131588 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-86plh_477e2e5e-7397-409e-9d1c-b35652715697/discovery/0.log" Sep 14 11:43:44.347688 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:44.347659 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f95fc98f9-kzb94_9aa22333-c697-43d6-ac9f-eb93f64b8801/kube-auth-proxy/0.log" Sep 14 11:43:44.585091 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:44.585067 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-p5srl_a6ecef0e-9b32-41ab-8452-618836054a5a/payload-pre-processing/0.log" Sep 14 11:43:44.725825 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:44.725797 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-mvnvh_d4eea439-1943-4f74-9067-d1bb67a438f1/payload-processing/0.log" Sep 14 11:43:44.849588 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:44.849518 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6f6c545fc8-77jgh_2baadd06-610f-4413-97c7-04f3a5c2039b/router/0.log" Sep 14 11:43:45.453472 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:45.453445 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-zckrp_e728a6b5-f5f2-4c35-bbee-2faa09e4059f/main/0.log" Sep 14 11:43:45.832434 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:45.832360 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-m49mq_518192ea-4063-4e8c-a600-6a85c83c2dcb/main/0.log" Sep 14 11:43:55.310193 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:55.310168 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2sk6p_07adbd4b-794d-4f1e-b566-02280ae8390a/global-pull-secret-syncer/0.log" Sep 14 11:43:55.742513 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:55.742483 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jfdk9_caa56301-04e7-4389-b0e8-b39bcc58b6ad/konnectivity-agent/0.log" Sep 14 11:43:55.949451 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:55.949423 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-230.ec2.internal_fc8ab4f3c567cd8f16d2622c33e7fe29/haproxy/0.log" Sep 14 11:43:59.696591 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:43:59.696554 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-zvw7m_66f6a622-980a-4372-b50c-fc633812faa9/manager/0.log" Sep 14 11:44:00.181737 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:00.181714 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-m29tk_30723dd6-3820-45a5-b47a-3ba0942e29b5/manager/0.log" Sep 14 11:44:01.831667 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:01.831591 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6406ef1e-5938-48da-8c40-54d4c0e717e9/alertmanager/0.log" Sep 14 11:44:01.897191 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:01.897166 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6406ef1e-5938-48da-8c40-54d4c0e717e9/config-reloader/0.log" Sep 14 11:44:01.960319 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:01.960290 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6406ef1e-5938-48da-8c40-54d4c0e717e9/kube-rbac-proxy-web/0.log" Sep 14 11:44:02.039923 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:02.039876 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6406ef1e-5938-48da-8c40-54d4c0e717e9/kube-rbac-proxy/0.log" Sep 14 11:44:02.185600 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:02.185574 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6406ef1e-5938-48da-8c40-54d4c0e717e9/kube-rbac-proxy-metric/0.log" Sep 14 11:44:02.405675 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:02.405655 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6406ef1e-5938-48da-8c40-54d4c0e717e9/prom-label-proxy/0.log" Sep 14 11:44:02.508822 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:02.508747 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6406ef1e-5938-48da-8c40-54d4c0e717e9/init-config-reloader/0.log" Sep 14 11:44:02.953923 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:02.953898 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5cf5d68bb4-nb8ds_92a227c2-cbea-4fc3-a721-b7fba16c0705/metrics-server/0.log" Sep 14 11:44:03.001277 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:03.001251 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-b6zt5_a3079479-adb1-43e3-9d0b-589673000bef/monitoring-plugin/0.log" Sep 14 11:44:03.324358 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:03.324257 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k25g8_6337f168-6fbc-4c83-ab67-9476b93b697c/node-exporter/0.log" Sep 14 11:44:03.368307 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:03.368257 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k25g8_6337f168-6fbc-4c83-ab67-9476b93b697c/kube-rbac-proxy/0.log" Sep 14 11:44:03.411846 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:03.411815 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k25g8_6337f168-6fbc-4c83-ab67-9476b93b697c/init-textfile/0.log" Sep 14 11:44:03.640208 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:03.640124 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_53829fff-bda5-4556-885c-6eb9465ecb04/prometheus/0.log" Sep 14 11:44:03.678301 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:03.678251 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_53829fff-bda5-4556-885c-6eb9465ecb04/config-reloader/0.log" Sep 14 11:44:03.709839 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:03.709807 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_53829fff-bda5-4556-885c-6eb9465ecb04/thanos-sidecar/0.log" Sep 14 11:44:03.748022 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:03.747997 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_53829fff-bda5-4556-885c-6eb9465ecb04/kube-rbac-proxy-web/0.log" Sep 14 11:44:03.790684 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:03.790662 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_53829fff-bda5-4556-885c-6eb9465ecb04/kube-rbac-proxy/0.log" Sep 14 11:44:03.833883 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:03.833858 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_53829fff-bda5-4556-885c-6eb9465ecb04/kube-rbac-proxy-thanos/0.log" Sep 14 11:44:03.869912 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:03.869867 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_53829fff-bda5-4556-885c-6eb9465ecb04/init-config-reloader/0.log" Sep 14 11:44:04.043476 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:04.043452 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-85fdf9d4cc-sk57l_560befdf-25ba-4c18-8946-30f75f0e53b1/telemeter-client/0.log" Sep 14 11:44:04.090614 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:04.090589 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-85fdf9d4cc-sk57l_560befdf-25ba-4c18-8946-30f75f0e53b1/reload/0.log" Sep 14 11:44:04.134578 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:04.134546 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-85fdf9d4cc-sk57l_560befdf-25ba-4c18-8946-30f75f0e53b1/kube-rbac-proxy/0.log" Sep 14 11:44:05.054703 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:05.054668 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zxfzq/perf-node-gather-daemonset-jrswf"] Sep 14 11:44:05.055145 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:05.055131 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be24615b-72c5-4376-a5b3-9f51d6df14d4" containerName="maas-api" Sep 14 11:44:05.055192 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:05.055147 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="be24615b-72c5-4376-a5b3-9f51d6df14d4" containerName="maas-api" Sep 14 11:44:05.055228 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:05.055216 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="be24615b-72c5-4376-a5b3-9f51d6df14d4" containerName="maas-api" Sep 14 11:44:05.058405 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:05.058388 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-jrswf" Sep 14 11:44:05.062567 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:05.062545 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zxfzq\"/\"kube-root-ca.crt\"" Sep 14 11:44:05.063219 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:05.063205 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zxfzq\"/\"default-dockercfg-2g5tp\"" Sep 14 11:44:05.063567 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:05.063554 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zxfzq\"/\"openshift-service-ca.crt\"" Sep 14 11:44:05.099074 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:05.099034 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zxfzq/perf-node-gather-daemonset-jrswf"] Sep 14 11:44:05.173023 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:05.172986 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmlxd\" (UniqueName: \"kubernetes.io/projected/a123f453-9f7f-4bb0-a8b7-8b5016d7d25b-kube-api-access-rmlxd\") pod \"perf-node-gather-daemonset-jrswf\" (UID: \"a123f453-9f7f-4bb0-a8b7-8b5016d7d25b\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-jrswf" Sep 14 11:44:05.173189 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:05.173043 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a123f453-9f7f-4bb0-a8b7-8b5016d7d25b-lib-modules\") pod \"perf-node-gather-daemonset-jrswf\" (UID: \"a123f453-9f7f-4bb0-a8b7-8b5016d7d25b\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-jrswf" Sep 14 11:44:05.173189 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:05.173111 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a123f453-9f7f-4bb0-a8b7-8b5016d7d25b-podres\") pod \"perf-node-gather-daemonset-jrswf\" (UID: \"a123f453-9f7f-4bb0-a8b7-8b5016d7d25b\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-jrswf" Sep 14 11:44:05.173301 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:05.173181 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a123f453-9f7f-4bb0-a8b7-8b5016d7d25b-sys\") pod \"perf-node-gather-daemonset-jrswf\" (UID: \"a123f453-9f7f-4bb0-a8b7-8b5016d7d25b\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-jrswf" Sep 14 11:44:05.173301 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:05.173216 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a123f453-9f7f-4bb0-a8b7-8b5016d7d25b-proc\") pod \"perf-node-gather-daemonset-jrswf\" (UID: \"a123f453-9f7f-4bb0-a8b7-8b5016d7d25b\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-jrswf" Sep 14 11:44:05.274707 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:05.274669 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a123f453-9f7f-4bb0-a8b7-8b5016d7d25b-proc\") pod \"perf-node-gather-daemonset-jrswf\" (UID: \"a123f453-9f7f-4bb0-a8b7-8b5016d7d25b\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-jrswf" Sep 14 11:44:05.274882 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:05.274738 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmlxd\" (UniqueName: \"kubernetes.io/projected/a123f453-9f7f-4bb0-a8b7-8b5016d7d25b-kube-api-access-rmlxd\") pod \"perf-node-gather-daemonset-jrswf\" (UID: \"a123f453-9f7f-4bb0-a8b7-8b5016d7d25b\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-jrswf" Sep 14 11:44:05.274882 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:05.274775 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a123f453-9f7f-4bb0-a8b7-8b5016d7d25b-lib-modules\") pod \"perf-node-gather-daemonset-jrswf\" (UID: \"a123f453-9f7f-4bb0-a8b7-8b5016d7d25b\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-jrswf" Sep 14 11:44:05.274882 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:05.274806 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a123f453-9f7f-4bb0-a8b7-8b5016d7d25b-proc\") pod \"perf-node-gather-daemonset-jrswf\" (UID: \"a123f453-9f7f-4bb0-a8b7-8b5016d7d25b\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-jrswf" Sep 14 11:44:05.274882 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:05.274838 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a123f453-9f7f-4bb0-a8b7-8b5016d7d25b-podres\") pod \"perf-node-gather-daemonset-jrswf\" (UID: \"a123f453-9f7f-4bb0-a8b7-8b5016d7d25b\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-jrswf" Sep 14 11:44:05.275047 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:05.274882 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a123f453-9f7f-4bb0-a8b7-8b5016d7d25b-sys\") pod \"perf-node-gather-daemonset-jrswf\" (UID: \"a123f453-9f7f-4bb0-a8b7-8b5016d7d25b\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-jrswf" Sep 14 11:44:05.275047 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:05.274927 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a123f453-9f7f-4bb0-a8b7-8b5016d7d25b-lib-modules\") pod \"perf-node-gather-daemonset-jrswf\" (UID: \"a123f453-9f7f-4bb0-a8b7-8b5016d7d25b\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-jrswf" Sep 14 11:44:05.275047 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:05.274968 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a123f453-9f7f-4bb0-a8b7-8b5016d7d25b-sys\") pod \"perf-node-gather-daemonset-jrswf\" (UID: \"a123f453-9f7f-4bb0-a8b7-8b5016d7d25b\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-jrswf" Sep 14 11:44:05.275047 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:05.275013 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a123f453-9f7f-4bb0-a8b7-8b5016d7d25b-podres\") pod \"perf-node-gather-daemonset-jrswf\" (UID: \"a123f453-9f7f-4bb0-a8b7-8b5016d7d25b\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-jrswf" Sep 14 11:44:05.285466 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:05.285436 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmlxd\" (UniqueName: \"kubernetes.io/projected/a123f453-9f7f-4bb0-a8b7-8b5016d7d25b-kube-api-access-rmlxd\") pod \"perf-node-gather-daemonset-jrswf\" (UID: \"a123f453-9f7f-4bb0-a8b7-8b5016d7d25b\") " pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-jrswf" Sep 14 11:44:05.368965 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:05.368874 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-jrswf" Sep 14 11:44:05.656702 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:05.656640 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zxfzq/perf-node-gather-daemonset-jrswf"] Sep 14 11:44:05.659146 ip-10-0-137-230 kubenswrapper[2574]: W0914 11:44:05.659120 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda123f453_9f7f_4bb0_a8b7_8b5016d7d25b.slice/crio-622f7aa4b3010bb64854e123e71b55af716f85c059d95763993d3381c93c213c WatchSource:0}: Error finding container 622f7aa4b3010bb64854e123e71b55af716f85c059d95763993d3381c93c213c: Status 404 returned error can't find the container with id 622f7aa4b3010bb64854e123e71b55af716f85c059d95763993d3381c93c213c Sep 14 11:44:05.668345 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:05.668321 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-jrswf" event={"ID":"a123f453-9f7f-4bb0-a8b7-8b5016d7d25b","Type":"ContainerStarted","Data":"622f7aa4b3010bb64854e123e71b55af716f85c059d95763993d3381c93c213c"} Sep 14 11:44:06.675054 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:06.675004 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-jrswf" event={"ID":"a123f453-9f7f-4bb0-a8b7-8b5016d7d25b","Type":"ContainerStarted","Data":"5cc8451751cdf42a47ff8e619a983cce3e9e76c28b6f0090621bbb0691f2dd47"} Sep 14 11:44:06.675504 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:06.675074 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-jrswf" Sep 14 11:44:06.696616 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:06.696568 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-jrswf" podStartSLOduration=1.6965529240000001 podStartE2EDuration="1.696552924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:44:06.695401498 +0000 UTC m=+2591.315614020" watchObservedRunningTime="2026-09-14 11:44:06.696552924 +0000 UTC m=+2591.316765447" Sep 14 11:44:07.068285 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:07.068196 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-78c6d7c756-btn66_8e64b36d-e059-413e-bca9-8d05a5850fd1/console/0.log" Sep 14 11:44:07.107974 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:07.107947 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-9rw9d_3b0d9445-30eb-4097-9f1c-1fb34c292559/download-server/0.log" Sep 14 11:44:09.263073 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:09.263040 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-md5h2_bc3a2da2-96cd-4397-a6f9-9e7088b04f1e/dns/0.log" Sep 14 11:44:09.305471 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:09.305448 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-md5h2_bc3a2da2-96cd-4397-a6f9-9e7088b04f1e/kube-rbac-proxy/0.log" Sep 14 11:44:09.455474 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:09.455444 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lcx2b_c961b4ad-d3a5-4032-aa9a-a96c167e3df8/dns-node-resolver/0.log" Sep 14 11:44:10.265995 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:10.265960 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-79cb87f565-swrl2_7d61446d-2f48-452e-bcc9-3f3495fc1544/registry/0.log" Sep 14 11:44:10.392385 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:10.392345 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sfnlb_978d6d43-2346-476b-9bff-274d816f1175/node-ca/0.log" Sep 14 11:44:12.054373 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:12.054338 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-86plh_477e2e5e-7397-409e-9d1c-b35652715697/discovery/0.log" Sep 14 11:44:12.141608 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:12.141581 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f95fc98f9-kzb94_9aa22333-c697-43d6-ac9f-eb93f64b8801/kube-auth-proxy/0.log" Sep 14 11:44:12.258721 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:12.258695 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-p5srl_a6ecef0e-9b32-41ab-8452-618836054a5a/payload-pre-processing/0.log" Sep 14 11:44:12.311972 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:12.311897 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-mvnvh_d4eea439-1943-4f74-9067-d1bb67a438f1/payload-processing/0.log" Sep 14 11:44:12.358023 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:12.357999 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6f6c545fc8-77jgh_2baadd06-610f-4413-97c7-04f3a5c2039b/router/0.log" Sep 14 11:44:12.691378 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:12.691344 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zxfzq/perf-node-gather-daemonset-jrswf" Sep 14 11:44:13.488761 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:13.488734 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6qbbf_4e9e0945-a3ff-449a-90a6-ddf38839b670/serve-healthcheck-canary/0.log" Sep 14 11:44:14.801156 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:14.801127 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bt4kc_8ad7a744-ff5a-49f7-9b99-e2bd045686c7/kube-rbac-proxy/0.log" Sep 14 11:44:14.973403 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:14.973330 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bt4kc_8ad7a744-ff5a-49f7-9b99-e2bd045686c7/exporter/0.log" Sep 14 11:44:15.057670 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:15.057641 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bt4kc_8ad7a744-ff5a-49f7-9b99-e2bd045686c7/extractor/0.log" Sep 14 11:44:15.254619 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:15.254544 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-86plh_477e2e5e-7397-409e-9d1c-b35652715697/discovery/0.log" Sep 14 11:44:15.388588 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:15.388563 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f95fc98f9-kzb94_9aa22333-c697-43d6-ac9f-eb93f64b8801/kube-auth-proxy/0.log" Sep 14 11:44:15.495031 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:15.494998 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-p5srl_a6ecef0e-9b32-41ab-8452-618836054a5a/payload-pre-processing/0.log" Sep 14 11:44:15.551021 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:15.550946 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-mvnvh_d4eea439-1943-4f74-9067-d1bb67a438f1/payload-processing/0.log" Sep 14 11:44:15.629896 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:15.629866 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6f6c545fc8-77jgh_2baadd06-610f-4413-97c7-04f3a5c2039b/router/0.log" Sep 14 11:44:17.653553 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:17.653461 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-86plh_477e2e5e-7397-409e-9d1c-b35652715697/discovery/0.log" Sep 14 11:44:17.716294 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:17.716255 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f95fc98f9-kzb94_9aa22333-c697-43d6-ac9f-eb93f64b8801/kube-auth-proxy/0.log" Sep 14 11:44:17.799766 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:17.799736 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-p5srl_a6ecef0e-9b32-41ab-8452-618836054a5a/payload-pre-processing/0.log" Sep 14 11:44:17.831325 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:17.831291 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-mvnvh_d4eea439-1943-4f74-9067-d1bb67a438f1/payload-processing/0.log" Sep 14 11:44:17.866853 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:17.866828 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6f6c545fc8-77jgh_2baadd06-610f-4413-97c7-04f3a5c2039b/router/0.log" Sep 14 11:44:18.605832 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:18.605804 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-796b4cc5fc-9pl9v_1eb2709a-547b-49ef-8ac6-0f05bece321b/manager/0.log" Sep 14 11:44:18.655548 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:18.655526 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-txc6g_2da68c9f-c790-4560-85bd-43dc1233bf8b/server/0.log" Sep 14 11:44:18.776343 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:18.776308 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-j8jfw_07d207b3-1e62-4a62-b090-3e56e9c10715/manager/0.log" Sep 14 11:44:19.108603 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:19.108571 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f85645c65-z2tnm_415f2f7b-4196-4d3d-b4ee-e237110a54bf/manager/0.log" Sep 14 11:44:19.230847 ip-10-0-137-230 kubenswrapper[2574]: I0914 11:44:19.230823 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f85645c65-z2tnm_415f2f7b-4196-4d3d-b4ee-e237110a54bf/copy-manifests/0.log"