Sep 14 10:48:46.164637 ip-10-0-137-53 systemd[1]: Starting Kubernetes Kubelet... Sep 14 10:48:46.577160 ip-10-0-137-53 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 14 10:48:46.577160 ip-10-0-137-53 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 14 10:48:46.577160 ip-10-0-137-53 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 14 10:48:46.577160 ip-10-0-137-53 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 14 10:48:46.577160 ip-10-0-137-53 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 14 10:48:46.580089 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.579995 2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 14 10:48:46.584272 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584255 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 10:48:46.584316 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584275 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 10:48:46.584316 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584280 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 10:48:46.584316 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584283 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 10:48:46.584316 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584286 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 10:48:46.584316 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584291 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 10:48:46.584316 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584295 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 10:48:46.584316 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584299 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 10:48:46.584316 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584302 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 10:48:46.584316 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584305 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 10:48:46.584316 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584308 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 10:48:46.584316 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584311 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 10:48:46.584316 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584314 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 10:48:46.584316 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584318 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 10:48:46.584316 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584321 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 10:48:46.584316 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584324 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 10:48:46.584733 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584327 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 10:48:46.584733 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584329 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 10:48:46.584733 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584332 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 10:48:46.584733 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584347 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 10:48:46.584733 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584350 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 10:48:46.584733 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584353 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 10:48:46.584733 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584356 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 10:48:46.584733 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584358 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 10:48:46.584733 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584361 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 10:48:46.584733 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584364 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 10:48:46.584733 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584366 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 10:48:46.584733 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584368 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 10:48:46.584733 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584371 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 10:48:46.584733 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584374 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 10:48:46.584733 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584376 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 10:48:46.584733 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584379 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 10:48:46.584733 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584382 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 10:48:46.584733 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584385 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 10:48:46.584733 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584388 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 10:48:46.584733 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584390 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 10:48:46.585196 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584393 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 10:48:46.585196 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584395 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 10:48:46.585196 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584398 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 10:48:46.585196 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584401 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 10:48:46.585196 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584403 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 10:48:46.585196 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584406 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 10:48:46.585196 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584408 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 10:48:46.585196 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584411 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 10:48:46.585196 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584414 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 10:48:46.585196 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584418 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 10:48:46.585196 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584420 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 10:48:46.585196 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584423 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 10:48:46.585196 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584425 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 10:48:46.585196 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584430 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 10:48:46.585196 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584433 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 10:48:46.585196 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584436 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 10:48:46.585196 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584438 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 10:48:46.585196 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584441 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 10:48:46.585196 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584443 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 10:48:46.585689 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584446 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 10:48:46.585689 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584448 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 10:48:46.585689 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584452 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 10:48:46.585689 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584454 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 10:48:46.585689 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584457 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 10:48:46.585689 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584459 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 10:48:46.585689 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584461 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 10:48:46.585689 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584464 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 10:48:46.585689 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584467 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 10:48:46.585689 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584470 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 10:48:46.585689 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584473 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 10:48:46.585689 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584476 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 10:48:46.585689 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584478 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 10:48:46.585689 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584482 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 10:48:46.585689 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584486 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 10:48:46.585689 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584489 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 10:48:46.585689 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584492 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 10:48:46.585689 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584495 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 10:48:46.585689 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584498 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 10:48:46.585689 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584501 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 10:48:46.586180 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584504 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 10:48:46.586180 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584506 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 10:48:46.586180 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584509 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 10:48:46.586180 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584511 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 10:48:46.586180 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584514 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 10:48:46.586180 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584516 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 10:48:46.586180 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584519 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 14 10:48:46.586180 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584522 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 10:48:46.586180 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584524 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 10:48:46.586180 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584527 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 10:48:46.586180 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584529 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 10:48:46.586180 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584532 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 10:48:46.586180 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584534 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 10:48:46.586180 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584537 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 10:48:46.586180 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584539 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 10:48:46.586180 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584542 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 10:48:46.586180 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584545 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 10:48:46.586180 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584547 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 10:48:46.586180 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584550 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 10:48:46.586180 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584552 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 10:48:46.586681 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584555 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 10:48:46.586681 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584557 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 10:48:46.586681 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584560 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 10:48:46.586681 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.584564 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 10:48:46.586681 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585304 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 10:48:46.586681 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585311 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 10:48:46.586681 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585314 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 10:48:46.586681 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585316 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 10:48:46.586681 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585319 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 10:48:46.586681 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585322 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 10:48:46.586681 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585324 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 10:48:46.586681 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585327 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 10:48:46.586681 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585330 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 10:48:46.586681 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585333 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 10:48:46.586681 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585351 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 10:48:46.586681 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585353 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 10:48:46.586681 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585356 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 10:48:46.586681 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585360 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 10:48:46.586681 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585365 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 10:48:46.587144 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585368 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 10:48:46.587144 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585371 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 10:48:46.587144 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585374 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 10:48:46.587144 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585377 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 10:48:46.587144 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585380 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 10:48:46.587144 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585382 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 10:48:46.587144 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585385 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 10:48:46.587144 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585387 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 10:48:46.587144 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585390 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 10:48:46.587144 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585392 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 10:48:46.587144 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585395 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 10:48:46.587144 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585397 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 10:48:46.587144 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585400 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 10:48:46.587144 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585402 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 10:48:46.587144 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585405 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 10:48:46.587144 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585407 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 10:48:46.587144 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585410 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 10:48:46.587144 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585413 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 10:48:46.587144 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585416 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 10:48:46.587144 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585418 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 10:48:46.587638 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585421 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 10:48:46.587638 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585423 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 10:48:46.587638 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585426 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 10:48:46.587638 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585428 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 10:48:46.587638 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585431 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 10:48:46.587638 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585433 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 10:48:46.587638 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585436 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 10:48:46.587638 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585438 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 10:48:46.587638 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585440 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 10:48:46.587638 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585443 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 10:48:46.587638 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585445 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 10:48:46.587638 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585448 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 10:48:46.587638 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585451 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 10:48:46.587638 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585454 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 10:48:46.587638 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585456 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 10:48:46.587638 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585459 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 10:48:46.587638 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585461 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 10:48:46.587638 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585464 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 10:48:46.587638 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585466 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 10:48:46.587638 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585469 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 10:48:46.588124 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585472 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 10:48:46.588124 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585475 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 10:48:46.588124 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585477 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 10:48:46.588124 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585481 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 10:48:46.588124 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585485 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 10:48:46.588124 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585488 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 10:48:46.588124 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585490 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 10:48:46.588124 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585493 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 10:48:46.588124 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585496 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 10:48:46.588124 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585499 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 10:48:46.588124 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585502 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 10:48:46.588124 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585504 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 10:48:46.588124 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585507 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 10:48:46.588124 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585509 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 10:48:46.588124 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585512 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 10:48:46.588124 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585514 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 10:48:46.588124 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585517 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 10:48:46.588124 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585520 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 10:48:46.588124 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585522 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 10:48:46.588124 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585525 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 10:48:46.588647 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585527 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 10:48:46.588647 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585529 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 10:48:46.588647 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585532 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 10:48:46.588647 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585534 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 10:48:46.588647 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585538 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 10:48:46.588647 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585540 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 10:48:46.588647 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585542 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 10:48:46.588647 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585545 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 10:48:46.588647 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585547 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 10:48:46.588647 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585550 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 10:48:46.588647 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585552 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 14 10:48:46.588647 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585555 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 10:48:46.588647 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585557 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 10:48:46.588647 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585559 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 10:48:46.588647 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585562 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 10:48:46.588647 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585565 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 10:48:46.588647 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585567 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 10:48:46.588647 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585569 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 10:48:46.588647 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585572 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 10:48:46.588647 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585578 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 10:48:46.589127 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585582 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 10:48:46.589127 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585585 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 10:48:46.589127 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585588 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 10:48:46.589127 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.585591 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 10:48:46.589127 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586045 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 10:48:46.589127 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586053 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 10:48:46.589127 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586057 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 10:48:46.589127 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586060 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 10:48:46.589127 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586063 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 10:48:46.589127 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586066 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 10:48:46.589127 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586069 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 10:48:46.589127 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586072 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 10:48:46.589127 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586075 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 10:48:46.589127 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586078 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 10:48:46.589127 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586080 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 10:48:46.589127 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586083 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 10:48:46.589127 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586085 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 10:48:46.589127 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586088 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 10:48:46.589127 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586090 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 10:48:46.589127 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586093 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 10:48:46.589637 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586095 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 10:48:46.589637 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586098 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 10:48:46.589637 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586100 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 10:48:46.589637 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586103 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 10:48:46.589637 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586105 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 10:48:46.589637 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586107 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 10:48:46.589637 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586110 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 10:48:46.589637 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586112 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 10:48:46.589637 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586115 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 10:48:46.589637 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586117 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 10:48:46.589637 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586120 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 10:48:46.589637 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586124 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 10:48:46.589637 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586127 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 10:48:46.589637 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586129 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 10:48:46.589637 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586132 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 10:48:46.589637 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586135 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 10:48:46.589637 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586137 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 10:48:46.589637 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586140 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 10:48:46.589637 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586142 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 10:48:46.590137 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586146 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 10:48:46.590137 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586149 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 10:48:46.590137 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586152 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 10:48:46.590137 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586155 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 10:48:46.590137 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586158 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 10:48:46.590137 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586160 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 10:48:46.590137 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586162 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 10:48:46.590137 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586165 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 10:48:46.590137 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586167 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 10:48:46.590137 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586171 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 10:48:46.590137 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586173 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 10:48:46.590137 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586175 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 10:48:46.590137 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586178 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 10:48:46.590137 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586180 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 10:48:46.590137 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586184 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 10:48:46.590137 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586188 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 10:48:46.590137 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586191 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 10:48:46.590137 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586193 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 10:48:46.590137 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586196 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 10:48:46.590616 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586199 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 10:48:46.590616 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586202 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 10:48:46.590616 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586205 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 10:48:46.590616 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586207 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 10:48:46.590616 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586210 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 10:48:46.590616 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586214 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 10:48:46.590616 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586216 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 10:48:46.590616 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586219 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 10:48:46.590616 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586221 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 10:48:46.590616 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586224 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 10:48:46.590616 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586232 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 10:48:46.590616 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586235 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 10:48:46.590616 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586238 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 10:48:46.590616 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586240 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 10:48:46.590616 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586243 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 10:48:46.590616 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586245 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 10:48:46.590616 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586248 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 10:48:46.590616 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586250 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 10:48:46.590616 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586253 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 10:48:46.590616 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586255 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 10:48:46.591147 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586258 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 10:48:46.591147 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586260 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 10:48:46.591147 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586263 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 10:48:46.591147 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586266 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 10:48:46.591147 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586269 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 10:48:46.591147 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586271 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 10:48:46.591147 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586274 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 10:48:46.591147 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586276 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 10:48:46.591147 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586279 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 10:48:46.591147 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586281 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 10:48:46.591147 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586284 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 10:48:46.591147 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586286 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 10:48:46.591147 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586288 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 10:48:46.591147 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586291 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 10:48:46.591147 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586293 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 10:48:46.591147 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586295 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 10:48:46.591147 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586298 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 14 10:48:46.591147 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586302 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 10:48:46.591147 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586305 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 10:48:46.591147 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586307 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 10:48:46.591643 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586310 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 10:48:46.591643 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586312 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 10:48:46.591643 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586315 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 10:48:46.591643 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586318 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 10:48:46.591643 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.586320 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 10:48:46.591643 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586411 2581 flags.go:64] FLAG: --address="0.0.0.0" Sep 14 10:48:46.591643 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586449 2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 14 10:48:46.591643 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586456 2581 flags.go:64] FLAG: --anonymous-auth="true" Sep 14 10:48:46.591643 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586461 2581 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 14 10:48:46.591643 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586465 2581 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 14 10:48:46.591643 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586468 2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 14 10:48:46.591643 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586475 2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 14 10:48:46.591643 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586482 2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 14 10:48:46.591643 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586487 2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 14 10:48:46.591643 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586492 2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 14 10:48:46.591643 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586497 2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 14 10:48:46.591643 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586503 2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 14 10:48:46.591643 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586507 2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 14 10:48:46.591643 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586510 2581 flags.go:64] FLAG: --cgroup-root="" Sep 14 10:48:46.591643 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586513 2581 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 14 10:48:46.591643 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586516 2581 flags.go:64] FLAG: --client-ca-file="" Sep 14 10:48:46.591643 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586519 2581 flags.go:64] FLAG: --cloud-provider="external" Sep 14 10:48:46.591643 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586522 2581 flags.go:64] FLAG: --cluster-dns="[]" Sep 14 10:48:46.591643 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586528 2581 flags.go:64] FLAG: --cluster-domain="" Sep 14 10:48:46.592207 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586530 2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 14 10:48:46.592207 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586533 2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 14 10:48:46.592207 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586536 2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 14 10:48:46.592207 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586540 2581 flags.go:64] FLAG: --container-log-max-files="5" Sep 14 10:48:46.592207 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586544 2581 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 14 10:48:46.592207 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586549 2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 14 10:48:46.592207 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586552 2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 14 10:48:46.592207 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586555 2581 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 14 10:48:46.592207 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586558 2581 flags.go:64] FLAG: --contention-profiling="false" Sep 14 10:48:46.592207 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586561 2581 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 14 10:48:46.592207 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586564 2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 14 10:48:46.592207 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586567 2581 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 14 10:48:46.592207 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586570 2581 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 14 10:48:46.592207 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586574 2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 14 10:48:46.592207 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586579 2581 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 14 10:48:46.592207 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586582 2581 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 14 10:48:46.592207 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586585 2581 flags.go:64] FLAG: --enable-load-reader="false" Sep 14 10:48:46.592207 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586588 2581 flags.go:64] FLAG: --enable-server="true" Sep 14 10:48:46.592207 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586590 2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 14 10:48:46.592207 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586595 2581 flags.go:64] FLAG: --event-burst="100" Sep 14 10:48:46.592207 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586598 2581 flags.go:64] FLAG: --event-qps="50" Sep 14 10:48:46.592207 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586601 2581 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 14 10:48:46.592207 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586604 2581 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 14 10:48:46.592207 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586607 2581 flags.go:64] FLAG: --eviction-hard="" Sep 14 10:48:46.592791 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586611 2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 14 10:48:46.592791 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586614 2581 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 14 10:48:46.592791 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586616 2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 14 10:48:46.592791 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586619 2581 flags.go:64] FLAG: --eviction-soft="" Sep 14 10:48:46.592791 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586622 2581 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 14 10:48:46.592791 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586625 2581 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 14 10:48:46.592791 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586628 2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 14 10:48:46.592791 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586631 2581 flags.go:64] FLAG: --experimental-mounter-path="" Sep 14 10:48:46.592791 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586634 2581 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 14 10:48:46.592791 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586637 2581 flags.go:64] FLAG: --fail-swap-on="true" Sep 14 10:48:46.592791 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586639 2581 flags.go:64] FLAG: --feature-gates="" Sep 14 10:48:46.592791 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586643 2581 flags.go:64] FLAG: --file-check-frequency="20s" Sep 14 10:48:46.592791 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586646 2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 14 10:48:46.592791 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586651 2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 14 10:48:46.592791 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586654 2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 14 10:48:46.592791 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586657 2581 flags.go:64] FLAG: --healthz-port="10248" Sep 14 10:48:46.592791 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586660 2581 flags.go:64] FLAG: --help="false" Sep 14 10:48:46.592791 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586663 2581 flags.go:64] FLAG: --hostname-override="ip-10-0-137-53.ec2.internal" Sep 14 10:48:46.592791 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586666 2581 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 14 10:48:46.592791 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586669 2581 flags.go:64] FLAG: --http-check-frequency="20s" Sep 14 10:48:46.592791 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586672 2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 14 10:48:46.592791 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586675 2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 14 10:48:46.592791 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586680 2581 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 14 10:48:46.593382 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586683 2581 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 14 10:48:46.593382 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586686 2581 flags.go:64] FLAG: --image-service-endpoint="" Sep 14 10:48:46.593382 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586689 2581 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 14 10:48:46.593382 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586691 2581 flags.go:64] FLAG: --kube-api-burst="100" Sep 14 10:48:46.593382 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586694 2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 14 10:48:46.593382 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586697 2581 flags.go:64] FLAG: --kube-api-qps="50" Sep 14 10:48:46.593382 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586700 2581 flags.go:64] FLAG: --kube-reserved="" Sep 14 10:48:46.593382 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586703 2581 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 14 10:48:46.593382 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586706 2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 14 10:48:46.593382 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586709 2581 flags.go:64] FLAG: --kubelet-cgroups="" Sep 14 10:48:46.593382 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586712 2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 14 10:48:46.593382 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586715 2581 flags.go:64] FLAG: --lock-file="" Sep 14 10:48:46.593382 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586717 2581 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 14 10:48:46.593382 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586720 2581 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 14 10:48:46.593382 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586723 2581 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 14 10:48:46.593382 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586729 2581 flags.go:64] FLAG: --log-json-split-stream="false" Sep 14 10:48:46.593382 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586732 2581 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 14 10:48:46.593382 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586734 2581 flags.go:64] FLAG: --log-text-split-stream="false" Sep 14 10:48:46.593382 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586737 2581 flags.go:64] FLAG: --logging-format="text" Sep 14 10:48:46.593382 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586740 2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 14 10:48:46.593382 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586743 2581 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 14 10:48:46.593382 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586746 2581 flags.go:64] FLAG: --manifest-url="" Sep 14 10:48:46.593382 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586749 2581 flags.go:64] FLAG: --manifest-url-header="" Sep 14 10:48:46.593382 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586753 2581 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 14 10:48:46.593382 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586764 2581 flags.go:64] FLAG: --max-open-files="1000000" Sep 14 10:48:46.593988 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586769 2581 flags.go:64] FLAG: --max-pods="110" Sep 14 10:48:46.593988 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586771 2581 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 14 10:48:46.593988 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586774 2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 14 10:48:46.593988 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586777 2581 flags.go:64] FLAG: --memory-manager-policy="None" Sep 14 10:48:46.593988 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586780 2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 14 10:48:46.593988 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586783 2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 14 10:48:46.593988 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586787 2581 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 14 10:48:46.593988 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586790 2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 14 10:48:46.593988 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586797 2581 flags.go:64] FLAG: --node-status-max-images="50" Sep 14 10:48:46.593988 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586800 2581 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 14 10:48:46.593988 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586803 2581 flags.go:64] FLAG: --oom-score-adj="-999" Sep 14 10:48:46.593988 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586806 2581 flags.go:64] FLAG: --pod-cidr="" Sep 14 10:48:46.593988 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586809 2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 14 10:48:46.593988 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586815 2581 flags.go:64] FLAG: --pod-manifest-path="" Sep 14 10:48:46.593988 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586818 2581 flags.go:64] FLAG: --pod-max-pids="-1" Sep 14 10:48:46.593988 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586821 2581 flags.go:64] FLAG: --pods-per-core="0" Sep 14 10:48:46.593988 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586824 2581 flags.go:64] FLAG: --port="10250" Sep 14 10:48:46.593988 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586827 2581 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 14 10:48:46.593988 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586830 2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0207a10dc563e27b7" Sep 14 10:48:46.593988 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586833 2581 flags.go:64] FLAG: --qos-reserved="" Sep 14 10:48:46.593988 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586836 2581 flags.go:64] FLAG: --read-only-port="10255" Sep 14 10:48:46.593988 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586839 2581 flags.go:64] FLAG: --register-node="true" Sep 14 10:48:46.593988 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586842 2581 flags.go:64] FLAG: --register-with-taints="" Sep 14 10:48:46.593988 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586846 2581 flags.go:64] FLAG: --registry-burst="10" Sep 14 10:48:46.594708 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586849 2581 flags.go:64] FLAG: --registry-qps="5" Sep 14 10:48:46.594708 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586852 2581 flags.go:64] FLAG: --reserved-cpus="" Sep 14 10:48:46.594708 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586855 2581 flags.go:64] FLAG: --reserved-memory="" Sep 14 10:48:46.594708 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586858 2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 14 10:48:46.594708 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586862 2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 14 10:48:46.594708 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586865 2581 flags.go:64] FLAG: --rotate-certificates="false" Sep 14 10:48:46.594708 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586868 2581 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 14 10:48:46.594708 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586871 2581 flags.go:64] FLAG: --runonce="false" Sep 14 10:48:46.594708 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586874 2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 14 10:48:46.594708 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586877 2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 14 10:48:46.594708 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586879 2581 flags.go:64] FLAG: --seccomp-default="false" Sep 14 10:48:46.594708 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586882 2581 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 14 10:48:46.594708 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586885 2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 14 10:48:46.594708 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586888 2581 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 14 10:48:46.594708 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586891 2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 14 10:48:46.594708 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586896 2581 flags.go:64] FLAG: --storage-driver-password="root" Sep 14 10:48:46.594708 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586899 2581 flags.go:64] FLAG: --storage-driver-secure="false" Sep 14 10:48:46.594708 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586902 2581 flags.go:64] FLAG: --storage-driver-table="stats" Sep 14 10:48:46.594708 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586905 2581 flags.go:64] FLAG: --storage-driver-user="root" Sep 14 10:48:46.594708 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586908 2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 14 10:48:46.594708 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586910 2581 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 14 10:48:46.594708 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586913 2581 flags.go:64] FLAG: --system-cgroups="" Sep 14 10:48:46.594708 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586917 2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 14 10:48:46.594708 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586922 2581 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 14 10:48:46.594708 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586925 2581 flags.go:64] FLAG: --tls-cert-file="" Sep 14 10:48:46.595305 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586927 2581 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 14 10:48:46.595305 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586932 2581 flags.go:64] FLAG: --tls-min-version="" Sep 14 10:48:46.595305 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586935 2581 flags.go:64] FLAG: --tls-private-key-file="" Sep 14 10:48:46.595305 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586938 2581 flags.go:64] FLAG: --topology-manager-policy="none" Sep 14 10:48:46.595305 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586941 2581 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 14 10:48:46.595305 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586944 2581 flags.go:64] FLAG: --topology-manager-scope="container" Sep 14 10:48:46.595305 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586946 2581 flags.go:64] FLAG: --v="2" Sep 14 10:48:46.595305 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586950 2581 flags.go:64] FLAG: --version="false" Sep 14 10:48:46.595305 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586955 2581 flags.go:64] FLAG: --vmodule="" Sep 14 10:48:46.595305 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586958 2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 14 10:48:46.595305 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.586962 2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 14 10:48:46.595305 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587080 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 10:48:46.595305 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587084 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 10:48:46.595305 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587087 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 10:48:46.595305 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587090 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 10:48:46.595305 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587093 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 10:48:46.595305 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587095 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 10:48:46.595305 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587100 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 10:48:46.595305 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587104 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 10:48:46.595305 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587107 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 10:48:46.595305 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587110 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 10:48:46.596077 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587114 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 10:48:46.596077 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587118 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 10:48:46.596077 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587121 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 10:48:46.596077 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587124 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 10:48:46.596077 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587127 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 10:48:46.596077 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587130 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 10:48:46.596077 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587133 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 10:48:46.596077 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587136 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 10:48:46.596077 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587138 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 10:48:46.596077 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587141 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 10:48:46.596077 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587144 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 10:48:46.596077 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587146 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 10:48:46.596077 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587149 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 10:48:46.596077 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587152 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 10:48:46.596077 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587155 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 10:48:46.596077 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587157 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 10:48:46.596077 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587160 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 10:48:46.596077 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587163 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 10:48:46.596077 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587165 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 10:48:46.596738 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587168 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 10:48:46.596738 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587170 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 10:48:46.596738 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587173 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 10:48:46.596738 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587176 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 10:48:46.596738 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587178 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 10:48:46.596738 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587181 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 10:48:46.596738 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587183 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 10:48:46.596738 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587187 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 10:48:46.596738 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587191 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 10:48:46.596738 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587193 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 10:48:46.596738 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587196 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 10:48:46.596738 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587199 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 10:48:46.596738 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587202 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 10:48:46.596738 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587204 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 10:48:46.596738 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587208 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 10:48:46.596738 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587211 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 10:48:46.596738 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587214 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 10:48:46.596738 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587217 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 10:48:46.596738 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587219 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 10:48:46.596738 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587222 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 10:48:46.597238 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587224 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 10:48:46.597238 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587227 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 10:48:46.597238 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587229 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 10:48:46.597238 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587232 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 10:48:46.597238 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587234 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 10:48:46.597238 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587237 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 10:48:46.597238 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587240 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 10:48:46.597238 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587242 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 10:48:46.597238 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587245 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 10:48:46.597238 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587247 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 10:48:46.597238 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587250 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 10:48:46.597238 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587252 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 10:48:46.597238 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587255 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 10:48:46.597238 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587257 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 10:48:46.597238 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587260 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 10:48:46.597238 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587262 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 10:48:46.597238 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587266 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 10:48:46.597238 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587269 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 10:48:46.597238 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587271 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 10:48:46.597238 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587274 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 10:48:46.597807 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587276 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 10:48:46.597807 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587279 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 10:48:46.597807 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587281 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 10:48:46.597807 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587284 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 10:48:46.597807 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587286 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 10:48:46.597807 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587289 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 10:48:46.597807 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587293 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 10:48:46.597807 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587295 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 10:48:46.597807 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587298 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 10:48:46.597807 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587301 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 10:48:46.597807 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587303 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 10:48:46.597807 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587306 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 10:48:46.597807 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587309 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 10:48:46.597807 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587311 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 10:48:46.597807 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587314 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 10:48:46.597807 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587316 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 10:48:46.597807 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587319 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 10:48:46.597807 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587321 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 10:48:46.597807 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587324 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 10:48:46.597807 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587326 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 10:48:46.598300 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587329 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 10:48:46.598300 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587353 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 10:48:46.598300 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587356 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 10:48:46.598300 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587359 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 10:48:46.598300 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587361 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 10:48:46.598300 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587364 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 10:48:46.598300 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587367 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 10:48:46.598300 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587370 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 10:48:46.598300 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587373 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 14 10:48:46.598300 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.587376 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 10:48:46.598300 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.588106 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 14 10:48:46.598300 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.597474 2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 14 10:48:46.598300 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.597493 2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 14 10:48:46.598671 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.598469 2581 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 14 10:48:46.598671 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.598478 2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 14 10:48:46.598671 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598535 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 10:48:46.598671 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598540 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 10:48:46.598671 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598543 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 10:48:46.598671 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598546 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 10:48:46.598671 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598549 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 10:48:46.598671 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598552 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 10:48:46.598671 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598555 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 10:48:46.598671 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598558 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 10:48:46.598671 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598561 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 10:48:46.598671 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598563 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 10:48:46.598671 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598566 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 10:48:46.598671 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598570 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 10:48:46.598671 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598574 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 10:48:46.598671 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598578 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 10:48:46.598671 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598581 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 10:48:46.598671 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598583 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 10:48:46.598671 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598586 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 10:48:46.599122 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598589 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 10:48:46.599122 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598591 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 10:48:46.599122 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598594 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 10:48:46.599122 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598597 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 10:48:46.599122 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598599 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 10:48:46.599122 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598602 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 10:48:46.599122 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598605 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 10:48:46.599122 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598608 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 10:48:46.599122 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598610 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 10:48:46.599122 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598613 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 10:48:46.599122 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598615 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 10:48:46.599122 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598619 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 10:48:46.599122 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598621 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 10:48:46.599122 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598624 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 10:48:46.599122 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598627 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 10:48:46.599122 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598629 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 10:48:46.599122 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598632 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 10:48:46.599122 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598634 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 10:48:46.599122 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598637 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 10:48:46.599122 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598640 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 10:48:46.599703 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598642 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 10:48:46.599703 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598645 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 10:48:46.599703 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598647 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 10:48:46.599703 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598650 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 10:48:46.599703 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598652 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 10:48:46.599703 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598655 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 10:48:46.599703 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598657 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 10:48:46.599703 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598660 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 10:48:46.599703 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598662 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 10:48:46.599703 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598665 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 10:48:46.599703 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598667 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 10:48:46.599703 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598670 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 10:48:46.599703 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598673 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 10:48:46.599703 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598675 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 14 10:48:46.599703 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598678 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 10:48:46.599703 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598681 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 10:48:46.599703 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598683 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 10:48:46.599703 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598687 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 10:48:46.599703 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598689 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 10:48:46.599703 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598692 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 10:48:46.600184 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598694 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 10:48:46.600184 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598697 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 10:48:46.600184 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598699 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 10:48:46.600184 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598702 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 10:48:46.600184 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598705 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 10:48:46.600184 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598708 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 10:48:46.600184 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598712 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 10:48:46.600184 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598716 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 10:48:46.600184 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598719 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 10:48:46.600184 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598721 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 10:48:46.600184 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598724 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 10:48:46.600184 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598727 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 10:48:46.600184 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598729 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 10:48:46.600184 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598732 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 10:48:46.600184 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598734 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 10:48:46.600184 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598737 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 10:48:46.600184 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598739 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 10:48:46.600184 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598742 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 10:48:46.600184 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598744 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 10:48:46.600184 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598747 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 10:48:46.600674 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598749 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 10:48:46.600674 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598753 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 10:48:46.600674 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598757 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 10:48:46.600674 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598760 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 10:48:46.600674 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598763 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 10:48:46.600674 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598766 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 10:48:46.600674 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598769 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 10:48:46.600674 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598771 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 10:48:46.600674 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598774 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 10:48:46.600674 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598777 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 10:48:46.600674 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598779 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 10:48:46.600674 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598782 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 10:48:46.600674 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598785 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 10:48:46.600674 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598787 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 10:48:46.600674 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598790 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 10:48:46.600674 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598792 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 10:48:46.600674 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598795 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 10:48:46.600674 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598797 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 10:48:46.600674 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598800 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 10:48:46.600674 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598803 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 10:48:46.601163 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598805 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 10:48:46.601163 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598808 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 10:48:46.601163 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.598813 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 14 10:48:46.601163 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598926 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 10:48:46.601163 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598931 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 10:48:46.601163 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598934 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 10:48:46.601163 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598937 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 10:48:46.601163 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598939 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 10:48:46.601163 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598942 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 10:48:46.601163 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598946 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 10:48:46.601163 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598951 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 10:48:46.601163 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598954 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 10:48:46.601163 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598957 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 10:48:46.601163 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598960 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 10:48:46.601529 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598962 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 10:48:46.601529 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598966 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 10:48:46.601529 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598968 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 10:48:46.601529 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598971 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 10:48:46.601529 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598974 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 10:48:46.601529 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598977 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 10:48:46.601529 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598979 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 10:48:46.601529 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598982 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 10:48:46.601529 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598984 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 10:48:46.601529 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598986 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 10:48:46.601529 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598989 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 10:48:46.601529 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598991 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 10:48:46.601529 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598994 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 10:48:46.601529 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598996 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 10:48:46.601529 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.598999 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 10:48:46.601529 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599002 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 10:48:46.601529 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599004 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 10:48:46.601529 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599006 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 10:48:46.601529 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599009 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 10:48:46.601529 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599012 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 10:48:46.602003 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599014 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 10:48:46.602003 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599017 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 10:48:46.602003 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599019 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 10:48:46.602003 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599022 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 10:48:46.602003 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599024 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 10:48:46.602003 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599028 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 10:48:46.602003 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599030 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 10:48:46.602003 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599032 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 10:48:46.602003 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599035 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 10:48:46.602003 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599037 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 10:48:46.602003 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599039 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 10:48:46.602003 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599042 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 10:48:46.602003 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599045 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 10:48:46.602003 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599048 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 10:48:46.602003 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599052 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 10:48:46.602003 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599055 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 10:48:46.602003 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599058 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 10:48:46.602003 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599060 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 10:48:46.602003 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599063 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 10:48:46.602003 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599066 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 10:48:46.602555 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599068 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 10:48:46.602555 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599071 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 10:48:46.602555 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599073 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 10:48:46.602555 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599076 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 10:48:46.602555 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599078 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 10:48:46.602555 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599081 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 10:48:46.602555 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599083 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 10:48:46.602555 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599086 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 10:48:46.602555 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599088 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 10:48:46.602555 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599091 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 10:48:46.602555 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599093 2581 feature_gate.go:328] unrecognized feature gate: Example Sep 14 10:48:46.602555 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599097 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 10:48:46.602555 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599100 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 10:48:46.602555 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599103 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 10:48:46.602555 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599106 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 10:48:46.602555 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599109 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 10:48:46.602555 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599111 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 10:48:46.602555 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599114 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 10:48:46.602555 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599116 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 10:48:46.602555 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599119 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 10:48:46.603065 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599121 2581 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 10:48:46.603065 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599124 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 10:48:46.603065 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599126 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 10:48:46.603065 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599129 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 10:48:46.603065 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599131 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 10:48:46.603065 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599134 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 10:48:46.603065 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599137 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 10:48:46.603065 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599140 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 10:48:46.603065 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599142 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 10:48:46.603065 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599145 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 10:48:46.603065 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599147 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 10:48:46.603065 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599149 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 10:48:46.603065 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599152 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 10:48:46.603065 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599154 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 10:48:46.603065 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599156 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 10:48:46.603065 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599159 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 10:48:46.603065 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599162 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 10:48:46.603065 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599164 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 10:48:46.603065 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599166 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 10:48:46.603065 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599169 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 10:48:46.603558 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599171 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 10:48:46.603558 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599174 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 10:48:46.603558 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599176 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 10:48:46.603558 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599179 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 10:48:46.603558 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599181 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 10:48:46.603558 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599183 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 10:48:46.603558 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599186 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 10:48:46.603558 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:46.599191 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 10:48:46.603558 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.599196 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 14 10:48:46.603558 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.599316 2581 server.go:962] "Client rotation is on, will bootstrap in background" Sep 14 10:48:46.603558 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.602802 2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 14 10:48:46.603912 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.603901 2581 server.go:1019] "Starting client certificate rotation" Sep 14 10:48:46.604018 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.604000 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 14 10:48:46.604054 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.604038 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 14 10:48:46.606808 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.606795 2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 14 10:48:46.608014 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.607997 2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 14 10:48:46.621213 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.621193 2581 log.go:25] "Validated CRI v1 runtime API" Sep 14 10:48:46.625549 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.625534 2581 log.go:25] "Validated CRI v1 image API" Sep 14 10:48:46.626512 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.626477 2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 14 10:48:46.630861 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.630841 2581 fs.go:135] Filesystem UUIDs: map[2585da33-c462-4aef-b9bf-52b863f35bf6:/dev/nvme0n1p4 6411ee7e-4abf-43ab-8307-511d2e8f1d27:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Sep 14 10:48:46.630932 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.630861 2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 14 10:48:46.632653 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.632635 2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 14 10:48:46.638634 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.638515 2581 manager.go:216] Machine: {Timestamp:2026-09-14 10:48:46.636980515 +0000 UTC m=+0.379382476 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3092085 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2cc0edbea1de35fd61c9a76a9d89c7 SystemUUID:ec2cc0ed-bea1-de35-fd61-c9a76a9d89c7 BootID:5093db9b-75e5-45c0-a19d-f38f53cd2204 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f4:76:20:39 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f4:76:20:39 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ca:8c:82:c6:e8:45 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 14 10:48:46.638634 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.638619 2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 14 10:48:46.638776 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.638709 2581 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 14 10:48:46.642405 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.642378 2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 14 10:48:46.642545 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.642409 2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-53.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 14 10:48:46.644005 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.643995 2581 topology_manager.go:138] "Creating topology manager with none policy" Sep 14 10:48:46.644042 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.644007 2581 container_manager_linux.go:309] "Creating device plugin manager" Sep 14 10:48:46.644042 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.644015 2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 14 10:48:46.644836 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.644826 2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 14 10:48:46.646114 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.646103 2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 14 10:48:46.650752 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.650740 2581 state_mem.go:36] "Initialized new in-memory state store" Sep 14 10:48:46.650872 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.650863 2581 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 14 10:48:46.650902 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.650894 2581 kubelet.go:486] "Attempting to sync node with API server" Sep 14 10:48:46.650929 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.650904 2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 14 10:48:46.650929 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.650919 2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 14 10:48:46.650929 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.650928 2581 kubelet.go:398] "Adding apiserver pod source" Sep 14 10:48:46.651020 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.650937 2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 14 10:48:46.651494 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.651481 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 14 10:48:46.651543 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.651504 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 14 10:48:46.654067 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.654048 2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 14 10:48:46.656943 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.656929 2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 14 10:48:46.657021 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.656960 2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 14 10:48:46.659410 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:46.659390 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 14 10:48:46.659410 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:46.659392 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-53.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 14 10:48:46.659727 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.659715 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 14 10:48:46.659758 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.659735 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 14 10:48:46.659758 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.659742 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 14 10:48:46.659758 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.659747 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 14 10:48:46.659758 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.659756 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 14 10:48:46.659861 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.659761 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 14 10:48:46.659861 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.659775 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 14 10:48:46.659861 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.659783 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 14 10:48:46.659861 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.659790 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 14 10:48:46.659861 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.659795 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 14 10:48:46.659861 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.659803 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 14 10:48:46.659861 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.659808 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 14 10:48:46.661607 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.661595 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 14 10:48:46.661643 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.661609 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 14 10:48:46.662294 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.662280 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g24xz" Sep 14 10:48:46.664568 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.664554 2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-53.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 14 10:48:46.665453 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.665441 2581 server.go:1268] "Started kubelet" Sep 14 10:48:46.665578 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.665521 2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 14 10:48:46.665703 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.665612 2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 14 10:48:46.665758 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.665746 2581 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 14 10:48:46.667417 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.667387 2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 14 10:48:46.667868 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.667854 2581 server.go:310] "Adding debug handlers to kubelet server" Sep 14 10:48:46.667918 ip-10-0-137-53 systemd[1]: Started Kubernetes Kubelet. Sep 14 10:48:46.670606 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.670586 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g24xz" Sep 14 10:48:46.671274 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:46.671252 2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 14 10:48:46.677869 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:46.676316 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-53.ec2.internal.18d52a4a8f0a6d3e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-53.ec2.internal,UID:ip-10-0-137-53.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-53.ec2.internal,},FirstTimestamp:2026-09-14 10:48:46.665411902 +0000 UTC m=+0.407813863,LastTimestamp:2026-09-14 10:48:46.665411902 +0000 UTC m=+0.407813863,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-53.ec2.internal,}" Sep 14 10:48:46.678470 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.678454 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 14 10:48:46.679136 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.679121 2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 14 10:48:46.679252 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:46.679223 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-53.ec2.internal\" not found" Sep 14 10:48:46.679913 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.679886 2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 14 10:48:46.679913 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.679896 2581 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 14 10:48:46.680051 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.679923 2581 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 14 10:48:46.680051 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.680016 2581 reconstruct.go:97] "Volume reconstruction finished" Sep 14 10:48:46.680051 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.680027 2581 reconciler.go:29] "Reconciler: start to sync state" Sep 14 10:48:46.680252 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.680240 2581 factory.go:153] Registering CRI-O factory Sep 14 10:48:46.680296 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.680256 2581 factory.go:223] Registration of the crio container factory successfully Sep 14 10:48:46.680350 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.680298 2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 14 10:48:46.680350 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.680304 2581 factory.go:55] Registering systemd factory Sep 14 10:48:46.680350 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.680309 2581 factory.go:223] Registration of the systemd container factory successfully Sep 14 10:48:46.680350 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.680322 2581 factory.go:103] Registering Raw factory Sep 14 10:48:46.680350 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.680331 2581 manager.go:1196] Started watching for new ooms in manager Sep 14 10:48:46.680882 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.680870 2581 manager.go:349] Starting recovery of all containers Sep 14 10:48:46.692314 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.692253 2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 10:48:46.692771 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.692760 2581 manager.go:354] Recovery completed Sep 14 10:48:46.694711 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:46.694691 2581 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-137-53.ec2.internal\" not found" node="ip-10-0-137-53.ec2.internal" Sep 14 10:48:46.697002 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.696989 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 10:48:46.699066 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.699047 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-53.ec2.internal" event="NodeHasSufficientMemory" Sep 14 10:48:46.699131 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.699083 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-53.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 10:48:46.699131 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.699094 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-53.ec2.internal" event="NodeHasSufficientPID" Sep 14 10:48:46.699613 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.699601 2581 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 14 10:48:46.699613 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.699614 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 10:48:46.699702 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.699612 2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 14 10:48:46.699702 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.699644 2581 state_mem.go:36] "Initialized new in-memory state store" Sep 14 10:48:46.701437 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.701423 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-53.ec2.internal" event="NodeHasSufficientMemory" Sep 14 10:48:46.701494 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.701448 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-53.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 10:48:46.701494 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.701459 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-53.ec2.internal" event="NodeHasSufficientPID" Sep 14 10:48:46.701736 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.701725 2581 policy_none.go:49] "None policy: Start" Sep 14 10:48:46.701771 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.701741 2581 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 14 10:48:46.701771 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.701751 2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 14 10:48:46.703797 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.703669 2581 policy_none.go:47] "Start" Sep 14 10:48:46.745191 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.745172 2581 manager.go:337] "Starting Device Plugin manager" Sep 14 10:48:46.764864 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:46.745220 2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 14 10:48:46.764864 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.745231 2581 server.go:87] "Starting device plugin registration server" Sep 14 10:48:46.764864 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.745675 2581 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 14 10:48:46.764864 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.745693 2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 14 10:48:46.764864 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.745815 2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 14 10:48:46.764864 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.745883 2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 14 10:48:46.764864 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.745889 2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 14 10:48:46.764864 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:46.746955 2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 14 10:48:46.764864 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:46.747006 2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-53.ec2.internal\" not found" Sep 14 10:48:46.823655 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.823622 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 14 10:48:46.824982 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.824959 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 14 10:48:46.825078 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.825007 2581 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 14 10:48:46.825078 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.825036 2581 kubelet.go:2448] "Starting kubelet main sync loop" Sep 14 10:48:46.825180 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:46.825127 2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 14 10:48:46.827480 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.827392 2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 10:48:46.846356 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.846312 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 10:48:46.847279 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.847253 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-53.ec2.internal" event="NodeHasSufficientMemory" Sep 14 10:48:46.847397 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.847295 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-53.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 10:48:46.847397 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.847308 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-53.ec2.internal" event="NodeHasSufficientPID" Sep 14 10:48:46.847397 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.847331 2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-53.ec2.internal" Sep 14 10:48:46.855626 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.855611 2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-53.ec2.internal" Sep 14 10:48:46.855692 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:46.855633 2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-53.ec2.internal\": node \"ip-10-0-137-53.ec2.internal\" not found" Sep 14 10:48:46.886842 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:46.886811 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-53.ec2.internal\" not found" Sep 14 10:48:46.925599 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.925552 2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-137-53.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-53.ec2.internal"] Sep 14 10:48:46.925684 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.925648 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 10:48:46.927446 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.927431 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-53.ec2.internal" event="NodeHasSufficientMemory" Sep 14 10:48:46.927497 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.927465 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-53.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 10:48:46.927497 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.927477 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-53.ec2.internal" event="NodeHasSufficientPID" Sep 14 10:48:46.929869 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.929856 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 10:48:46.930042 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.930022 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-53.ec2.internal" Sep 14 10:48:46.930091 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.930060 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 10:48:46.930631 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.930617 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-53.ec2.internal" event="NodeHasSufficientMemory" Sep 14 10:48:46.930697 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.930648 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-53.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 10:48:46.930697 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.930662 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-53.ec2.internal" event="NodeHasSufficientPID" Sep 14 10:48:46.930697 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.930670 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-53.ec2.internal" event="NodeHasSufficientMemory" Sep 14 10:48:46.930697 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.930687 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-53.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 10:48:46.930697 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.930696 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-53.ec2.internal" event="NodeHasSufficientPID" Sep 14 10:48:46.932973 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.932958 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-53.ec2.internal" Sep 14 10:48:46.933035 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.932982 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 10:48:46.933664 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.933650 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-53.ec2.internal" event="NodeHasSufficientMemory" Sep 14 10:48:46.933742 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.933676 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-53.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 10:48:46.933742 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.933688 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-53.ec2.internal" event="NodeHasSufficientPID" Sep 14 10:48:46.949907 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:46.949887 2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-53.ec2.internal\" not found" node="ip-10-0-137-53.ec2.internal" Sep 14 10:48:46.954156 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:46.954139 2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-53.ec2.internal\" not found" node="ip-10-0-137-53.ec2.internal" Sep 14 10:48:46.981950 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.981924 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3ed51660b0e0cea0ccab4eff146b4e99-config\") pod \"kube-apiserver-proxy-ip-10-0-137-53.ec2.internal\" (UID: \"3ed51660b0e0cea0ccab4eff146b4e99\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-53.ec2.internal" Sep 14 10:48:46.982042 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.981954 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ed4e451484feca11635366f26cff0482-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-53.ec2.internal\" (UID: \"ed4e451484feca11635366f26cff0482\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-53.ec2.internal" Sep 14 10:48:46.982042 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:46.981979 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed4e451484feca11635366f26cff0482-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-53.ec2.internal\" (UID: \"ed4e451484feca11635366f26cff0482\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-53.ec2.internal" Sep 14 10:48:46.986996 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:46.986984 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-53.ec2.internal\" not found" Sep 14 10:48:47.082503 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:47.082419 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ed4e451484feca11635366f26cff0482-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-53.ec2.internal\" (UID: \"ed4e451484feca11635366f26cff0482\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-53.ec2.internal" Sep 14 10:48:47.082503 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:47.082469 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed4e451484feca11635366f26cff0482-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-53.ec2.internal\" (UID: \"ed4e451484feca11635366f26cff0482\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-53.ec2.internal" Sep 14 10:48:47.082503 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:47.082488 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3ed51660b0e0cea0ccab4eff146b4e99-config\") pod \"kube-apiserver-proxy-ip-10-0-137-53.ec2.internal\" (UID: \"3ed51660b0e0cea0ccab4eff146b4e99\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-53.ec2.internal" Sep 14 10:48:47.082503 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:47.082488 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ed4e451484feca11635366f26cff0482-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-53.ec2.internal\" (UID: \"ed4e451484feca11635366f26cff0482\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-53.ec2.internal" Sep 14 10:48:47.082503 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:47.082506 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed4e451484feca11635366f26cff0482-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-53.ec2.internal\" (UID: \"ed4e451484feca11635366f26cff0482\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-53.ec2.internal" Sep 14 10:48:47.082717 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:47.082545 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3ed51660b0e0cea0ccab4eff146b4e99-config\") pod \"kube-apiserver-proxy-ip-10-0-137-53.ec2.internal\" (UID: \"3ed51660b0e0cea0ccab4eff146b4e99\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-53.ec2.internal" Sep 14 10:48:47.087532 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:47.087518 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-53.ec2.internal\" not found" Sep 14 10:48:47.188238 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:47.188200 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-53.ec2.internal\" not found" Sep 14 10:48:47.252436 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:47.252416 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-53.ec2.internal" Sep 14 10:48:47.257499 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:47.257176 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-53.ec2.internal" Sep 14 10:48:47.289232 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:47.289201 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-53.ec2.internal\" not found" Sep 14 10:48:47.389819 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:47.389737 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-53.ec2.internal\" not found" Sep 14 10:48:47.490271 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:47.490223 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-53.ec2.internal\" not found" Sep 14 10:48:47.590924 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:47.590890 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-53.ec2.internal\" not found" Sep 14 10:48:47.604072 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:47.604044 2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 14 10:48:47.604261 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:47.604236 2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 14 10:48:47.604314 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:47.604232 2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 14 10:48:47.677010 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:47.676975 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-13 10:43:46 +0000 UTC" deadline="2028-04-29 11:15:32.79155021 +0000 UTC" Sep 14 10:48:47.677010 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:47.677005 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14232h26m45.114548637s" Sep 14 10:48:47.691155 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:47.691125 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-53.ec2.internal\" not found" Sep 14 10:48:47.704967 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:47.704945 2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 10:48:47.780049 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:47.780003 2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-53.ec2.internal" Sep 14 10:48:47.787561 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:47.787542 2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 14 10:48:47.788869 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:47.788856 2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-53.ec2.internal" Sep 14 10:48:47.799780 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:47.799759 2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 14 10:48:47.817506 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:47.817469 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ed51660b0e0cea0ccab4eff146b4e99.slice/crio-b8019fa8697451fbbd03775642ee5773ec5fd1d674ed4a33c1f6a40c9208073c WatchSource:0}: Error finding container b8019fa8697451fbbd03775642ee5773ec5fd1d674ed4a33c1f6a40c9208073c: Status 404 returned error can't find the container with id b8019fa8697451fbbd03775642ee5773ec5fd1d674ed4a33c1f6a40c9208073c Sep 14 10:48:47.817981 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:47.817958 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded4e451484feca11635366f26cff0482.slice/crio-2f45b07160e6900d9a902f3e042c8127b96905cca7526ab9435c4da3534254d5 WatchSource:0}: Error finding container 2f45b07160e6900d9a902f3e042c8127b96905cca7526ab9435c4da3534254d5: Status 404 returned error can't find the container with id 2f45b07160e6900d9a902f3e042c8127b96905cca7526ab9435c4da3534254d5 Sep 14 10:48:47.821967 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:47.821952 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 10:48:47.829646 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:47.828670 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-53.ec2.internal" event={"ID":"ed4e451484feca11635366f26cff0482","Type":"ContainerStarted","Data":"2f45b07160e6900d9a902f3e042c8127b96905cca7526ab9435c4da3534254d5"} Sep 14 10:48:47.831035 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:47.831006 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-53.ec2.internal" event={"ID":"3ed51660b0e0cea0ccab4eff146b4e99","Type":"ContainerStarted","Data":"b8019fa8697451fbbd03775642ee5773ec5fd1d674ed4a33c1f6a40c9208073c"} Sep 14 10:48:48.187801 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.187763 2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 10:48:48.652493 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.652451 2581 apiserver.go:52] "Watching apiserver" Sep 14 10:48:48.663140 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.663111 2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 14 10:48:48.663661 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.663632 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-js9xt","openshift-dns/node-resolver-5nq5v","openshift-multus/multus-additional-cni-plugins-nfqwx","openshift-network-diagnostics/network-check-target-tw9zw","openshift-network-operator/iptables-alerter-lh4nz","openshift-ovn-kubernetes/ovnkube-node-bmb78","kube-system/global-pull-secret-syncer-b62cb","kube-system/kube-apiserver-proxy-ip-10-0-137-53.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b98zp","openshift-cluster-node-tuning-operator/tuned-6xfsw","openshift-image-registry/node-ca-rfppn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-53.ec2.internal","openshift-multus/multus-zblrl","openshift-multus/network-metrics-daemon-29jrs"] Sep 14 10:48:48.666473 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.666428 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-js9xt" Sep 14 10:48:48.669785 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.669757 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5nq5v" Sep 14 10:48:48.671974 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.671952 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nfqwx" Sep 14 10:48:48.673200 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.673177 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 14 10:48:48.673470 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.673452 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 14 10:48:48.676547 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.676523 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw9zw" Sep 14 10:48:48.676660 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:48.676592 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tw9zw" podUID="54a7a889-af23-436c-ac67-8b45c1b39194" Sep 14 10:48:48.676660 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.676622 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lh4nz" Sep 14 10:48:48.677077 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.677061 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 14 10:48:48.677148 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.677091 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6z8b5\"" Sep 14 10:48:48.680852 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.678497 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.680852 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.679270 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 14 10:48:48.680852 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.679366 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 14 10:48:48.680852 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.679439 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 14 10:48:48.680852 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.679828 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-s8fkn\"" Sep 14 10:48:48.680852 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.679981 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 14 10:48:48.680852 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.680063 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8hqh4\"" Sep 14 10:48:48.680852 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.680134 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 14 10:48:48.681543 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.681440 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b62cb" Sep 14 10:48:48.681543 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:48.681506 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b62cb" podUID="6f7b7b9c-eb99-4eb6-b619-8db12670cef0" Sep 14 10:48:48.682920 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.682898 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 14 10:48:48.683200 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.683182 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 14 10:48:48.683307 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.683292 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 14 10:48:48.683680 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.683661 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b98zp" Sep 14 10:48:48.685597 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.685576 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 14 10:48:48.686590 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.686567 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" Sep 14 10:48:48.691764 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.690508 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rfppn" Sep 14 10:48:48.691764 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.690713 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 14 10:48:48.691764 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.691319 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 14 10:48:48.691957 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.691780 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 14 10:48:48.691957 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.691903 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-r758b\"" Sep 14 10:48:48.692139 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.692115 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 14 10:48:48.692543 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.692521 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 14 10:48:48.692740 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.692298 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 14 10:48:48.692964 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.692943 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v84zt\" (UniqueName: \"kubernetes.io/projected/ceba844b-2781-40ea-96e0-9eacd1e13ef6-kube-api-access-v84zt\") pod \"node-resolver-5nq5v\" (UID: \"ceba844b-2781-40ea-96e0-9eacd1e13ef6\") " pod="openshift-dns/node-resolver-5nq5v" Sep 14 10:48:48.693043 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.692988 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-host-kubelet\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.693043 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.693020 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-host-cni-netd\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.693159 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.693052 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-x9vrl\"" Sep 14 10:48:48.693213 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.693056 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5cc7f97c-133f-4c01-a394-1e06cb81d372-agent-certs\") pod \"konnectivity-agent-js9xt\" (UID: \"5cc7f97c-133f-4c01-a394-1e06cb81d372\") " pod="kube-system/konnectivity-agent-js9xt" Sep 14 10:48:48.693268 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.693224 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f44ee762-76b2-4d0a-aabd-8b68c6aaffdb-cni-binary-copy\") pod \"multus-additional-cni-plugins-nfqwx\" (UID: \"f44ee762-76b2-4d0a-aabd-8b68c6aaffdb\") " pod="openshift-multus/multus-additional-cni-plugins-nfqwx" Sep 14 10:48:48.693323 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.693261 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f44ee762-76b2-4d0a-aabd-8b68c6aaffdb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nfqwx\" (UID: \"f44ee762-76b2-4d0a-aabd-8b68c6aaffdb\") " pod="openshift-multus/multus-additional-cni-plugins-nfqwx" Sep 14 10:48:48.693363 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.693322 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-host-run-netns\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.693363 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.693335 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 14 10:48:48.693471 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.693359 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-var-lib-openvswitch\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.693471 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.693440 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6f7b7b9c-eb99-4eb6-b619-8db12670cef0-dbus\") pod \"global-pull-secret-syncer-b62cb\" (UID: \"6f7b7b9c-eb99-4eb6-b619-8db12670cef0\") " pod="kube-system/global-pull-secret-syncer-b62cb" Sep 14 10:48:48.693535 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.693479 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.693535 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.693514 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d152261f-2616-406f-b9df-22a9614deafe-ovn-node-metrics-cert\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.693600 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.693541 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d152261f-2616-406f-b9df-22a9614deafe-ovnkube-script-lib\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.693600 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.693577 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5cc7f97c-133f-4c01-a394-1e06cb81d372-konnectivity-ca\") pod \"konnectivity-agent-js9xt\" (UID: \"5cc7f97c-133f-4c01-a394-1e06cb81d372\") " pod="kube-system/konnectivity-agent-js9xt" Sep 14 10:48:48.693659 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.693611 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh7zw\" (UniqueName: \"kubernetes.io/projected/54a7a889-af23-436c-ac67-8b45c1b39194-kube-api-access-mh7zw\") pod \"network-check-target-tw9zw\" (UID: \"54a7a889-af23-436c-ac67-8b45c1b39194\") " pod="openshift-network-diagnostics/network-check-target-tw9zw" Sep 14 10:48:48.693659 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.693636 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xrhwl\"" Sep 14 10:48:48.693659 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.693643 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ceba844b-2781-40ea-96e0-9eacd1e13ef6-hosts-file\") pod \"node-resolver-5nq5v\" (UID: \"ceba844b-2781-40ea-96e0-9eacd1e13ef6\") " pod="openshift-dns/node-resolver-5nq5v" Sep 14 10:48:48.693751 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.693673 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-run-systemd\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.693751 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.693699 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-run-ovn\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.693751 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.693725 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-log-socket\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.693751 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.693744 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-host-cni-bin\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.693891 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.693763 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f44ee762-76b2-4d0a-aabd-8b68c6aaffdb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nfqwx\" (UID: \"f44ee762-76b2-4d0a-aabd-8b68c6aaffdb\") " pod="openshift-multus/multus-additional-cni-plugins-nfqwx" Sep 14 10:48:48.693891 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.693806 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d152261f-2616-406f-b9df-22a9614deafe-env-overrides\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.694061 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.693987 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 14 10:48:48.694127 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.694069 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f7b7b9c-eb99-4eb6-b619-8db12670cef0-original-pull-secret\") pod \"global-pull-secret-syncer-b62cb\" (UID: \"6f7b7b9c-eb99-4eb6-b619-8db12670cef0\") " pod="kube-system/global-pull-secret-syncer-b62cb" Sep 14 10:48:48.694184 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.694126 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bd4585d7-6ffb-42c6-b71e-478af8e902cc-iptables-alerter-script\") pod \"iptables-alerter-lh4nz\" (UID: \"bd4585d7-6ffb-42c6-b71e-478af8e902cc\") " pod="openshift-network-operator/iptables-alerter-lh4nz" Sep 14 10:48:48.694184 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.694167 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd4585d7-6ffb-42c6-b71e-478af8e902cc-host-slash\") pod \"iptables-alerter-lh4nz\" (UID: \"bd4585d7-6ffb-42c6-b71e-478af8e902cc\") " pod="openshift-network-operator/iptables-alerter-lh4nz" Sep 14 10:48:48.694292 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.694213 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-run-openvswitch\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.694345 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.694289 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-host-run-ovn-kubernetes\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.694399 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.694371 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fpvg\" (UniqueName: \"kubernetes.io/projected/d152261f-2616-406f-b9df-22a9614deafe-kube-api-access-6fpvg\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.694484 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.694446 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f44ee762-76b2-4d0a-aabd-8b68c6aaffdb-cnibin\") pod \"multus-additional-cni-plugins-nfqwx\" (UID: \"f44ee762-76b2-4d0a-aabd-8b68c6aaffdb\") " pod="openshift-multus/multus-additional-cni-plugins-nfqwx" Sep 14 10:48:48.694536 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.694511 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6f7b7b9c-eb99-4eb6-b619-8db12670cef0-kubelet-config\") pod \"global-pull-secret-syncer-b62cb\" (UID: \"6f7b7b9c-eb99-4eb6-b619-8db12670cef0\") " pod="kube-system/global-pull-secret-syncer-b62cb" Sep 14 10:48:48.694585 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.694541 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4mvw\" (UniqueName: \"kubernetes.io/projected/bd4585d7-6ffb-42c6-b71e-478af8e902cc-kube-api-access-l4mvw\") pod \"iptables-alerter-lh4nz\" (UID: \"bd4585d7-6ffb-42c6-b71e-478af8e902cc\") " pod="openshift-network-operator/iptables-alerter-lh4nz" Sep 14 10:48:48.694585 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.694572 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-etc-openvswitch\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.694683 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.694637 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-node-log\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.694683 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.694670 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f44ee762-76b2-4d0a-aabd-8b68c6aaffdb-system-cni-dir\") pod \"multus-additional-cni-plugins-nfqwx\" (UID: \"f44ee762-76b2-4d0a-aabd-8b68c6aaffdb\") " pod="openshift-multus/multus-additional-cni-plugins-nfqwx" Sep 14 10:48:48.694776 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.694703 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f44ee762-76b2-4d0a-aabd-8b68c6aaffdb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nfqwx\" (UID: \"f44ee762-76b2-4d0a-aabd-8b68c6aaffdb\") " pod="openshift-multus/multus-additional-cni-plugins-nfqwx" Sep 14 10:48:48.694776 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.694739 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql2lb\" (UniqueName: \"kubernetes.io/projected/f44ee762-76b2-4d0a-aabd-8b68c6aaffdb-kube-api-access-ql2lb\") pod \"multus-additional-cni-plugins-nfqwx\" (UID: \"f44ee762-76b2-4d0a-aabd-8b68c6aaffdb\") " pod="openshift-multus/multus-additional-cni-plugins-nfqwx" Sep 14 10:48:48.694965 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.694940 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ceba844b-2781-40ea-96e0-9eacd1e13ef6-tmp-dir\") pod \"node-resolver-5nq5v\" (UID: \"ceba844b-2781-40ea-96e0-9eacd1e13ef6\") " pod="openshift-dns/node-resolver-5nq5v" Sep 14 10:48:48.695046 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.694997 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-systemd-units\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.695046 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.695025 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-host-slash\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.695155 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.695113 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d152261f-2616-406f-b9df-22a9614deafe-ovnkube-config\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.695229 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.695211 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.699186 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.698552 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 14 10:48:48.699186 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.698577 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29jrs" Sep 14 10:48:48.699186 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:48.698691 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29jrs" podUID="04ff0710-e09c-48e9-9769-7ffce3645b66" Sep 14 10:48:48.699186 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.698766 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 14 10:48:48.699186 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.698801 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 14 10:48:48.699186 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.698905 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 14 10:48:48.699186 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.698931 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 14 10:48:48.699186 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.699004 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 14 10:48:48.699186 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.698552 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lhzpd\"" Sep 14 10:48:48.699186 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.699143 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qc5gr\"" Sep 14 10:48:48.701896 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.701873 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 14 10:48:48.704328 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.704307 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-82hqr\"" Sep 14 10:48:48.704453 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.704330 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 14 10:48:48.710006 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.709982 2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 14 10:48:48.736020 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.736002 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-j8mf7" Sep 14 10:48:48.743088 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.742760 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-j8mf7" Sep 14 10:48:48.781040 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.781012 2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 14 10:48:48.795968 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.795931 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e95ca08f-2a1d-45b7-b0de-959c0b546a18-multus-daemon-config\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.796148 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.795979 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/07adfe8a-0f38-499a-b903-f8de96458395-etc-selinux\") pod \"aws-ebs-csi-driver-node-b98zp\" (UID: \"07adfe8a-0f38-499a-b903-f8de96458395\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b98zp" Sep 14 10:48:48.796148 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.796012 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-host-kubelet\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.796148 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.796043 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5cc7f97c-133f-4c01-a394-1e06cb81d372-agent-certs\") pod \"konnectivity-agent-js9xt\" (UID: \"5cc7f97c-133f-4c01-a394-1e06cb81d372\") " pod="kube-system/konnectivity-agent-js9xt" Sep 14 10:48:48.796148 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.796070 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f44ee762-76b2-4d0a-aabd-8b68c6aaffdb-cni-binary-copy\") pod \"multus-additional-cni-plugins-nfqwx\" (UID: \"f44ee762-76b2-4d0a-aabd-8b68c6aaffdb\") " pod="openshift-multus/multus-additional-cni-plugins-nfqwx" Sep 14 10:48:48.796148 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.796095 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/07adfe8a-0f38-499a-b903-f8de96458395-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b98zp\" (UID: \"07adfe8a-0f38-499a-b903-f8de96458395\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b98zp" Sep 14 10:48:48.796148 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.796120 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e95ca08f-2a1d-45b7-b0de-959c0b546a18-host-var-lib-cni-multus\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.796148 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.796133 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-host-kubelet\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.796507 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.796147 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e95ca08f-2a1d-45b7-b0de-959c0b546a18-host-var-lib-kubelet\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.796507 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.796192 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e95ca08f-2a1d-45b7-b0de-959c0b546a18-etc-kubernetes\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.796507 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.796218 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3a0b37cc-bde2-4894-930d-d4d9b9345568-etc-sysconfig\") pod \"tuned-6xfsw\" (UID: \"3a0b37cc-bde2-4894-930d-d4d9b9345568\") " pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" Sep 14 10:48:48.796507 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.796252 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a0b37cc-bde2-4894-930d-d4d9b9345568-var-lib-kubelet\") pod \"tuned-6xfsw\" (UID: \"3a0b37cc-bde2-4894-930d-d4d9b9345568\") " pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" Sep 14 10:48:48.796507 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.796278 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8czk5\" (UniqueName: \"kubernetes.io/projected/3a0b37cc-bde2-4894-930d-d4d9b9345568-kube-api-access-8czk5\") pod \"tuned-6xfsw\" (UID: \"3a0b37cc-bde2-4894-930d-d4d9b9345568\") " pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" Sep 14 10:48:48.796507 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.796305 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d152261f-2616-406f-b9df-22a9614deafe-ovn-node-metrics-cert\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.796507 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.796330 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d152261f-2616-406f-b9df-22a9614deafe-ovnkube-script-lib\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.796507 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.796355 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5cc7f97c-133f-4c01-a394-1e06cb81d372-konnectivity-ca\") pod \"konnectivity-agent-js9xt\" (UID: \"5cc7f97c-133f-4c01-a394-1e06cb81d372\") " pod="kube-system/konnectivity-agent-js9xt" Sep 14 10:48:48.796507 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.796397 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e95ca08f-2a1d-45b7-b0de-959c0b546a18-multus-cni-dir\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.796507 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.796423 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3a0b37cc-bde2-4894-930d-d4d9b9345568-run\") pod \"tuned-6xfsw\" (UID: \"3a0b37cc-bde2-4894-930d-d4d9b9345568\") " pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" Sep 14 10:48:48.796507 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.796447 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5b6399c5-592d-463e-a9a4-337834d10604-host\") pod \"node-ca-rfppn\" (UID: \"5b6399c5-592d-463e-a9a4-337834d10604\") " pod="openshift-image-registry/node-ca-rfppn" Sep 14 10:48:48.796507 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.796474 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ceba844b-2781-40ea-96e0-9eacd1e13ef6-hosts-file\") pod \"node-resolver-5nq5v\" (UID: \"ceba844b-2781-40ea-96e0-9eacd1e13ef6\") " pod="openshift-dns/node-resolver-5nq5v" Sep 14 10:48:48.796507 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.796500 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-run-ovn\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.797084 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.796524 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e95ca08f-2a1d-45b7-b0de-959c0b546a18-cni-binary-copy\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.797084 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.796547 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3a0b37cc-bde2-4894-930d-d4d9b9345568-etc-sysctl-conf\") pod \"tuned-6xfsw\" (UID: \"3a0b37cc-bde2-4894-930d-d4d9b9345568\") " pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" Sep 14 10:48:48.797084 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.796572 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d152261f-2616-406f-b9df-22a9614deafe-env-overrides\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.797084 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.796596 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3a0b37cc-bde2-4894-930d-d4d9b9345568-etc-tuned\") pod \"tuned-6xfsw\" (UID: \"3a0b37cc-bde2-4894-930d-d4d9b9345568\") " pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" Sep 14 10:48:48.797084 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.796618 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f7b7b9c-eb99-4eb6-b619-8db12670cef0-original-pull-secret\") pod \"global-pull-secret-syncer-b62cb\" (UID: \"6f7b7b9c-eb99-4eb6-b619-8db12670cef0\") " pod="kube-system/global-pull-secret-syncer-b62cb" Sep 14 10:48:48.797084 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.796654 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bd4585d7-6ffb-42c6-b71e-478af8e902cc-iptables-alerter-script\") pod \"iptables-alerter-lh4nz\" (UID: \"bd4585d7-6ffb-42c6-b71e-478af8e902cc\") " pod="openshift-network-operator/iptables-alerter-lh4nz" Sep 14 10:48:48.797084 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.796674 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd4585d7-6ffb-42c6-b71e-478af8e902cc-host-slash\") pod \"iptables-alerter-lh4nz\" (UID: \"bd4585d7-6ffb-42c6-b71e-478af8e902cc\") " pod="openshift-network-operator/iptables-alerter-lh4nz" Sep 14 10:48:48.797084 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.796692 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e95ca08f-2a1d-45b7-b0de-959c0b546a18-host-run-k8s-cni-cncf-io\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.797084 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.796707 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3a0b37cc-bde2-4894-930d-d4d9b9345568-etc-sysctl-d\") pod \"tuned-6xfsw\" (UID: \"3a0b37cc-bde2-4894-930d-d4d9b9345568\") " pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" Sep 14 10:48:48.797084 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.796745 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6f7b7b9c-eb99-4eb6-b619-8db12670cef0-kubelet-config\") pod \"global-pull-secret-syncer-b62cb\" (UID: \"6f7b7b9c-eb99-4eb6-b619-8db12670cef0\") " pod="kube-system/global-pull-secret-syncer-b62cb" Sep 14 10:48:48.797084 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.796762 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-etc-openvswitch\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.797084 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.796778 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-node-log\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.797084 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.796802 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f44ee762-76b2-4d0a-aabd-8b68c6aaffdb-system-cni-dir\") pod \"multus-additional-cni-plugins-nfqwx\" (UID: \"f44ee762-76b2-4d0a-aabd-8b68c6aaffdb\") " pod="openshift-multus/multus-additional-cni-plugins-nfqwx" Sep 14 10:48:48.797084 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.796826 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e95ca08f-2a1d-45b7-b0de-959c0b546a18-host-run-multus-certs\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.797084 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.796843 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04ff0710-e09c-48e9-9769-7ffce3645b66-metrics-certs\") pod \"network-metrics-daemon-29jrs\" (UID: \"04ff0710-e09c-48e9-9769-7ffce3645b66\") " pod="openshift-multus/network-metrics-daemon-29jrs" Sep 14 10:48:48.797084 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.796870 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ceba844b-2781-40ea-96e0-9eacd1e13ef6-tmp-dir\") pod \"node-resolver-5nq5v\" (UID: \"ceba844b-2781-40ea-96e0-9eacd1e13ef6\") " pod="openshift-dns/node-resolver-5nq5v" Sep 14 10:48:48.797084 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.796893 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-systemd-units\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.797753 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.796911 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a0b37cc-bde2-4894-930d-d4d9b9345568-lib-modules\") pod \"tuned-6xfsw\" (UID: \"3a0b37cc-bde2-4894-930d-d4d9b9345568\") " pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" Sep 14 10:48:48.797753 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.796925 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a0b37cc-bde2-4894-930d-d4d9b9345568-host\") pod \"tuned-6xfsw\" (UID: \"3a0b37cc-bde2-4894-930d-d4d9b9345568\") " pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" Sep 14 10:48:48.797753 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.796941 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v84zt\" (UniqueName: \"kubernetes.io/projected/ceba844b-2781-40ea-96e0-9eacd1e13ef6-kube-api-access-v84zt\") pod \"node-resolver-5nq5v\" (UID: \"ceba844b-2781-40ea-96e0-9eacd1e13ef6\") " pod="openshift-dns/node-resolver-5nq5v" Sep 14 10:48:48.797753 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.796957 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-host-cni-netd\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.797753 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.796973 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f44ee762-76b2-4d0a-aabd-8b68c6aaffdb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nfqwx\" (UID: \"f44ee762-76b2-4d0a-aabd-8b68c6aaffdb\") " pod="openshift-multus/multus-additional-cni-plugins-nfqwx" Sep 14 10:48:48.797753 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.797011 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e95ca08f-2a1d-45b7-b0de-959c0b546a18-multus-conf-dir\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.797753 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.797014 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f44ee762-76b2-4d0a-aabd-8b68c6aaffdb-cni-binary-copy\") pod \"multus-additional-cni-plugins-nfqwx\" (UID: \"f44ee762-76b2-4d0a-aabd-8b68c6aaffdb\") " pod="openshift-multus/multus-additional-cni-plugins-nfqwx" Sep 14 10:48:48.797753 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.797028 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/07adfe8a-0f38-499a-b903-f8de96458395-registration-dir\") pod \"aws-ebs-csi-driver-node-b98zp\" (UID: \"07adfe8a-0f38-499a-b903-f8de96458395\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b98zp" Sep 14 10:48:48.797753 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.797045 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5b6399c5-592d-463e-a9a4-337834d10604-serviceca\") pod \"node-ca-rfppn\" (UID: \"5b6399c5-592d-463e-a9a4-337834d10604\") " pod="openshift-image-registry/node-ca-rfppn" Sep 14 10:48:48.797753 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.797068 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-host-run-netns\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.797753 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.797085 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-var-lib-openvswitch\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.797753 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.797101 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td5f9\" (UniqueName: \"kubernetes.io/projected/04ff0710-e09c-48e9-9769-7ffce3645b66-kube-api-access-td5f9\") pod \"network-metrics-daemon-29jrs\" (UID: \"04ff0710-e09c-48e9-9769-7ffce3645b66\") " pod="openshift-multus/network-metrics-daemon-29jrs" Sep 14 10:48:48.797753 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.797122 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/07adfe8a-0f38-499a-b903-f8de96458395-device-dir\") pod \"aws-ebs-csi-driver-node-b98zp\" (UID: \"07adfe8a-0f38-499a-b903-f8de96458395\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b98zp" Sep 14 10:48:48.797753 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.797143 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqx5x\" (UniqueName: \"kubernetes.io/projected/07adfe8a-0f38-499a-b903-f8de96458395-kube-api-access-jqx5x\") pod \"aws-ebs-csi-driver-node-b98zp\" (UID: \"07adfe8a-0f38-499a-b903-f8de96458395\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b98zp" Sep 14 10:48:48.797753 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.797166 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a0b37cc-bde2-4894-930d-d4d9b9345568-tmp\") pod \"tuned-6xfsw\" (UID: \"3a0b37cc-bde2-4894-930d-d4d9b9345568\") " pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" Sep 14 10:48:48.797753 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.797188 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6f7b7b9c-eb99-4eb6-b619-8db12670cef0-dbus\") pod \"global-pull-secret-syncer-b62cb\" (UID: \"6f7b7b9c-eb99-4eb6-b619-8db12670cef0\") " pod="kube-system/global-pull-secret-syncer-b62cb" Sep 14 10:48:48.798543 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.797205 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.798543 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.797222 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e95ca08f-2a1d-45b7-b0de-959c0b546a18-cnibin\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.798543 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.797237 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e95ca08f-2a1d-45b7-b0de-959c0b546a18-hostroot\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.798543 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.797236 2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 14 10:48:48.798543 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.797258 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/07adfe8a-0f38-499a-b903-f8de96458395-sys-fs\") pod \"aws-ebs-csi-driver-node-b98zp\" (UID: \"07adfe8a-0f38-499a-b903-f8de96458395\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b98zp" Sep 14 10:48:48.798543 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.797276 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh7zw\" (UniqueName: \"kubernetes.io/projected/54a7a889-af23-436c-ac67-8b45c1b39194-kube-api-access-mh7zw\") pod \"network-check-target-tw9zw\" (UID: \"54a7a889-af23-436c-ac67-8b45c1b39194\") " pod="openshift-network-diagnostics/network-check-target-tw9zw" Sep 14 10:48:48.798543 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.797292 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-run-systemd\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.798543 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.797309 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-log-socket\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.798543 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.797325 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-host-cni-bin\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.798543 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.797356 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f44ee762-76b2-4d0a-aabd-8b68c6aaffdb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nfqwx\" (UID: \"f44ee762-76b2-4d0a-aabd-8b68c6aaffdb\") " pod="openshift-multus/multus-additional-cni-plugins-nfqwx" Sep 14 10:48:48.798543 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.797379 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a0b37cc-bde2-4894-930d-d4d9b9345568-etc-kubernetes\") pod \"tuned-6xfsw\" (UID: \"3a0b37cc-bde2-4894-930d-d4d9b9345568\") " pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" Sep 14 10:48:48.798543 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.797401 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e95ca08f-2a1d-45b7-b0de-959c0b546a18-multus-socket-dir-parent\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.798543 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.797439 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3a0b37cc-bde2-4894-930d-d4d9b9345568-etc-modprobe-d\") pod \"tuned-6xfsw\" (UID: \"3a0b37cc-bde2-4894-930d-d4d9b9345568\") " pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" Sep 14 10:48:48.798543 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.797466 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3a0b37cc-bde2-4894-930d-d4d9b9345568-etc-systemd\") pod \"tuned-6xfsw\" (UID: \"3a0b37cc-bde2-4894-930d-d4d9b9345568\") " pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" Sep 14 10:48:48.798543 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.797495 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-run-openvswitch\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.798543 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.797533 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ceba844b-2781-40ea-96e0-9eacd1e13ef6-hosts-file\") pod \"node-resolver-5nq5v\" (UID: \"ceba844b-2781-40ea-96e0-9eacd1e13ef6\") " pod="openshift-dns/node-resolver-5nq5v" Sep 14 10:48:48.798543 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.797625 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-run-ovn\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.798543 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:48.797730 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 10:48:48.799496 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.797743 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6f7b7b9c-eb99-4eb6-b619-8db12670cef0-dbus\") pod \"global-pull-secret-syncer-b62cb\" (UID: \"6f7b7b9c-eb99-4eb6-b619-8db12670cef0\") " pod="kube-system/global-pull-secret-syncer-b62cb" Sep 14 10:48:48.799496 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.797770 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-host-run-ovn-kubernetes\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.799496 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:48.797816 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f7b7b9c-eb99-4eb6-b619-8db12670cef0-original-pull-secret podName:6f7b7b9c-eb99-4eb6-b619-8db12670cef0 nodeName:}" failed. No retries permitted until 2026-09-14 10:48:49.297789403 +0000 UTC m=+3.040191352 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f7b7b9c-eb99-4eb6-b619-8db12670cef0-original-pull-secret") pod "global-pull-secret-syncer-b62cb" (UID: "6f7b7b9c-eb99-4eb6-b619-8db12670cef0") : object "kube-system"/"original-pull-secret" not registered Sep 14 10:48:48.799496 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.797822 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-run-openvswitch\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.799496 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.797821 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.799496 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.797877 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6f7b7b9c-eb99-4eb6-b619-8db12670cef0-kubelet-config\") pod \"global-pull-secret-syncer-b62cb\" (UID: \"6f7b7b9c-eb99-4eb6-b619-8db12670cef0\") " pod="kube-system/global-pull-secret-syncer-b62cb" Sep 14 10:48:48.799496 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.797943 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-var-lib-openvswitch\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.799496 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.797934 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-host-run-netns\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.799496 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.797732 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-host-run-ovn-kubernetes\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.799496 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.798004 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-host-cni-bin\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.799496 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.798008 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fpvg\" (UniqueName: \"kubernetes.io/projected/d152261f-2616-406f-b9df-22a9614deafe-kube-api-access-6fpvg\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.799496 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.798040 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f44ee762-76b2-4d0a-aabd-8b68c6aaffdb-cnibin\") pod \"multus-additional-cni-plugins-nfqwx\" (UID: \"f44ee762-76b2-4d0a-aabd-8b68c6aaffdb\") " pod="openshift-multus/multus-additional-cni-plugins-nfqwx" Sep 14 10:48:48.799496 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.798068 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f44ee762-76b2-4d0a-aabd-8b68c6aaffdb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nfqwx\" (UID: \"f44ee762-76b2-4d0a-aabd-8b68c6aaffdb\") " pod="openshift-multus/multus-additional-cni-plugins-nfqwx" Sep 14 10:48:48.799496 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.798102 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e95ca08f-2a1d-45b7-b0de-959c0b546a18-host-run-netns\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.799496 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.798129 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnzhg\" (UniqueName: \"kubernetes.io/projected/e95ca08f-2a1d-45b7-b0de-959c0b546a18-kube-api-access-tnzhg\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.799496 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.798156 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/07adfe8a-0f38-499a-b903-f8de96458395-socket-dir\") pod \"aws-ebs-csi-driver-node-b98zp\" (UID: \"07adfe8a-0f38-499a-b903-f8de96458395\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b98zp" Sep 14 10:48:48.800328 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.798171 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d152261f-2616-406f-b9df-22a9614deafe-ovnkube-script-lib\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.800328 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.798188 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4mvw\" (UniqueName: \"kubernetes.io/projected/bd4585d7-6ffb-42c6-b71e-478af8e902cc-kube-api-access-l4mvw\") pod \"iptables-alerter-lh4nz\" (UID: \"bd4585d7-6ffb-42c6-b71e-478af8e902cc\") " pod="openshift-network-operator/iptables-alerter-lh4nz" Sep 14 10:48:48.800328 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.798226 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql2lb\" (UniqueName: \"kubernetes.io/projected/f44ee762-76b2-4d0a-aabd-8b68c6aaffdb-kube-api-access-ql2lb\") pod \"multus-additional-cni-plugins-nfqwx\" (UID: \"f44ee762-76b2-4d0a-aabd-8b68c6aaffdb\") " pod="openshift-multus/multus-additional-cni-plugins-nfqwx" Sep 14 10:48:48.800328 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.798245 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5cc7f97c-133f-4c01-a394-1e06cb81d372-konnectivity-ca\") pod \"konnectivity-agent-js9xt\" (UID: \"5cc7f97c-133f-4c01-a394-1e06cb81d372\") " pod="kube-system/konnectivity-agent-js9xt" Sep 14 10:48:48.800328 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.798263 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a0b37cc-bde2-4894-930d-d4d9b9345568-sys\") pod \"tuned-6xfsw\" (UID: \"3a0b37cc-bde2-4894-930d-d4d9b9345568\") " pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" Sep 14 10:48:48.800328 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.798293 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsx7m\" (UniqueName: \"kubernetes.io/projected/5b6399c5-592d-463e-a9a4-337834d10604-kube-api-access-fsx7m\") pod \"node-ca-rfppn\" (UID: \"5b6399c5-592d-463e-a9a4-337834d10604\") " pod="openshift-image-registry/node-ca-rfppn" Sep 14 10:48:48.800328 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.798302 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd4585d7-6ffb-42c6-b71e-478af8e902cc-host-slash\") pod \"iptables-alerter-lh4nz\" (UID: \"bd4585d7-6ffb-42c6-b71e-478af8e902cc\") " pod="openshift-network-operator/iptables-alerter-lh4nz" Sep 14 10:48:48.800328 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.798324 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-host-slash\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.800328 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.798333 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f44ee762-76b2-4d0a-aabd-8b68c6aaffdb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nfqwx\" (UID: \"f44ee762-76b2-4d0a-aabd-8b68c6aaffdb\") " pod="openshift-multus/multus-additional-cni-plugins-nfqwx" Sep 14 10:48:48.800328 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.798351 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-host-cni-netd\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.800328 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.798355 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d152261f-2616-406f-b9df-22a9614deafe-ovnkube-config\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.800328 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.798439 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-log-socket\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.800328 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.798461 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-etc-openvswitch\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.800328 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.798486 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-run-systemd\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.800328 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.798516 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e95ca08f-2a1d-45b7-b0de-959c0b546a18-system-cni-dir\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.800328 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.798539 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-node-log\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.800328 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.798548 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e95ca08f-2a1d-45b7-b0de-959c0b546a18-host-var-lib-cni-bin\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.801278 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.798571 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bd4585d7-6ffb-42c6-b71e-478af8e902cc-iptables-alerter-script\") pod \"iptables-alerter-lh4nz\" (UID: \"bd4585d7-6ffb-42c6-b71e-478af8e902cc\") " pod="openshift-network-operator/iptables-alerter-lh4nz" Sep 14 10:48:48.801278 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.798599 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f44ee762-76b2-4d0a-aabd-8b68c6aaffdb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nfqwx\" (UID: \"f44ee762-76b2-4d0a-aabd-8b68c6aaffdb\") " pod="openshift-multus/multus-additional-cni-plugins-nfqwx" Sep 14 10:48:48.801278 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.798617 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-systemd-units\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.801278 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.798669 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d152261f-2616-406f-b9df-22a9614deafe-host-slash\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.801278 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.798714 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ceba844b-2781-40ea-96e0-9eacd1e13ef6-tmp-dir\") pod \"node-resolver-5nq5v\" (UID: \"ceba844b-2781-40ea-96e0-9eacd1e13ef6\") " pod="openshift-dns/node-resolver-5nq5v" Sep 14 10:48:48.801278 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.798769 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f44ee762-76b2-4d0a-aabd-8b68c6aaffdb-system-cni-dir\") pod \"multus-additional-cni-plugins-nfqwx\" (UID: \"f44ee762-76b2-4d0a-aabd-8b68c6aaffdb\") " pod="openshift-multus/multus-additional-cni-plugins-nfqwx" Sep 14 10:48:48.801278 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.798780 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f44ee762-76b2-4d0a-aabd-8b68c6aaffdb-cnibin\") pod \"multus-additional-cni-plugins-nfqwx\" (UID: \"f44ee762-76b2-4d0a-aabd-8b68c6aaffdb\") " pod="openshift-multus/multus-additional-cni-plugins-nfqwx" Sep 14 10:48:48.801278 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.798851 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d152261f-2616-406f-b9df-22a9614deafe-env-overrides\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.801278 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.799047 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f44ee762-76b2-4d0a-aabd-8b68c6aaffdb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nfqwx\" (UID: \"f44ee762-76b2-4d0a-aabd-8b68c6aaffdb\") " pod="openshift-multus/multus-additional-cni-plugins-nfqwx" Sep 14 10:48:48.801278 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.799247 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d152261f-2616-406f-b9df-22a9614deafe-ovnkube-config\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.804727 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:48.804699 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 10:48:48.804727 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:48.804728 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 10:48:48.804915 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:48.804745 2581 projected.go:196] Error preparing data for projected volume kube-api-access-mh7zw for pod openshift-network-diagnostics/network-check-target-tw9zw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 10:48:48.804915 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:48.804841 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/54a7a889-af23-436c-ac67-8b45c1b39194-kube-api-access-mh7zw podName:54a7a889-af23-436c-ac67-8b45c1b39194 nodeName:}" failed. No retries permitted until 2026-09-14 10:48:49.304799233 +0000 UTC m=+3.047201199 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mh7zw" (UniqueName: "kubernetes.io/projected/54a7a889-af23-436c-ac67-8b45c1b39194-kube-api-access-mh7zw") pod "network-check-target-tw9zw" (UID: "54a7a889-af23-436c-ac67-8b45c1b39194") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 10:48:48.806319 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.806289 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d152261f-2616-406f-b9df-22a9614deafe-ovn-node-metrics-cert\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.807650 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.807621 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v84zt\" (UniqueName: \"kubernetes.io/projected/ceba844b-2781-40ea-96e0-9eacd1e13ef6-kube-api-access-v84zt\") pod \"node-resolver-5nq5v\" (UID: \"ceba844b-2781-40ea-96e0-9eacd1e13ef6\") " pod="openshift-dns/node-resolver-5nq5v" Sep 14 10:48:48.809418 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.809349 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5cc7f97c-133f-4c01-a394-1e06cb81d372-agent-certs\") pod \"konnectivity-agent-js9xt\" (UID: \"5cc7f97c-133f-4c01-a394-1e06cb81d372\") " pod="kube-system/konnectivity-agent-js9xt" Sep 14 10:48:48.809506 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.809353 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fpvg\" (UniqueName: \"kubernetes.io/projected/d152261f-2616-406f-b9df-22a9614deafe-kube-api-access-6fpvg\") pod \"ovnkube-node-bmb78\" (UID: \"d152261f-2616-406f-b9df-22a9614deafe\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:48.810006 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.809985 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql2lb\" (UniqueName: \"kubernetes.io/projected/f44ee762-76b2-4d0a-aabd-8b68c6aaffdb-kube-api-access-ql2lb\") pod \"multus-additional-cni-plugins-nfqwx\" (UID: \"f44ee762-76b2-4d0a-aabd-8b68c6aaffdb\") " pod="openshift-multus/multus-additional-cni-plugins-nfqwx" Sep 14 10:48:48.811500 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.811474 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4mvw\" (UniqueName: \"kubernetes.io/projected/bd4585d7-6ffb-42c6-b71e-478af8e902cc-kube-api-access-l4mvw\") pod \"iptables-alerter-lh4nz\" (UID: \"bd4585d7-6ffb-42c6-b71e-478af8e902cc\") " pod="openshift-network-operator/iptables-alerter-lh4nz" Sep 14 10:48:48.817432 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.817410 2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 10:48:48.849821 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.849787 2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 10:48:48.899304 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.899261 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3a0b37cc-bde2-4894-930d-d4d9b9345568-etc-tuned\") pod \"tuned-6xfsw\" (UID: \"3a0b37cc-bde2-4894-930d-d4d9b9345568\") " pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" Sep 14 10:48:48.899483 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.899334 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e95ca08f-2a1d-45b7-b0de-959c0b546a18-host-run-k8s-cni-cncf-io\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.899483 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.899382 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3a0b37cc-bde2-4894-930d-d4d9b9345568-etc-sysctl-d\") pod \"tuned-6xfsw\" (UID: \"3a0b37cc-bde2-4894-930d-d4d9b9345568\") " pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" Sep 14 10:48:48.899483 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.899413 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e95ca08f-2a1d-45b7-b0de-959c0b546a18-host-run-multus-certs\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.899483 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.899419 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e95ca08f-2a1d-45b7-b0de-959c0b546a18-host-run-k8s-cni-cncf-io\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.899483 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.899440 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04ff0710-e09c-48e9-9769-7ffce3645b66-metrics-certs\") pod \"network-metrics-daemon-29jrs\" (UID: \"04ff0710-e09c-48e9-9769-7ffce3645b66\") " pod="openshift-multus/network-metrics-daemon-29jrs" Sep 14 10:48:48.899685 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.899498 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e95ca08f-2a1d-45b7-b0de-959c0b546a18-host-run-multus-certs\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.899685 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:48.899547 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 10:48:48.899685 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:48.899620 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04ff0710-e09c-48e9-9769-7ffce3645b66-metrics-certs podName:04ff0710-e09c-48e9-9769-7ffce3645b66 nodeName:}" failed. No retries permitted until 2026-09-14 10:48:49.399597064 +0000 UTC m=+3.141999035 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/04ff0710-e09c-48e9-9769-7ffce3645b66-metrics-certs") pod "network-metrics-daemon-29jrs" (UID: "04ff0710-e09c-48e9-9769-7ffce3645b66") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 10:48:48.899685 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.899620 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3a0b37cc-bde2-4894-930d-d4d9b9345568-etc-sysctl-d\") pod \"tuned-6xfsw\" (UID: \"3a0b37cc-bde2-4894-930d-d4d9b9345568\") " pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" Sep 14 10:48:48.899902 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.899749 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a0b37cc-bde2-4894-930d-d4d9b9345568-lib-modules\") pod \"tuned-6xfsw\" (UID: \"3a0b37cc-bde2-4894-930d-d4d9b9345568\") " pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" Sep 14 10:48:48.899902 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.899778 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a0b37cc-bde2-4894-930d-d4d9b9345568-host\") pod \"tuned-6xfsw\" (UID: \"3a0b37cc-bde2-4894-930d-d4d9b9345568\") " pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" Sep 14 10:48:48.899902 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.899806 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e95ca08f-2a1d-45b7-b0de-959c0b546a18-multus-conf-dir\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.899902 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.899832 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/07adfe8a-0f38-499a-b903-f8de96458395-registration-dir\") pod \"aws-ebs-csi-driver-node-b98zp\" (UID: \"07adfe8a-0f38-499a-b903-f8de96458395\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b98zp" Sep 14 10:48:48.899902 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.899856 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5b6399c5-592d-463e-a9a4-337834d10604-serviceca\") pod \"node-ca-rfppn\" (UID: \"5b6399c5-592d-463e-a9a4-337834d10604\") " pod="openshift-image-registry/node-ca-rfppn" Sep 14 10:48:48.899902 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.899885 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td5f9\" (UniqueName: \"kubernetes.io/projected/04ff0710-e09c-48e9-9769-7ffce3645b66-kube-api-access-td5f9\") pod \"network-metrics-daemon-29jrs\" (UID: \"04ff0710-e09c-48e9-9769-7ffce3645b66\") " pod="openshift-multus/network-metrics-daemon-29jrs" Sep 14 10:48:48.900200 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.899889 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a0b37cc-bde2-4894-930d-d4d9b9345568-host\") pod \"tuned-6xfsw\" (UID: \"3a0b37cc-bde2-4894-930d-d4d9b9345568\") " pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" Sep 14 10:48:48.900200 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.899909 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/07adfe8a-0f38-499a-b903-f8de96458395-device-dir\") pod \"aws-ebs-csi-driver-node-b98zp\" (UID: \"07adfe8a-0f38-499a-b903-f8de96458395\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b98zp" Sep 14 10:48:48.900200 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.899960 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqx5x\" (UniqueName: \"kubernetes.io/projected/07adfe8a-0f38-499a-b903-f8de96458395-kube-api-access-jqx5x\") pod \"aws-ebs-csi-driver-node-b98zp\" (UID: \"07adfe8a-0f38-499a-b903-f8de96458395\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b98zp" Sep 14 10:48:48.900200 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.899988 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a0b37cc-bde2-4894-930d-d4d9b9345568-tmp\") pod \"tuned-6xfsw\" (UID: \"3a0b37cc-bde2-4894-930d-d4d9b9345568\") " pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" Sep 14 10:48:48.900200 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.899996 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e95ca08f-2a1d-45b7-b0de-959c0b546a18-multus-conf-dir\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.900200 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.900017 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e95ca08f-2a1d-45b7-b0de-959c0b546a18-cnibin\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.900200 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.900046 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e95ca08f-2a1d-45b7-b0de-959c0b546a18-hostroot\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.900200 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.900062 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e95ca08f-2a1d-45b7-b0de-959c0b546a18-cnibin\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.900200 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.900072 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/07adfe8a-0f38-499a-b903-f8de96458395-sys-fs\") pod \"aws-ebs-csi-driver-node-b98zp\" (UID: \"07adfe8a-0f38-499a-b903-f8de96458395\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b98zp" Sep 14 10:48:48.900200 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.900116 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a0b37cc-bde2-4894-930d-d4d9b9345568-etc-kubernetes\") pod \"tuned-6xfsw\" (UID: \"3a0b37cc-bde2-4894-930d-d4d9b9345568\") " pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" Sep 14 10:48:48.900200 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.900141 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e95ca08f-2a1d-45b7-b0de-959c0b546a18-multus-socket-dir-parent\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.900200 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.900164 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a0b37cc-bde2-4894-930d-d4d9b9345568-lib-modules\") pod \"tuned-6xfsw\" (UID: \"3a0b37cc-bde2-4894-930d-d4d9b9345568\") " pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" Sep 14 10:48:48.900200 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.900166 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3a0b37cc-bde2-4894-930d-d4d9b9345568-etc-modprobe-d\") pod \"tuned-6xfsw\" (UID: \"3a0b37cc-bde2-4894-930d-d4d9b9345568\") " pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" Sep 14 10:48:48.900200 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.900202 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3a0b37cc-bde2-4894-930d-d4d9b9345568-etc-systemd\") pod \"tuned-6xfsw\" (UID: \"3a0b37cc-bde2-4894-930d-d4d9b9345568\") " pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" Sep 14 10:48:48.900963 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.900223 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e95ca08f-2a1d-45b7-b0de-959c0b546a18-host-run-netns\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.900963 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.900239 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnzhg\" (UniqueName: \"kubernetes.io/projected/e95ca08f-2a1d-45b7-b0de-959c0b546a18-kube-api-access-tnzhg\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.900963 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.900258 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/07adfe8a-0f38-499a-b903-f8de96458395-socket-dir\") pod \"aws-ebs-csi-driver-node-b98zp\" (UID: \"07adfe8a-0f38-499a-b903-f8de96458395\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b98zp" Sep 14 10:48:48.900963 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.900283 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a0b37cc-bde2-4894-930d-d4d9b9345568-sys\") pod \"tuned-6xfsw\" (UID: \"3a0b37cc-bde2-4894-930d-d4d9b9345568\") " pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" Sep 14 10:48:48.900963 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.900299 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsx7m\" (UniqueName: \"kubernetes.io/projected/5b6399c5-592d-463e-a9a4-337834d10604-kube-api-access-fsx7m\") pod \"node-ca-rfppn\" (UID: \"5b6399c5-592d-463e-a9a4-337834d10604\") " pod="openshift-image-registry/node-ca-rfppn" Sep 14 10:48:48.900963 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.900302 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/07adfe8a-0f38-499a-b903-f8de96458395-registration-dir\") pod \"aws-ebs-csi-driver-node-b98zp\" (UID: \"07adfe8a-0f38-499a-b903-f8de96458395\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b98zp" Sep 14 10:48:48.900963 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.900317 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e95ca08f-2a1d-45b7-b0de-959c0b546a18-system-cni-dir\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.900963 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.900334 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e95ca08f-2a1d-45b7-b0de-959c0b546a18-host-var-lib-cni-bin\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.900963 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.900354 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e95ca08f-2a1d-45b7-b0de-959c0b546a18-multus-daemon-config\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.900963 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.900401 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/07adfe8a-0f38-499a-b903-f8de96458395-etc-selinux\") pod \"aws-ebs-csi-driver-node-b98zp\" (UID: \"07adfe8a-0f38-499a-b903-f8de96458395\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b98zp" Sep 14 10:48:48.900963 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.900439 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/07adfe8a-0f38-499a-b903-f8de96458395-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b98zp\" (UID: \"07adfe8a-0f38-499a-b903-f8de96458395\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b98zp" Sep 14 10:48:48.900963 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.900467 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e95ca08f-2a1d-45b7-b0de-959c0b546a18-host-var-lib-cni-multus\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.900963 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.900497 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e95ca08f-2a1d-45b7-b0de-959c0b546a18-host-var-lib-kubelet\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.900963 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.900522 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e95ca08f-2a1d-45b7-b0de-959c0b546a18-etc-kubernetes\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.900963 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.900546 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3a0b37cc-bde2-4894-930d-d4d9b9345568-etc-sysconfig\") pod \"tuned-6xfsw\" (UID: \"3a0b37cc-bde2-4894-930d-d4d9b9345568\") " pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" Sep 14 10:48:48.900963 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.900569 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a0b37cc-bde2-4894-930d-d4d9b9345568-var-lib-kubelet\") pod \"tuned-6xfsw\" (UID: \"3a0b37cc-bde2-4894-930d-d4d9b9345568\") " pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" Sep 14 10:48:48.900963 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.900597 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8czk5\" (UniqueName: \"kubernetes.io/projected/3a0b37cc-bde2-4894-930d-d4d9b9345568-kube-api-access-8czk5\") pod \"tuned-6xfsw\" (UID: \"3a0b37cc-bde2-4894-930d-d4d9b9345568\") " pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" Sep 14 10:48:48.901651 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.900627 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e95ca08f-2a1d-45b7-b0de-959c0b546a18-multus-cni-dir\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.901651 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.900653 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3a0b37cc-bde2-4894-930d-d4d9b9345568-run\") pod \"tuned-6xfsw\" (UID: \"3a0b37cc-bde2-4894-930d-d4d9b9345568\") " pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" Sep 14 10:48:48.901651 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.900680 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5b6399c5-592d-463e-a9a4-337834d10604-host\") pod \"node-ca-rfppn\" (UID: \"5b6399c5-592d-463e-a9a4-337834d10604\") " pod="openshift-image-registry/node-ca-rfppn" Sep 14 10:48:48.901651 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.900708 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e95ca08f-2a1d-45b7-b0de-959c0b546a18-cni-binary-copy\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.901651 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.900735 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3a0b37cc-bde2-4894-930d-d4d9b9345568-etc-sysctl-conf\") pod \"tuned-6xfsw\" (UID: \"3a0b37cc-bde2-4894-930d-d4d9b9345568\") " pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" Sep 14 10:48:48.901651 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.900760 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5b6399c5-592d-463e-a9a4-337834d10604-serviceca\") pod \"node-ca-rfppn\" (UID: \"5b6399c5-592d-463e-a9a4-337834d10604\") " pod="openshift-image-registry/node-ca-rfppn" Sep 14 10:48:48.901651 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.900763 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/07adfe8a-0f38-499a-b903-f8de96458395-sys-fs\") pod \"aws-ebs-csi-driver-node-b98zp\" (UID: \"07adfe8a-0f38-499a-b903-f8de96458395\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b98zp" Sep 14 10:48:48.901651 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.900846 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e95ca08f-2a1d-45b7-b0de-959c0b546a18-multus-socket-dir-parent\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.901651 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.900894 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e95ca08f-2a1d-45b7-b0de-959c0b546a18-hostroot\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.901651 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.900985 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3a0b37cc-bde2-4894-930d-d4d9b9345568-etc-sysctl-conf\") pod \"tuned-6xfsw\" (UID: \"3a0b37cc-bde2-4894-930d-d4d9b9345568\") " pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" Sep 14 10:48:48.901651 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.901056 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a0b37cc-bde2-4894-930d-d4d9b9345568-etc-kubernetes\") pod \"tuned-6xfsw\" (UID: \"3a0b37cc-bde2-4894-930d-d4d9b9345568\") " pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" Sep 14 10:48:48.901651 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.901069 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/07adfe8a-0f38-499a-b903-f8de96458395-etc-selinux\") pod \"aws-ebs-csi-driver-node-b98zp\" (UID: \"07adfe8a-0f38-499a-b903-f8de96458395\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b98zp" Sep 14 10:48:48.901651 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.901126 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/07adfe8a-0f38-499a-b903-f8de96458395-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b98zp\" (UID: \"07adfe8a-0f38-499a-b903-f8de96458395\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b98zp" Sep 14 10:48:48.901651 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.901172 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e95ca08f-2a1d-45b7-b0de-959c0b546a18-host-var-lib-cni-multus\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.901651 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.901219 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e95ca08f-2a1d-45b7-b0de-959c0b546a18-host-var-lib-kubelet\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.901651 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.901271 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e95ca08f-2a1d-45b7-b0de-959c0b546a18-etc-kubernetes\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.901651 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.901326 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3a0b37cc-bde2-4894-930d-d4d9b9345568-etc-sysconfig\") pod \"tuned-6xfsw\" (UID: \"3a0b37cc-bde2-4894-930d-d4d9b9345568\") " pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" Sep 14 10:48:48.901651 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.901379 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a0b37cc-bde2-4894-930d-d4d9b9345568-var-lib-kubelet\") pod \"tuned-6xfsw\" (UID: \"3a0b37cc-bde2-4894-930d-d4d9b9345568\") " pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" Sep 14 10:48:48.902697 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.899963 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/07adfe8a-0f38-499a-b903-f8de96458395-device-dir\") pod \"aws-ebs-csi-driver-node-b98zp\" (UID: \"07adfe8a-0f38-499a-b903-f8de96458395\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b98zp" Sep 14 10:48:48.902697 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.901405 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e95ca08f-2a1d-45b7-b0de-959c0b546a18-multus-daemon-config\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.902697 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.900258 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3a0b37cc-bde2-4894-930d-d4d9b9345568-etc-modprobe-d\") pod \"tuned-6xfsw\" (UID: \"3a0b37cc-bde2-4894-930d-d4d9b9345568\") " pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" Sep 14 10:48:48.902697 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.901515 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a0b37cc-bde2-4894-930d-d4d9b9345568-sys\") pod \"tuned-6xfsw\" (UID: \"3a0b37cc-bde2-4894-930d-d4d9b9345568\") " pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" Sep 14 10:48:48.902697 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.901649 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e95ca08f-2a1d-45b7-b0de-959c0b546a18-multus-cni-dir\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.902697 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.901703 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3a0b37cc-bde2-4894-930d-d4d9b9345568-run\") pod \"tuned-6xfsw\" (UID: \"3a0b37cc-bde2-4894-930d-d4d9b9345568\") " pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" Sep 14 10:48:48.902697 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.901747 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5b6399c5-592d-463e-a9a4-337834d10604-host\") pod \"node-ca-rfppn\" (UID: \"5b6399c5-592d-463e-a9a4-337834d10604\") " pod="openshift-image-registry/node-ca-rfppn" Sep 14 10:48:48.902697 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.901782 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e95ca08f-2a1d-45b7-b0de-959c0b546a18-system-cni-dir\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.902697 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.901795 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3a0b37cc-bde2-4894-930d-d4d9b9345568-etc-systemd\") pod \"tuned-6xfsw\" (UID: \"3a0b37cc-bde2-4894-930d-d4d9b9345568\") " pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" Sep 14 10:48:48.902697 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.901838 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e95ca08f-2a1d-45b7-b0de-959c0b546a18-host-var-lib-cni-bin\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.902697 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.901855 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e95ca08f-2a1d-45b7-b0de-959c0b546a18-host-run-netns\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.902697 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.902045 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/07adfe8a-0f38-499a-b903-f8de96458395-socket-dir\") pod \"aws-ebs-csi-driver-node-b98zp\" (UID: \"07adfe8a-0f38-499a-b903-f8de96458395\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b98zp" Sep 14 10:48:48.902697 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.902233 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e95ca08f-2a1d-45b7-b0de-959c0b546a18-cni-binary-copy\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.904295 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.904271 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a0b37cc-bde2-4894-930d-d4d9b9345568-tmp\") pod \"tuned-6xfsw\" (UID: \"3a0b37cc-bde2-4894-930d-d4d9b9345568\") " pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" Sep 14 10:48:48.904398 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.904334 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3a0b37cc-bde2-4894-930d-d4d9b9345568-etc-tuned\") pod \"tuned-6xfsw\" (UID: \"3a0b37cc-bde2-4894-930d-d4d9b9345568\") " pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" Sep 14 10:48:48.912007 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.911153 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnzhg\" (UniqueName: \"kubernetes.io/projected/e95ca08f-2a1d-45b7-b0de-959c0b546a18-kube-api-access-tnzhg\") pod \"multus-zblrl\" (UID: \"e95ca08f-2a1d-45b7-b0de-959c0b546a18\") " pod="openshift-multus/multus-zblrl" Sep 14 10:48:48.912007 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.911579 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8czk5\" (UniqueName: \"kubernetes.io/projected/3a0b37cc-bde2-4894-930d-d4d9b9345568-kube-api-access-8czk5\") pod \"tuned-6xfsw\" (UID: \"3a0b37cc-bde2-4894-930d-d4d9b9345568\") " pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" Sep 14 10:48:48.912007 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.911957 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsx7m\" (UniqueName: \"kubernetes.io/projected/5b6399c5-592d-463e-a9a4-337834d10604-kube-api-access-fsx7m\") pod \"node-ca-rfppn\" (UID: \"5b6399c5-592d-463e-a9a4-337834d10604\") " pod="openshift-image-registry/node-ca-rfppn" Sep 14 10:48:48.914590 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.914266 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-td5f9\" (UniqueName: \"kubernetes.io/projected/04ff0710-e09c-48e9-9769-7ffce3645b66-kube-api-access-td5f9\") pod \"network-metrics-daemon-29jrs\" (UID: \"04ff0710-e09c-48e9-9769-7ffce3645b66\") " pod="openshift-multus/network-metrics-daemon-29jrs" Sep 14 10:48:48.914590 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.914449 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqx5x\" (UniqueName: \"kubernetes.io/projected/07adfe8a-0f38-499a-b903-f8de96458395-kube-api-access-jqx5x\") pod \"aws-ebs-csi-driver-node-b98zp\" (UID: \"07adfe8a-0f38-499a-b903-f8de96458395\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b98zp" Sep 14 10:48:48.986889 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:48.986855 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-js9xt" Sep 14 10:48:49.000485 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:49.000459 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5nq5v" Sep 14 10:48:49.011567 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:49.011540 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rfppn" Sep 14 10:48:49.019481 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:49.019459 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nfqwx" Sep 14 10:48:49.030459 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:49.030437 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lh4nz" Sep 14 10:48:49.038575 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:49.038549 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:48:49.050546 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:49.050517 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b98zp" Sep 14 10:48:49.060548 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:49.060524 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" Sep 14 10:48:49.067504 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:49.067473 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zblrl" Sep 14 10:48:49.305202 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:49.305103 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f7b7b9c-eb99-4eb6-b619-8db12670cef0-original-pull-secret\") pod \"global-pull-secret-syncer-b62cb\" (UID: \"6f7b7b9c-eb99-4eb6-b619-8db12670cef0\") " pod="kube-system/global-pull-secret-syncer-b62cb" Sep 14 10:48:49.305202 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:49.305184 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh7zw\" (UniqueName: \"kubernetes.io/projected/54a7a889-af23-436c-ac67-8b45c1b39194-kube-api-access-mh7zw\") pod \"network-check-target-tw9zw\" (UID: \"54a7a889-af23-436c-ac67-8b45c1b39194\") " pod="openshift-network-diagnostics/network-check-target-tw9zw" Sep 14 10:48:49.305477 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:49.305291 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 10:48:49.305477 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:49.305328 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 10:48:49.305477 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:49.305344 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 10:48:49.305477 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:49.305357 2581 projected.go:196] Error preparing data for projected volume kube-api-access-mh7zw for pod openshift-network-diagnostics/network-check-target-tw9zw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 10:48:49.305477 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:49.305396 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f7b7b9c-eb99-4eb6-b619-8db12670cef0-original-pull-secret podName:6f7b7b9c-eb99-4eb6-b619-8db12670cef0 nodeName:}" failed. No retries permitted until 2026-09-14 10:48:50.305371932 +0000 UTC m=+4.047773895 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f7b7b9c-eb99-4eb6-b619-8db12670cef0-original-pull-secret") pod "global-pull-secret-syncer-b62cb" (UID: "6f7b7b9c-eb99-4eb6-b619-8db12670cef0") : object "kube-system"/"original-pull-secret" not registered Sep 14 10:48:49.305477 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:49.305435 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/54a7a889-af23-436c-ac67-8b45c1b39194-kube-api-access-mh7zw podName:54a7a889-af23-436c-ac67-8b45c1b39194 nodeName:}" failed. No retries permitted until 2026-09-14 10:48:50.305415837 +0000 UTC m=+4.047817802 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mh7zw" (UniqueName: "kubernetes.io/projected/54a7a889-af23-436c-ac67-8b45c1b39194-kube-api-access-mh7zw") pod "network-check-target-tw9zw" (UID: "54a7a889-af23-436c-ac67-8b45c1b39194") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 10:48:49.405544 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:49.405502 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04ff0710-e09c-48e9-9769-7ffce3645b66-metrics-certs\") pod \"network-metrics-daemon-29jrs\" (UID: \"04ff0710-e09c-48e9-9769-7ffce3645b66\") " pod="openshift-multus/network-metrics-daemon-29jrs" Sep 14 10:48:49.405740 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:49.405650 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 10:48:49.405740 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:49.405716 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04ff0710-e09c-48e9-9769-7ffce3645b66-metrics-certs podName:04ff0710-e09c-48e9-9769-7ffce3645b66 nodeName:}" failed. No retries permitted until 2026-09-14 10:48:50.405701172 +0000 UTC m=+4.148103139 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/04ff0710-e09c-48e9-9769-7ffce3645b66-metrics-certs") pod "network-metrics-daemon-29jrs" (UID: "04ff0710-e09c-48e9-9769-7ffce3645b66") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 10:48:49.746859 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:49.746758 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-13 10:43:48 +0000 UTC" deadline="2028-05-21 17:15:03.901574586 +0000 UTC" Sep 14 10:48:49.746859 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:49.746804 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14766h26m14.15477677s" Sep 14 10:48:49.825829 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:49.825780 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29jrs" Sep 14 10:48:49.826022 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:49.825948 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29jrs" podUID="04ff0710-e09c-48e9-9769-7ffce3645b66" Sep 14 10:48:49.949606 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:49.949367 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07adfe8a_0f38_499a_b903_f8de96458395.slice/crio-472384b3d1b03f4a72914750efcf3701810449d65dbcf6edd15d7ebc9995104c WatchSource:0}: Error finding container 472384b3d1b03f4a72914750efcf3701810449d65dbcf6edd15d7ebc9995104c: Status 404 returned error can't find the container with id 472384b3d1b03f4a72914750efcf3701810449d65dbcf6edd15d7ebc9995104c Sep 14 10:48:49.952066 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:49.952044 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a0b37cc_bde2_4894_930d_d4d9b9345568.slice/crio-d0eafe83b733e6170b53630f92d4ee8d9d855509aea5b9ea05b4256a94c6f1c5 WatchSource:0}: Error finding container d0eafe83b733e6170b53630f92d4ee8d9d855509aea5b9ea05b4256a94c6f1c5: Status 404 returned error can't find the container with id d0eafe83b733e6170b53630f92d4ee8d9d855509aea5b9ea05b4256a94c6f1c5 Sep 14 10:48:49.953778 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:49.953758 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode95ca08f_2a1d_45b7_b0de_959c0b546a18.slice/crio-831e9a0614dc83acd76bba8ed4e64aca5b3cdd3e96075113380cff31c09f7fa9 WatchSource:0}: Error finding container 831e9a0614dc83acd76bba8ed4e64aca5b3cdd3e96075113380cff31c09f7fa9: Status 404 returned error can't find the container with id 831e9a0614dc83acd76bba8ed4e64aca5b3cdd3e96075113380cff31c09f7fa9 Sep 14 10:48:49.954220 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:49.954182 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd152261f_2616_406f_b9df_22a9614deafe.slice/crio-e905358abca664e5c085e06345b3a64ed32e95d6c7ec6df702e1feafd1fa6140 WatchSource:0}: Error finding container e905358abca664e5c085e06345b3a64ed32e95d6c7ec6df702e1feafd1fa6140: Status 404 returned error can't find the container with id e905358abca664e5c085e06345b3a64ed32e95d6c7ec6df702e1feafd1fa6140 Sep 14 10:48:49.956288 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:49.955664 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cc7f97c_133f_4c01_a394_1e06cb81d372.slice/crio-ef7ee9e8713d48b8a027c04d2622d30224ecce1281a171e5bc40ca9c5f741848 WatchSource:0}: Error finding container ef7ee9e8713d48b8a027c04d2622d30224ecce1281a171e5bc40ca9c5f741848: Status 404 returned error can't find the container with id ef7ee9e8713d48b8a027c04d2622d30224ecce1281a171e5bc40ca9c5f741848 Sep 14 10:48:49.980395 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:49.980369 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd4585d7_6ffb_42c6_b71e_478af8e902cc.slice/crio-a9890c61d3d90632fd94ea9b7f0e9ac425cefadae3ff7ef17e659c624233282f WatchSource:0}: Error finding container a9890c61d3d90632fd94ea9b7f0e9ac425cefadae3ff7ef17e659c624233282f: Status 404 returned error can't find the container with id a9890c61d3d90632fd94ea9b7f0e9ac425cefadae3ff7ef17e659c624233282f Sep 14 10:48:49.981937 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:49.981906 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceba844b_2781_40ea_96e0_9eacd1e13ef6.slice/crio-8ddd5de1db69398aba64478c9f6932ba2384c9ec862adef0d09ca3fbe772cf7e WatchSource:0}: Error finding container 8ddd5de1db69398aba64478c9f6932ba2384c9ec862adef0d09ca3fbe772cf7e: Status 404 returned error can't find the container with id 8ddd5de1db69398aba64478c9f6932ba2384c9ec862adef0d09ca3fbe772cf7e Sep 14 10:48:49.982470 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:48:49.982449 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b6399c5_592d_463e_a9a4_337834d10604.slice/crio-c25037441915220080bcc5625dad438889fdbfcda500c635f720a3e659b04f1c WatchSource:0}: Error finding container c25037441915220080bcc5625dad438889fdbfcda500c635f720a3e659b04f1c: Status 404 returned error can't find the container with id c25037441915220080bcc5625dad438889fdbfcda500c635f720a3e659b04f1c Sep 14 10:48:50.313718 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:50.313629 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f7b7b9c-eb99-4eb6-b619-8db12670cef0-original-pull-secret\") pod \"global-pull-secret-syncer-b62cb\" (UID: \"6f7b7b9c-eb99-4eb6-b619-8db12670cef0\") " pod="kube-system/global-pull-secret-syncer-b62cb" Sep 14 10:48:50.313718 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:50.313685 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh7zw\" (UniqueName: \"kubernetes.io/projected/54a7a889-af23-436c-ac67-8b45c1b39194-kube-api-access-mh7zw\") pod \"network-check-target-tw9zw\" (UID: \"54a7a889-af23-436c-ac67-8b45c1b39194\") " pod="openshift-network-diagnostics/network-check-target-tw9zw" Sep 14 10:48:50.313885 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:50.313782 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 10:48:50.313885 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:50.313789 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 10:48:50.313885 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:50.313807 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 10:48:50.313885 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:50.313816 2581 projected.go:196] Error preparing data for projected volume kube-api-access-mh7zw for pod openshift-network-diagnostics/network-check-target-tw9zw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 10:48:50.313885 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:50.313854 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f7b7b9c-eb99-4eb6-b619-8db12670cef0-original-pull-secret podName:6f7b7b9c-eb99-4eb6-b619-8db12670cef0 nodeName:}" failed. No retries permitted until 2026-09-14 10:48:52.313836013 +0000 UTC m=+6.056237961 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f7b7b9c-eb99-4eb6-b619-8db12670cef0-original-pull-secret") pod "global-pull-secret-syncer-b62cb" (UID: "6f7b7b9c-eb99-4eb6-b619-8db12670cef0") : object "kube-system"/"original-pull-secret" not registered Sep 14 10:48:50.313885 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:50.313867 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/54a7a889-af23-436c-ac67-8b45c1b39194-kube-api-access-mh7zw podName:54a7a889-af23-436c-ac67-8b45c1b39194 nodeName:}" failed. No retries permitted until 2026-09-14 10:48:52.313861436 +0000 UTC m=+6.056263384 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mh7zw" (UniqueName: "kubernetes.io/projected/54a7a889-af23-436c-ac67-8b45c1b39194-kube-api-access-mh7zw") pod "network-check-target-tw9zw" (UID: "54a7a889-af23-436c-ac67-8b45c1b39194") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 10:48:50.414220 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:50.414188 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04ff0710-e09c-48e9-9769-7ffce3645b66-metrics-certs\") pod \"network-metrics-daemon-29jrs\" (UID: \"04ff0710-e09c-48e9-9769-7ffce3645b66\") " pod="openshift-multus/network-metrics-daemon-29jrs" Sep 14 10:48:50.414376 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:50.414300 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 10:48:50.414376 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:50.414373 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04ff0710-e09c-48e9-9769-7ffce3645b66-metrics-certs podName:04ff0710-e09c-48e9-9769-7ffce3645b66 nodeName:}" failed. No retries permitted until 2026-09-14 10:48:52.414359033 +0000 UTC m=+6.156760982 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/04ff0710-e09c-48e9-9769-7ffce3645b66-metrics-certs") pod "network-metrics-daemon-29jrs" (UID: "04ff0710-e09c-48e9-9769-7ffce3645b66") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 10:48:50.747237 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:50.747121 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-13 10:43:48 +0000 UTC" deadline="2028-05-26 07:29:49.391285646 +0000 UTC" Sep 14 10:48:50.747237 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:50.747185 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14876h40m58.644104733s" Sep 14 10:48:50.828507 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:50.828474 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw9zw" Sep 14 10:48:50.828662 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:50.828607 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tw9zw" podUID="54a7a889-af23-436c-ac67-8b45c1b39194" Sep 14 10:48:50.828731 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:50.828692 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b62cb" Sep 14 10:48:50.828780 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:50.828763 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b62cb" podUID="6f7b7b9c-eb99-4eb6-b619-8db12670cef0" Sep 14 10:48:50.857197 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:50.856790 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-53.ec2.internal" event={"ID":"3ed51660b0e0cea0ccab4eff146b4e99","Type":"ContainerStarted","Data":"dce6d572efbdb1759290d4b020e1e93c74d1d4447579b26abfb4d6655152a63b"} Sep 14 10:48:50.864866 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:50.864714 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lh4nz" event={"ID":"bd4585d7-6ffb-42c6-b71e-478af8e902cc","Type":"ContainerStarted","Data":"a9890c61d3d90632fd94ea9b7f0e9ac425cefadae3ff7ef17e659c624233282f"} Sep 14 10:48:50.871723 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:50.871670 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-53.ec2.internal" podStartSLOduration=3.871650508 podStartE2EDuration="3.871650508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 10:48:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:48:50.870896439 +0000 UTC m=+4.613298411" watchObservedRunningTime="2026-09-14 10:48:50.871650508 +0000 UTC m=+4.614052479" Sep 14 10:48:50.880356 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:50.880305 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b98zp" event={"ID":"07adfe8a-0f38-499a-b903-f8de96458395","Type":"ContainerStarted","Data":"472384b3d1b03f4a72914750efcf3701810449d65dbcf6edd15d7ebc9995104c"} Sep 14 10:48:50.892684 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:50.891827 2581 generic.go:356] "Generic (PLEG): container finished" podID="ed4e451484feca11635366f26cff0482" containerID="ec1831db7dbcd431256ae29b076240263fa3d1a02ea0f26027502b6a86832f29" exitCode=0 Sep 14 10:48:50.892684 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:50.891914 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-53.ec2.internal" event={"ID":"ed4e451484feca11635366f26cff0482","Type":"ContainerDied","Data":"ec1831db7dbcd431256ae29b076240263fa3d1a02ea0f26027502b6a86832f29"} Sep 14 10:48:50.897554 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:50.897525 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5nq5v" event={"ID":"ceba844b-2781-40ea-96e0-9eacd1e13ef6","Type":"ContainerStarted","Data":"8ddd5de1db69398aba64478c9f6932ba2384c9ec862adef0d09ca3fbe772cf7e"} Sep 14 10:48:50.916869 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:50.916787 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rfppn" event={"ID":"5b6399c5-592d-463e-a9a4-337834d10604","Type":"ContainerStarted","Data":"c25037441915220080bcc5625dad438889fdbfcda500c635f720a3e659b04f1c"} Sep 14 10:48:50.931945 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:50.931913 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nfqwx" event={"ID":"f44ee762-76b2-4d0a-aabd-8b68c6aaffdb","Type":"ContainerStarted","Data":"f67a3c3abf51fa3a74c5dafd8403f7791420bf7fd99fcb44ef2841ef22da6a7b"} Sep 14 10:48:50.940142 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:50.940109 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-js9xt" event={"ID":"5cc7f97c-133f-4c01-a394-1e06cb81d372","Type":"ContainerStarted","Data":"ef7ee9e8713d48b8a027c04d2622d30224ecce1281a171e5bc40ca9c5f741848"} Sep 14 10:48:50.958533 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:50.958440 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zblrl" event={"ID":"e95ca08f-2a1d-45b7-b0de-959c0b546a18","Type":"ContainerStarted","Data":"831e9a0614dc83acd76bba8ed4e64aca5b3cdd3e96075113380cff31c09f7fa9"} Sep 14 10:48:50.976857 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:50.976796 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" event={"ID":"3a0b37cc-bde2-4894-930d-d4d9b9345568","Type":"ContainerStarted","Data":"d0eafe83b733e6170b53630f92d4ee8d9d855509aea5b9ea05b4256a94c6f1c5"} Sep 14 10:48:50.985925 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:50.985880 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" event={"ID":"d152261f-2616-406f-b9df-22a9614deafe","Type":"ContainerStarted","Data":"e905358abca664e5c085e06345b3a64ed32e95d6c7ec6df702e1feafd1fa6140"} Sep 14 10:48:51.825976 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:51.825942 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29jrs" Sep 14 10:48:51.826592 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:51.826094 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29jrs" podUID="04ff0710-e09c-48e9-9769-7ffce3645b66" Sep 14 10:48:52.026821 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:52.026785 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-53.ec2.internal" event={"ID":"ed4e451484feca11635366f26cff0482","Type":"ContainerStarted","Data":"5c1712486ede16edc6e51f329070a4a2dbe62625a3b3b2b3779f81bc23a706ba"} Sep 14 10:48:52.045598 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:52.045245 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-53.ec2.internal" podStartSLOduration=5.045222493 podStartE2EDuration="5.045222493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 10:48:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:48:52.043462074 +0000 UTC m=+5.785864045" watchObservedRunningTime="2026-09-14 10:48:52.045222493 +0000 UTC m=+5.787624465" Sep 14 10:48:52.327629 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:52.327586 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh7zw\" (UniqueName: \"kubernetes.io/projected/54a7a889-af23-436c-ac67-8b45c1b39194-kube-api-access-mh7zw\") pod \"network-check-target-tw9zw\" (UID: \"54a7a889-af23-436c-ac67-8b45c1b39194\") " pod="openshift-network-diagnostics/network-check-target-tw9zw" Sep 14 10:48:52.327811 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:52.327681 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f7b7b9c-eb99-4eb6-b619-8db12670cef0-original-pull-secret\") pod \"global-pull-secret-syncer-b62cb\" (UID: \"6f7b7b9c-eb99-4eb6-b619-8db12670cef0\") " pod="kube-system/global-pull-secret-syncer-b62cb" Sep 14 10:48:52.327869 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:52.327829 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 10:48:52.327929 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:52.327894 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f7b7b9c-eb99-4eb6-b619-8db12670cef0-original-pull-secret podName:6f7b7b9c-eb99-4eb6-b619-8db12670cef0 nodeName:}" failed. No retries permitted until 2026-09-14 10:48:56.327876671 +0000 UTC m=+10.070278636 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f7b7b9c-eb99-4eb6-b619-8db12670cef0-original-pull-secret") pod "global-pull-secret-syncer-b62cb" (UID: "6f7b7b9c-eb99-4eb6-b619-8db12670cef0") : object "kube-system"/"original-pull-secret" not registered Sep 14 10:48:52.328388 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:52.328368 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 10:48:52.328479 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:52.328396 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 10:48:52.328479 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:52.328408 2581 projected.go:196] Error preparing data for projected volume kube-api-access-mh7zw for pod openshift-network-diagnostics/network-check-target-tw9zw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 10:48:52.328479 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:52.328454 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/54a7a889-af23-436c-ac67-8b45c1b39194-kube-api-access-mh7zw podName:54a7a889-af23-436c-ac67-8b45c1b39194 nodeName:}" failed. No retries permitted until 2026-09-14 10:48:56.328439846 +0000 UTC m=+10.070841809 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mh7zw" (UniqueName: "kubernetes.io/projected/54a7a889-af23-436c-ac67-8b45c1b39194-kube-api-access-mh7zw") pod "network-check-target-tw9zw" (UID: "54a7a889-af23-436c-ac67-8b45c1b39194") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 10:48:52.428771 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:52.428729 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04ff0710-e09c-48e9-9769-7ffce3645b66-metrics-certs\") pod \"network-metrics-daemon-29jrs\" (UID: \"04ff0710-e09c-48e9-9769-7ffce3645b66\") " pod="openshift-multus/network-metrics-daemon-29jrs" Sep 14 10:48:52.428969 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:52.428951 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 10:48:52.429034 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:52.429024 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04ff0710-e09c-48e9-9769-7ffce3645b66-metrics-certs podName:04ff0710-e09c-48e9-9769-7ffce3645b66 nodeName:}" failed. No retries permitted until 2026-09-14 10:48:56.429004005 +0000 UTC m=+10.171405966 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/04ff0710-e09c-48e9-9769-7ffce3645b66-metrics-certs") pod "network-metrics-daemon-29jrs" (UID: "04ff0710-e09c-48e9-9769-7ffce3645b66") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 10:48:52.826601 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:52.826448 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw9zw" Sep 14 10:48:52.826601 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:52.826584 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tw9zw" podUID="54a7a889-af23-436c-ac67-8b45c1b39194" Sep 14 10:48:52.827119 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:52.826962 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b62cb" Sep 14 10:48:52.827119 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:52.827051 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b62cb" podUID="6f7b7b9c-eb99-4eb6-b619-8db12670cef0" Sep 14 10:48:53.826408 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:53.825830 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29jrs" Sep 14 10:48:53.826408 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:53.825989 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29jrs" podUID="04ff0710-e09c-48e9-9769-7ffce3645b66" Sep 14 10:48:54.828374 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:54.827729 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw9zw" Sep 14 10:48:54.828374 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:54.827856 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tw9zw" podUID="54a7a889-af23-436c-ac67-8b45c1b39194" Sep 14 10:48:54.828374 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:54.828215 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b62cb" Sep 14 10:48:54.828374 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:54.828306 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b62cb" podUID="6f7b7b9c-eb99-4eb6-b619-8db12670cef0" Sep 14 10:48:55.826271 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:55.825795 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29jrs" Sep 14 10:48:55.826271 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:55.825931 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29jrs" podUID="04ff0710-e09c-48e9-9769-7ffce3645b66" Sep 14 10:48:56.362243 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:56.362208 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f7b7b9c-eb99-4eb6-b619-8db12670cef0-original-pull-secret\") pod \"global-pull-secret-syncer-b62cb\" (UID: \"6f7b7b9c-eb99-4eb6-b619-8db12670cef0\") " pod="kube-system/global-pull-secret-syncer-b62cb" Sep 14 10:48:56.362710 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:56.362274 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh7zw\" (UniqueName: \"kubernetes.io/projected/54a7a889-af23-436c-ac67-8b45c1b39194-kube-api-access-mh7zw\") pod \"network-check-target-tw9zw\" (UID: \"54a7a889-af23-436c-ac67-8b45c1b39194\") " pod="openshift-network-diagnostics/network-check-target-tw9zw" Sep 14 10:48:56.362710 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:56.362399 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 10:48:56.362710 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:56.362436 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 10:48:56.362710 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:56.362454 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 10:48:56.362710 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:56.362466 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f7b7b9c-eb99-4eb6-b619-8db12670cef0-original-pull-secret podName:6f7b7b9c-eb99-4eb6-b619-8db12670cef0 nodeName:}" failed. No retries permitted until 2026-09-14 10:49:04.362451832 +0000 UTC m=+18.104853795 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f7b7b9c-eb99-4eb6-b619-8db12670cef0-original-pull-secret") pod "global-pull-secret-syncer-b62cb" (UID: "6f7b7b9c-eb99-4eb6-b619-8db12670cef0") : object "kube-system"/"original-pull-secret" not registered Sep 14 10:48:56.362710 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:56.362467 2581 projected.go:196] Error preparing data for projected volume kube-api-access-mh7zw for pod openshift-network-diagnostics/network-check-target-tw9zw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 10:48:56.362710 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:56.362522 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/54a7a889-af23-436c-ac67-8b45c1b39194-kube-api-access-mh7zw podName:54a7a889-af23-436c-ac67-8b45c1b39194 nodeName:}" failed. No retries permitted until 2026-09-14 10:49:04.362507245 +0000 UTC m=+18.104909215 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mh7zw" (UniqueName: "kubernetes.io/projected/54a7a889-af23-436c-ac67-8b45c1b39194-kube-api-access-mh7zw") pod "network-check-target-tw9zw" (UID: "54a7a889-af23-436c-ac67-8b45c1b39194") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 10:48:56.463393 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:56.463265 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04ff0710-e09c-48e9-9769-7ffce3645b66-metrics-certs\") pod \"network-metrics-daemon-29jrs\" (UID: \"04ff0710-e09c-48e9-9769-7ffce3645b66\") " pod="openshift-multus/network-metrics-daemon-29jrs" Sep 14 10:48:56.463635 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:56.463453 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 10:48:56.463635 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:56.463528 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04ff0710-e09c-48e9-9769-7ffce3645b66-metrics-certs podName:04ff0710-e09c-48e9-9769-7ffce3645b66 nodeName:}" failed. No retries permitted until 2026-09-14 10:49:04.46350842 +0000 UTC m=+18.205910382 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/04ff0710-e09c-48e9-9769-7ffce3645b66-metrics-certs") pod "network-metrics-daemon-29jrs" (UID: "04ff0710-e09c-48e9-9769-7ffce3645b66") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 10:48:56.826414 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:56.826317 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw9zw" Sep 14 10:48:56.826576 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:56.826441 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tw9zw" podUID="54a7a889-af23-436c-ac67-8b45c1b39194" Sep 14 10:48:56.826792 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:56.826778 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b62cb" Sep 14 10:48:56.826867 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:56.826842 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b62cb" podUID="6f7b7b9c-eb99-4eb6-b619-8db12670cef0" Sep 14 10:48:57.825986 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:57.825526 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29jrs" Sep 14 10:48:57.825986 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:57.825655 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29jrs" podUID="04ff0710-e09c-48e9-9769-7ffce3645b66" Sep 14 10:48:58.825994 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:58.825953 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b62cb" Sep 14 10:48:58.826467 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:58.826002 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw9zw" Sep 14 10:48:58.826467 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:58.826103 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b62cb" podUID="6f7b7b9c-eb99-4eb6-b619-8db12670cef0" Sep 14 10:48:58.826467 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:58.826226 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tw9zw" podUID="54a7a889-af23-436c-ac67-8b45c1b39194" Sep 14 10:48:59.826064 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:48:59.826032 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29jrs" Sep 14 10:48:59.826540 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:48:59.826152 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29jrs" podUID="04ff0710-e09c-48e9-9769-7ffce3645b66" Sep 14 10:49:00.826422 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:00.826386 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw9zw" Sep 14 10:49:00.826422 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:00.826416 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b62cb" Sep 14 10:49:00.826890 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:00.826515 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tw9zw" podUID="54a7a889-af23-436c-ac67-8b45c1b39194" Sep 14 10:49:00.826890 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:00.826640 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b62cb" podUID="6f7b7b9c-eb99-4eb6-b619-8db12670cef0" Sep 14 10:49:01.825903 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:01.825844 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29jrs" Sep 14 10:49:01.826113 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:01.825978 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29jrs" podUID="04ff0710-e09c-48e9-9769-7ffce3645b66" Sep 14 10:49:02.825568 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:02.825529 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw9zw" Sep 14 10:49:02.826032 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:02.825664 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tw9zw" podUID="54a7a889-af23-436c-ac67-8b45c1b39194" Sep 14 10:49:02.826032 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:02.825726 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b62cb" Sep 14 10:49:02.826032 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:02.825863 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b62cb" podUID="6f7b7b9c-eb99-4eb6-b619-8db12670cef0" Sep 14 10:49:03.825621 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:03.825582 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29jrs" Sep 14 10:49:03.826189 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:03.825701 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29jrs" podUID="04ff0710-e09c-48e9-9769-7ffce3645b66" Sep 14 10:49:04.426445 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:04.426407 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh7zw\" (UniqueName: \"kubernetes.io/projected/54a7a889-af23-436c-ac67-8b45c1b39194-kube-api-access-mh7zw\") pod \"network-check-target-tw9zw\" (UID: \"54a7a889-af23-436c-ac67-8b45c1b39194\") " pod="openshift-network-diagnostics/network-check-target-tw9zw" Sep 14 10:49:04.426684 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:04.426473 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f7b7b9c-eb99-4eb6-b619-8db12670cef0-original-pull-secret\") pod \"global-pull-secret-syncer-b62cb\" (UID: \"6f7b7b9c-eb99-4eb6-b619-8db12670cef0\") " pod="kube-system/global-pull-secret-syncer-b62cb" Sep 14 10:49:04.426684 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:04.426572 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 10:49:04.426684 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:04.426574 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 10:49:04.426684 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:04.426589 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 10:49:04.426684 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:04.426603 2581 projected.go:196] Error preparing data for projected volume kube-api-access-mh7zw for pod openshift-network-diagnostics/network-check-target-tw9zw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 10:49:04.426684 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:04.426644 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f7b7b9c-eb99-4eb6-b619-8db12670cef0-original-pull-secret podName:6f7b7b9c-eb99-4eb6-b619-8db12670cef0 nodeName:}" failed. No retries permitted until 2026-09-14 10:49:20.426623119 +0000 UTC m=+34.169025081 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f7b7b9c-eb99-4eb6-b619-8db12670cef0-original-pull-secret") pod "global-pull-secret-syncer-b62cb" (UID: "6f7b7b9c-eb99-4eb6-b619-8db12670cef0") : object "kube-system"/"original-pull-secret" not registered Sep 14 10:49:04.426684 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:04.426663 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/54a7a889-af23-436c-ac67-8b45c1b39194-kube-api-access-mh7zw podName:54a7a889-af23-436c-ac67-8b45c1b39194 nodeName:}" failed. No retries permitted until 2026-09-14 10:49:20.426654162 +0000 UTC m=+34.169056115 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mh7zw" (UniqueName: "kubernetes.io/projected/54a7a889-af23-436c-ac67-8b45c1b39194-kube-api-access-mh7zw") pod "network-check-target-tw9zw" (UID: "54a7a889-af23-436c-ac67-8b45c1b39194") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 10:49:04.527619 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:04.527586 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04ff0710-e09c-48e9-9769-7ffce3645b66-metrics-certs\") pod \"network-metrics-daemon-29jrs\" (UID: \"04ff0710-e09c-48e9-9769-7ffce3645b66\") " pod="openshift-multus/network-metrics-daemon-29jrs" Sep 14 10:49:04.527794 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:04.527708 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 10:49:04.527794 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:04.527758 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04ff0710-e09c-48e9-9769-7ffce3645b66-metrics-certs podName:04ff0710-e09c-48e9-9769-7ffce3645b66 nodeName:}" failed. No retries permitted until 2026-09-14 10:49:20.527744998 +0000 UTC m=+34.270146959 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/04ff0710-e09c-48e9-9769-7ffce3645b66-metrics-certs") pod "network-metrics-daemon-29jrs" (UID: "04ff0710-e09c-48e9-9769-7ffce3645b66") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 10:49:04.826088 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:04.826045 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw9zw" Sep 14 10:49:04.826574 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:04.826181 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tw9zw" podUID="54a7a889-af23-436c-ac67-8b45c1b39194" Sep 14 10:49:04.826574 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:04.826541 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b62cb" Sep 14 10:49:04.826724 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:04.826631 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b62cb" podUID="6f7b7b9c-eb99-4eb6-b619-8db12670cef0" Sep 14 10:49:05.825910 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:05.825691 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29jrs" Sep 14 10:49:05.826068 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:05.826018 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29jrs" podUID="04ff0710-e09c-48e9-9769-7ffce3645b66" Sep 14 10:49:06.826534 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:06.826495 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw9zw" Sep 14 10:49:06.826992 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:06.826619 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tw9zw" podUID="54a7a889-af23-436c-ac67-8b45c1b39194" Sep 14 10:49:06.826992 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:06.826677 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b62cb" Sep 14 10:49:06.826992 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:06.826813 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b62cb" podUID="6f7b7b9c-eb99-4eb6-b619-8db12670cef0" Sep 14 10:49:07.826151 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:07.826113 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29jrs" Sep 14 10:49:07.826367 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:07.826246 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29jrs" podUID="04ff0710-e09c-48e9-9769-7ffce3645b66" Sep 14 10:49:08.826125 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:08.825859 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw9zw" Sep 14 10:49:08.827008 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:08.825877 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b62cb" Sep 14 10:49:08.827008 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:08.826234 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tw9zw" podUID="54a7a889-af23-436c-ac67-8b45c1b39194" Sep 14 10:49:08.827008 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:08.826304 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b62cb" podUID="6f7b7b9c-eb99-4eb6-b619-8db12670cef0" Sep 14 10:49:09.065762 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:09.065683 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5nq5v" event={"ID":"ceba844b-2781-40ea-96e0-9eacd1e13ef6","Type":"ContainerStarted","Data":"04749ebc761a62b39ab38ac1e3dbf2a445c27e42b0e03e4f95837dbfaf99288b"} Sep 14 10:49:09.067894 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:09.067866 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rfppn" event={"ID":"5b6399c5-592d-463e-a9a4-337834d10604","Type":"ContainerStarted","Data":"f4e7022ea76aa9e0c78f4dbfa0239e55b5e30c0689bc4184e8b5deeea8971c85"} Sep 14 10:49:09.072797 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:09.072766 2581 generic.go:356] "Generic (PLEG): container finished" podID="f44ee762-76b2-4d0a-aabd-8b68c6aaffdb" containerID="ac12687eee2a9fd4b078b221632bd18724434b0117b13c3013808e66b44a28e9" exitCode=0 Sep 14 10:49:09.072978 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:09.072857 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nfqwx" event={"ID":"f44ee762-76b2-4d0a-aabd-8b68c6aaffdb","Type":"ContainerDied","Data":"ac12687eee2a9fd4b078b221632bd18724434b0117b13c3013808e66b44a28e9"} Sep 14 10:49:09.074716 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:09.074686 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-js9xt" event={"ID":"5cc7f97c-133f-4c01-a394-1e06cb81d372","Type":"ContainerStarted","Data":"15cba978a2b9254a4b31ab5980ca73bbdb64104fec2999277a08e94cac088b67"} Sep 14 10:49:09.076797 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:09.076768 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zblrl" event={"ID":"e95ca08f-2a1d-45b7-b0de-959c0b546a18","Type":"ContainerStarted","Data":"977a5b03b3908f477e4f555081caa240d7cecad27612eabcb6b63b5387063b4e"} Sep 14 10:49:09.078447 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:09.078425 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" event={"ID":"3a0b37cc-bde2-4894-930d-d4d9b9345568","Type":"ContainerStarted","Data":"7cbf24ff72a6760d35103cae6bf5e67161d699971cdb59a8d427eeb00f0aac1e"} Sep 14 10:49:09.082213 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:09.082177 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" event={"ID":"d152261f-2616-406f-b9df-22a9614deafe","Type":"ContainerStarted","Data":"fae68bf8b411b9c0c9b0c03e3e0d0457380d86183d6027ac7f27afb6ed4a28d8"} Sep 14 10:49:09.082323 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:09.082245 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" event={"ID":"d152261f-2616-406f-b9df-22a9614deafe","Type":"ContainerStarted","Data":"206821238c89646e39f32f795ec1038602f2d447552ac6058516eb32f414bf86"} Sep 14 10:49:09.082323 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:09.082275 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" event={"ID":"d152261f-2616-406f-b9df-22a9614deafe","Type":"ContainerStarted","Data":"808fa3e92ee0d269f11e3c1ccb1f35d69b4f9bf787e744f3470b637cc1b409fa"} Sep 14 10:49:09.082323 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:09.082308 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" event={"ID":"d152261f-2616-406f-b9df-22a9614deafe","Type":"ContainerStarted","Data":"4015cd2c6499f2a1d75e3062fce91edeab06e6c6536395dc7cf5ae737e850b61"} Sep 14 10:49:09.082510 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:09.082355 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" event={"ID":"d152261f-2616-406f-b9df-22a9614deafe","Type":"ContainerStarted","Data":"9db88944e1ebcb63f8663b25896c2b3915a3fd1521dd9ac4c45b0f33a2d692bd"} Sep 14 10:49:09.082510 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:09.082384 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" event={"ID":"d152261f-2616-406f-b9df-22a9614deafe","Type":"ContainerStarted","Data":"f00190c17b59ef70688eb322198ffce37c5ea4fa78fb8403a3418fcf92339ce3"} Sep 14 10:49:09.083916 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:09.083891 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b98zp" event={"ID":"07adfe8a-0f38-499a-b903-f8de96458395","Type":"ContainerStarted","Data":"1f171b18efad2f017140ad5581578b0e3b86d336b176841054b923088c51d811"} Sep 14 10:49:09.085101 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:09.085055 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5nq5v" podStartSLOduration=3.970635126 podStartE2EDuration="22.08503859s" totalImagesPullingTime="18.114403464s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 10:48:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:49:09.084136223 +0000 UTC m=+22.826538194" watchObservedRunningTime="2026-09-14 10:49:09.08503859 +0000 UTC m=+22.827440562" Sep 14 10:49:09.129018 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:09.128967 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6xfsw" podStartSLOduration=3.982045344 podStartE2EDuration="22.128951567s" totalImagesPullingTime="18.146906223s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 10:48:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:49:09.128509716 +0000 UTC m=+22.870911685" watchObservedRunningTime="2026-09-14 10:49:09.128951567 +0000 UTC m=+22.871353536" Sep 14 10:49:09.145234 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:09.145187 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rfppn" podStartSLOduration=4.033697414 podStartE2EDuration="22.145171954s" totalImagesPullingTime="18.11147454s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 10:48:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:49:09.144991653 +0000 UTC m=+22.887393620" watchObservedRunningTime="2026-09-14 10:49:09.145171954 +0000 UTC m=+22.887573924" Sep 14 10:49:09.162512 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:09.162471 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-js9xt" podStartSLOduration=11.374839645 podStartE2EDuration="22.16245481s" totalImagesPullingTime="10.787615165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:48:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:49:09.162275371 +0000 UTC m=+22.904677341" watchObservedRunningTime="2026-09-14 10:49:09.16245481 +0000 UTC m=+22.904856782" Sep 14 10:49:09.179819 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:09.179772 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-zblrl" podStartSLOduration=3.988214204 podStartE2EDuration="22.179756243s" totalImagesPullingTime="18.191542039s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 10:48:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:49:09.179039221 +0000 UTC m=+22.921441191" watchObservedRunningTime="2026-09-14 10:49:09.179756243 +0000 UTC m=+22.922158213" Sep 14 10:49:09.325155 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:09.325075 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-js9xt_5cc7f97c-133f-4c01-a394-1e06cb81d372/konnectivity-agent/0.log" limit=1024 Sep 14 10:49:09.555537 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:09.555506 2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 14 10:49:09.762537 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:09.762218 2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-14T10:49:09.555530077Z","UUID":"9b42625a-19b3-4165-843f-3b93c884b35f","Handler":null,"Name":"","Endpoint":""} Sep 14 10:49:09.763944 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:09.763917 2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 14 10:49:09.764081 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:09.763956 2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 14 10:49:09.826112 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:09.826082 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29jrs" Sep 14 10:49:09.826289 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:09.826206 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29jrs" podUID="04ff0710-e09c-48e9-9769-7ffce3645b66" Sep 14 10:49:10.089440 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:10.089397 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lh4nz" event={"ID":"bd4585d7-6ffb-42c6-b71e-478af8e902cc","Type":"ContainerStarted","Data":"3ad206f07591c8a3214c67777a3b9b3a969c705cb5b6a98db0f3a0b1179764fa"} Sep 14 10:49:10.094120 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:10.094088 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b98zp" event={"ID":"07adfe8a-0f38-499a-b903-f8de96458395","Type":"ContainerStarted","Data":"34df418932b5bd39e3b4b8f66fda29098c086c390ddbd2b0a827dc71cfb86fec"} Sep 14 10:49:10.500361 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:10.500224 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-js9xt_5cc7f97c-133f-4c01-a394-1e06cb81d372/konnectivity-agent/0.log" limit=1024 Sep 14 10:49:10.825990 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:10.825963 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b62cb" Sep 14 10:49:10.826104 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:10.826083 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b62cb" podUID="6f7b7b9c-eb99-4eb6-b619-8db12670cef0" Sep 14 10:49:10.826286 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:10.826252 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw9zw" Sep 14 10:49:10.826415 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:10.826390 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tw9zw" podUID="54a7a889-af23-436c-ac67-8b45c1b39194" Sep 14 10:49:11.100881 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:11.100840 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" event={"ID":"d152261f-2616-406f-b9df-22a9614deafe","Type":"ContainerStarted","Data":"9bd2a6e0b1e6c2c8d113f80cef2e51ea25adb2a226ed3d9da9494f5aece25b32"} Sep 14 10:49:11.103432 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:11.103397 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b98zp" event={"ID":"07adfe8a-0f38-499a-b903-f8de96458395","Type":"ContainerStarted","Data":"f03f53efc3d1ad691303c43389ad331b1f3201ea56bac85cc56d70399d0f7971"} Sep 14 10:49:11.124232 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:11.124175 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b98zp" podStartSLOduration=3.753329612 podStartE2EDuration="24.124158154s" totalImagesPullingTime="20.370828542s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 10:48:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:49:11.123997848 +0000 UTC m=+24.866399821" watchObservedRunningTime="2026-09-14 10:49:11.124158154 +0000 UTC m=+24.866560123" Sep 14 10:49:11.124873 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:11.124827 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lh4nz" podStartSLOduration=6.013193601 podStartE2EDuration="24.124817841s" totalImagesPullingTime="18.11162424s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 10:48:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:49:10.105502535 +0000 UTC m=+23.847904506" watchObservedRunningTime="2026-09-14 10:49:11.124817841 +0000 UTC m=+24.867219815" Sep 14 10:49:11.825432 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:11.825400 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29jrs" Sep 14 10:49:11.825656 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:11.825516 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29jrs" podUID="04ff0710-e09c-48e9-9769-7ffce3645b66" Sep 14 10:49:11.851209 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:11.851185 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-js9xt" Sep 14 10:49:12.433572 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:12.433537 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-js9xt" Sep 14 10:49:12.434265 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:12.434236 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-js9xt" Sep 14 10:49:12.828715 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:12.828687 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b62cb" Sep 14 10:49:12.828715 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:12.828703 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw9zw" Sep 14 10:49:12.828926 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:12.828812 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b62cb" podUID="6f7b7b9c-eb99-4eb6-b619-8db12670cef0" Sep 14 10:49:12.828982 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:12.828918 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tw9zw" podUID="54a7a889-af23-436c-ac67-8b45c1b39194" Sep 14 10:49:13.108399 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:13.108313 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-js9xt" Sep 14 10:49:13.172224 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:13.172176 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-js9xt_5cc7f97c-133f-4c01-a394-1e06cb81d372/konnectivity-agent/0.log" limit=1024 Sep 14 10:49:13.826320 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:13.826127 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29jrs" Sep 14 10:49:13.826551 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:13.826445 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29jrs" podUID="04ff0710-e09c-48e9-9769-7ffce3645b66" Sep 14 10:49:14.113495 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:14.113456 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" event={"ID":"d152261f-2616-406f-b9df-22a9614deafe","Type":"ContainerStarted","Data":"0bc52b9b3153814de1fb52b86579fc6e57694881e69083a3d40363b83199adcc"} Sep 14 10:49:14.113960 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:14.113702 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:49:14.115416 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:14.115392 2581 generic.go:356] "Generic (PLEG): container finished" podID="f44ee762-76b2-4d0a-aabd-8b68c6aaffdb" containerID="c69517d979d37029c3e01fd0d3e8176a007c52b0013f2a6f88b1ccb1bc8d5768" exitCode=0 Sep 14 10:49:14.115543 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:14.115475 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nfqwx" event={"ID":"f44ee762-76b2-4d0a-aabd-8b68c6aaffdb","Type":"ContainerDied","Data":"c69517d979d37029c3e01fd0d3e8176a007c52b0013f2a6f88b1ccb1bc8d5768"} Sep 14 10:49:14.129717 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:14.129695 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:49:14.139708 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:14.139655 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" podStartSLOduration=8.943486053 podStartE2EDuration="27.139639419s" totalImagesPullingTime="18.196153366s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 10:48:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:49:14.139038904 +0000 UTC m=+27.881440878" watchObservedRunningTime="2026-09-14 10:49:14.139639419 +0000 UTC m=+27.882041389" Sep 14 10:49:14.825426 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:14.825392 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw9zw" Sep 14 10:49:14.825679 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:14.825514 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tw9zw" podUID="54a7a889-af23-436c-ac67-8b45c1b39194" Sep 14 10:49:14.825818 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:14.825407 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b62cb" Sep 14 10:49:14.825917 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:14.825896 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b62cb" podUID="6f7b7b9c-eb99-4eb6-b619-8db12670cef0" Sep 14 10:49:15.119680 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:15.119568 2581 generic.go:356] "Generic (PLEG): container finished" podID="f44ee762-76b2-4d0a-aabd-8b68c6aaffdb" containerID="0f029094d0d4a33ce990c53e948abb4db9938e1b1c8312d5bdd98d2917e71ecf" exitCode=0 Sep 14 10:49:15.119680 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:15.119659 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nfqwx" event={"ID":"f44ee762-76b2-4d0a-aabd-8b68c6aaffdb","Type":"ContainerDied","Data":"0f029094d0d4a33ce990c53e948abb4db9938e1b1c8312d5bdd98d2917e71ecf"} Sep 14 10:49:15.120352 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:15.120120 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:49:15.120352 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:15.120209 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:49:15.137865 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:15.137836 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:49:15.177496 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:15.177455 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-js9xt_5cc7f97c-133f-4c01-a394-1e06cb81d372/konnectivity-agent/0.log" limit=1024 Sep 14 10:49:15.250309 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:15.250271 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-js9xt_5cc7f97c-133f-4c01-a394-1e06cb81d372/konnectivity-agent/0.log" limit=1024 Sep 14 10:49:15.263319 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:15.263295 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b62cb"] Sep 14 10:49:15.263493 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:15.263442 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b62cb" Sep 14 10:49:15.263572 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:15.263551 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b62cb" podUID="6f7b7b9c-eb99-4eb6-b619-8db12670cef0" Sep 14 10:49:15.266045 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:15.266018 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-29jrs"] Sep 14 10:49:15.266160 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:15.266136 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29jrs" Sep 14 10:49:15.266240 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:15.266221 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29jrs" podUID="04ff0710-e09c-48e9-9769-7ffce3645b66" Sep 14 10:49:15.281209 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:15.281179 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tw9zw"] Sep 14 10:49:15.281365 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:15.281294 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw9zw" Sep 14 10:49:15.281437 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:15.281417 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tw9zw" podUID="54a7a889-af23-436c-ac67-8b45c1b39194" Sep 14 10:49:15.315429 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:15.315403 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-js9xt_5cc7f97c-133f-4c01-a394-1e06cb81d372/konnectivity-agent/0.log" limit=1024 Sep 14 10:49:16.123777 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:16.123574 2581 generic.go:356] "Generic (PLEG): container finished" podID="f44ee762-76b2-4d0a-aabd-8b68c6aaffdb" containerID="6fba99219dcf53f452b8e176879a6e1dd24e8a8a82b042a7efac38de933c3164" exitCode=0 Sep 14 10:49:16.124169 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:16.123658 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nfqwx" event={"ID":"f44ee762-76b2-4d0a-aabd-8b68c6aaffdb","Type":"ContainerDied","Data":"6fba99219dcf53f452b8e176879a6e1dd24e8a8a82b042a7efac38de933c3164"} Sep 14 10:49:16.828167 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:16.828126 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw9zw" Sep 14 10:49:16.828358 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:16.828273 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tw9zw" podUID="54a7a889-af23-436c-ac67-8b45c1b39194" Sep 14 10:49:16.828773 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:16.828746 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b62cb" Sep 14 10:49:16.828908 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:16.828885 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b62cb" podUID="6f7b7b9c-eb99-4eb6-b619-8db12670cef0" Sep 14 10:49:16.828967 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:16.828960 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29jrs" Sep 14 10:49:16.829090 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:16.829074 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29jrs" podUID="04ff0710-e09c-48e9-9769-7ffce3645b66" Sep 14 10:49:18.825925 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:18.825877 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw9zw" Sep 14 10:49:18.826416 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:18.826015 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29jrs" Sep 14 10:49:18.826416 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:18.826024 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tw9zw" podUID="54a7a889-af23-436c-ac67-8b45c1b39194" Sep 14 10:49:18.826416 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:18.825906 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b62cb" Sep 14 10:49:18.826416 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:18.826102 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29jrs" podUID="04ff0710-e09c-48e9-9769-7ffce3645b66" Sep 14 10:49:18.826416 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:18.826185 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b62cb" podUID="6f7b7b9c-eb99-4eb6-b619-8db12670cef0" Sep 14 10:49:20.440351 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:20.440298 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f7b7b9c-eb99-4eb6-b619-8db12670cef0-original-pull-secret\") pod \"global-pull-secret-syncer-b62cb\" (UID: \"6f7b7b9c-eb99-4eb6-b619-8db12670cef0\") " pod="kube-system/global-pull-secret-syncer-b62cb" Sep 14 10:49:20.440973 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:20.440367 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh7zw\" (UniqueName: \"kubernetes.io/projected/54a7a889-af23-436c-ac67-8b45c1b39194-kube-api-access-mh7zw\") pod \"network-check-target-tw9zw\" (UID: \"54a7a889-af23-436c-ac67-8b45c1b39194\") " pod="openshift-network-diagnostics/network-check-target-tw9zw" Sep 14 10:49:20.440973 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:20.440465 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 10:49:20.440973 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:20.440485 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 10:49:20.440973 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:20.440505 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 10:49:20.440973 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:20.440519 2581 projected.go:196] Error preparing data for projected volume kube-api-access-mh7zw for pod openshift-network-diagnostics/network-check-target-tw9zw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 10:49:20.440973 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:20.440545 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f7b7b9c-eb99-4eb6-b619-8db12670cef0-original-pull-secret podName:6f7b7b9c-eb99-4eb6-b619-8db12670cef0 nodeName:}" failed. No retries permitted until 2026-09-14 10:49:52.440527139 +0000 UTC m=+66.182929114 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f7b7b9c-eb99-4eb6-b619-8db12670cef0-original-pull-secret") pod "global-pull-secret-syncer-b62cb" (UID: "6f7b7b9c-eb99-4eb6-b619-8db12670cef0") : object "kube-system"/"original-pull-secret" not registered Sep 14 10:49:20.440973 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:20.440570 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/54a7a889-af23-436c-ac67-8b45c1b39194-kube-api-access-mh7zw podName:54a7a889-af23-436c-ac67-8b45c1b39194 nodeName:}" failed. No retries permitted until 2026-09-14 10:49:52.440553489 +0000 UTC m=+66.182955443 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mh7zw" (UniqueName: "kubernetes.io/projected/54a7a889-af23-436c-ac67-8b45c1b39194-kube-api-access-mh7zw") pod "network-check-target-tw9zw" (UID: "54a7a889-af23-436c-ac67-8b45c1b39194") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 10:49:20.541638 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:20.541608 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04ff0710-e09c-48e9-9769-7ffce3645b66-metrics-certs\") pod \"network-metrics-daemon-29jrs\" (UID: \"04ff0710-e09c-48e9-9769-7ffce3645b66\") " pod="openshift-multus/network-metrics-daemon-29jrs" Sep 14 10:49:20.541797 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:20.541730 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 10:49:20.541797 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:20.541784 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04ff0710-e09c-48e9-9769-7ffce3645b66-metrics-certs podName:04ff0710-e09c-48e9-9769-7ffce3645b66 nodeName:}" failed. No retries permitted until 2026-09-14 10:49:52.541771071 +0000 UTC m=+66.284173019 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/04ff0710-e09c-48e9-9769-7ffce3645b66-metrics-certs") pod "network-metrics-daemon-29jrs" (UID: "04ff0710-e09c-48e9-9769-7ffce3645b66") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 10:49:20.826007 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:20.825972 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b62cb" Sep 14 10:49:20.826129 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:20.826014 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29jrs" Sep 14 10:49:20.826129 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:20.826095 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-b62cb" podUID="6f7b7b9c-eb99-4eb6-b619-8db12670cef0" Sep 14 10:49:20.826129 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:20.825993 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw9zw" Sep 14 10:49:20.826281 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:20.826201 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-29jrs" podUID="04ff0710-e09c-48e9-9769-7ffce3645b66" Sep 14 10:49:20.826352 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:20.826280 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tw9zw" podUID="54a7a889-af23-436c-ac67-8b45c1b39194" Sep 14 10:49:20.946565 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:20.946477 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-js9xt_5cc7f97c-133f-4c01-a394-1e06cb81d372/konnectivity-agent/0.log" limit=1024 Sep 14 10:49:21.045445 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:21.045408 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-js9xt_5cc7f97c-133f-4c01-a394-1e06cb81d372/konnectivity-agent/0.log" limit=1024 Sep 14 10:49:21.102110 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:21.102084 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-53.ec2.internal" event="NodeReady" Sep 14 10:49:21.102248 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:21.102205 2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 14 10:49:21.140167 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:21.140135 2581 generic.go:356] "Generic (PLEG): container finished" podID="f44ee762-76b2-4d0a-aabd-8b68c6aaffdb" containerID="2167fbc74adc06b35834902550d1b575bfad512d1ff7c6e394b0617dcb1d9d61" exitCode=0 Sep 14 10:49:21.140315 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:21.140191 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nfqwx" event={"ID":"f44ee762-76b2-4d0a-aabd-8b68c6aaffdb","Type":"ContainerDied","Data":"2167fbc74adc06b35834902550d1b575bfad512d1ff7c6e394b0617dcb1d9d61"} Sep 14 10:49:21.146605 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:21.146580 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-js9xt_5cc7f97c-133f-4c01-a394-1e06cb81d372/konnectivity-agent/0.log" limit=1024 Sep 14 10:49:21.159512 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:21.159486 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zp5bd"] Sep 14 10:49:21.178025 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:21.177999 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wrtgl"] Sep 14 10:49:21.178132 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:21.178114 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zp5bd" Sep 14 10:49:21.180750 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:21.180496 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 14 10:49:21.180750 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:21.180675 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qfwcj\"" Sep 14 10:49:21.181273 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:21.181090 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 14 10:49:21.202462 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:21.202403 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zp5bd"] Sep 14 10:49:21.202462 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:21.202449 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wrtgl"] Sep 14 10:49:21.202615 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:21.202517 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wrtgl" Sep 14 10:49:21.205139 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:21.205118 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 14 10:49:21.205230 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:21.205212 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 14 10:49:21.205387 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:21.205371 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 14 10:49:21.205471 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:21.205456 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5cjtt\"" Sep 14 10:49:21.246410 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:21.246381 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/305e2ad1-1b29-4c6b-92e9-b21fa53847bc-metrics-tls\") pod \"dns-default-zp5bd\" (UID: \"305e2ad1-1b29-4c6b-92e9-b21fa53847bc\") " pod="openshift-dns/dns-default-zp5bd" Sep 14 10:49:21.246516 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:21.246433 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/305e2ad1-1b29-4c6b-92e9-b21fa53847bc-config-volume\") pod \"dns-default-zp5bd\" (UID: \"305e2ad1-1b29-4c6b-92e9-b21fa53847bc\") " pod="openshift-dns/dns-default-zp5bd" Sep 14 10:49:21.246579 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:21.246559 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq8jq\" (UniqueName: \"kubernetes.io/projected/305e2ad1-1b29-4c6b-92e9-b21fa53847bc-kube-api-access-rq8jq\") pod \"dns-default-zp5bd\" (UID: \"305e2ad1-1b29-4c6b-92e9-b21fa53847bc\") " pod="openshift-dns/dns-default-zp5bd" Sep 14 10:49:21.246625 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:21.246598 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/305e2ad1-1b29-4c6b-92e9-b21fa53847bc-tmp-dir\") pod \"dns-default-zp5bd\" (UID: \"305e2ad1-1b29-4c6b-92e9-b21fa53847bc\") " pod="openshift-dns/dns-default-zp5bd" Sep 14 10:49:21.256549 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:21.256526 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-js9xt_5cc7f97c-133f-4c01-a394-1e06cb81d372/konnectivity-agent/0.log" limit=1024 Sep 14 10:49:21.347000 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:21.346974 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/305e2ad1-1b29-4c6b-92e9-b21fa53847bc-metrics-tls\") pod \"dns-default-zp5bd\" (UID: \"305e2ad1-1b29-4c6b-92e9-b21fa53847bc\") " pod="openshift-dns/dns-default-zp5bd" Sep 14 10:49:21.347109 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:21.347012 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/305e2ad1-1b29-4c6b-92e9-b21fa53847bc-config-volume\") pod \"dns-default-zp5bd\" (UID: \"305e2ad1-1b29-4c6b-92e9-b21fa53847bc\") " pod="openshift-dns/dns-default-zp5bd" Sep 14 10:49:21.347109 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:21.347043 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68hzr\" (UniqueName: \"kubernetes.io/projected/3112f5c3-15ae-4606-8da4-c066b3c92af8-kube-api-access-68hzr\") pod \"ingress-canary-wrtgl\" (UID: \"3112f5c3-15ae-4606-8da4-c066b3c92af8\") " pod="openshift-ingress-canary/ingress-canary-wrtgl" Sep 14 10:49:21.347109 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:21.347070 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq8jq\" (UniqueName: \"kubernetes.io/projected/305e2ad1-1b29-4c6b-92e9-b21fa53847bc-kube-api-access-rq8jq\") pod \"dns-default-zp5bd\" (UID: \"305e2ad1-1b29-4c6b-92e9-b21fa53847bc\") " pod="openshift-dns/dns-default-zp5bd" Sep 14 10:49:21.347109 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:21.347091 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/305e2ad1-1b29-4c6b-92e9-b21fa53847bc-tmp-dir\") pod \"dns-default-zp5bd\" (UID: \"305e2ad1-1b29-4c6b-92e9-b21fa53847bc\") " pod="openshift-dns/dns-default-zp5bd" Sep 14 10:49:21.347109 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:21.347107 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3112f5c3-15ae-4606-8da4-c066b3c92af8-cert\") pod \"ingress-canary-wrtgl\" (UID: \"3112f5c3-15ae-4606-8da4-c066b3c92af8\") " pod="openshift-ingress-canary/ingress-canary-wrtgl" Sep 14 10:49:21.347375 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:21.347139 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 10:49:21.347375 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:21.347212 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/305e2ad1-1b29-4c6b-92e9-b21fa53847bc-metrics-tls podName:305e2ad1-1b29-4c6b-92e9-b21fa53847bc nodeName:}" failed. No retries permitted until 2026-09-14 10:49:21.84719285 +0000 UTC m=+35.589594805 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/305e2ad1-1b29-4c6b-92e9-b21fa53847bc-metrics-tls") pod "dns-default-zp5bd" (UID: "305e2ad1-1b29-4c6b-92e9-b21fa53847bc") : secret "dns-default-metrics-tls" not found Sep 14 10:49:21.347626 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:21.347606 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/305e2ad1-1b29-4c6b-92e9-b21fa53847bc-config-volume\") pod \"dns-default-zp5bd\" (UID: \"305e2ad1-1b29-4c6b-92e9-b21fa53847bc\") " pod="openshift-dns/dns-default-zp5bd" Sep 14 10:49:21.353719 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:21.353692 2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-js9xt_5cc7f97c-133f-4c01-a394-1e06cb81d372/konnectivity-agent/0.log" limit=1024 Sep 14 10:49:21.360107 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:21.359972 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/305e2ad1-1b29-4c6b-92e9-b21fa53847bc-tmp-dir\") pod \"dns-default-zp5bd\" (UID: \"305e2ad1-1b29-4c6b-92e9-b21fa53847bc\") " pod="openshift-dns/dns-default-zp5bd" Sep 14 10:49:21.364955 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:21.364936 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq8jq\" (UniqueName: \"kubernetes.io/projected/305e2ad1-1b29-4c6b-92e9-b21fa53847bc-kube-api-access-rq8jq\") pod \"dns-default-zp5bd\" (UID: \"305e2ad1-1b29-4c6b-92e9-b21fa53847bc\") " pod="openshift-dns/dns-default-zp5bd" Sep 14 10:49:21.448054 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:21.448015 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68hzr\" (UniqueName: \"kubernetes.io/projected/3112f5c3-15ae-4606-8da4-c066b3c92af8-kube-api-access-68hzr\") pod \"ingress-canary-wrtgl\" (UID: \"3112f5c3-15ae-4606-8da4-c066b3c92af8\") " pod="openshift-ingress-canary/ingress-canary-wrtgl" Sep 14 10:49:21.448530 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:21.448063 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3112f5c3-15ae-4606-8da4-c066b3c92af8-cert\") pod \"ingress-canary-wrtgl\" (UID: \"3112f5c3-15ae-4606-8da4-c066b3c92af8\") " pod="openshift-ingress-canary/ingress-canary-wrtgl" Sep 14 10:49:21.450696 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:21.450680 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3112f5c3-15ae-4606-8da4-c066b3c92af8-cert\") pod \"ingress-canary-wrtgl\" (UID: \"3112f5c3-15ae-4606-8da4-c066b3c92af8\") " pod="openshift-ingress-canary/ingress-canary-wrtgl" Sep 14 10:49:21.456828 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:21.456785 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68hzr\" (UniqueName: \"kubernetes.io/projected/3112f5c3-15ae-4606-8da4-c066b3c92af8-kube-api-access-68hzr\") pod \"ingress-canary-wrtgl\" (UID: \"3112f5c3-15ae-4606-8da4-c066b3c92af8\") " pod="openshift-ingress-canary/ingress-canary-wrtgl" Sep 14 10:49:21.511857 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:21.511825 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wrtgl" Sep 14 10:49:21.821613 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:21.821582 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wrtgl"] Sep 14 10:49:21.825281 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:49:21.825256 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3112f5c3_15ae_4606_8da4_c066b3c92af8.slice/crio-1e7637182f261bf251524720e116ce885edb6875e06a213a6760bff559565836 WatchSource:0}: Error finding container 1e7637182f261bf251524720e116ce885edb6875e06a213a6760bff559565836: Status 404 returned error can't find the container with id 1e7637182f261bf251524720e116ce885edb6875e06a213a6760bff559565836 Sep 14 10:49:21.851133 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:21.851110 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/305e2ad1-1b29-4c6b-92e9-b21fa53847bc-metrics-tls\") pod \"dns-default-zp5bd\" (UID: \"305e2ad1-1b29-4c6b-92e9-b21fa53847bc\") " pod="openshift-dns/dns-default-zp5bd" Sep 14 10:49:21.851265 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:21.851233 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 10:49:21.851304 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:21.851284 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/305e2ad1-1b29-4c6b-92e9-b21fa53847bc-metrics-tls podName:305e2ad1-1b29-4c6b-92e9-b21fa53847bc nodeName:}" failed. No retries permitted until 2026-09-14 10:49:22.851269448 +0000 UTC m=+36.593671395 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/305e2ad1-1b29-4c6b-92e9-b21fa53847bc-metrics-tls") pod "dns-default-zp5bd" (UID: "305e2ad1-1b29-4c6b-92e9-b21fa53847bc") : secret "dns-default-metrics-tls" not found Sep 14 10:49:22.143490 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:22.143401 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wrtgl" event={"ID":"3112f5c3-15ae-4606-8da4-c066b3c92af8","Type":"ContainerStarted","Data":"1e7637182f261bf251524720e116ce885edb6875e06a213a6760bff559565836"} Sep 14 10:49:22.146430 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:22.146403 2581 generic.go:356] "Generic (PLEG): container finished" podID="f44ee762-76b2-4d0a-aabd-8b68c6aaffdb" containerID="67de328b180b1480c91f4bafb83d476b1802d0a0bcce5547a7c6c7d1b28c65c6" exitCode=0 Sep 14 10:49:22.146554 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:22.146451 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nfqwx" event={"ID":"f44ee762-76b2-4d0a-aabd-8b68c6aaffdb","Type":"ContainerDied","Data":"67de328b180b1480c91f4bafb83d476b1802d0a0bcce5547a7c6c7d1b28c65c6"} Sep 14 10:49:22.828989 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:22.828961 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b62cb" Sep 14 10:49:22.829876 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:22.828961 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29jrs" Sep 14 10:49:22.829876 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:22.828961 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw9zw" Sep 14 10:49:22.832818 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:22.832799 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 14 10:49:22.833523 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:22.833500 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 14 10:49:22.833621 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:22.833530 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fkf2p\"" Sep 14 10:49:22.833996 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:22.833977 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bckh8\"" Sep 14 10:49:22.834057 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:22.834015 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 14 10:49:22.834460 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:22.834410 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 14 10:49:22.861696 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:22.861672 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/305e2ad1-1b29-4c6b-92e9-b21fa53847bc-metrics-tls\") pod \"dns-default-zp5bd\" (UID: \"305e2ad1-1b29-4c6b-92e9-b21fa53847bc\") " pod="openshift-dns/dns-default-zp5bd" Sep 14 10:49:22.861827 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:22.861811 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 10:49:22.861900 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:22.861889 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/305e2ad1-1b29-4c6b-92e9-b21fa53847bc-metrics-tls podName:305e2ad1-1b29-4c6b-92e9-b21fa53847bc nodeName:}" failed. No retries permitted until 2026-09-14 10:49:24.861862709 +0000 UTC m=+38.604264673 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/305e2ad1-1b29-4c6b-92e9-b21fa53847bc-metrics-tls") pod "dns-default-zp5bd" (UID: "305e2ad1-1b29-4c6b-92e9-b21fa53847bc") : secret "dns-default-metrics-tls" not found Sep 14 10:49:23.153913 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:23.153827 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nfqwx" event={"ID":"f44ee762-76b2-4d0a-aabd-8b68c6aaffdb","Type":"ContainerStarted","Data":"57a93c609841ef4438bcc50636ca562e261c66651b38afb92f9b52b68577fae2"} Sep 14 10:49:23.181074 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:23.181014 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nfqwx" podStartSLOduration=7.715522957 podStartE2EDuration="36.180998568s" totalImagesPullingTime="28.465475611s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 10:48:47 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:49:23.179908018 +0000 UTC m=+36.922309998" watchObservedRunningTime="2026-09-14 10:49:23.180998568 +0000 UTC m=+36.923400538" Sep 14 10:49:24.876411 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:24.876200 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/305e2ad1-1b29-4c6b-92e9-b21fa53847bc-metrics-tls\") pod \"dns-default-zp5bd\" (UID: \"305e2ad1-1b29-4c6b-92e9-b21fa53847bc\") " pod="openshift-dns/dns-default-zp5bd" Sep 14 10:49:24.876758 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:24.876352 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 10:49:24.876758 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:24.876486 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/305e2ad1-1b29-4c6b-92e9-b21fa53847bc-metrics-tls podName:305e2ad1-1b29-4c6b-92e9-b21fa53847bc nodeName:}" failed. No retries permitted until 2026-09-14 10:49:28.87647151 +0000 UTC m=+42.618873474 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/305e2ad1-1b29-4c6b-92e9-b21fa53847bc-metrics-tls") pod "dns-default-zp5bd" (UID: "305e2ad1-1b29-4c6b-92e9-b21fa53847bc") : secret "dns-default-metrics-tls" not found Sep 14 10:49:25.159906 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:25.159814 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wrtgl" event={"ID":"3112f5c3-15ae-4606-8da4-c066b3c92af8","Type":"ContainerStarted","Data":"f89b0f6fc4ec465ab9a4d5fd4e3c8c6787d813fc21d0e838f8c49de498fef315"} Sep 14 10:49:25.175574 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:25.175526 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wrtgl" podStartSLOduration=1.695569957 podStartE2EDuration="4.175510191s" totalImagesPullingTime="2.479940234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:49:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:49:25.174566455 +0000 UTC m=+38.916968425" watchObservedRunningTime="2026-09-14 10:49:25.175510191 +0000 UTC m=+38.917912161" Sep 14 10:49:28.902561 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:28.902526 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/305e2ad1-1b29-4c6b-92e9-b21fa53847bc-metrics-tls\") pod \"dns-default-zp5bd\" (UID: \"305e2ad1-1b29-4c6b-92e9-b21fa53847bc\") " pod="openshift-dns/dns-default-zp5bd" Sep 14 10:49:28.902944 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:28.902646 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 10:49:28.902944 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:28.902698 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/305e2ad1-1b29-4c6b-92e9-b21fa53847bc-metrics-tls podName:305e2ad1-1b29-4c6b-92e9-b21fa53847bc nodeName:}" failed. No retries permitted until 2026-09-14 10:49:36.902685446 +0000 UTC m=+50.645087394 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/305e2ad1-1b29-4c6b-92e9-b21fa53847bc-metrics-tls") pod "dns-default-zp5bd" (UID: "305e2ad1-1b29-4c6b-92e9-b21fa53847bc") : secret "dns-default-metrics-tls" not found Sep 14 10:49:36.956396 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:36.956334 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/305e2ad1-1b29-4c6b-92e9-b21fa53847bc-metrics-tls\") pod \"dns-default-zp5bd\" (UID: \"305e2ad1-1b29-4c6b-92e9-b21fa53847bc\") " pod="openshift-dns/dns-default-zp5bd" Sep 14 10:49:36.956931 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:36.956461 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 10:49:36.956931 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:36.956523 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/305e2ad1-1b29-4c6b-92e9-b21fa53847bc-metrics-tls podName:305e2ad1-1b29-4c6b-92e9-b21fa53847bc nodeName:}" failed. No retries permitted until 2026-09-14 10:49:52.956509194 +0000 UTC m=+66.698911142 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/305e2ad1-1b29-4c6b-92e9-b21fa53847bc-metrics-tls") pod "dns-default-zp5bd" (UID: "305e2ad1-1b29-4c6b-92e9-b21fa53847bc") : secret "dns-default-metrics-tls" not found Sep 14 10:49:43.846642 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:43.846604 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-bmlrv"] Sep 14 10:49:43.885076 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:43.885039 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7cf5b5b9d-2v58p"] Sep 14 10:49:43.885222 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:43.885173 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bmlrv" Sep 14 10:49:43.889243 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:43.889218 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 14 10:49:43.890180 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:43.890163 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 14 10:49:43.890384 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:43.890368 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 14 10:49:43.890461 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:43.890418 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-clglc\"" Sep 14 10:49:43.890504 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:43.890491 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 14 10:49:43.903031 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:43.903000 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-bmlrv"] Sep 14 10:49:43.903151 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:43.903039 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7cf5b5b9d-2v58p"] Sep 14 10:49:43.903151 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:43.903113 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cf5b5b9d-2v58p" Sep 14 10:49:43.905476 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:43.905454 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 14 10:49:43.905576 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:43.905567 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 14 10:49:43.905738 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:43.905725 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xzrxf\"" Sep 14 10:49:43.906135 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:43.906119 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 14 10:49:43.910648 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:43.910629 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 14 10:49:43.969686 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:43.969652 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-r5rfv"] Sep 14 10:49:43.990509 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:43.990484 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-9tvnj"] Sep 14 10:49:43.990670 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:43.990622 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-r5rfv" Sep 14 10:49:43.992912 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:43.992887 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 14 10:49:43.993042 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:43.992955 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-xxpwm\"" Sep 14 10:49:43.993042 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:43.992897 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 14 10:49:44.006231 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.006205 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cf34b271-8c37-42fc-9bb6-0ff0a45af426-image-registry-private-configuration\") pod \"image-registry-7cf5b5b9d-2v58p\" (UID: \"cf34b271-8c37-42fc-9bb6-0ff0a45af426\") " pod="openshift-image-registry/image-registry-7cf5b5b9d-2v58p" Sep 14 10:49:44.006328 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.006234 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cf34b271-8c37-42fc-9bb6-0ff0a45af426-ca-trust-extracted\") pod \"image-registry-7cf5b5b9d-2v58p\" (UID: \"cf34b271-8c37-42fc-9bb6-0ff0a45af426\") " pod="openshift-image-registry/image-registry-7cf5b5b9d-2v58p" Sep 14 10:49:44.006328 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.006264 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/923b7386-9d29-4313-9a63-d4f43e4c2932-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-bmlrv\" (UID: \"923b7386-9d29-4313-9a63-d4f43e4c2932\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bmlrv" Sep 14 10:49:44.006328 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.006289 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cf34b271-8c37-42fc-9bb6-0ff0a45af426-bound-sa-token\") pod \"image-registry-7cf5b5b9d-2v58p\" (UID: \"cf34b271-8c37-42fc-9bb6-0ff0a45af426\") " pod="openshift-image-registry/image-registry-7cf5b5b9d-2v58p" Sep 14 10:49:44.006452 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.006362 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/923b7386-9d29-4313-9a63-d4f43e4c2932-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-bmlrv\" (UID: \"923b7386-9d29-4313-9a63-d4f43e4c2932\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bmlrv" Sep 14 10:49:44.006452 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.006386 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cf34b271-8c37-42fc-9bb6-0ff0a45af426-registry-certificates\") pod \"image-registry-7cf5b5b9d-2v58p\" (UID: \"cf34b271-8c37-42fc-9bb6-0ff0a45af426\") " pod="openshift-image-registry/image-registry-7cf5b5b9d-2v58p" Sep 14 10:49:44.006452 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.006418 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf34b271-8c37-42fc-9bb6-0ff0a45af426-registry-tls\") pod \"image-registry-7cf5b5b9d-2v58p\" (UID: \"cf34b271-8c37-42fc-9bb6-0ff0a45af426\") " pod="openshift-image-registry/image-registry-7cf5b5b9d-2v58p" Sep 14 10:49:44.006452 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.006434 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cf34b271-8c37-42fc-9bb6-0ff0a45af426-trusted-ca\") pod \"image-registry-7cf5b5b9d-2v58p\" (UID: \"cf34b271-8c37-42fc-9bb6-0ff0a45af426\") " pod="openshift-image-registry/image-registry-7cf5b5b9d-2v58p" Sep 14 10:49:44.006569 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.006453 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cf34b271-8c37-42fc-9bb6-0ff0a45af426-installation-pull-secrets\") pod \"image-registry-7cf5b5b9d-2v58p\" (UID: \"cf34b271-8c37-42fc-9bb6-0ff0a45af426\") " pod="openshift-image-registry/image-registry-7cf5b5b9d-2v58p" Sep 14 10:49:44.006569 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.006481 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-275v9\" (UniqueName: \"kubernetes.io/projected/923b7386-9d29-4313-9a63-d4f43e4c2932-kube-api-access-275v9\") pod \"cluster-monitoring-operator-6cd954b897-bmlrv\" (UID: \"923b7386-9d29-4313-9a63-d4f43e4c2932\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bmlrv" Sep 14 10:49:44.006628 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.006560 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t57g\" (UniqueName: \"kubernetes.io/projected/cf34b271-8c37-42fc-9bb6-0ff0a45af426-kube-api-access-9t57g\") pod \"image-registry-7cf5b5b9d-2v58p\" (UID: \"cf34b271-8c37-42fc-9bb6-0ff0a45af426\") " pod="openshift-image-registry/image-registry-7cf5b5b9d-2v58p" Sep 14 10:49:44.013333 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.013313 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-r5rfv"] Sep 14 10:49:44.013454 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.013358 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-9tvnj"] Sep 14 10:49:44.013454 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.013446 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-9tvnj" Sep 14 10:49:44.015873 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.015856 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 14 10:49:44.016012 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.015859 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-mzbxd\"" Sep 14 10:49:44.016012 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.015899 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 14 10:49:44.016125 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.016060 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 14 10:49:44.016196 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.016124 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 14 10:49:44.021800 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.021780 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 14 10:49:44.063569 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.063539 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-g76j6"] Sep 14 10:49:44.082236 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.082201 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-g76j6"] Sep 14 10:49:44.082391 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.082322 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-g76j6" Sep 14 10:49:44.084730 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.084709 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 14 10:49:44.085060 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.085029 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 14 10:49:44.085060 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.085041 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-rzzt5\"" Sep 14 10:49:44.085226 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.085166 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 14 10:49:44.085226 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.085168 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 14 10:49:44.106928 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.106861 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cf34b271-8c37-42fc-9bb6-0ff0a45af426-image-registry-private-configuration\") pod \"image-registry-7cf5b5b9d-2v58p\" (UID: \"cf34b271-8c37-42fc-9bb6-0ff0a45af426\") " pod="openshift-image-registry/image-registry-7cf5b5b9d-2v58p" Sep 14 10:49:44.106928 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.106893 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cf34b271-8c37-42fc-9bb6-0ff0a45af426-ca-trust-extracted\") pod \"image-registry-7cf5b5b9d-2v58p\" (UID: \"cf34b271-8c37-42fc-9bb6-0ff0a45af426\") " pod="openshift-image-registry/image-registry-7cf5b5b9d-2v58p" Sep 14 10:49:44.106928 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.106920 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e2424929-e528-4e66-bba3-04fe380324af-snapshots\") pod \"insights-operator-5844b856db-9tvnj\" (UID: \"e2424929-e528-4e66-bba3-04fe380324af\") " pod="openshift-insights/insights-operator-5844b856db-9tvnj" Sep 14 10:49:44.107141 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.106955 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/923b7386-9d29-4313-9a63-d4f43e4c2932-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-bmlrv\" (UID: \"923b7386-9d29-4313-9a63-d4f43e4c2932\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bmlrv" Sep 14 10:49:44.107141 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.106979 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cf34b271-8c37-42fc-9bb6-0ff0a45af426-bound-sa-token\") pod \"image-registry-7cf5b5b9d-2v58p\" (UID: \"cf34b271-8c37-42fc-9bb6-0ff0a45af426\") " pod="openshift-image-registry/image-registry-7cf5b5b9d-2v58p" Sep 14 10:49:44.107141 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.107004 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvv4v\" (UniqueName: \"kubernetes.io/projected/854f87cd-8f75-45dc-9518-3dd0242fb0dc-kube-api-access-rvv4v\") pod \"volume-data-source-validator-86695b7cc7-r5rfv\" (UID: \"854f87cd-8f75-45dc-9518-3dd0242fb0dc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-r5rfv" Sep 14 10:49:44.107141 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.107043 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/923b7386-9d29-4313-9a63-d4f43e4c2932-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-bmlrv\" (UID: \"923b7386-9d29-4313-9a63-d4f43e4c2932\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bmlrv" Sep 14 10:49:44.107419 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.107395 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cf34b271-8c37-42fc-9bb6-0ff0a45af426-registry-certificates\") pod \"image-registry-7cf5b5b9d-2v58p\" (UID: \"cf34b271-8c37-42fc-9bb6-0ff0a45af426\") " pod="openshift-image-registry/image-registry-7cf5b5b9d-2v58p" Sep 14 10:49:44.107501 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:44.107442 2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 14 10:49:44.107501 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.107463 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf34b271-8c37-42fc-9bb6-0ff0a45af426-registry-tls\") pod \"image-registry-7cf5b5b9d-2v58p\" (UID: \"cf34b271-8c37-42fc-9bb6-0ff0a45af426\") " pod="openshift-image-registry/image-registry-7cf5b5b9d-2v58p" Sep 14 10:49:44.107590 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.107499 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cf34b271-8c37-42fc-9bb6-0ff0a45af426-trusted-ca\") pod \"image-registry-7cf5b5b9d-2v58p\" (UID: \"cf34b271-8c37-42fc-9bb6-0ff0a45af426\") " pod="openshift-image-registry/image-registry-7cf5b5b9d-2v58p" Sep 14 10:49:44.107590 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.107537 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2424929-e528-4e66-bba3-04fe380324af-trusted-ca-bundle\") pod \"insights-operator-5844b856db-9tvnj\" (UID: \"e2424929-e528-4e66-bba3-04fe380324af\") " pod="openshift-insights/insights-operator-5844b856db-9tvnj" Sep 14 10:49:44.107590 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.107559 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cf34b271-8c37-42fc-9bb6-0ff0a45af426-ca-trust-extracted\") pod \"image-registry-7cf5b5b9d-2v58p\" (UID: \"cf34b271-8c37-42fc-9bb6-0ff0a45af426\") " pod="openshift-image-registry/image-registry-7cf5b5b9d-2v58p" Sep 14 10:49:44.107764 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.107568 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e2424929-e528-4e66-bba3-04fe380324af-serving-cert\") pod \"insights-operator-5844b856db-9tvnj\" (UID: \"e2424929-e528-4e66-bba3-04fe380324af\") " pod="openshift-insights/insights-operator-5844b856db-9tvnj" Sep 14 10:49:44.107833 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:44.107816 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 10:49:44.107893 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:44.107837 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cf5b5b9d-2v58p: secret "image-registry-tls" not found Sep 14 10:49:44.108937 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:44.108287 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/923b7386-9d29-4313-9a63-d4f43e4c2932-cluster-monitoring-operator-tls podName:923b7386-9d29-4313-9a63-d4f43e4c2932 nodeName:}" failed. No retries permitted until 2026-09-14 10:49:44.608250693 +0000 UTC m=+58.350652658 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/923b7386-9d29-4313-9a63-d4f43e4c2932-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-bmlrv" (UID: "923b7386-9d29-4313-9a63-d4f43e4c2932") : secret "cluster-monitoring-operator-tls" not found Sep 14 10:49:44.108937 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:44.108586 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf34b271-8c37-42fc-9bb6-0ff0a45af426-registry-tls podName:cf34b271-8c37-42fc-9bb6-0ff0a45af426 nodeName:}" failed. No retries permitted until 2026-09-14 10:49:44.60855703 +0000 UTC m=+58.350958991 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cf34b271-8c37-42fc-9bb6-0ff0a45af426-registry-tls") pod "image-registry-7cf5b5b9d-2v58p" (UID: "cf34b271-8c37-42fc-9bb6-0ff0a45af426") : secret "image-registry-tls" not found Sep 14 10:49:44.108937 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.107635 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cf34b271-8c37-42fc-9bb6-0ff0a45af426-installation-pull-secrets\") pod \"image-registry-7cf5b5b9d-2v58p\" (UID: \"cf34b271-8c37-42fc-9bb6-0ff0a45af426\") " pod="openshift-image-registry/image-registry-7cf5b5b9d-2v58p" Sep 14 10:49:44.108937 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.108915 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2424929-e528-4e66-bba3-04fe380324af-tmp\") pod \"insights-operator-5844b856db-9tvnj\" (UID: \"e2424929-e528-4e66-bba3-04fe380324af\") " pod="openshift-insights/insights-operator-5844b856db-9tvnj" Sep 14 10:49:44.109220 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.108950 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2424929-e528-4e66-bba3-04fe380324af-service-ca-bundle\") pod \"insights-operator-5844b856db-9tvnj\" (UID: \"e2424929-e528-4e66-bba3-04fe380324af\") " pod="openshift-insights/insights-operator-5844b856db-9tvnj" Sep 14 10:49:44.109220 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.109029 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cf34b271-8c37-42fc-9bb6-0ff0a45af426-registry-certificates\") pod \"image-registry-7cf5b5b9d-2v58p\" (UID: \"cf34b271-8c37-42fc-9bb6-0ff0a45af426\") " pod="openshift-image-registry/image-registry-7cf5b5b9d-2v58p" Sep 14 10:49:44.109220 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.109079 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-275v9\" (UniqueName: \"kubernetes.io/projected/923b7386-9d29-4313-9a63-d4f43e4c2932-kube-api-access-275v9\") pod \"cluster-monitoring-operator-6cd954b897-bmlrv\" (UID: \"923b7386-9d29-4313-9a63-d4f43e4c2932\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bmlrv" Sep 14 10:49:44.109220 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.109117 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qfp2\" (UniqueName: \"kubernetes.io/projected/e2424929-e528-4e66-bba3-04fe380324af-kube-api-access-5qfp2\") pod \"insights-operator-5844b856db-9tvnj\" (UID: \"e2424929-e528-4e66-bba3-04fe380324af\") " pod="openshift-insights/insights-operator-5844b856db-9tvnj" Sep 14 10:49:44.109457 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.109317 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t57g\" (UniqueName: \"kubernetes.io/projected/cf34b271-8c37-42fc-9bb6-0ff0a45af426-kube-api-access-9t57g\") pod \"image-registry-7cf5b5b9d-2v58p\" (UID: \"cf34b271-8c37-42fc-9bb6-0ff0a45af426\") " pod="openshift-image-registry/image-registry-7cf5b5b9d-2v58p" Sep 14 10:49:44.110152 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.110130 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/923b7386-9d29-4313-9a63-d4f43e4c2932-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-bmlrv\" (UID: \"923b7386-9d29-4313-9a63-d4f43e4c2932\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bmlrv" Sep 14 10:49:44.112997 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.112974 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cf34b271-8c37-42fc-9bb6-0ff0a45af426-installation-pull-secrets\") pod \"image-registry-7cf5b5b9d-2v58p\" (UID: \"cf34b271-8c37-42fc-9bb6-0ff0a45af426\") " pod="openshift-image-registry/image-registry-7cf5b5b9d-2v58p" Sep 14 10:49:44.113105 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.113030 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cf34b271-8c37-42fc-9bb6-0ff0a45af426-image-registry-private-configuration\") pod \"image-registry-7cf5b5b9d-2v58p\" (UID: \"cf34b271-8c37-42fc-9bb6-0ff0a45af426\") " pod="openshift-image-registry/image-registry-7cf5b5b9d-2v58p" Sep 14 10:49:44.113495 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.113474 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cf34b271-8c37-42fc-9bb6-0ff0a45af426-trusted-ca\") pod \"image-registry-7cf5b5b9d-2v58p\" (UID: \"cf34b271-8c37-42fc-9bb6-0ff0a45af426\") " pod="openshift-image-registry/image-registry-7cf5b5b9d-2v58p" Sep 14 10:49:44.115711 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.115684 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cf34b271-8c37-42fc-9bb6-0ff0a45af426-bound-sa-token\") pod \"image-registry-7cf5b5b9d-2v58p\" (UID: \"cf34b271-8c37-42fc-9bb6-0ff0a45af426\") " pod="openshift-image-registry/image-registry-7cf5b5b9d-2v58p" Sep 14 10:49:44.118647 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.118627 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t57g\" (UniqueName: \"kubernetes.io/projected/cf34b271-8c37-42fc-9bb6-0ff0a45af426-kube-api-access-9t57g\") pod \"image-registry-7cf5b5b9d-2v58p\" (UID: \"cf34b271-8c37-42fc-9bb6-0ff0a45af426\") " pod="openshift-image-registry/image-registry-7cf5b5b9d-2v58p" Sep 14 10:49:44.125779 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.125756 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-275v9\" (UniqueName: \"kubernetes.io/projected/923b7386-9d29-4313-9a63-d4f43e4c2932-kube-api-access-275v9\") pod \"cluster-monitoring-operator-6cd954b897-bmlrv\" (UID: \"923b7386-9d29-4313-9a63-d4f43e4c2932\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bmlrv" Sep 14 10:49:44.210290 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.210256 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0cedfa94-2214-4710-86cb-448821993687-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-g76j6\" (UID: \"0cedfa94-2214-4710-86cb-448821993687\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-g76j6" Sep 14 10:49:44.210475 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.210306 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2424929-e528-4e66-bba3-04fe380324af-trusted-ca-bundle\") pod \"insights-operator-5844b856db-9tvnj\" (UID: \"e2424929-e528-4e66-bba3-04fe380324af\") " pod="openshift-insights/insights-operator-5844b856db-9tvnj" Sep 14 10:49:44.210475 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.210322 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e2424929-e528-4e66-bba3-04fe380324af-serving-cert\") pod \"insights-operator-5844b856db-9tvnj\" (UID: \"e2424929-e528-4e66-bba3-04fe380324af\") " pod="openshift-insights/insights-operator-5844b856db-9tvnj" Sep 14 10:49:44.210475 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.210379 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2424929-e528-4e66-bba3-04fe380324af-tmp\") pod \"insights-operator-5844b856db-9tvnj\" (UID: \"e2424929-e528-4e66-bba3-04fe380324af\") " pod="openshift-insights/insights-operator-5844b856db-9tvnj" Sep 14 10:49:44.210475 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.210399 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2424929-e528-4e66-bba3-04fe380324af-service-ca-bundle\") pod \"insights-operator-5844b856db-9tvnj\" (UID: \"e2424929-e528-4e66-bba3-04fe380324af\") " pod="openshift-insights/insights-operator-5844b856db-9tvnj" Sep 14 10:49:44.210475 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.210416 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0cedfa94-2214-4710-86cb-448821993687-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-g76j6\" (UID: \"0cedfa94-2214-4710-86cb-448821993687\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-g76j6" Sep 14 10:49:44.210475 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.210454 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qfp2\" (UniqueName: \"kubernetes.io/projected/e2424929-e528-4e66-bba3-04fe380324af-kube-api-access-5qfp2\") pod \"insights-operator-5844b856db-9tvnj\" (UID: \"e2424929-e528-4e66-bba3-04fe380324af\") " pod="openshift-insights/insights-operator-5844b856db-9tvnj" Sep 14 10:49:44.210750 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.210630 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e2424929-e528-4e66-bba3-04fe380324af-snapshots\") pod \"insights-operator-5844b856db-9tvnj\" (UID: \"e2424929-e528-4e66-bba3-04fe380324af\") " pod="openshift-insights/insights-operator-5844b856db-9tvnj" Sep 14 10:49:44.210750 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.210655 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvjzq\" (UniqueName: \"kubernetes.io/projected/0cedfa94-2214-4710-86cb-448821993687-kube-api-access-zvjzq\") pod \"kube-storage-version-migrator-operator-6c8f84f897-g76j6\" (UID: \"0cedfa94-2214-4710-86cb-448821993687\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-g76j6" Sep 14 10:49:44.210750 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.210704 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvv4v\" (UniqueName: \"kubernetes.io/projected/854f87cd-8f75-45dc-9518-3dd0242fb0dc-kube-api-access-rvv4v\") pod \"volume-data-source-validator-86695b7cc7-r5rfv\" (UID: \"854f87cd-8f75-45dc-9518-3dd0242fb0dc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-r5rfv" Sep 14 10:49:44.210887 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.210783 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2424929-e528-4e66-bba3-04fe380324af-tmp\") pod \"insights-operator-5844b856db-9tvnj\" (UID: \"e2424929-e528-4e66-bba3-04fe380324af\") " pod="openshift-insights/insights-operator-5844b856db-9tvnj" Sep 14 10:49:44.211044 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.211026 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2424929-e528-4e66-bba3-04fe380324af-service-ca-bundle\") pod \"insights-operator-5844b856db-9tvnj\" (UID: \"e2424929-e528-4e66-bba3-04fe380324af\") " pod="openshift-insights/insights-operator-5844b856db-9tvnj" Sep 14 10:49:44.211298 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.211270 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2424929-e528-4e66-bba3-04fe380324af-trusted-ca-bundle\") pod \"insights-operator-5844b856db-9tvnj\" (UID: \"e2424929-e528-4e66-bba3-04fe380324af\") " pod="openshift-insights/insights-operator-5844b856db-9tvnj" Sep 14 10:49:44.211437 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.211277 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e2424929-e528-4e66-bba3-04fe380324af-snapshots\") pod \"insights-operator-5844b856db-9tvnj\" (UID: \"e2424929-e528-4e66-bba3-04fe380324af\") " pod="openshift-insights/insights-operator-5844b856db-9tvnj" Sep 14 10:49:44.213139 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.213123 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e2424929-e528-4e66-bba3-04fe380324af-serving-cert\") pod \"insights-operator-5844b856db-9tvnj\" (UID: \"e2424929-e528-4e66-bba3-04fe380324af\") " pod="openshift-insights/insights-operator-5844b856db-9tvnj" Sep 14 10:49:44.219025 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.219005 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvv4v\" (UniqueName: \"kubernetes.io/projected/854f87cd-8f75-45dc-9518-3dd0242fb0dc-kube-api-access-rvv4v\") pod \"volume-data-source-validator-86695b7cc7-r5rfv\" (UID: \"854f87cd-8f75-45dc-9518-3dd0242fb0dc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-r5rfv" Sep 14 10:49:44.219095 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.219067 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qfp2\" (UniqueName: \"kubernetes.io/projected/e2424929-e528-4e66-bba3-04fe380324af-kube-api-access-5qfp2\") pod \"insights-operator-5844b856db-9tvnj\" (UID: \"e2424929-e528-4e66-bba3-04fe380324af\") " pod="openshift-insights/insights-operator-5844b856db-9tvnj" Sep 14 10:49:44.298758 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.298725 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-r5rfv" Sep 14 10:49:44.311023 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.310990 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvjzq\" (UniqueName: \"kubernetes.io/projected/0cedfa94-2214-4710-86cb-448821993687-kube-api-access-zvjzq\") pod \"kube-storage-version-migrator-operator-6c8f84f897-g76j6\" (UID: \"0cedfa94-2214-4710-86cb-448821993687\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-g76j6" Sep 14 10:49:44.311156 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.311060 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0cedfa94-2214-4710-86cb-448821993687-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-g76j6\" (UID: \"0cedfa94-2214-4710-86cb-448821993687\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-g76j6" Sep 14 10:49:44.311156 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.311114 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0cedfa94-2214-4710-86cb-448821993687-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-g76j6\" (UID: \"0cedfa94-2214-4710-86cb-448821993687\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-g76j6" Sep 14 10:49:44.311594 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.311574 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0cedfa94-2214-4710-86cb-448821993687-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-g76j6\" (UID: \"0cedfa94-2214-4710-86cb-448821993687\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-g76j6" Sep 14 10:49:44.313395 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.313378 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0cedfa94-2214-4710-86cb-448821993687-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-g76j6\" (UID: \"0cedfa94-2214-4710-86cb-448821993687\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-g76j6" Sep 14 10:49:44.319999 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.319979 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvjzq\" (UniqueName: \"kubernetes.io/projected/0cedfa94-2214-4710-86cb-448821993687-kube-api-access-zvjzq\") pod \"kube-storage-version-migrator-operator-6c8f84f897-g76j6\" (UID: \"0cedfa94-2214-4710-86cb-448821993687\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-g76j6" Sep 14 10:49:44.322913 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.322897 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-9tvnj" Sep 14 10:49:44.390934 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.390689 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-g76j6" Sep 14 10:49:44.595206 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.595174 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-r5rfv"] Sep 14 10:49:44.598127 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:49:44.598098 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod854f87cd_8f75_45dc_9518_3dd0242fb0dc.slice/crio-cc0bb6d86523e3ff2b79ddb4f61ca9dbd30012baf33ec80ab71a4e59740051f0 WatchSource:0}: Error finding container cc0bb6d86523e3ff2b79ddb4f61ca9dbd30012baf33ec80ab71a4e59740051f0: Status 404 returned error can't find the container with id cc0bb6d86523e3ff2b79ddb4f61ca9dbd30012baf33ec80ab71a4e59740051f0 Sep 14 10:49:44.613139 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.613071 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/923b7386-9d29-4313-9a63-d4f43e4c2932-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-bmlrv\" (UID: \"923b7386-9d29-4313-9a63-d4f43e4c2932\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bmlrv" Sep 14 10:49:44.613139 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.613133 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf34b271-8c37-42fc-9bb6-0ff0a45af426-registry-tls\") pod \"image-registry-7cf5b5b9d-2v58p\" (UID: \"cf34b271-8c37-42fc-9bb6-0ff0a45af426\") " pod="openshift-image-registry/image-registry-7cf5b5b9d-2v58p" Sep 14 10:49:44.613290 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:44.613268 2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 14 10:49:44.613353 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:44.613290 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 10:49:44.613353 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:44.613302 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cf5b5b9d-2v58p: secret "image-registry-tls" not found Sep 14 10:49:44.613421 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:44.613358 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/923b7386-9d29-4313-9a63-d4f43e4c2932-cluster-monitoring-operator-tls podName:923b7386-9d29-4313-9a63-d4f43e4c2932 nodeName:}" failed. No retries permitted until 2026-09-14 10:49:45.613319678 +0000 UTC m=+59.355721643 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/923b7386-9d29-4313-9a63-d4f43e4c2932-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-bmlrv" (UID: "923b7386-9d29-4313-9a63-d4f43e4c2932") : secret "cluster-monitoring-operator-tls" not found Sep 14 10:49:44.613421 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:44.613373 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf34b271-8c37-42fc-9bb6-0ff0a45af426-registry-tls podName:cf34b271-8c37-42fc-9bb6-0ff0a45af426 nodeName:}" failed. No retries permitted until 2026-09-14 10:49:45.613365391 +0000 UTC m=+59.355767342 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cf34b271-8c37-42fc-9bb6-0ff0a45af426-registry-tls") pod "image-registry-7cf5b5b9d-2v58p" (UID: "cf34b271-8c37-42fc-9bb6-0ff0a45af426") : secret "image-registry-tls" not found Sep 14 10:49:44.637094 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.637066 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-9tvnj"] Sep 14 10:49:44.639977 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:49:44.639950 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2424929_e528_4e66_bba3_04fe380324af.slice/crio-6b953021ad2f6273487ca52b97e23a939053b163825d80ad9ffaafb28c061669 WatchSource:0}: Error finding container 6b953021ad2f6273487ca52b97e23a939053b163825d80ad9ffaafb28c061669: Status 404 returned error can't find the container with id 6b953021ad2f6273487ca52b97e23a939053b163825d80ad9ffaafb28c061669 Sep 14 10:49:44.677045 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:44.677019 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-g76j6"] Sep 14 10:49:44.680141 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:49:44.680119 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cedfa94_2214_4710_86cb_448821993687.slice/crio-ea85465a0b80e379aa51c0c34d677cb889aeed58d0c139abfd9f5cf9b3a373bb WatchSource:0}: Error finding container ea85465a0b80e379aa51c0c34d677cb889aeed58d0c139abfd9f5cf9b3a373bb: Status 404 returned error can't find the container with id ea85465a0b80e379aa51c0c34d677cb889aeed58d0c139abfd9f5cf9b3a373bb Sep 14 10:49:45.205811 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:45.205764 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-9tvnj" event={"ID":"e2424929-e528-4e66-bba3-04fe380324af","Type":"ContainerStarted","Data":"6b953021ad2f6273487ca52b97e23a939053b163825d80ad9ffaafb28c061669"} Sep 14 10:49:45.207144 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:45.207115 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-g76j6" event={"ID":"0cedfa94-2214-4710-86cb-448821993687","Type":"ContainerStarted","Data":"ea85465a0b80e379aa51c0c34d677cb889aeed58d0c139abfd9f5cf9b3a373bb"} Sep 14 10:49:45.208679 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:45.208655 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-r5rfv" event={"ID":"854f87cd-8f75-45dc-9518-3dd0242fb0dc","Type":"ContainerStarted","Data":"cc0bb6d86523e3ff2b79ddb4f61ca9dbd30012baf33ec80ab71a4e59740051f0"} Sep 14 10:49:45.622418 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:45.622167 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/923b7386-9d29-4313-9a63-d4f43e4c2932-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-bmlrv\" (UID: \"923b7386-9d29-4313-9a63-d4f43e4c2932\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bmlrv" Sep 14 10:49:45.622418 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:45.622240 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf34b271-8c37-42fc-9bb6-0ff0a45af426-registry-tls\") pod \"image-registry-7cf5b5b9d-2v58p\" (UID: \"cf34b271-8c37-42fc-9bb6-0ff0a45af426\") " pod="openshift-image-registry/image-registry-7cf5b5b9d-2v58p" Sep 14 10:49:45.622418 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:45.622284 2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 14 10:49:45.622418 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:45.622329 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 10:49:45.622418 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:45.622360 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cf5b5b9d-2v58p: secret "image-registry-tls" not found Sep 14 10:49:45.622418 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:45.622376 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/923b7386-9d29-4313-9a63-d4f43e4c2932-cluster-monitoring-operator-tls podName:923b7386-9d29-4313-9a63-d4f43e4c2932 nodeName:}" failed. No retries permitted until 2026-09-14 10:49:47.622357297 +0000 UTC m=+61.364759263 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/923b7386-9d29-4313-9a63-d4f43e4c2932-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-bmlrv" (UID: "923b7386-9d29-4313-9a63-d4f43e4c2932") : secret "cluster-monitoring-operator-tls" not found Sep 14 10:49:45.622418 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:45.622395 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf34b271-8c37-42fc-9bb6-0ff0a45af426-registry-tls podName:cf34b271-8c37-42fc-9bb6-0ff0a45af426 nodeName:}" failed. No retries permitted until 2026-09-14 10:49:47.622384613 +0000 UTC m=+61.364786567 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cf34b271-8c37-42fc-9bb6-0ff0a45af426-registry-tls") pod "image-registry-7cf5b5b9d-2v58p" (UID: "cf34b271-8c37-42fc-9bb6-0ff0a45af426") : secret "image-registry-tls" not found Sep 14 10:49:46.213431 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:46.213323 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-r5rfv" event={"ID":"854f87cd-8f75-45dc-9518-3dd0242fb0dc","Type":"ContainerStarted","Data":"7a7fa7b7d2fdf85dc20416f585e2854e7c791dd3cbae1b6cfadbb74c251164af"} Sep 14 10:49:46.228931 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:46.228869 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-r5rfv" podStartSLOduration=1.964881815 podStartE2EDuration="3.228853819s" totalImagesPullingTime="1.263972004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:49:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:49:46.228294196 +0000 UTC m=+59.970696167" watchObservedRunningTime="2026-09-14 10:49:46.228853819 +0000 UTC m=+59.971255789" Sep 14 10:49:47.137918 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:47.137883 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bmb78" Sep 14 10:49:47.639824 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:47.639785 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf34b271-8c37-42fc-9bb6-0ff0a45af426-registry-tls\") pod \"image-registry-7cf5b5b9d-2v58p\" (UID: \"cf34b271-8c37-42fc-9bb6-0ff0a45af426\") " pod="openshift-image-registry/image-registry-7cf5b5b9d-2v58p" Sep 14 10:49:47.640438 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:47.639882 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/923b7386-9d29-4313-9a63-d4f43e4c2932-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-bmlrv\" (UID: \"923b7386-9d29-4313-9a63-d4f43e4c2932\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bmlrv" Sep 14 10:49:47.640438 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:47.639960 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 10:49:47.640438 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:47.639971 2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 14 10:49:47.640438 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:47.639982 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cf5b5b9d-2v58p: secret "image-registry-tls" not found Sep 14 10:49:47.640438 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:47.640030 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/923b7386-9d29-4313-9a63-d4f43e4c2932-cluster-monitoring-operator-tls podName:923b7386-9d29-4313-9a63-d4f43e4c2932 nodeName:}" failed. No retries permitted until 2026-09-14 10:49:51.640015914 +0000 UTC m=+65.382417861 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/923b7386-9d29-4313-9a63-d4f43e4c2932-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-bmlrv" (UID: "923b7386-9d29-4313-9a63-d4f43e4c2932") : secret "cluster-monitoring-operator-tls" not found Sep 14 10:49:47.640438 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:47.640044 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf34b271-8c37-42fc-9bb6-0ff0a45af426-registry-tls podName:cf34b271-8c37-42fc-9bb6-0ff0a45af426 nodeName:}" failed. No retries permitted until 2026-09-14 10:49:51.640037376 +0000 UTC m=+65.382439324 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cf34b271-8c37-42fc-9bb6-0ff0a45af426-registry-tls") pod "image-registry-7cf5b5b9d-2v58p" (UID: "cf34b271-8c37-42fc-9bb6-0ff0a45af426") : secret "image-registry-tls" not found Sep 14 10:49:48.223826 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:48.223787 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-9tvnj" event={"ID":"e2424929-e528-4e66-bba3-04fe380324af","Type":"ContainerStarted","Data":"dd8e321eede23339c1f6bf93195914f66e4a39195bea2cc3a1cb727d962a994d"} Sep 14 10:49:48.225458 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:48.225435 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-g76j6" event={"ID":"0cedfa94-2214-4710-86cb-448821993687","Type":"ContainerStarted","Data":"d970e3e67c1d7ad067f3f777a882da071a5ef58cf489e7dd12010e025df44863"} Sep 14 10:49:48.243566 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:48.243508 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-9tvnj" podStartSLOduration=2.732127936 podStartE2EDuration="5.243493028s" totalImagesPullingTime="2.511365092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:49:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:49:48.24275235 +0000 UTC m=+61.985154320" watchObservedRunningTime="2026-09-14 10:49:48.243493028 +0000 UTC m=+61.985894992" Sep 14 10:49:48.264616 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:48.264554 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-g76j6" podStartSLOduration=1.7903901690000001 podStartE2EDuration="4.264531722s" totalImagesPullingTime="2.474141553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:49:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:49:48.263957927 +0000 UTC m=+62.006359901" watchObservedRunningTime="2026-09-14 10:49:48.264531722 +0000 UTC m=+62.006933717" Sep 14 10:49:48.443783 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:48.443746 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4mmlv"] Sep 14 10:49:48.447283 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:48.447262 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4mmlv" Sep 14 10:49:48.449840 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:48.449815 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 14 10:49:48.450269 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:48.450241 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 14 10:49:48.452282 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:48.452266 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6n6bp\"" Sep 14 10:49:48.461757 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:48.461734 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4mmlv"] Sep 14 10:49:48.547996 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:48.547963 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8073aabc-9fac-40a5-bfaa-b150a5623e1c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4mmlv\" (UID: \"8073aabc-9fac-40a5-bfaa-b150a5623e1c\") " pod="openshift-insights/insights-runtime-extractor-4mmlv" Sep 14 10:49:48.548169 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:48.548011 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8073aabc-9fac-40a5-bfaa-b150a5623e1c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4mmlv\" (UID: \"8073aabc-9fac-40a5-bfaa-b150a5623e1c\") " pod="openshift-insights/insights-runtime-extractor-4mmlv" Sep 14 10:49:48.548169 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:48.548045 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8bbd\" (UniqueName: \"kubernetes.io/projected/8073aabc-9fac-40a5-bfaa-b150a5623e1c-kube-api-access-c8bbd\") pod \"insights-runtime-extractor-4mmlv\" (UID: \"8073aabc-9fac-40a5-bfaa-b150a5623e1c\") " pod="openshift-insights/insights-runtime-extractor-4mmlv" Sep 14 10:49:48.548169 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:48.548149 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8073aabc-9fac-40a5-bfaa-b150a5623e1c-crio-socket\") pod \"insights-runtime-extractor-4mmlv\" (UID: \"8073aabc-9fac-40a5-bfaa-b150a5623e1c\") " pod="openshift-insights/insights-runtime-extractor-4mmlv" Sep 14 10:49:48.548284 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:48.548191 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8073aabc-9fac-40a5-bfaa-b150a5623e1c-data-volume\") pod \"insights-runtime-extractor-4mmlv\" (UID: \"8073aabc-9fac-40a5-bfaa-b150a5623e1c\") " pod="openshift-insights/insights-runtime-extractor-4mmlv" Sep 14 10:49:48.648706 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:48.648665 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8073aabc-9fac-40a5-bfaa-b150a5623e1c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4mmlv\" (UID: \"8073aabc-9fac-40a5-bfaa-b150a5623e1c\") " pod="openshift-insights/insights-runtime-extractor-4mmlv" Sep 14 10:49:48.648706 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:48.648711 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8073aabc-9fac-40a5-bfaa-b150a5623e1c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4mmlv\" (UID: \"8073aabc-9fac-40a5-bfaa-b150a5623e1c\") " pod="openshift-insights/insights-runtime-extractor-4mmlv" Sep 14 10:49:48.649159 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:48.648772 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8bbd\" (UniqueName: \"kubernetes.io/projected/8073aabc-9fac-40a5-bfaa-b150a5623e1c-kube-api-access-c8bbd\") pod \"insights-runtime-extractor-4mmlv\" (UID: \"8073aabc-9fac-40a5-bfaa-b150a5623e1c\") " pod="openshift-insights/insights-runtime-extractor-4mmlv" Sep 14 10:49:48.649159 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:48.648786 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 14 10:49:48.649159 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:48.648841 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8073aabc-9fac-40a5-bfaa-b150a5623e1c-kube-rbac-proxy-cm podName:8073aabc-9fac-40a5-bfaa-b150a5623e1c nodeName:}" failed. No retries permitted until 2026-09-14 10:49:49.14882688 +0000 UTC m=+62.891228845 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8073aabc-9fac-40a5-bfaa-b150a5623e1c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4mmlv" (UID: "8073aabc-9fac-40a5-bfaa-b150a5623e1c") : configmap "kube-rbac-proxy" not found Sep 14 10:49:48.649159 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:48.648852 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8073aabc-9fac-40a5-bfaa-b150a5623e1c-crio-socket\") pod \"insights-runtime-extractor-4mmlv\" (UID: \"8073aabc-9fac-40a5-bfaa-b150a5623e1c\") " pod="openshift-insights/insights-runtime-extractor-4mmlv" Sep 14 10:49:48.649159 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:48.648950 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 14 10:49:48.649159 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:48.649003 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8073aabc-9fac-40a5-bfaa-b150a5623e1c-data-volume\") pod \"insights-runtime-extractor-4mmlv\" (UID: \"8073aabc-9fac-40a5-bfaa-b150a5623e1c\") " pod="openshift-insights/insights-runtime-extractor-4mmlv" Sep 14 10:49:48.649159 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:48.649011 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8073aabc-9fac-40a5-bfaa-b150a5623e1c-insights-runtime-extractor-tls podName:8073aabc-9fac-40a5-bfaa-b150a5623e1c nodeName:}" failed. No retries permitted until 2026-09-14 10:49:49.148993391 +0000 UTC m=+62.891395341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8073aabc-9fac-40a5-bfaa-b150a5623e1c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4mmlv" (UID: "8073aabc-9fac-40a5-bfaa-b150a5623e1c") : secret "insights-runtime-extractor-tls" not found Sep 14 10:49:48.649159 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:48.649042 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8073aabc-9fac-40a5-bfaa-b150a5623e1c-crio-socket\") pod \"insights-runtime-extractor-4mmlv\" (UID: \"8073aabc-9fac-40a5-bfaa-b150a5623e1c\") " pod="openshift-insights/insights-runtime-extractor-4mmlv" Sep 14 10:49:48.649495 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:48.649384 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8073aabc-9fac-40a5-bfaa-b150a5623e1c-data-volume\") pod \"insights-runtime-extractor-4mmlv\" (UID: \"8073aabc-9fac-40a5-bfaa-b150a5623e1c\") " pod="openshift-insights/insights-runtime-extractor-4mmlv" Sep 14 10:49:48.660011 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:48.659978 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8bbd\" (UniqueName: \"kubernetes.io/projected/8073aabc-9fac-40a5-bfaa-b150a5623e1c-kube-api-access-c8bbd\") pod \"insights-runtime-extractor-4mmlv\" (UID: \"8073aabc-9fac-40a5-bfaa-b150a5623e1c\") " pod="openshift-insights/insights-runtime-extractor-4mmlv" Sep 14 10:49:49.152708 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:49.152652 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8073aabc-9fac-40a5-bfaa-b150a5623e1c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4mmlv\" (UID: \"8073aabc-9fac-40a5-bfaa-b150a5623e1c\") " pod="openshift-insights/insights-runtime-extractor-4mmlv" Sep 14 10:49:49.152708 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:49.152719 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8073aabc-9fac-40a5-bfaa-b150a5623e1c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4mmlv\" (UID: \"8073aabc-9fac-40a5-bfaa-b150a5623e1c\") " pod="openshift-insights/insights-runtime-extractor-4mmlv" Sep 14 10:49:49.152936 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:49.152774 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 14 10:49:49.152936 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:49.152811 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 14 10:49:49.152936 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:49.152834 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8073aabc-9fac-40a5-bfaa-b150a5623e1c-kube-rbac-proxy-cm podName:8073aabc-9fac-40a5-bfaa-b150a5623e1c nodeName:}" failed. No retries permitted until 2026-09-14 10:49:50.15282048 +0000 UTC m=+63.895222428 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8073aabc-9fac-40a5-bfaa-b150a5623e1c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4mmlv" (UID: "8073aabc-9fac-40a5-bfaa-b150a5623e1c") : configmap "kube-rbac-proxy" not found Sep 14 10:49:49.152936 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:49.152886 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8073aabc-9fac-40a5-bfaa-b150a5623e1c-insights-runtime-extractor-tls podName:8073aabc-9fac-40a5-bfaa-b150a5623e1c nodeName:}" failed. No retries permitted until 2026-09-14 10:49:50.152868762 +0000 UTC m=+63.895270724 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8073aabc-9fac-40a5-bfaa-b150a5623e1c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4mmlv" (UID: "8073aabc-9fac-40a5-bfaa-b150a5623e1c") : secret "insights-runtime-extractor-tls" not found Sep 14 10:49:50.160852 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:50.160807 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8073aabc-9fac-40a5-bfaa-b150a5623e1c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4mmlv\" (UID: \"8073aabc-9fac-40a5-bfaa-b150a5623e1c\") " pod="openshift-insights/insights-runtime-extractor-4mmlv" Sep 14 10:49:50.161234 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:50.160861 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8073aabc-9fac-40a5-bfaa-b150a5623e1c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4mmlv\" (UID: \"8073aabc-9fac-40a5-bfaa-b150a5623e1c\") " pod="openshift-insights/insights-runtime-extractor-4mmlv" Sep 14 10:49:50.161234 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:50.160976 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 14 10:49:50.161234 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:50.161048 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8073aabc-9fac-40a5-bfaa-b150a5623e1c-insights-runtime-extractor-tls podName:8073aabc-9fac-40a5-bfaa-b150a5623e1c nodeName:}" failed. No retries permitted until 2026-09-14 10:49:52.161032322 +0000 UTC m=+65.903434275 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8073aabc-9fac-40a5-bfaa-b150a5623e1c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4mmlv" (UID: "8073aabc-9fac-40a5-bfaa-b150a5623e1c") : secret "insights-runtime-extractor-tls" not found Sep 14 10:49:50.161234 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:50.160977 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 14 10:49:50.161234 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:50.161111 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8073aabc-9fac-40a5-bfaa-b150a5623e1c-kube-rbac-proxy-cm podName:8073aabc-9fac-40a5-bfaa-b150a5623e1c nodeName:}" failed. No retries permitted until 2026-09-14 10:49:52.16109712 +0000 UTC m=+65.903499085 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8073aabc-9fac-40a5-bfaa-b150a5623e1c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4mmlv" (UID: "8073aabc-9fac-40a5-bfaa-b150a5623e1c") : configmap "kube-rbac-proxy" not found Sep 14 10:49:50.249273 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:50.249242 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5nq5v_ceba844b-2781-40ea-96e0-9eacd1e13ef6/dns-node-resolver/0.log" Sep 14 10:49:51.446680 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:51.446654 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rfppn_5b6399c5-592d-463e-a9a4-337834d10604/node-ca/0.log" Sep 14 10:49:51.672946 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:51.672905 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf34b271-8c37-42fc-9bb6-0ff0a45af426-registry-tls\") pod \"image-registry-7cf5b5b9d-2v58p\" (UID: \"cf34b271-8c37-42fc-9bb6-0ff0a45af426\") " pod="openshift-image-registry/image-registry-7cf5b5b9d-2v58p" Sep 14 10:49:51.673122 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:51.672986 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/923b7386-9d29-4313-9a63-d4f43e4c2932-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-bmlrv\" (UID: \"923b7386-9d29-4313-9a63-d4f43e4c2932\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bmlrv" Sep 14 10:49:51.673122 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:51.673045 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 10:49:51.673122 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:51.673062 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cf5b5b9d-2v58p: secret "image-registry-tls" not found Sep 14 10:49:51.673122 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:51.673114 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf34b271-8c37-42fc-9bb6-0ff0a45af426-registry-tls podName:cf34b271-8c37-42fc-9bb6-0ff0a45af426 nodeName:}" failed. No retries permitted until 2026-09-14 10:49:59.673099096 +0000 UTC m=+73.415501044 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cf34b271-8c37-42fc-9bb6-0ff0a45af426-registry-tls") pod "image-registry-7cf5b5b9d-2v58p" (UID: "cf34b271-8c37-42fc-9bb6-0ff0a45af426") : secret "image-registry-tls" not found Sep 14 10:49:51.673286 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:51.673119 2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 14 10:49:51.673286 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:51.673167 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/923b7386-9d29-4313-9a63-d4f43e4c2932-cluster-monitoring-operator-tls podName:923b7386-9d29-4313-9a63-d4f43e4c2932 nodeName:}" failed. No retries permitted until 2026-09-14 10:49:59.673155138 +0000 UTC m=+73.415557112 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/923b7386-9d29-4313-9a63-d4f43e4c2932-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-bmlrv" (UID: "923b7386-9d29-4313-9a63-d4f43e4c2932") : secret "cluster-monitoring-operator-tls" not found Sep 14 10:49:52.175973 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:52.175936 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8073aabc-9fac-40a5-bfaa-b150a5623e1c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4mmlv\" (UID: \"8073aabc-9fac-40a5-bfaa-b150a5623e1c\") " pod="openshift-insights/insights-runtime-extractor-4mmlv" Sep 14 10:49:52.175973 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:52.175983 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8073aabc-9fac-40a5-bfaa-b150a5623e1c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4mmlv\" (UID: \"8073aabc-9fac-40a5-bfaa-b150a5623e1c\") " pod="openshift-insights/insights-runtime-extractor-4mmlv" Sep 14 10:49:52.176208 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:52.176051 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 14 10:49:52.176208 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:52.176090 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 14 10:49:52.176208 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:52.176105 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8073aabc-9fac-40a5-bfaa-b150a5623e1c-kube-rbac-proxy-cm podName:8073aabc-9fac-40a5-bfaa-b150a5623e1c nodeName:}" failed. No retries permitted until 2026-09-14 10:49:56.176091591 +0000 UTC m=+69.918493539 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8073aabc-9fac-40a5-bfaa-b150a5623e1c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4mmlv" (UID: "8073aabc-9fac-40a5-bfaa-b150a5623e1c") : configmap "kube-rbac-proxy" not found Sep 14 10:49:52.176208 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:52.176159 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8073aabc-9fac-40a5-bfaa-b150a5623e1c-insights-runtime-extractor-tls podName:8073aabc-9fac-40a5-bfaa-b150a5623e1c nodeName:}" failed. No retries permitted until 2026-09-14 10:49:56.176146348 +0000 UTC m=+69.918548299 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8073aabc-9fac-40a5-bfaa-b150a5623e1c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4mmlv" (UID: "8073aabc-9fac-40a5-bfaa-b150a5623e1c") : secret "insights-runtime-extractor-tls" not found Sep 14 10:49:52.448295 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:52.448214 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wrtgl_3112f5c3-15ae-4606-8da4-c066b3c92af8/serve-healthcheck-canary/0.log" Sep 14 10:49:52.478611 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:52.478574 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh7zw\" (UniqueName: \"kubernetes.io/projected/54a7a889-af23-436c-ac67-8b45c1b39194-kube-api-access-mh7zw\") pod \"network-check-target-tw9zw\" (UID: \"54a7a889-af23-436c-ac67-8b45c1b39194\") " pod="openshift-network-diagnostics/network-check-target-tw9zw" Sep 14 10:49:52.478744 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:52.478661 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f7b7b9c-eb99-4eb6-b619-8db12670cef0-original-pull-secret\") pod \"global-pull-secret-syncer-b62cb\" (UID: \"6f7b7b9c-eb99-4eb6-b619-8db12670cef0\") " pod="kube-system/global-pull-secret-syncer-b62cb" Sep 14 10:49:52.481301 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:52.481277 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 14 10:49:52.481442 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:52.481303 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 14 10:49:52.491436 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:52.491405 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 14 10:49:52.491674 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:52.491658 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f7b7b9c-eb99-4eb6-b619-8db12670cef0-original-pull-secret\") pod \"global-pull-secret-syncer-b62cb\" (UID: \"6f7b7b9c-eb99-4eb6-b619-8db12670cef0\") " pod="kube-system/global-pull-secret-syncer-b62cb" Sep 14 10:49:52.501723 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:52.501699 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh7zw\" (UniqueName: \"kubernetes.io/projected/54a7a889-af23-436c-ac67-8b45c1b39194-kube-api-access-mh7zw\") pod \"network-check-target-tw9zw\" (UID: \"54a7a889-af23-436c-ac67-8b45c1b39194\") " pod="openshift-network-diagnostics/network-check-target-tw9zw" Sep 14 10:49:52.540058 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:52.540021 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b62cb" Sep 14 10:49:52.556964 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:52.556943 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fkf2p\"" Sep 14 10:49:52.565436 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:52.565410 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tw9zw" Sep 14 10:49:52.580146 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:52.579679 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04ff0710-e09c-48e9-9769-7ffce3645b66-metrics-certs\") pod \"network-metrics-daemon-29jrs\" (UID: \"04ff0710-e09c-48e9-9769-7ffce3645b66\") " pod="openshift-multus/network-metrics-daemon-29jrs" Sep 14 10:49:52.582512 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:52.582436 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 14 10:49:52.591668 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:52.591171 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 14 10:49:52.591668 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:52.591255 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04ff0710-e09c-48e9-9769-7ffce3645b66-metrics-certs podName:04ff0710-e09c-48e9-9769-7ffce3645b66 nodeName:}" failed. No retries permitted until 2026-09-14 10:50:56.591230933 +0000 UTC m=+130.333632897 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/04ff0710-e09c-48e9-9769-7ffce3645b66-metrics-certs") pod "network-metrics-daemon-29jrs" (UID: "04ff0710-e09c-48e9-9769-7ffce3645b66") : secret "metrics-daemon-secret" not found Sep 14 10:49:52.815230 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:52.815198 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b62cb"] Sep 14 10:49:52.820254 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:49:52.820226 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f7b7b9c_eb99_4eb6_b619_8db12670cef0.slice/crio-0b7e8cb853b531cc477233f18174d3ec9b2a762a2c51e1c11dfcf33993157502 WatchSource:0}: Error finding container 0b7e8cb853b531cc477233f18174d3ec9b2a762a2c51e1c11dfcf33993157502: Status 404 returned error can't find the container with id 0b7e8cb853b531cc477233f18174d3ec9b2a762a2c51e1c11dfcf33993157502 Sep 14 10:49:52.847250 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:52.847216 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tw9zw"] Sep 14 10:49:52.850074 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:49:52.850049 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54a7a889_af23_436c_ac67_8b45c1b39194.slice/crio-21cfe86dede7dba3bf1853c267860e012076f21850fdd53efe16c1b973ec9bf0 WatchSource:0}: Error finding container 21cfe86dede7dba3bf1853c267860e012076f21850fdd53efe16c1b973ec9bf0: Status 404 returned error can't find the container with id 21cfe86dede7dba3bf1853c267860e012076f21850fdd53efe16c1b973ec9bf0 Sep 14 10:49:52.982149 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:52.982108 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/305e2ad1-1b29-4c6b-92e9-b21fa53847bc-metrics-tls\") pod \"dns-default-zp5bd\" (UID: \"305e2ad1-1b29-4c6b-92e9-b21fa53847bc\") " pod="openshift-dns/dns-default-zp5bd" Sep 14 10:49:52.982312 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:52.982265 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 10:49:52.982388 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:52.982333 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/305e2ad1-1b29-4c6b-92e9-b21fa53847bc-metrics-tls podName:305e2ad1-1b29-4c6b-92e9-b21fa53847bc nodeName:}" failed. No retries permitted until 2026-09-14 10:50:24.982315669 +0000 UTC m=+98.724717620 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/305e2ad1-1b29-4c6b-92e9-b21fa53847bc-metrics-tls") pod "dns-default-zp5bd" (UID: "305e2ad1-1b29-4c6b-92e9-b21fa53847bc") : secret "dns-default-metrics-tls" not found Sep 14 10:49:53.239069 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:53.238974 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tw9zw" event={"ID":"54a7a889-af23-436c-ac67-8b45c1b39194","Type":"ContainerStarted","Data":"21cfe86dede7dba3bf1853c267860e012076f21850fdd53efe16c1b973ec9bf0"} Sep 14 10:49:53.240071 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:53.240049 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b62cb" event={"ID":"6f7b7b9c-eb99-4eb6-b619-8db12670cef0","Type":"ContainerStarted","Data":"0b7e8cb853b531cc477233f18174d3ec9b2a762a2c51e1c11dfcf33993157502"} Sep 14 10:49:56.211294 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:56.211250 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8073aabc-9fac-40a5-bfaa-b150a5623e1c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4mmlv\" (UID: \"8073aabc-9fac-40a5-bfaa-b150a5623e1c\") " pod="openshift-insights/insights-runtime-extractor-4mmlv" Sep 14 10:49:56.211294 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:56.211298 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8073aabc-9fac-40a5-bfaa-b150a5623e1c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4mmlv\" (UID: \"8073aabc-9fac-40a5-bfaa-b150a5623e1c\") " pod="openshift-insights/insights-runtime-extractor-4mmlv" Sep 14 10:49:56.211767 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:56.211436 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 14 10:49:56.211767 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:56.211498 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8073aabc-9fac-40a5-bfaa-b150a5623e1c-insights-runtime-extractor-tls podName:8073aabc-9fac-40a5-bfaa-b150a5623e1c nodeName:}" failed. No retries permitted until 2026-09-14 10:50:04.211483691 +0000 UTC m=+77.953885639 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8073aabc-9fac-40a5-bfaa-b150a5623e1c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4mmlv" (UID: "8073aabc-9fac-40a5-bfaa-b150a5623e1c") : secret "insights-runtime-extractor-tls" not found Sep 14 10:49:56.211767 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:56.211524 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 14 10:49:56.211767 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:56.211599 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8073aabc-9fac-40a5-bfaa-b150a5623e1c-kube-rbac-proxy-cm podName:8073aabc-9fac-40a5-bfaa-b150a5623e1c nodeName:}" failed. No retries permitted until 2026-09-14 10:50:04.21158061 +0000 UTC m=+77.953982572 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8073aabc-9fac-40a5-bfaa-b150a5623e1c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4mmlv" (UID: "8073aabc-9fac-40a5-bfaa-b150a5623e1c") : configmap "kube-rbac-proxy" not found Sep 14 10:49:58.256577 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:58.256531 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b62cb" event={"ID":"6f7b7b9c-eb99-4eb6-b619-8db12670cef0","Type":"ContainerStarted","Data":"2ce6e330e11a80ee75534f98cad6d466b56d6f9d8c37fb7ea816a366f5dd434e"} Sep 14 10:49:58.258068 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:58.258047 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tw9zw" event={"ID":"54a7a889-af23-436c-ac67-8b45c1b39194","Type":"ContainerStarted","Data":"896a46fc9a63586cc7234d5a247d64f595c3747e0c2b4315cff0cf632953ef32"} Sep 14 10:49:58.258182 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:58.258171 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tw9zw" Sep 14 10:49:58.274274 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:58.274220 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-b62cb" podStartSLOduration=66.460704244 podStartE2EDuration="1m11.27420807s" totalImagesPullingTime="4.813503826s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 10:48:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:49:58.273780369 +0000 UTC m=+72.016182339" watchObservedRunningTime="2026-09-14 10:49:58.27420807 +0000 UTC m=+72.016610039" Sep 14 10:49:58.293512 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:58.293448 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tw9zw" podStartSLOduration=66.524777801 podStartE2EDuration="1m11.293431037s" totalImagesPullingTime="4.768653236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:48:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:49:58.292584291 +0000 UTC m=+72.034986262" watchObservedRunningTime="2026-09-14 10:49:58.293431037 +0000 UTC m=+72.035833007" Sep 14 10:49:59.744160 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:59.744119 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf34b271-8c37-42fc-9bb6-0ff0a45af426-registry-tls\") pod \"image-registry-7cf5b5b9d-2v58p\" (UID: \"cf34b271-8c37-42fc-9bb6-0ff0a45af426\") " pod="openshift-image-registry/image-registry-7cf5b5b9d-2v58p" Sep 14 10:49:59.744574 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:49:59.744182 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/923b7386-9d29-4313-9a63-d4f43e4c2932-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-bmlrv\" (UID: \"923b7386-9d29-4313-9a63-d4f43e4c2932\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bmlrv" Sep 14 10:49:59.744574 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:59.744286 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 10:49:59.744574 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:59.744301 2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 14 10:49:59.744574 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:59.744308 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cf5b5b9d-2v58p: secret "image-registry-tls" not found Sep 14 10:49:59.744574 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:59.744373 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/923b7386-9d29-4313-9a63-d4f43e4c2932-cluster-monitoring-operator-tls podName:923b7386-9d29-4313-9a63-d4f43e4c2932 nodeName:}" failed. No retries permitted until 2026-09-14 10:50:15.744358707 +0000 UTC m=+89.486760655 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/923b7386-9d29-4313-9a63-d4f43e4c2932-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-bmlrv" (UID: "923b7386-9d29-4313-9a63-d4f43e4c2932") : secret "cluster-monitoring-operator-tls" not found Sep 14 10:49:59.744574 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:49:59.744387 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf34b271-8c37-42fc-9bb6-0ff0a45af426-registry-tls podName:cf34b271-8c37-42fc-9bb6-0ff0a45af426 nodeName:}" failed. No retries permitted until 2026-09-14 10:50:15.744379958 +0000 UTC m=+89.486781906 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cf34b271-8c37-42fc-9bb6-0ff0a45af426-registry-tls") pod "image-registry-7cf5b5b9d-2v58p" (UID: "cf34b271-8c37-42fc-9bb6-0ff0a45af426") : secret "image-registry-tls" not found Sep 14 10:50:04.279230 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:04.279192 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8073aabc-9fac-40a5-bfaa-b150a5623e1c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4mmlv\" (UID: \"8073aabc-9fac-40a5-bfaa-b150a5623e1c\") " pod="openshift-insights/insights-runtime-extractor-4mmlv" Sep 14 10:50:04.279750 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:04.279239 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8073aabc-9fac-40a5-bfaa-b150a5623e1c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4mmlv\" (UID: \"8073aabc-9fac-40a5-bfaa-b150a5623e1c\") " pod="openshift-insights/insights-runtime-extractor-4mmlv" Sep 14 10:50:04.279750 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:50:04.279333 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 14 10:50:04.279750 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:50:04.279406 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8073aabc-9fac-40a5-bfaa-b150a5623e1c-insights-runtime-extractor-tls podName:8073aabc-9fac-40a5-bfaa-b150a5623e1c nodeName:}" failed. No retries permitted until 2026-09-14 10:50:20.279390679 +0000 UTC m=+94.021792627 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8073aabc-9fac-40a5-bfaa-b150a5623e1c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4mmlv" (UID: "8073aabc-9fac-40a5-bfaa-b150a5623e1c") : secret "insights-runtime-extractor-tls" not found Sep 14 10:50:04.279928 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:04.279867 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8073aabc-9fac-40a5-bfaa-b150a5623e1c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4mmlv\" (UID: \"8073aabc-9fac-40a5-bfaa-b150a5623e1c\") " pod="openshift-insights/insights-runtime-extractor-4mmlv" Sep 14 10:50:13.568510 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.568392 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-rfn74"] Sep 14 10:50:13.574941 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.574911 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f9b5877f6-qx4p2"] Sep 14 10:50:13.575090 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.575031 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-rfn74" Sep 14 10:50:13.577628 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.577606 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 14 10:50:13.577628 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.577618 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f9b5877f6-qx4p2" Sep 14 10:50:13.577852 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.577737 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-jq8lf\"" Sep 14 10:50:13.578559 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.578538 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 14 10:50:13.580613 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.580592 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-nzwrf\"" Sep 14 10:50:13.580771 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.580748 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 14 10:50:13.580903 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.580821 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 14 10:50:13.581141 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.581128 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 14 10:50:13.581248 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.581231 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 14 10:50:13.589007 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.588949 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f9b5877f6-qx4p2"] Sep 14 10:50:13.589998 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.589970 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-rfn74"] Sep 14 10:50:13.606718 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.606693 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7cf5b5b9d-2v58p"] Sep 14 10:50:13.606941 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:50:13.606919 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-7cf5b5b9d-2v58p" podUID="cf34b271-8c37-42fc-9bb6-0ff0a45af426" Sep 14 10:50:13.647587 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.647558 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-78c748dd77-fzgm4"] Sep 14 10:50:13.650534 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.650519 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78c748dd77-fzgm4" Sep 14 10:50:13.653746 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.653725 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6ebfcbad-f943-4bfa-8ace-27cca1543c26-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-rfn74\" (UID: \"6ebfcbad-f943-4bfa-8ace-27cca1543c26\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-rfn74" Sep 14 10:50:13.653858 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.653756 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8vt2\" (UniqueName: \"kubernetes.io/projected/fcad83b2-1320-4263-be0c-06c5471233f1-kube-api-access-j8vt2\") pod \"managed-serviceaccount-addon-agent-f9b5877f6-qx4p2\" (UID: \"fcad83b2-1320-4263-be0c-06c5471233f1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f9b5877f6-qx4p2" Sep 14 10:50:13.653858 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.653775 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6ebfcbad-f943-4bfa-8ace-27cca1543c26-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-rfn74\" (UID: \"6ebfcbad-f943-4bfa-8ace-27cca1543c26\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-rfn74" Sep 14 10:50:13.653858 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.653806 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fcad83b2-1320-4263-be0c-06c5471233f1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-f9b5877f6-qx4p2\" (UID: \"fcad83b2-1320-4263-be0c-06c5471233f1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f9b5877f6-qx4p2" Sep 14 10:50:13.667937 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.667908 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78c748dd77-fzgm4"] Sep 14 10:50:13.755036 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.755001 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/227e5dcb-0c45-4294-b6db-c6fc6edac409-ca-trust-extracted\") pod \"image-registry-78c748dd77-fzgm4\" (UID: \"227e5dcb-0c45-4294-b6db-c6fc6edac409\") " pod="openshift-image-registry/image-registry-78c748dd77-fzgm4" Sep 14 10:50:13.755036 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.755040 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/227e5dcb-0c45-4294-b6db-c6fc6edac409-image-registry-private-configuration\") pod \"image-registry-78c748dd77-fzgm4\" (UID: \"227e5dcb-0c45-4294-b6db-c6fc6edac409\") " pod="openshift-image-registry/image-registry-78c748dd77-fzgm4" Sep 14 10:50:13.755246 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.755132 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/227e5dcb-0c45-4294-b6db-c6fc6edac409-bound-sa-token\") pod \"image-registry-78c748dd77-fzgm4\" (UID: \"227e5dcb-0c45-4294-b6db-c6fc6edac409\") " pod="openshift-image-registry/image-registry-78c748dd77-fzgm4" Sep 14 10:50:13.755246 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.755165 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6dx4\" (UniqueName: \"kubernetes.io/projected/227e5dcb-0c45-4294-b6db-c6fc6edac409-kube-api-access-c6dx4\") pod \"image-registry-78c748dd77-fzgm4\" (UID: \"227e5dcb-0c45-4294-b6db-c6fc6edac409\") " pod="openshift-image-registry/image-registry-78c748dd77-fzgm4" Sep 14 10:50:13.755246 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.755189 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6ebfcbad-f943-4bfa-8ace-27cca1543c26-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-rfn74\" (UID: \"6ebfcbad-f943-4bfa-8ace-27cca1543c26\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-rfn74" Sep 14 10:50:13.755246 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.755235 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8vt2\" (UniqueName: \"kubernetes.io/projected/fcad83b2-1320-4263-be0c-06c5471233f1-kube-api-access-j8vt2\") pod \"managed-serviceaccount-addon-agent-f9b5877f6-qx4p2\" (UID: \"fcad83b2-1320-4263-be0c-06c5471233f1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f9b5877f6-qx4p2" Sep 14 10:50:13.755411 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.755266 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6ebfcbad-f943-4bfa-8ace-27cca1543c26-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-rfn74\" (UID: \"6ebfcbad-f943-4bfa-8ace-27cca1543c26\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-rfn74" Sep 14 10:50:13.755411 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.755292 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/227e5dcb-0c45-4294-b6db-c6fc6edac409-registry-certificates\") pod \"image-registry-78c748dd77-fzgm4\" (UID: \"227e5dcb-0c45-4294-b6db-c6fc6edac409\") " pod="openshift-image-registry/image-registry-78c748dd77-fzgm4" Sep 14 10:50:13.755411 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.755331 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/227e5dcb-0c45-4294-b6db-c6fc6edac409-registry-tls\") pod \"image-registry-78c748dd77-fzgm4\" (UID: \"227e5dcb-0c45-4294-b6db-c6fc6edac409\") " pod="openshift-image-registry/image-registry-78c748dd77-fzgm4" Sep 14 10:50:13.755411 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.755385 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fcad83b2-1320-4263-be0c-06c5471233f1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-f9b5877f6-qx4p2\" (UID: \"fcad83b2-1320-4263-be0c-06c5471233f1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f9b5877f6-qx4p2" Sep 14 10:50:13.755574 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.755408 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/227e5dcb-0c45-4294-b6db-c6fc6edac409-trusted-ca\") pod \"image-registry-78c748dd77-fzgm4\" (UID: \"227e5dcb-0c45-4294-b6db-c6fc6edac409\") " pod="openshift-image-registry/image-registry-78c748dd77-fzgm4" Sep 14 10:50:13.755574 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.755474 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/227e5dcb-0c45-4294-b6db-c6fc6edac409-installation-pull-secrets\") pod \"image-registry-78c748dd77-fzgm4\" (UID: \"227e5dcb-0c45-4294-b6db-c6fc6edac409\") " pod="openshift-image-registry/image-registry-78c748dd77-fzgm4" Sep 14 10:50:13.755929 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.755911 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6ebfcbad-f943-4bfa-8ace-27cca1543c26-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-rfn74\" (UID: \"6ebfcbad-f943-4bfa-8ace-27cca1543c26\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-rfn74" Sep 14 10:50:13.758095 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.758070 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6ebfcbad-f943-4bfa-8ace-27cca1543c26-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-rfn74\" (UID: \"6ebfcbad-f943-4bfa-8ace-27cca1543c26\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-rfn74" Sep 14 10:50:13.758187 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.758152 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fcad83b2-1320-4263-be0c-06c5471233f1-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-f9b5877f6-qx4p2\" (UID: \"fcad83b2-1320-4263-be0c-06c5471233f1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f9b5877f6-qx4p2" Sep 14 10:50:13.767389 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.767359 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8vt2\" (UniqueName: \"kubernetes.io/projected/fcad83b2-1320-4263-be0c-06c5471233f1-kube-api-access-j8vt2\") pod \"managed-serviceaccount-addon-agent-f9b5877f6-qx4p2\" (UID: \"fcad83b2-1320-4263-be0c-06c5471233f1\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f9b5877f6-qx4p2" Sep 14 10:50:13.856620 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.856529 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/227e5dcb-0c45-4294-b6db-c6fc6edac409-registry-tls\") pod \"image-registry-78c748dd77-fzgm4\" (UID: \"227e5dcb-0c45-4294-b6db-c6fc6edac409\") " pod="openshift-image-registry/image-registry-78c748dd77-fzgm4" Sep 14 10:50:13.856620 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.856569 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/227e5dcb-0c45-4294-b6db-c6fc6edac409-trusted-ca\") pod \"image-registry-78c748dd77-fzgm4\" (UID: \"227e5dcb-0c45-4294-b6db-c6fc6edac409\") " pod="openshift-image-registry/image-registry-78c748dd77-fzgm4" Sep 14 10:50:13.856805 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.856714 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/227e5dcb-0c45-4294-b6db-c6fc6edac409-installation-pull-secrets\") pod \"image-registry-78c748dd77-fzgm4\" (UID: \"227e5dcb-0c45-4294-b6db-c6fc6edac409\") " pod="openshift-image-registry/image-registry-78c748dd77-fzgm4" Sep 14 10:50:13.856805 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.856749 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/227e5dcb-0c45-4294-b6db-c6fc6edac409-ca-trust-extracted\") pod \"image-registry-78c748dd77-fzgm4\" (UID: \"227e5dcb-0c45-4294-b6db-c6fc6edac409\") " pod="openshift-image-registry/image-registry-78c748dd77-fzgm4" Sep 14 10:50:13.856805 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.856774 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/227e5dcb-0c45-4294-b6db-c6fc6edac409-image-registry-private-configuration\") pod \"image-registry-78c748dd77-fzgm4\" (UID: \"227e5dcb-0c45-4294-b6db-c6fc6edac409\") " pod="openshift-image-registry/image-registry-78c748dd77-fzgm4" Sep 14 10:50:13.856926 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.856828 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/227e5dcb-0c45-4294-b6db-c6fc6edac409-bound-sa-token\") pod \"image-registry-78c748dd77-fzgm4\" (UID: \"227e5dcb-0c45-4294-b6db-c6fc6edac409\") " pod="openshift-image-registry/image-registry-78c748dd77-fzgm4" Sep 14 10:50:13.856926 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.856855 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6dx4\" (UniqueName: \"kubernetes.io/projected/227e5dcb-0c45-4294-b6db-c6fc6edac409-kube-api-access-c6dx4\") pod \"image-registry-78c748dd77-fzgm4\" (UID: \"227e5dcb-0c45-4294-b6db-c6fc6edac409\") " pod="openshift-image-registry/image-registry-78c748dd77-fzgm4" Sep 14 10:50:13.856926 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.856895 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/227e5dcb-0c45-4294-b6db-c6fc6edac409-registry-certificates\") pod \"image-registry-78c748dd77-fzgm4\" (UID: \"227e5dcb-0c45-4294-b6db-c6fc6edac409\") " pod="openshift-image-registry/image-registry-78c748dd77-fzgm4" Sep 14 10:50:13.857162 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.857134 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/227e5dcb-0c45-4294-b6db-c6fc6edac409-ca-trust-extracted\") pod \"image-registry-78c748dd77-fzgm4\" (UID: \"227e5dcb-0c45-4294-b6db-c6fc6edac409\") " pod="openshift-image-registry/image-registry-78c748dd77-fzgm4" Sep 14 10:50:13.857608 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.857588 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/227e5dcb-0c45-4294-b6db-c6fc6edac409-registry-certificates\") pod \"image-registry-78c748dd77-fzgm4\" (UID: \"227e5dcb-0c45-4294-b6db-c6fc6edac409\") " pod="openshift-image-registry/image-registry-78c748dd77-fzgm4" Sep 14 10:50:13.857771 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.857755 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/227e5dcb-0c45-4294-b6db-c6fc6edac409-trusted-ca\") pod \"image-registry-78c748dd77-fzgm4\" (UID: \"227e5dcb-0c45-4294-b6db-c6fc6edac409\") " pod="openshift-image-registry/image-registry-78c748dd77-fzgm4" Sep 14 10:50:13.859252 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.859236 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/227e5dcb-0c45-4294-b6db-c6fc6edac409-installation-pull-secrets\") pod \"image-registry-78c748dd77-fzgm4\" (UID: \"227e5dcb-0c45-4294-b6db-c6fc6edac409\") " pod="openshift-image-registry/image-registry-78c748dd77-fzgm4" Sep 14 10:50:13.859708 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.859685 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/227e5dcb-0c45-4294-b6db-c6fc6edac409-image-registry-private-configuration\") pod \"image-registry-78c748dd77-fzgm4\" (UID: \"227e5dcb-0c45-4294-b6db-c6fc6edac409\") " pod="openshift-image-registry/image-registry-78c748dd77-fzgm4" Sep 14 10:50:13.859912 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.859892 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/227e5dcb-0c45-4294-b6db-c6fc6edac409-registry-tls\") pod \"image-registry-78c748dd77-fzgm4\" (UID: \"227e5dcb-0c45-4294-b6db-c6fc6edac409\") " pod="openshift-image-registry/image-registry-78c748dd77-fzgm4" Sep 14 10:50:13.868018 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.867994 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/227e5dcb-0c45-4294-b6db-c6fc6edac409-bound-sa-token\") pod \"image-registry-78c748dd77-fzgm4\" (UID: \"227e5dcb-0c45-4294-b6db-c6fc6edac409\") " pod="openshift-image-registry/image-registry-78c748dd77-fzgm4" Sep 14 10:50:13.868546 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.868530 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6dx4\" (UniqueName: \"kubernetes.io/projected/227e5dcb-0c45-4294-b6db-c6fc6edac409-kube-api-access-c6dx4\") pod \"image-registry-78c748dd77-fzgm4\" (UID: \"227e5dcb-0c45-4294-b6db-c6fc6edac409\") " pod="openshift-image-registry/image-registry-78c748dd77-fzgm4" Sep 14 10:50:13.885423 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.885399 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-rfn74" Sep 14 10:50:13.903802 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.903781 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f9b5877f6-qx4p2" Sep 14 10:50:13.961522 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.961489 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xzrxf\"" Sep 14 10:50:13.969173 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:13.969152 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78c748dd77-fzgm4" Sep 14 10:50:14.159915 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:14.159838 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-rfn74"] Sep 14 10:50:14.163013 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:50:14.162988 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ebfcbad_f943_4bfa_8ace_27cca1543c26.slice/crio-be57d1c102b2dd3f75642ec7f8bdbfb4f8487c8ad089111aeda157ffccc0a4c2 WatchSource:0}: Error finding container be57d1c102b2dd3f75642ec7f8bdbfb4f8487c8ad089111aeda157ffccc0a4c2: Status 404 returned error can't find the container with id be57d1c102b2dd3f75642ec7f8bdbfb4f8487c8ad089111aeda157ffccc0a4c2 Sep 14 10:50:14.188798 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:14.188771 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f9b5877f6-qx4p2"] Sep 14 10:50:14.191697 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:50:14.191673 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfcad83b2_1320_4263_be0c_06c5471233f1.slice/crio-d9a4b3790b186e73fcd6de35309aa02dfbfe4d53430bf455682111075a488663 WatchSource:0}: Error finding container d9a4b3790b186e73fcd6de35309aa02dfbfe4d53430bf455682111075a488663: Status 404 returned error can't find the container with id d9a4b3790b186e73fcd6de35309aa02dfbfe4d53430bf455682111075a488663 Sep 14 10:50:14.243023 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:14.242989 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78c748dd77-fzgm4"] Sep 14 10:50:14.245632 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:50:14.245603 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod227e5dcb_0c45_4294_b6db_c6fc6edac409.slice/crio-57c01420e4601daed6aabb3bfb2764d8093c0987fedb3f4e41ffe33435b76748 WatchSource:0}: Error finding container 57c01420e4601daed6aabb3bfb2764d8093c0987fedb3f4e41ffe33435b76748: Status 404 returned error can't find the container with id 57c01420e4601daed6aabb3bfb2764d8093c0987fedb3f4e41ffe33435b76748 Sep 14 10:50:14.313030 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:14.313001 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f9b5877f6-qx4p2" event={"ID":"fcad83b2-1320-4263-be0c-06c5471233f1","Type":"ContainerStarted","Data":"d9a4b3790b186e73fcd6de35309aa02dfbfe4d53430bf455682111075a488663"} Sep 14 10:50:14.314520 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:14.314496 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78c748dd77-fzgm4" event={"ID":"227e5dcb-0c45-4294-b6db-c6fc6edac409","Type":"ContainerStarted","Data":"3680ca72fc8161a88c78608a069bfc7fb07f219d54d4499d1be20f8575e2808b"} Sep 14 10:50:14.314624 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:14.314543 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-78c748dd77-fzgm4" event={"ID":"227e5dcb-0c45-4294-b6db-c6fc6edac409","Type":"ContainerStarted","Data":"57c01420e4601daed6aabb3bfb2764d8093c0987fedb3f4e41ffe33435b76748"} Sep 14 10:50:14.314624 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:14.314593 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-78c748dd77-fzgm4" Sep 14 10:50:14.315793 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:14.315768 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-rfn74" event={"ID":"6ebfcbad-f943-4bfa-8ace-27cca1543c26","Type":"ContainerStarted","Data":"be57d1c102b2dd3f75642ec7f8bdbfb4f8487c8ad089111aeda157ffccc0a4c2"} Sep 14 10:50:14.315793 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:14.315781 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cf5b5b9d-2v58p" Sep 14 10:50:14.322352 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:14.322322 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cf5b5b9d-2v58p" Sep 14 10:50:14.338356 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:14.338296 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-78c748dd77-fzgm4" podStartSLOduration=1.338277359 podStartE2EDuration="1.338277359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:50:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:50:14.337242444 +0000 UTC m=+88.079644414" watchObservedRunningTime="2026-09-14 10:50:14.338277359 +0000 UTC m=+88.080679330" Sep 14 10:50:14.461886 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:14.461790 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cf34b271-8c37-42fc-9bb6-0ff0a45af426-registry-certificates\") pod \"cf34b271-8c37-42fc-9bb6-0ff0a45af426\" (UID: \"cf34b271-8c37-42fc-9bb6-0ff0a45af426\") " Sep 14 10:50:14.461886 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:14.461844 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cf34b271-8c37-42fc-9bb6-0ff0a45af426-bound-sa-token\") pod \"cf34b271-8c37-42fc-9bb6-0ff0a45af426\" (UID: \"cf34b271-8c37-42fc-9bb6-0ff0a45af426\") " Sep 14 10:50:14.461886 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:14.461869 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cf34b271-8c37-42fc-9bb6-0ff0a45af426-trusted-ca\") pod \"cf34b271-8c37-42fc-9bb6-0ff0a45af426\" (UID: \"cf34b271-8c37-42fc-9bb6-0ff0a45af426\") " Sep 14 10:50:14.462228 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:14.461945 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cf34b271-8c37-42fc-9bb6-0ff0a45af426-ca-trust-extracted\") pod \"cf34b271-8c37-42fc-9bb6-0ff0a45af426\" (UID: \"cf34b271-8c37-42fc-9bb6-0ff0a45af426\") " Sep 14 10:50:14.462228 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:14.461986 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cf34b271-8c37-42fc-9bb6-0ff0a45af426-image-registry-private-configuration\") pod \"cf34b271-8c37-42fc-9bb6-0ff0a45af426\" (UID: \"cf34b271-8c37-42fc-9bb6-0ff0a45af426\") " Sep 14 10:50:14.462228 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:14.462023 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9t57g\" (UniqueName: \"kubernetes.io/projected/cf34b271-8c37-42fc-9bb6-0ff0a45af426-kube-api-access-9t57g\") pod \"cf34b271-8c37-42fc-9bb6-0ff0a45af426\" (UID: \"cf34b271-8c37-42fc-9bb6-0ff0a45af426\") " Sep 14 10:50:14.462228 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:14.462076 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cf34b271-8c37-42fc-9bb6-0ff0a45af426-installation-pull-secrets\") pod \"cf34b271-8c37-42fc-9bb6-0ff0a45af426\" (UID: \"cf34b271-8c37-42fc-9bb6-0ff0a45af426\") " Sep 14 10:50:14.462228 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:14.462146 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf34b271-8c37-42fc-9bb6-0ff0a45af426-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "cf34b271-8c37-42fc-9bb6-0ff0a45af426" (UID: "cf34b271-8c37-42fc-9bb6-0ff0a45af426"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:50:14.462503 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:14.462264 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf34b271-8c37-42fc-9bb6-0ff0a45af426-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "cf34b271-8c37-42fc-9bb6-0ff0a45af426" (UID: "cf34b271-8c37-42fc-9bb6-0ff0a45af426"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 10:50:14.462503 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:14.462384 2581 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cf34b271-8c37-42fc-9bb6-0ff0a45af426-registry-certificates\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:50:14.462503 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:14.462402 2581 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cf34b271-8c37-42fc-9bb6-0ff0a45af426-ca-trust-extracted\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:50:14.462503 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:14.462407 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf34b271-8c37-42fc-9bb6-0ff0a45af426-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "cf34b271-8c37-42fc-9bb6-0ff0a45af426" (UID: "cf34b271-8c37-42fc-9bb6-0ff0a45af426"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:50:14.464609 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:14.464586 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf34b271-8c37-42fc-9bb6-0ff0a45af426-kube-api-access-9t57g" (OuterVolumeSpecName: "kube-api-access-9t57g") pod "cf34b271-8c37-42fc-9bb6-0ff0a45af426" (UID: "cf34b271-8c37-42fc-9bb6-0ff0a45af426"). InnerVolumeSpecName "kube-api-access-9t57g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:50:14.464728 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:14.464689 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf34b271-8c37-42fc-9bb6-0ff0a45af426-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "cf34b271-8c37-42fc-9bb6-0ff0a45af426" (UID: "cf34b271-8c37-42fc-9bb6-0ff0a45af426"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:50:14.464836 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:14.464816 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf34b271-8c37-42fc-9bb6-0ff0a45af426-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "cf34b271-8c37-42fc-9bb6-0ff0a45af426" (UID: "cf34b271-8c37-42fc-9bb6-0ff0a45af426"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 10:50:14.464881 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:14.464858 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf34b271-8c37-42fc-9bb6-0ff0a45af426-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "cf34b271-8c37-42fc-9bb6-0ff0a45af426" (UID: "cf34b271-8c37-42fc-9bb6-0ff0a45af426"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 10:50:14.563637 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:14.563605 2581 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cf34b271-8c37-42fc-9bb6-0ff0a45af426-bound-sa-token\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:50:14.563637 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:14.563630 2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cf34b271-8c37-42fc-9bb6-0ff0a45af426-trusted-ca\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:50:14.563637 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:14.563642 2581 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cf34b271-8c37-42fc-9bb6-0ff0a45af426-image-registry-private-configuration\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:50:14.563855 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:14.563651 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t57g\" (UniqueName: \"kubernetes.io/projected/cf34b271-8c37-42fc-9bb6-0ff0a45af426-kube-api-access-9t57g\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:50:14.563855 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:14.563660 2581 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cf34b271-8c37-42fc-9bb6-0ff0a45af426-installation-pull-secrets\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:50:15.321978 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:15.321539 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cf5b5b9d-2v58p" Sep 14 10:50:15.356235 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:15.356183 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7cf5b5b9d-2v58p"] Sep 14 10:50:15.365314 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:15.365271 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7cf5b5b9d-2v58p"] Sep 14 10:50:15.473797 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:15.473764 2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cf34b271-8c37-42fc-9bb6-0ff0a45af426-registry-tls\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:50:15.776139 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:15.776103 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/923b7386-9d29-4313-9a63-d4f43e4c2932-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-bmlrv\" (UID: \"923b7386-9d29-4313-9a63-d4f43e4c2932\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bmlrv" Sep 14 10:50:15.778993 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:15.778966 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/923b7386-9d29-4313-9a63-d4f43e4c2932-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-bmlrv\" (UID: \"923b7386-9d29-4313-9a63-d4f43e4c2932\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bmlrv" Sep 14 10:50:15.999266 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:15.999191 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-clglc\"" Sep 14 10:50:16.004792 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:16.004770 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bmlrv" Sep 14 10:50:16.326800 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:16.326758 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-rfn74" event={"ID":"6ebfcbad-f943-4bfa-8ace-27cca1543c26","Type":"ContainerStarted","Data":"7214dc078d274f1926e2583c2bea197d13a3d6e0014031ae4696d4c626a54c69"} Sep 14 10:50:16.343217 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:16.343163 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-rfn74" podStartSLOduration=1.980771774 podStartE2EDuration="3.343142808s" totalImagesPullingTime="1.362371034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:50:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:50:16.34270807 +0000 UTC m=+90.085110041" watchObservedRunningTime="2026-09-14 10:50:16.343142808 +0000 UTC m=+90.085544779" Sep 14 10:50:16.830146 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:16.830125 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf34b271-8c37-42fc-9bb6-0ff0a45af426" path="/var/lib/kubelet/pods/cf34b271-8c37-42fc-9bb6-0ff0a45af426/volumes" Sep 14 10:50:17.020447 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:17.020365 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-bmlrv"] Sep 14 10:50:17.023668 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:50:17.023637 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod923b7386_9d29_4313_9a63_d4f43e4c2932.slice/crio-be73107f7295aa0ff1c7eec68cadc2890db9fe7511bd4f2a395fd15a85e84b02 WatchSource:0}: Error finding container be73107f7295aa0ff1c7eec68cadc2890db9fe7511bd4f2a395fd15a85e84b02: Status 404 returned error can't find the container with id be73107f7295aa0ff1c7eec68cadc2890db9fe7511bd4f2a395fd15a85e84b02 Sep 14 10:50:17.331883 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:17.331852 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f9b5877f6-qx4p2" event={"ID":"fcad83b2-1320-4263-be0c-06c5471233f1","Type":"ContainerStarted","Data":"25433f72aa07c3c48ac93d2365c46a0b51249f6c286e2f1acc6aac8e74f7eb89"} Sep 14 10:50:17.332991 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:17.332970 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bmlrv" event={"ID":"923b7386-9d29-4313-9a63-d4f43e4c2932","Type":"ContainerStarted","Data":"be73107f7295aa0ff1c7eec68cadc2890db9fe7511bd4f2a395fd15a85e84b02"} Sep 14 10:50:19.342218 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:19.342181 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bmlrv" event={"ID":"923b7386-9d29-4313-9a63-d4f43e4c2932","Type":"ContainerStarted","Data":"44061ade5c7aab351bec273f83e5a6b50bd49da2a9624a2ca47a28e20a10a4ea"} Sep 14 10:50:19.361876 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:19.361827 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bmlrv" podStartSLOduration=34.482458659 podStartE2EDuration="36.361811128s" totalImagesPullingTime="1.879352469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:49:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:50:19.360917103 +0000 UTC m=+93.103319075" watchObservedRunningTime="2026-09-14 10:50:19.361811128 +0000 UTC m=+93.104213098" Sep 14 10:50:19.362038 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:19.361937 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f9b5877f6-qx4p2" podStartSLOduration=3.7278501139999998 podStartE2EDuration="6.361931608s" totalImagesPullingTime="2.634081494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:50:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:50:17.348228509 +0000 UTC m=+91.090630478" watchObservedRunningTime="2026-09-14 10:50:19.361931608 +0000 UTC m=+93.104333609" Sep 14 10:50:20.314206 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:20.314168 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8073aabc-9fac-40a5-bfaa-b150a5623e1c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4mmlv\" (UID: \"8073aabc-9fac-40a5-bfaa-b150a5623e1c\") " pod="openshift-insights/insights-runtime-extractor-4mmlv" Sep 14 10:50:20.316653 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:20.316623 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8073aabc-9fac-40a5-bfaa-b150a5623e1c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4mmlv\" (UID: \"8073aabc-9fac-40a5-bfaa-b150a5623e1c\") " pod="openshift-insights/insights-runtime-extractor-4mmlv" Sep 14 10:50:20.556513 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:20.556459 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4mmlv" Sep 14 10:50:20.828695 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:50:20.828667 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8073aabc_9fac_40a5_bfaa_b150a5623e1c.slice/crio-a90c5fab746367d2b988805d0484db6916ce7959d17311aed2bee4bb696f7c48 WatchSource:0}: Error finding container a90c5fab746367d2b988805d0484db6916ce7959d17311aed2bee4bb696f7c48: Status 404 returned error can't find the container with id a90c5fab746367d2b988805d0484db6916ce7959d17311aed2bee4bb696f7c48 Sep 14 10:50:20.829849 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:20.829827 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4mmlv"] Sep 14 10:50:21.349627 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:21.349595 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4mmlv" event={"ID":"8073aabc-9fac-40a5-bfaa-b150a5623e1c","Type":"ContainerStarted","Data":"c1a5f8e3b67615a981f73410d934c103db19f4d79b4bc20f110377911e3ca7eb"} Sep 14 10:50:21.349798 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:21.349655 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4mmlv" event={"ID":"8073aabc-9fac-40a5-bfaa-b150a5623e1c","Type":"ContainerStarted","Data":"a90c5fab746367d2b988805d0484db6916ce7959d17311aed2bee4bb696f7c48"} Sep 14 10:50:21.521934 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:21.521904 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7fdd647dcf-88ncj"] Sep 14 10:50:21.524645 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:21.524627 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fdd647dcf-88ncj" Sep 14 10:50:21.528402 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:21.528186 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 14 10:50:21.528402 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:21.528221 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 14 10:50:21.528402 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:21.528244 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 14 10:50:21.528402 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:21.528250 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-75bjs\"" Sep 14 10:50:21.528402 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:21.528277 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 14 10:50:21.528402 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:21.528304 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 14 10:50:21.528402 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:21.528332 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 14 10:50:21.528402 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:21.528280 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 14 10:50:21.539126 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:21.539105 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fdd647dcf-88ncj"] Sep 14 10:50:21.623819 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:21.623728 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bac0f1cd-ab44-4309-8587-f6aca897c7f3-service-ca\") pod \"console-7fdd647dcf-88ncj\" (UID: \"bac0f1cd-ab44-4309-8587-f6aca897c7f3\") " pod="openshift-console/console-7fdd647dcf-88ncj" Sep 14 10:50:21.623819 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:21.623767 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bac0f1cd-ab44-4309-8587-f6aca897c7f3-console-config\") pod \"console-7fdd647dcf-88ncj\" (UID: \"bac0f1cd-ab44-4309-8587-f6aca897c7f3\") " pod="openshift-console/console-7fdd647dcf-88ncj" Sep 14 10:50:21.623819 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:21.623787 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bac0f1cd-ab44-4309-8587-f6aca897c7f3-console-serving-cert\") pod \"console-7fdd647dcf-88ncj\" (UID: \"bac0f1cd-ab44-4309-8587-f6aca897c7f3\") " pod="openshift-console/console-7fdd647dcf-88ncj" Sep 14 10:50:21.623819 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:21.623810 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bac0f1cd-ab44-4309-8587-f6aca897c7f3-oauth-serving-cert\") pod \"console-7fdd647dcf-88ncj\" (UID: \"bac0f1cd-ab44-4309-8587-f6aca897c7f3\") " pod="openshift-console/console-7fdd647dcf-88ncj" Sep 14 10:50:21.624297 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:21.623859 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bac0f1cd-ab44-4309-8587-f6aca897c7f3-console-oauth-config\") pod \"console-7fdd647dcf-88ncj\" (UID: \"bac0f1cd-ab44-4309-8587-f6aca897c7f3\") " pod="openshift-console/console-7fdd647dcf-88ncj" Sep 14 10:50:21.624297 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:21.623897 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbztg\" (UniqueName: \"kubernetes.io/projected/bac0f1cd-ab44-4309-8587-f6aca897c7f3-kube-api-access-zbztg\") pod \"console-7fdd647dcf-88ncj\" (UID: \"bac0f1cd-ab44-4309-8587-f6aca897c7f3\") " pod="openshift-console/console-7fdd647dcf-88ncj" Sep 14 10:50:21.724530 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:21.724491 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bac0f1cd-ab44-4309-8587-f6aca897c7f3-service-ca\") pod \"console-7fdd647dcf-88ncj\" (UID: \"bac0f1cd-ab44-4309-8587-f6aca897c7f3\") " pod="openshift-console/console-7fdd647dcf-88ncj" Sep 14 10:50:21.724724 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:21.724549 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bac0f1cd-ab44-4309-8587-f6aca897c7f3-console-config\") pod \"console-7fdd647dcf-88ncj\" (UID: \"bac0f1cd-ab44-4309-8587-f6aca897c7f3\") " pod="openshift-console/console-7fdd647dcf-88ncj" Sep 14 10:50:21.724724 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:21.724593 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bac0f1cd-ab44-4309-8587-f6aca897c7f3-console-serving-cert\") pod \"console-7fdd647dcf-88ncj\" (UID: \"bac0f1cd-ab44-4309-8587-f6aca897c7f3\") " pod="openshift-console/console-7fdd647dcf-88ncj" Sep 14 10:50:21.724724 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:21.724628 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bac0f1cd-ab44-4309-8587-f6aca897c7f3-oauth-serving-cert\") pod \"console-7fdd647dcf-88ncj\" (UID: \"bac0f1cd-ab44-4309-8587-f6aca897c7f3\") " pod="openshift-console/console-7fdd647dcf-88ncj" Sep 14 10:50:21.724724 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:21.724656 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bac0f1cd-ab44-4309-8587-f6aca897c7f3-console-oauth-config\") pod \"console-7fdd647dcf-88ncj\" (UID: \"bac0f1cd-ab44-4309-8587-f6aca897c7f3\") " pod="openshift-console/console-7fdd647dcf-88ncj" Sep 14 10:50:21.724724 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:21.724682 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbztg\" (UniqueName: \"kubernetes.io/projected/bac0f1cd-ab44-4309-8587-f6aca897c7f3-kube-api-access-zbztg\") pod \"console-7fdd647dcf-88ncj\" (UID: \"bac0f1cd-ab44-4309-8587-f6aca897c7f3\") " pod="openshift-console/console-7fdd647dcf-88ncj" Sep 14 10:50:21.725358 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:21.725293 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bac0f1cd-ab44-4309-8587-f6aca897c7f3-console-config\") pod \"console-7fdd647dcf-88ncj\" (UID: \"bac0f1cd-ab44-4309-8587-f6aca897c7f3\") " pod="openshift-console/console-7fdd647dcf-88ncj" Sep 14 10:50:21.725358 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:21.725295 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bac0f1cd-ab44-4309-8587-f6aca897c7f3-service-ca\") pod \"console-7fdd647dcf-88ncj\" (UID: \"bac0f1cd-ab44-4309-8587-f6aca897c7f3\") " pod="openshift-console/console-7fdd647dcf-88ncj" Sep 14 10:50:21.725803 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:21.725710 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bac0f1cd-ab44-4309-8587-f6aca897c7f3-oauth-serving-cert\") pod \"console-7fdd647dcf-88ncj\" (UID: \"bac0f1cd-ab44-4309-8587-f6aca897c7f3\") " pod="openshift-console/console-7fdd647dcf-88ncj" Sep 14 10:50:21.727797 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:21.727756 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bac0f1cd-ab44-4309-8587-f6aca897c7f3-console-serving-cert\") pod \"console-7fdd647dcf-88ncj\" (UID: \"bac0f1cd-ab44-4309-8587-f6aca897c7f3\") " pod="openshift-console/console-7fdd647dcf-88ncj" Sep 14 10:50:21.727904 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:21.727840 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bac0f1cd-ab44-4309-8587-f6aca897c7f3-console-oauth-config\") pod \"console-7fdd647dcf-88ncj\" (UID: \"bac0f1cd-ab44-4309-8587-f6aca897c7f3\") " pod="openshift-console/console-7fdd647dcf-88ncj" Sep 14 10:50:21.733591 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:21.733566 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbztg\" (UniqueName: \"kubernetes.io/projected/bac0f1cd-ab44-4309-8587-f6aca897c7f3-kube-api-access-zbztg\") pod \"console-7fdd647dcf-88ncj\" (UID: \"bac0f1cd-ab44-4309-8587-f6aca897c7f3\") " pod="openshift-console/console-7fdd647dcf-88ncj" Sep 14 10:50:21.833391 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:21.833325 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fdd647dcf-88ncj" Sep 14 10:50:22.115232 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:22.115200 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fdd647dcf-88ncj"] Sep 14 10:50:22.118177 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:50:22.118144 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbac0f1cd_ab44_4309_8587_f6aca897c7f3.slice/crio-2c3d333f418a2f34de8d96f7b33dd8d10bf698be614136f12822e242388f5bb1 WatchSource:0}: Error finding container 2c3d333f418a2f34de8d96f7b33dd8d10bf698be614136f12822e242388f5bb1: Status 404 returned error can't find the container with id 2c3d333f418a2f34de8d96f7b33dd8d10bf698be614136f12822e242388f5bb1 Sep 14 10:50:22.354706 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:22.354670 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fdd647dcf-88ncj" event={"ID":"bac0f1cd-ab44-4309-8587-f6aca897c7f3","Type":"ContainerStarted","Data":"2c3d333f418a2f34de8d96f7b33dd8d10bf698be614136f12822e242388f5bb1"} Sep 14 10:50:22.356758 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:22.356731 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4mmlv" event={"ID":"8073aabc-9fac-40a5-bfaa-b150a5623e1c","Type":"ContainerStarted","Data":"bc8a62d87ca551c9c578edda5022a94edfefb3dd69e3a77991e4d59e01caedad"} Sep 14 10:50:22.538070 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:22.538037 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-6z6p6"] Sep 14 10:50:22.541283 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:22.541257 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-6z6p6" Sep 14 10:50:22.545081 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:22.544938 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 14 10:50:22.545081 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:22.544992 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-6mcxd\"" Sep 14 10:50:22.545285 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:22.545265 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 14 10:50:22.545450 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:22.545432 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 14 10:50:22.552705 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:22.551832 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-6z6p6"] Sep 14 10:50:22.633188 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:22.633106 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/57632e3a-064f-4b7a-9063-e254aa47b7f9-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-6z6p6\" (UID: \"57632e3a-064f-4b7a-9063-e254aa47b7f9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-6z6p6" Sep 14 10:50:22.633188 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:22.633160 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2mm2\" (UniqueName: \"kubernetes.io/projected/57632e3a-064f-4b7a-9063-e254aa47b7f9-kube-api-access-n2mm2\") pod \"prometheus-operator-68bcd7f987-6z6p6\" (UID: \"57632e3a-064f-4b7a-9063-e254aa47b7f9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-6z6p6" Sep 14 10:50:22.633616 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:22.633262 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/57632e3a-064f-4b7a-9063-e254aa47b7f9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-6z6p6\" (UID: \"57632e3a-064f-4b7a-9063-e254aa47b7f9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-6z6p6" Sep 14 10:50:22.633616 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:22.633353 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57632e3a-064f-4b7a-9063-e254aa47b7f9-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-6z6p6\" (UID: \"57632e3a-064f-4b7a-9063-e254aa47b7f9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-6z6p6" Sep 14 10:50:22.734093 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:22.734052 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/57632e3a-064f-4b7a-9063-e254aa47b7f9-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-6z6p6\" (UID: \"57632e3a-064f-4b7a-9063-e254aa47b7f9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-6z6p6" Sep 14 10:50:22.734240 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:22.734106 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2mm2\" (UniqueName: \"kubernetes.io/projected/57632e3a-064f-4b7a-9063-e254aa47b7f9-kube-api-access-n2mm2\") pod \"prometheus-operator-68bcd7f987-6z6p6\" (UID: \"57632e3a-064f-4b7a-9063-e254aa47b7f9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-6z6p6" Sep 14 10:50:22.734240 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:22.734191 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/57632e3a-064f-4b7a-9063-e254aa47b7f9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-6z6p6\" (UID: \"57632e3a-064f-4b7a-9063-e254aa47b7f9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-6z6p6" Sep 14 10:50:22.734240 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:22.734225 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57632e3a-064f-4b7a-9063-e254aa47b7f9-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-6z6p6\" (UID: \"57632e3a-064f-4b7a-9063-e254aa47b7f9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-6z6p6" Sep 14 10:50:22.735016 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:22.734968 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57632e3a-064f-4b7a-9063-e254aa47b7f9-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-6z6p6\" (UID: \"57632e3a-064f-4b7a-9063-e254aa47b7f9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-6z6p6" Sep 14 10:50:22.737204 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:22.737174 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/57632e3a-064f-4b7a-9063-e254aa47b7f9-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-6z6p6\" (UID: \"57632e3a-064f-4b7a-9063-e254aa47b7f9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-6z6p6" Sep 14 10:50:22.737352 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:22.737319 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/57632e3a-064f-4b7a-9063-e254aa47b7f9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-6z6p6\" (UID: \"57632e3a-064f-4b7a-9063-e254aa47b7f9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-6z6p6" Sep 14 10:50:22.742739 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:22.742693 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2mm2\" (UniqueName: \"kubernetes.io/projected/57632e3a-064f-4b7a-9063-e254aa47b7f9-kube-api-access-n2mm2\") pod \"prometheus-operator-68bcd7f987-6z6p6\" (UID: \"57632e3a-064f-4b7a-9063-e254aa47b7f9\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-6z6p6" Sep 14 10:50:22.855971 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:22.855937 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-6z6p6" Sep 14 10:50:23.523852 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:23.523813 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-6z6p6"] Sep 14 10:50:23.527578 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:50:23.527540 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57632e3a_064f_4b7a_9063_e254aa47b7f9.slice/crio-9f30007bf16b297998846cfa0c775b706ea955df2b1e5f5ffd3797a78f9743b5 WatchSource:0}: Error finding container 9f30007bf16b297998846cfa0c775b706ea955df2b1e5f5ffd3797a78f9743b5: Status 404 returned error can't find the container with id 9f30007bf16b297998846cfa0c775b706ea955df2b1e5f5ffd3797a78f9743b5 Sep 14 10:50:24.367255 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:24.367213 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4mmlv" event={"ID":"8073aabc-9fac-40a5-bfaa-b150a5623e1c","Type":"ContainerStarted","Data":"a9e31d9143f121ef4f8fe6119e7abdcae482aa6d6e7397a927cb521f076f1fd2"} Sep 14 10:50:24.368564 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:24.368540 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-6z6p6" event={"ID":"57632e3a-064f-4b7a-9063-e254aa47b7f9","Type":"ContainerStarted","Data":"9f30007bf16b297998846cfa0c775b706ea955df2b1e5f5ffd3797a78f9743b5"} Sep 14 10:50:24.389163 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:24.389104 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4mmlv" podStartSLOduration=34.028027759 podStartE2EDuration="36.389083849s" totalImagesPullingTime="2.36105609s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 10:49:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:50:24.387770363 +0000 UTC m=+98.130172334" watchObservedRunningTime="2026-09-14 10:50:24.389083849 +0000 UTC m=+98.131485821" Sep 14 10:50:25.054241 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:25.054154 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/305e2ad1-1b29-4c6b-92e9-b21fa53847bc-metrics-tls\") pod \"dns-default-zp5bd\" (UID: \"305e2ad1-1b29-4c6b-92e9-b21fa53847bc\") " pod="openshift-dns/dns-default-zp5bd" Sep 14 10:50:25.056839 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:25.056809 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/305e2ad1-1b29-4c6b-92e9-b21fa53847bc-metrics-tls\") pod \"dns-default-zp5bd\" (UID: \"305e2ad1-1b29-4c6b-92e9-b21fa53847bc\") " pod="openshift-dns/dns-default-zp5bd" Sep 14 10:50:25.097577 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:25.097540 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qfwcj\"" Sep 14 10:50:25.105679 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:25.105654 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zp5bd" Sep 14 10:50:25.373782 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:25.373700 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fdd647dcf-88ncj" event={"ID":"bac0f1cd-ab44-4309-8587-f6aca897c7f3","Type":"ContainerStarted","Data":"9e691c9ed21a86961aa672375dc1af124103118dd6a675d1474760b2404146a1"} Sep 14 10:50:25.394142 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:25.394112 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zp5bd"] Sep 14 10:50:25.396936 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:25.396889 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7fdd647dcf-88ncj" podStartSLOduration=1.5523036019999998 podStartE2EDuration="4.396874128s" totalImagesPullingTime="2.844570526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:50:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:50:25.395296468 +0000 UTC m=+99.137698437" watchObservedRunningTime="2026-09-14 10:50:25.396874128 +0000 UTC m=+99.139276097" Sep 14 10:50:25.683210 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:50:25.683147 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod305e2ad1_1b29_4c6b_92e9_b21fa53847bc.slice/crio-936816e44074f42d6cc9a42802561753c50ea1b2ede5a56b72926fa564d2d657 WatchSource:0}: Error finding container 936816e44074f42d6cc9a42802561753c50ea1b2ede5a56b72926fa564d2d657: Status 404 returned error can't find the container with id 936816e44074f42d6cc9a42802561753c50ea1b2ede5a56b72926fa564d2d657 Sep 14 10:50:26.378320 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:26.378279 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zp5bd" event={"ID":"305e2ad1-1b29-4c6b-92e9-b21fa53847bc","Type":"ContainerStarted","Data":"936816e44074f42d6cc9a42802561753c50ea1b2ede5a56b72926fa564d2d657"} Sep 14 10:50:26.379940 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:26.379914 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-6z6p6" event={"ID":"57632e3a-064f-4b7a-9063-e254aa47b7f9","Type":"ContainerStarted","Data":"11782318ed1001b33cec7121ed7cac0eb76172b83e191ae75e2116b47bd4e6c2"} Sep 14 10:50:26.380044 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:26.379972 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-6z6p6" event={"ID":"57632e3a-064f-4b7a-9063-e254aa47b7f9","Type":"ContainerStarted","Data":"a0c0342a08a0cd628c6203f13de4cc952600d2389e2f716d88e36316ed0bd7d4"} Sep 14 10:50:26.397157 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:26.397103 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-6z6p6" podStartSLOduration=2.19910752 podStartE2EDuration="4.397083747s" totalImagesPullingTime="2.197976227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:50:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:50:26.396739132 +0000 UTC m=+100.139141103" watchObservedRunningTime="2026-09-14 10:50:26.397083747 +0000 UTC m=+100.139485718" Sep 14 10:50:28.339755 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:28.339714 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bm7pz"] Sep 14 10:50:28.343145 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:28.343121 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bm7pz" Sep 14 10:50:28.355406 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:50:28.355376 2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"node-exporter-bm7pz\" is forbidden: User \"system:node:ip-10-0-137-53.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-137-53.ec2.internal' and this object" podUID="1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46" pod="openshift-monitoring/node-exporter-bm7pz" Sep 14 10:50:28.356398 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:50:28.356375 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"node-exporter-kube-rbac-proxy-config\" is forbidden: User \"system:node:ip-10-0-137-53.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-137-53.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" type="*v1.Secret" Sep 14 10:50:28.356484 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:50:28.356464 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"node-exporter-dockercfg-86qr8\" is forbidden: User \"system:node:ip-10-0-137-53.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-137-53.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-86qr8\"" type="*v1.Secret" Sep 14 10:50:28.356792 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:50:28.356769 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"node-exporter-tls\" is forbidden: User \"system:node:ip-10-0-137-53.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-137-53.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" type="*v1.Secret" Sep 14 10:50:28.356861 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:50:28.356835 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"node-exporter-accelerators-collector-config\" is forbidden: User \"system:node:ip-10-0-137-53.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-137-53.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" type="*v1.ConfigMap" Sep 14 10:50:28.388734 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:28.388704 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zp5bd" event={"ID":"305e2ad1-1b29-4c6b-92e9-b21fa53847bc","Type":"ContainerStarted","Data":"7a612b4e89d44e28efbbaf709604c08499bee85c321a1c9a28e7c9ec8819f762"} Sep 14 10:50:28.388869 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:28.388760 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zp5bd" event={"ID":"305e2ad1-1b29-4c6b-92e9-b21fa53847bc","Type":"ContainerStarted","Data":"71eac8efab4f03cc2e1e5d846d6d5f3543ed04ef6478e0f94a9e3f7604396424"} Sep 14 10:50:28.388869 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:28.388799 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zp5bd" Sep 14 10:50:28.436282 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:28.436230 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zp5bd" podStartSLOduration=65.428260725 podStartE2EDuration="1m7.43621652s" totalImagesPullingTime="2.007955795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:49:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:50:28.434827069 +0000 UTC m=+102.177229061" watchObservedRunningTime="2026-09-14 10:50:28.43621652 +0000 UTC m=+102.178618489" Sep 14 10:50:28.487328 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:28.487298 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46-node-exporter-textfile\") pod \"node-exporter-bm7pz\" (UID: \"1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46\") " pod="openshift-monitoring/node-exporter-bm7pz" Sep 14 10:50:28.487514 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:28.487379 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46-node-exporter-wtmp\") pod \"node-exporter-bm7pz\" (UID: \"1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46\") " pod="openshift-monitoring/node-exporter-bm7pz" Sep 14 10:50:28.487514 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:28.487441 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46-metrics-client-ca\") pod \"node-exporter-bm7pz\" (UID: \"1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46\") " pod="openshift-monitoring/node-exporter-bm7pz" Sep 14 10:50:28.487514 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:28.487491 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46-node-exporter-accelerators-collector-config\") pod \"node-exporter-bm7pz\" (UID: \"1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46\") " pod="openshift-monitoring/node-exporter-bm7pz" Sep 14 10:50:28.487662 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:28.487519 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46-sys\") pod \"node-exporter-bm7pz\" (UID: \"1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46\") " pod="openshift-monitoring/node-exporter-bm7pz" Sep 14 10:50:28.487662 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:28.487542 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bm7pz\" (UID: \"1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46\") " pod="openshift-monitoring/node-exporter-bm7pz" Sep 14 10:50:28.487662 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:28.487609 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46-root\") pod \"node-exporter-bm7pz\" (UID: \"1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46\") " pod="openshift-monitoring/node-exporter-bm7pz" Sep 14 10:50:28.487781 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:28.487675 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pjfl\" (UniqueName: \"kubernetes.io/projected/1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46-kube-api-access-9pjfl\") pod \"node-exporter-bm7pz\" (UID: \"1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46\") " pod="openshift-monitoring/node-exporter-bm7pz" Sep 14 10:50:28.487781 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:28.487762 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46-node-exporter-tls\") pod \"node-exporter-bm7pz\" (UID: \"1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46\") " pod="openshift-monitoring/node-exporter-bm7pz" Sep 14 10:50:28.588965 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:28.588931 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46-node-exporter-accelerators-collector-config\") pod \"node-exporter-bm7pz\" (UID: \"1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46\") " pod="openshift-monitoring/node-exporter-bm7pz" Sep 14 10:50:28.588965 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:28.588967 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46-sys\") pod \"node-exporter-bm7pz\" (UID: \"1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46\") " pod="openshift-monitoring/node-exporter-bm7pz" Sep 14 10:50:28.589192 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:28.588982 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bm7pz\" (UID: \"1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46\") " pod="openshift-monitoring/node-exporter-bm7pz" Sep 14 10:50:28.589192 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:28.589015 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46-root\") pod \"node-exporter-bm7pz\" (UID: \"1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46\") " pod="openshift-monitoring/node-exporter-bm7pz" Sep 14 10:50:28.589192 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:28.589056 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46-sys\") pod \"node-exporter-bm7pz\" (UID: \"1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46\") " pod="openshift-monitoring/node-exporter-bm7pz" Sep 14 10:50:28.589192 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:28.589065 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pjfl\" (UniqueName: \"kubernetes.io/projected/1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46-kube-api-access-9pjfl\") pod \"node-exporter-bm7pz\" (UID: \"1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46\") " pod="openshift-monitoring/node-exporter-bm7pz" Sep 14 10:50:28.589192 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:28.589107 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46-node-exporter-tls\") pod \"node-exporter-bm7pz\" (UID: \"1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46\") " pod="openshift-monitoring/node-exporter-bm7pz" Sep 14 10:50:28.589192 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:28.589119 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46-root\") pod \"node-exporter-bm7pz\" (UID: \"1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46\") " pod="openshift-monitoring/node-exporter-bm7pz" Sep 14 10:50:28.589192 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:28.589143 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46-node-exporter-textfile\") pod \"node-exporter-bm7pz\" (UID: \"1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46\") " pod="openshift-monitoring/node-exporter-bm7pz" Sep 14 10:50:28.589192 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:28.589170 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46-node-exporter-wtmp\") pod \"node-exporter-bm7pz\" (UID: \"1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46\") " pod="openshift-monitoring/node-exporter-bm7pz" Sep 14 10:50:28.589491 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:28.589204 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46-metrics-client-ca\") pod \"node-exporter-bm7pz\" (UID: \"1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46\") " pod="openshift-monitoring/node-exporter-bm7pz" Sep 14 10:50:28.589491 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:28.589368 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46-node-exporter-wtmp\") pod \"node-exporter-bm7pz\" (UID: \"1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46\") " pod="openshift-monitoring/node-exporter-bm7pz" Sep 14 10:50:28.589556 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:28.589522 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46-node-exporter-textfile\") pod \"node-exporter-bm7pz\" (UID: \"1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46\") " pod="openshift-monitoring/node-exporter-bm7pz" Sep 14 10:50:28.589794 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:28.589755 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46-metrics-client-ca\") pod \"node-exporter-bm7pz\" (UID: \"1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46\") " pod="openshift-monitoring/node-exporter-bm7pz" Sep 14 10:50:28.609475 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:28.609449 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pjfl\" (UniqueName: \"kubernetes.io/projected/1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46-kube-api-access-9pjfl\") pod \"node-exporter-bm7pz\" (UID: \"1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46\") " pod="openshift-monitoring/node-exporter-bm7pz" Sep 14 10:50:29.264494 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:29.264466 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tw9zw" Sep 14 10:50:29.499278 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:29.499247 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 14 10:50:29.509994 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:29.509972 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46-node-exporter-accelerators-collector-config\") pod \"node-exporter-bm7pz\" (UID: \"1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46\") " pod="openshift-monitoring/node-exporter-bm7pz" Sep 14 10:50:29.589999 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:50:29.589968 2581 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-kube-rbac-proxy-config: failed to sync secret cache: timed out waiting for the condition Sep 14 10:50:29.590173 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:50:29.590035 2581 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: failed to sync secret cache: timed out waiting for the condition Sep 14 10:50:29.590173 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:50:29.590053 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46-node-exporter-kube-rbac-proxy-config podName:1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46 nodeName:}" failed. No retries permitted until 2026-09-14 10:50:30.090034825 +0000 UTC m=+103.832436784 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-kube-rbac-proxy-config" (UniqueName: "kubernetes.io/secret/1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46-node-exporter-kube-rbac-proxy-config") pod "node-exporter-bm7pz" (UID: "1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46") : failed to sync secret cache: timed out waiting for the condition Sep 14 10:50:29.590173 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:50:29.590083 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46-node-exporter-tls podName:1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46 nodeName:}" failed. No retries permitted until 2026-09-14 10:50:30.090070442 +0000 UTC m=+103.832472389 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46-node-exporter-tls") pod "node-exporter-bm7pz" (UID: "1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46") : failed to sync secret cache: timed out waiting for the condition Sep 14 10:50:29.694979 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:29.694948 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-86qr8\"" Sep 14 10:50:29.736263 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:29.736228 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 14 10:50:29.957286 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:29.957214 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 14 10:50:30.101677 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:30.101642 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46-node-exporter-tls\") pod \"node-exporter-bm7pz\" (UID: \"1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46\") " pod="openshift-monitoring/node-exporter-bm7pz" Sep 14 10:50:30.101821 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:30.101706 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bm7pz\" (UID: \"1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46\") " pod="openshift-monitoring/node-exporter-bm7pz" Sep 14 10:50:30.104162 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:30.104136 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bm7pz\" (UID: \"1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46\") " pod="openshift-monitoring/node-exporter-bm7pz" Sep 14 10:50:30.104285 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:30.104196 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46-node-exporter-tls\") pod \"node-exporter-bm7pz\" (UID: \"1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46\") " pod="openshift-monitoring/node-exporter-bm7pz" Sep 14 10:50:30.151477 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:30.151444 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bm7pz" Sep 14 10:50:30.164689 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:50:30.164657 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f4d9097_9d34_44ff_a9d1_cff6c8cb6e46.slice/crio-22cf734cfbb9c8afd10341bb39fc3baf2b4265113814ef51040ea3cc432361e1 WatchSource:0}: Error finding container 22cf734cfbb9c8afd10341bb39fc3baf2b4265113814ef51040ea3cc432361e1: Status 404 returned error can't find the container with id 22cf734cfbb9c8afd10341bb39fc3baf2b4265113814ef51040ea3cc432361e1 Sep 14 10:50:30.396199 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:30.396150 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bm7pz" event={"ID":"1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46","Type":"ContainerStarted","Data":"22cf734cfbb9c8afd10341bb39fc3baf2b4265113814ef51040ea3cc432361e1"} Sep 14 10:50:31.546664 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:31.546638 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-84b587f84f-bbcq9"] Sep 14 10:50:31.578666 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:31.578637 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-84b587f84f-bbcq9" Sep 14 10:50:31.589781 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:31.589758 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 14 10:50:31.591599 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:31.591581 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 14 10:50:31.594721 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:31.594701 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 14 10:50:31.594820 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:31.594748 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-cjrh00kb74egu\"" Sep 14 10:50:31.594820 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:31.594768 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 14 10:50:31.595369 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:31.595354 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 14 10:50:31.609591 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:31.609567 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-vw4fx\"" Sep 14 10:50:31.611918 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:31.611896 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-84b587f84f-bbcq9"] Sep 14 10:50:31.717034 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:31.717002 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0ae46dae-1601-4782-b36e-3f824d74c904-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84b587f84f-bbcq9\" (UID: \"0ae46dae-1601-4782-b36e-3f824d74c904\") " pod="openshift-monitoring/thanos-querier-84b587f84f-bbcq9" Sep 14 10:50:31.717159 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:31.717053 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0ae46dae-1601-4782-b36e-3f824d74c904-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84b587f84f-bbcq9\" (UID: \"0ae46dae-1601-4782-b36e-3f824d74c904\") " pod="openshift-monitoring/thanos-querier-84b587f84f-bbcq9" Sep 14 10:50:31.717159 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:31.717096 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0ae46dae-1601-4782-b36e-3f824d74c904-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84b587f84f-bbcq9\" (UID: \"0ae46dae-1601-4782-b36e-3f824d74c904\") " pod="openshift-monitoring/thanos-querier-84b587f84f-bbcq9" Sep 14 10:50:31.717236 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:31.717178 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0ae46dae-1601-4782-b36e-3f824d74c904-secret-thanos-querier-tls\") pod \"thanos-querier-84b587f84f-bbcq9\" (UID: \"0ae46dae-1601-4782-b36e-3f824d74c904\") " pod="openshift-monitoring/thanos-querier-84b587f84f-bbcq9" Sep 14 10:50:31.717236 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:31.717210 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0ae46dae-1601-4782-b36e-3f824d74c904-secret-grpc-tls\") pod \"thanos-querier-84b587f84f-bbcq9\" (UID: \"0ae46dae-1601-4782-b36e-3f824d74c904\") " pod="openshift-monitoring/thanos-querier-84b587f84f-bbcq9" Sep 14 10:50:31.717303 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:31.717233 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cfvr\" (UniqueName: \"kubernetes.io/projected/0ae46dae-1601-4782-b36e-3f824d74c904-kube-api-access-2cfvr\") pod \"thanos-querier-84b587f84f-bbcq9\" (UID: \"0ae46dae-1601-4782-b36e-3f824d74c904\") " pod="openshift-monitoring/thanos-querier-84b587f84f-bbcq9" Sep 14 10:50:31.717303 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:31.717292 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ae46dae-1601-4782-b36e-3f824d74c904-metrics-client-ca\") pod \"thanos-querier-84b587f84f-bbcq9\" (UID: \"0ae46dae-1601-4782-b36e-3f824d74c904\") " pod="openshift-monitoring/thanos-querier-84b587f84f-bbcq9" Sep 14 10:50:31.717399 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:31.717323 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0ae46dae-1601-4782-b36e-3f824d74c904-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84b587f84f-bbcq9\" (UID: \"0ae46dae-1601-4782-b36e-3f824d74c904\") " pod="openshift-monitoring/thanos-querier-84b587f84f-bbcq9" Sep 14 10:50:31.817746 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:31.817718 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0ae46dae-1601-4782-b36e-3f824d74c904-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84b587f84f-bbcq9\" (UID: \"0ae46dae-1601-4782-b36e-3f824d74c904\") " pod="openshift-monitoring/thanos-querier-84b587f84f-bbcq9" Sep 14 10:50:31.817941 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:31.817772 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0ae46dae-1601-4782-b36e-3f824d74c904-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84b587f84f-bbcq9\" (UID: \"0ae46dae-1601-4782-b36e-3f824d74c904\") " pod="openshift-monitoring/thanos-querier-84b587f84f-bbcq9" Sep 14 10:50:31.817941 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:31.817793 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0ae46dae-1601-4782-b36e-3f824d74c904-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84b587f84f-bbcq9\" (UID: \"0ae46dae-1601-4782-b36e-3f824d74c904\") " pod="openshift-monitoring/thanos-querier-84b587f84f-bbcq9" Sep 14 10:50:31.817941 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:31.817814 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0ae46dae-1601-4782-b36e-3f824d74c904-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84b587f84f-bbcq9\" (UID: \"0ae46dae-1601-4782-b36e-3f824d74c904\") " pod="openshift-monitoring/thanos-querier-84b587f84f-bbcq9" Sep 14 10:50:31.817941 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:31.817848 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0ae46dae-1601-4782-b36e-3f824d74c904-secret-thanos-querier-tls\") pod \"thanos-querier-84b587f84f-bbcq9\" (UID: \"0ae46dae-1601-4782-b36e-3f824d74c904\") " pod="openshift-monitoring/thanos-querier-84b587f84f-bbcq9" Sep 14 10:50:31.818154 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:31.817971 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0ae46dae-1601-4782-b36e-3f824d74c904-secret-grpc-tls\") pod \"thanos-querier-84b587f84f-bbcq9\" (UID: \"0ae46dae-1601-4782-b36e-3f824d74c904\") " pod="openshift-monitoring/thanos-querier-84b587f84f-bbcq9" Sep 14 10:50:31.818154 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:31.818004 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cfvr\" (UniqueName: \"kubernetes.io/projected/0ae46dae-1601-4782-b36e-3f824d74c904-kube-api-access-2cfvr\") pod \"thanos-querier-84b587f84f-bbcq9\" (UID: \"0ae46dae-1601-4782-b36e-3f824d74c904\") " pod="openshift-monitoring/thanos-querier-84b587f84f-bbcq9" Sep 14 10:50:31.818154 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:31.818055 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ae46dae-1601-4782-b36e-3f824d74c904-metrics-client-ca\") pod \"thanos-querier-84b587f84f-bbcq9\" (UID: \"0ae46dae-1601-4782-b36e-3f824d74c904\") " pod="openshift-monitoring/thanos-querier-84b587f84f-bbcq9" Sep 14 10:50:31.818813 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:31.818761 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ae46dae-1601-4782-b36e-3f824d74c904-metrics-client-ca\") pod \"thanos-querier-84b587f84f-bbcq9\" (UID: \"0ae46dae-1601-4782-b36e-3f824d74c904\") " pod="openshift-monitoring/thanos-querier-84b587f84f-bbcq9" Sep 14 10:50:31.820688 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:31.820660 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0ae46dae-1601-4782-b36e-3f824d74c904-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84b587f84f-bbcq9\" (UID: \"0ae46dae-1601-4782-b36e-3f824d74c904\") " pod="openshift-monitoring/thanos-querier-84b587f84f-bbcq9" Sep 14 10:50:31.820798 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:31.820777 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0ae46dae-1601-4782-b36e-3f824d74c904-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84b587f84f-bbcq9\" (UID: \"0ae46dae-1601-4782-b36e-3f824d74c904\") " pod="openshift-monitoring/thanos-querier-84b587f84f-bbcq9" Sep 14 10:50:31.821320 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:31.821295 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0ae46dae-1601-4782-b36e-3f824d74c904-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84b587f84f-bbcq9\" (UID: \"0ae46dae-1601-4782-b36e-3f824d74c904\") " pod="openshift-monitoring/thanos-querier-84b587f84f-bbcq9" Sep 14 10:50:31.821440 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:31.821394 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0ae46dae-1601-4782-b36e-3f824d74c904-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84b587f84f-bbcq9\" (UID: \"0ae46dae-1601-4782-b36e-3f824d74c904\") " pod="openshift-monitoring/thanos-querier-84b587f84f-bbcq9" Sep 14 10:50:31.821656 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:31.821631 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0ae46dae-1601-4782-b36e-3f824d74c904-secret-thanos-querier-tls\") pod \"thanos-querier-84b587f84f-bbcq9\" (UID: \"0ae46dae-1601-4782-b36e-3f824d74c904\") " pod="openshift-monitoring/thanos-querier-84b587f84f-bbcq9" Sep 14 10:50:31.821819 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:31.821798 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0ae46dae-1601-4782-b36e-3f824d74c904-secret-grpc-tls\") pod \"thanos-querier-84b587f84f-bbcq9\" (UID: \"0ae46dae-1601-4782-b36e-3f824d74c904\") " pod="openshift-monitoring/thanos-querier-84b587f84f-bbcq9" Sep 14 10:50:31.830632 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:31.830611 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cfvr\" (UniqueName: \"kubernetes.io/projected/0ae46dae-1601-4782-b36e-3f824d74c904-kube-api-access-2cfvr\") pod \"thanos-querier-84b587f84f-bbcq9\" (UID: \"0ae46dae-1601-4782-b36e-3f824d74c904\") " pod="openshift-monitoring/thanos-querier-84b587f84f-bbcq9" Sep 14 10:50:31.834387 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:31.834363 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7fdd647dcf-88ncj" Sep 14 10:50:31.834468 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:31.834431 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7fdd647dcf-88ncj" Sep 14 10:50:31.839559 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:31.839541 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7fdd647dcf-88ncj" Sep 14 10:50:31.887860 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:31.887819 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-84b587f84f-bbcq9" Sep 14 10:50:32.228315 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:32.228243 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-84b587f84f-bbcq9"] Sep 14 10:50:32.231138 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:50:32.231117 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ae46dae_1601_4782_b36e_3f824d74c904.slice/crio-34bc878128959f25feaf3519d0742e04064fb8d7ee6ae161bb91d12765eabdbd WatchSource:0}: Error finding container 34bc878128959f25feaf3519d0742e04064fb8d7ee6ae161bb91d12765eabdbd: Status 404 returned error can't find the container with id 34bc878128959f25feaf3519d0742e04064fb8d7ee6ae161bb91d12765eabdbd Sep 14 10:50:32.405950 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:32.405916 2581 generic.go:356] "Generic (PLEG): container finished" podID="1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46" containerID="64cccdab038d4166cfdff8a71bcf08c1e95e8f7f437d19da51c47fa35906e4cc" exitCode=0 Sep 14 10:50:32.406122 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:32.405985 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bm7pz" event={"ID":"1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46","Type":"ContainerDied","Data":"64cccdab038d4166cfdff8a71bcf08c1e95e8f7f437d19da51c47fa35906e4cc"} Sep 14 10:50:32.407215 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:32.407190 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84b587f84f-bbcq9" event={"ID":"0ae46dae-1601-4782-b36e-3f824d74c904","Type":"ContainerStarted","Data":"34bc878128959f25feaf3519d0742e04064fb8d7ee6ae161bb91d12765eabdbd"} Sep 14 10:50:32.411143 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:32.411121 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7fdd647dcf-88ncj" Sep 14 10:50:33.039536 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:33.039503 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7d4578988b-8clwn"] Sep 14 10:50:33.060059 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:33.060023 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7d4578988b-8clwn" Sep 14 10:50:33.069004 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:33.068972 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 14 10:50:33.069142 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:33.069058 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 14 10:50:33.069331 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:33.069305 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 14 10:50:33.069557 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:33.069403 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-2zmkh\"" Sep 14 10:50:33.069557 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:33.069408 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5stdt4rnjjmu3\"" Sep 14 10:50:33.069557 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:33.069408 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 14 10:50:33.230650 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:33.230614 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/191cacad-284e-4312-9ef5-ede80f847126-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d4578988b-8clwn\" (UID: \"191cacad-284e-4312-9ef5-ede80f847126\") " pod="openshift-monitoring/metrics-server-7d4578988b-8clwn" Sep 14 10:50:33.230841 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:33.230664 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/191cacad-284e-4312-9ef5-ede80f847126-secret-metrics-server-tls\") pod \"metrics-server-7d4578988b-8clwn\" (UID: \"191cacad-284e-4312-9ef5-ede80f847126\") " pod="openshift-monitoring/metrics-server-7d4578988b-8clwn" Sep 14 10:50:33.230841 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:33.230697 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/191cacad-284e-4312-9ef5-ede80f847126-audit-log\") pod \"metrics-server-7d4578988b-8clwn\" (UID: \"191cacad-284e-4312-9ef5-ede80f847126\") " pod="openshift-monitoring/metrics-server-7d4578988b-8clwn" Sep 14 10:50:33.230841 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:33.230801 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/191cacad-284e-4312-9ef5-ede80f847126-metrics-server-audit-profiles\") pod \"metrics-server-7d4578988b-8clwn\" (UID: \"191cacad-284e-4312-9ef5-ede80f847126\") " pod="openshift-monitoring/metrics-server-7d4578988b-8clwn" Sep 14 10:50:33.231000 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:33.230954 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8nxp\" (UniqueName: \"kubernetes.io/projected/191cacad-284e-4312-9ef5-ede80f847126-kube-api-access-t8nxp\") pod \"metrics-server-7d4578988b-8clwn\" (UID: \"191cacad-284e-4312-9ef5-ede80f847126\") " pod="openshift-monitoring/metrics-server-7d4578988b-8clwn" Sep 14 10:50:33.231042 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:33.231018 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/191cacad-284e-4312-9ef5-ede80f847126-secret-metrics-server-client-certs\") pod \"metrics-server-7d4578988b-8clwn\" (UID: \"191cacad-284e-4312-9ef5-ede80f847126\") " pod="openshift-monitoring/metrics-server-7d4578988b-8clwn" Sep 14 10:50:33.231088 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:33.231058 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/191cacad-284e-4312-9ef5-ede80f847126-client-ca-bundle\") pod \"metrics-server-7d4578988b-8clwn\" (UID: \"191cacad-284e-4312-9ef5-ede80f847126\") " pod="openshift-monitoring/metrics-server-7d4578988b-8clwn" Sep 14 10:50:33.259798 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:33.259763 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7d4578988b-8clwn"] Sep 14 10:50:33.332371 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:33.332259 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/191cacad-284e-4312-9ef5-ede80f847126-metrics-server-audit-profiles\") pod \"metrics-server-7d4578988b-8clwn\" (UID: \"191cacad-284e-4312-9ef5-ede80f847126\") " pod="openshift-monitoring/metrics-server-7d4578988b-8clwn" Sep 14 10:50:33.332371 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:33.332330 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8nxp\" (UniqueName: \"kubernetes.io/projected/191cacad-284e-4312-9ef5-ede80f847126-kube-api-access-t8nxp\") pod \"metrics-server-7d4578988b-8clwn\" (UID: \"191cacad-284e-4312-9ef5-ede80f847126\") " pod="openshift-monitoring/metrics-server-7d4578988b-8clwn" Sep 14 10:50:33.332604 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:33.332390 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/191cacad-284e-4312-9ef5-ede80f847126-secret-metrics-server-client-certs\") pod \"metrics-server-7d4578988b-8clwn\" (UID: \"191cacad-284e-4312-9ef5-ede80f847126\") " pod="openshift-monitoring/metrics-server-7d4578988b-8clwn" Sep 14 10:50:33.332604 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:33.332415 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/191cacad-284e-4312-9ef5-ede80f847126-client-ca-bundle\") pod \"metrics-server-7d4578988b-8clwn\" (UID: \"191cacad-284e-4312-9ef5-ede80f847126\") " pod="openshift-monitoring/metrics-server-7d4578988b-8clwn" Sep 14 10:50:33.332604 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:33.332448 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/191cacad-284e-4312-9ef5-ede80f847126-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d4578988b-8clwn\" (UID: \"191cacad-284e-4312-9ef5-ede80f847126\") " pod="openshift-monitoring/metrics-server-7d4578988b-8clwn" Sep 14 10:50:33.332604 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:33.332499 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/191cacad-284e-4312-9ef5-ede80f847126-secret-metrics-server-tls\") pod \"metrics-server-7d4578988b-8clwn\" (UID: \"191cacad-284e-4312-9ef5-ede80f847126\") " pod="openshift-monitoring/metrics-server-7d4578988b-8clwn" Sep 14 10:50:33.332604 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:33.332536 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/191cacad-284e-4312-9ef5-ede80f847126-audit-log\") pod \"metrics-server-7d4578988b-8clwn\" (UID: \"191cacad-284e-4312-9ef5-ede80f847126\") " pod="openshift-monitoring/metrics-server-7d4578988b-8clwn" Sep 14 10:50:33.333278 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:33.332993 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/191cacad-284e-4312-9ef5-ede80f847126-audit-log\") pod \"metrics-server-7d4578988b-8clwn\" (UID: \"191cacad-284e-4312-9ef5-ede80f847126\") " pod="openshift-monitoring/metrics-server-7d4578988b-8clwn" Sep 14 10:50:33.333278 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:33.333271 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/191cacad-284e-4312-9ef5-ede80f847126-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d4578988b-8clwn\" (UID: \"191cacad-284e-4312-9ef5-ede80f847126\") " pod="openshift-monitoring/metrics-server-7d4578988b-8clwn" Sep 14 10:50:33.333745 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:33.333724 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/191cacad-284e-4312-9ef5-ede80f847126-metrics-server-audit-profiles\") pod \"metrics-server-7d4578988b-8clwn\" (UID: \"191cacad-284e-4312-9ef5-ede80f847126\") " pod="openshift-monitoring/metrics-server-7d4578988b-8clwn" Sep 14 10:50:33.335528 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:33.335504 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/191cacad-284e-4312-9ef5-ede80f847126-client-ca-bundle\") pod \"metrics-server-7d4578988b-8clwn\" (UID: \"191cacad-284e-4312-9ef5-ede80f847126\") " pod="openshift-monitoring/metrics-server-7d4578988b-8clwn" Sep 14 10:50:33.335876 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:33.335857 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/191cacad-284e-4312-9ef5-ede80f847126-secret-metrics-server-client-certs\") pod \"metrics-server-7d4578988b-8clwn\" (UID: \"191cacad-284e-4312-9ef5-ede80f847126\") " pod="openshift-monitoring/metrics-server-7d4578988b-8clwn" Sep 14 10:50:33.336108 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:33.336091 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/191cacad-284e-4312-9ef5-ede80f847126-secret-metrics-server-tls\") pod \"metrics-server-7d4578988b-8clwn\" (UID: \"191cacad-284e-4312-9ef5-ede80f847126\") " pod="openshift-monitoring/metrics-server-7d4578988b-8clwn" Sep 14 10:50:33.356317 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:33.356284 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8nxp\" (UniqueName: \"kubernetes.io/projected/191cacad-284e-4312-9ef5-ede80f847126-kube-api-access-t8nxp\") pod \"metrics-server-7d4578988b-8clwn\" (UID: \"191cacad-284e-4312-9ef5-ede80f847126\") " pod="openshift-monitoring/metrics-server-7d4578988b-8clwn" Sep 14 10:50:33.371035 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:33.371007 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7d4578988b-8clwn" Sep 14 10:50:33.413268 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:33.413234 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bm7pz" event={"ID":"1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46","Type":"ContainerStarted","Data":"c5b19a9aea49f2a04463291670e70cd8d7cc65bf811274ab2633a6952322d076"} Sep 14 10:50:33.413453 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:33.413289 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bm7pz" event={"ID":"1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46","Type":"ContainerStarted","Data":"f00f51f32c04333d324cfcb3b745c0ffb6fc51ad9a919b3e7a6ebd9d91543675"} Sep 14 10:50:33.474217 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:33.474141 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bm7pz" podStartSLOduration=4.07916904 podStartE2EDuration="5.474117584s" totalImagesPullingTime="1.315847097s" totalInitContainerRuntime="79.101447ms" isStatefulPod=true podCreationTimestamp="2026-09-14 10:50:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:50:33.473639134 +0000 UTC m=+107.216041105" watchObservedRunningTime="2026-09-14 10:50:33.474117584 +0000 UTC m=+107.216519551" Sep 14 10:50:33.730536 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:33.730440 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7d4578988b-8clwn"] Sep 14 10:50:34.032017 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:50:34.031986 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod191cacad_284e_4312_9ef5_ede80f847126.slice/crio-77cea03e3d8643eff29de010b2106732184439c3c2f7e490e0a1db064261a43d WatchSource:0}: Error finding container 77cea03e3d8643eff29de010b2106732184439c3c2f7e490e0a1db064261a43d: Status 404 returned error can't find the container with id 77cea03e3d8643eff29de010b2106732184439c3c2f7e490e0a1db064261a43d Sep 14 10:50:34.161166 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:34.161137 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5799fbfd5d-272jz"] Sep 14 10:50:34.165834 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:34.165811 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5799fbfd5d-272jz" Sep 14 10:50:34.175323 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:34.175294 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 14 10:50:34.194419 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:34.194388 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5799fbfd5d-272jz"] Sep 14 10:50:34.343617 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:34.343532 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d42234d-ca6b-4e18-af83-1c6c0022cb09-console-oauth-config\") pod \"console-5799fbfd5d-272jz\" (UID: \"8d42234d-ca6b-4e18-af83-1c6c0022cb09\") " pod="openshift-console/console-5799fbfd5d-272jz" Sep 14 10:50:34.343617 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:34.343568 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d42234d-ca6b-4e18-af83-1c6c0022cb09-console-serving-cert\") pod \"console-5799fbfd5d-272jz\" (UID: \"8d42234d-ca6b-4e18-af83-1c6c0022cb09\") " pod="openshift-console/console-5799fbfd5d-272jz" Sep 14 10:50:34.343617 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:34.343597 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d42234d-ca6b-4e18-af83-1c6c0022cb09-oauth-serving-cert\") pod \"console-5799fbfd5d-272jz\" (UID: \"8d42234d-ca6b-4e18-af83-1c6c0022cb09\") " pod="openshift-console/console-5799fbfd5d-272jz" Sep 14 10:50:34.343850 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:34.343662 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d42234d-ca6b-4e18-af83-1c6c0022cb09-service-ca\") pod \"console-5799fbfd5d-272jz\" (UID: \"8d42234d-ca6b-4e18-af83-1c6c0022cb09\") " pod="openshift-console/console-5799fbfd5d-272jz" Sep 14 10:50:34.343850 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:34.343727 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d42234d-ca6b-4e18-af83-1c6c0022cb09-trusted-ca-bundle\") pod \"console-5799fbfd5d-272jz\" (UID: \"8d42234d-ca6b-4e18-af83-1c6c0022cb09\") " pod="openshift-console/console-5799fbfd5d-272jz" Sep 14 10:50:34.343850 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:34.343764 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d42234d-ca6b-4e18-af83-1c6c0022cb09-console-config\") pod \"console-5799fbfd5d-272jz\" (UID: \"8d42234d-ca6b-4e18-af83-1c6c0022cb09\") " pod="openshift-console/console-5799fbfd5d-272jz" Sep 14 10:50:34.343850 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:34.343789 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2l2d\" (UniqueName: \"kubernetes.io/projected/8d42234d-ca6b-4e18-af83-1c6c0022cb09-kube-api-access-l2l2d\") pod \"console-5799fbfd5d-272jz\" (UID: \"8d42234d-ca6b-4e18-af83-1c6c0022cb09\") " pod="openshift-console/console-5799fbfd5d-272jz" Sep 14 10:50:34.419072 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:34.419041 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84b587f84f-bbcq9" event={"ID":"0ae46dae-1601-4782-b36e-3f824d74c904","Type":"ContainerStarted","Data":"70ad16122c837f832b1561a8439979be5ddd8f8f052cf0d6c18868f5132c01b5"} Sep 14 10:50:34.419256 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:34.419108 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84b587f84f-bbcq9" event={"ID":"0ae46dae-1601-4782-b36e-3f824d74c904","Type":"ContainerStarted","Data":"be7eca4efe4950f89fb25db13b66c3beeb88186ec440f24b1f79f496fab43cd4"} Sep 14 10:50:34.419256 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:34.419150 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84b587f84f-bbcq9" event={"ID":"0ae46dae-1601-4782-b36e-3f824d74c904","Type":"ContainerStarted","Data":"9eaf9234762c7280afdd887b22171b314bab176539dd46a772cecfbf2d50a4be"} Sep 14 10:50:34.420244 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:34.420221 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7d4578988b-8clwn" event={"ID":"191cacad-284e-4312-9ef5-ede80f847126","Type":"ContainerStarted","Data":"77cea03e3d8643eff29de010b2106732184439c3c2f7e490e0a1db064261a43d"} Sep 14 10:50:34.444961 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:34.444930 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d42234d-ca6b-4e18-af83-1c6c0022cb09-oauth-serving-cert\") pod \"console-5799fbfd5d-272jz\" (UID: \"8d42234d-ca6b-4e18-af83-1c6c0022cb09\") " pod="openshift-console/console-5799fbfd5d-272jz" Sep 14 10:50:34.445307 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:34.444971 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d42234d-ca6b-4e18-af83-1c6c0022cb09-service-ca\") pod \"console-5799fbfd5d-272jz\" (UID: \"8d42234d-ca6b-4e18-af83-1c6c0022cb09\") " pod="openshift-console/console-5799fbfd5d-272jz" Sep 14 10:50:34.445307 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:34.445013 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d42234d-ca6b-4e18-af83-1c6c0022cb09-trusted-ca-bundle\") pod \"console-5799fbfd5d-272jz\" (UID: \"8d42234d-ca6b-4e18-af83-1c6c0022cb09\") " pod="openshift-console/console-5799fbfd5d-272jz" Sep 14 10:50:34.445307 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:34.445050 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d42234d-ca6b-4e18-af83-1c6c0022cb09-console-config\") pod \"console-5799fbfd5d-272jz\" (UID: \"8d42234d-ca6b-4e18-af83-1c6c0022cb09\") " pod="openshift-console/console-5799fbfd5d-272jz" Sep 14 10:50:34.445307 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:34.445082 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2l2d\" (UniqueName: \"kubernetes.io/projected/8d42234d-ca6b-4e18-af83-1c6c0022cb09-kube-api-access-l2l2d\") pod \"console-5799fbfd5d-272jz\" (UID: \"8d42234d-ca6b-4e18-af83-1c6c0022cb09\") " pod="openshift-console/console-5799fbfd5d-272jz" Sep 14 10:50:34.445307 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:34.445115 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d42234d-ca6b-4e18-af83-1c6c0022cb09-console-oauth-config\") pod \"console-5799fbfd5d-272jz\" (UID: \"8d42234d-ca6b-4e18-af83-1c6c0022cb09\") " pod="openshift-console/console-5799fbfd5d-272jz" Sep 14 10:50:34.445307 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:34.445145 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d42234d-ca6b-4e18-af83-1c6c0022cb09-console-serving-cert\") pod \"console-5799fbfd5d-272jz\" (UID: \"8d42234d-ca6b-4e18-af83-1c6c0022cb09\") " pod="openshift-console/console-5799fbfd5d-272jz" Sep 14 10:50:34.446150 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:34.445730 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d42234d-ca6b-4e18-af83-1c6c0022cb09-oauth-serving-cert\") pod \"console-5799fbfd5d-272jz\" (UID: \"8d42234d-ca6b-4e18-af83-1c6c0022cb09\") " pod="openshift-console/console-5799fbfd5d-272jz" Sep 14 10:50:34.446150 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:34.446064 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d42234d-ca6b-4e18-af83-1c6c0022cb09-service-ca\") pod \"console-5799fbfd5d-272jz\" (UID: \"8d42234d-ca6b-4e18-af83-1c6c0022cb09\") " pod="openshift-console/console-5799fbfd5d-272jz" Sep 14 10:50:34.446463 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:34.446444 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d42234d-ca6b-4e18-af83-1c6c0022cb09-console-config\") pod \"console-5799fbfd5d-272jz\" (UID: \"8d42234d-ca6b-4e18-af83-1c6c0022cb09\") " pod="openshift-console/console-5799fbfd5d-272jz" Sep 14 10:50:34.446920 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:34.446888 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d42234d-ca6b-4e18-af83-1c6c0022cb09-trusted-ca-bundle\") pod \"console-5799fbfd5d-272jz\" (UID: \"8d42234d-ca6b-4e18-af83-1c6c0022cb09\") " pod="openshift-console/console-5799fbfd5d-272jz" Sep 14 10:50:34.448179 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:34.448155 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d42234d-ca6b-4e18-af83-1c6c0022cb09-console-oauth-config\") pod \"console-5799fbfd5d-272jz\" (UID: \"8d42234d-ca6b-4e18-af83-1c6c0022cb09\") " pod="openshift-console/console-5799fbfd5d-272jz" Sep 14 10:50:34.448281 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:34.448217 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d42234d-ca6b-4e18-af83-1c6c0022cb09-console-serving-cert\") pod \"console-5799fbfd5d-272jz\" (UID: \"8d42234d-ca6b-4e18-af83-1c6c0022cb09\") " pod="openshift-console/console-5799fbfd5d-272jz" Sep 14 10:50:34.459503 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:34.459477 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2l2d\" (UniqueName: \"kubernetes.io/projected/8d42234d-ca6b-4e18-af83-1c6c0022cb09-kube-api-access-l2l2d\") pod \"console-5799fbfd5d-272jz\" (UID: \"8d42234d-ca6b-4e18-af83-1c6c0022cb09\") " pod="openshift-console/console-5799fbfd5d-272jz" Sep 14 10:50:34.498316 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:34.498270 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5799fbfd5d-272jz" Sep 14 10:50:34.797918 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:34.797871 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5799fbfd5d-272jz"] Sep 14 10:50:34.802301 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:50:34.802269 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d42234d_ca6b_4e18_af83_1c6c0022cb09.slice/crio-548558dc7c8545839a9006394fcda56e44db1554771a19a64c451247e714159b WatchSource:0}: Error finding container 548558dc7c8545839a9006394fcda56e44db1554771a19a64c451247e714159b: Status 404 returned error can't find the container with id 548558dc7c8545839a9006394fcda56e44db1554771a19a64c451247e714159b Sep 14 10:50:35.211539 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.210952 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 14 10:50:35.215767 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.215738 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.226519 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.225820 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 14 10:50:35.226519 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.225858 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 14 10:50:35.226519 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.225820 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 14 10:50:35.227281 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.227257 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 14 10:50:35.227539 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.227512 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 14 10:50:35.227648 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.227601 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 14 10:50:35.227873 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.227749 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 14 10:50:35.229144 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.229076 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5r9hs63c49ru6\"" Sep 14 10:50:35.229383 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.229332 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 14 10:50:35.229632 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.229613 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 14 10:50:35.229936 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.229814 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 14 10:50:35.229936 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.229819 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 14 10:50:35.230235 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.229274 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 14 10:50:35.236800 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.232564 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 14 10:50:35.236800 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.232775 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lrckz\"" Sep 14 10:50:35.236800 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.235163 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 14 10:50:35.242038 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.242005 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 14 10:50:35.252329 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.252272 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.252524 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.252377 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c1186989-a7ab-41d3-a493-18c002acf360-config-out\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.252524 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.252410 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.252524 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.252452 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.252524 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.252500 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c1186989-a7ab-41d3-a493-18c002acf360-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.252745 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.252566 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-config\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.327038 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.327008 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-78c748dd77-fzgm4" Sep 14 10:50:35.354114 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.354077 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c1186989-a7ab-41d3-a493-18c002acf360-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.354301 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.354144 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-config\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.354301 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.354189 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c1186989-a7ab-41d3-a493-18c002acf360-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.354301 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.354228 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1186989-a7ab-41d3-a493-18c002acf360-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.354301 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.354288 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1186989-a7ab-41d3-a493-18c002acf360-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.354549 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.354330 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.354549 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.354374 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c1186989-a7ab-41d3-a493-18c002acf360-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.354549 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.354416 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.354549 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.354458 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.354549 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.354488 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1186989-a7ab-41d3-a493-18c002acf360-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.354549 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.354513 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c1186989-a7ab-41d3-a493-18c002acf360-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.354796 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.354551 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c1186989-a7ab-41d3-a493-18c002acf360-config-out\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.354796 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.354576 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.354796 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.354598 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.354796 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.354623 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1186989-a7ab-41d3-a493-18c002acf360-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.354796 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.354646 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-web-config\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.354796 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.354692 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.354796 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.354719 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z5d5\" (UniqueName: \"kubernetes.io/projected/c1186989-a7ab-41d3-a493-18c002acf360-kube-api-access-5z5d5\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.354796 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.354764 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c1186989-a7ab-41d3-a493-18c002acf360-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.354796 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.354787 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.358961 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.358767 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-config\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.358961 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.358917 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.359264 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.359227 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.360354 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.360314 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c1186989-a7ab-41d3-a493-18c002acf360-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.361120 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.361097 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c1186989-a7ab-41d3-a493-18c002acf360-config-out\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.361819 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.361795 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.429520 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.428700 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5799fbfd5d-272jz" event={"ID":"8d42234d-ca6b-4e18-af83-1c6c0022cb09","Type":"ContainerStarted","Data":"69b81209404157385f4ec28384284dad15ab3ebcac81aa5ca0cd708e77b11208"} Sep 14 10:50:35.429520 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.428763 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5799fbfd5d-272jz" event={"ID":"8d42234d-ca6b-4e18-af83-1c6c0022cb09","Type":"ContainerStarted","Data":"548558dc7c8545839a9006394fcda56e44db1554771a19a64c451247e714159b"} Sep 14 10:50:35.450598 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.450533 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5799fbfd5d-272jz" podStartSLOduration=1.450509765 podStartE2EDuration="1.450509765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:50:35.449446162 +0000 UTC m=+109.191848134" watchObservedRunningTime="2026-09-14 10:50:35.450509765 +0000 UTC m=+109.192911736" Sep 14 10:50:35.456286 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.456244 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.456286 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.456288 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1186989-a7ab-41d3-a493-18c002acf360-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.456948 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.456314 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-web-config\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.456948 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.456405 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z5d5\" (UniqueName: \"kubernetes.io/projected/c1186989-a7ab-41d3-a493-18c002acf360-kube-api-access-5z5d5\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.456948 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.456449 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.456948 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.456525 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c1186989-a7ab-41d3-a493-18c002acf360-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.456948 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.456576 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c1186989-a7ab-41d3-a493-18c002acf360-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.456948 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.456623 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1186989-a7ab-41d3-a493-18c002acf360-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.456948 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.456681 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1186989-a7ab-41d3-a493-18c002acf360-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.456948 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.456710 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c1186989-a7ab-41d3-a493-18c002acf360-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.456948 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.456741 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.456948 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.456806 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.456948 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.456838 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1186989-a7ab-41d3-a493-18c002acf360-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.456948 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.456862 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c1186989-a7ab-41d3-a493-18c002acf360-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.457542 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.457268 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c1186989-a7ab-41d3-a493-18c002acf360-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.457672 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.457644 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c1186989-a7ab-41d3-a493-18c002acf360-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.459153 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.458436 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1186989-a7ab-41d3-a493-18c002acf360-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.459153 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.458782 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1186989-a7ab-41d3-a493-18c002acf360-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.459313 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.459250 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1186989-a7ab-41d3-a493-18c002acf360-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.460024 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.459975 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c1186989-a7ab-41d3-a493-18c002acf360-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.460654 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.460634 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1186989-a7ab-41d3-a493-18c002acf360-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.460855 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.460800 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c1186989-a7ab-41d3-a493-18c002acf360-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.461083 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.460890 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.461840 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.461784 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.462197 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.462174 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-web-config\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.462740 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.462713 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.462820 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.462805 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.471503 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.471427 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z5d5\" (UniqueName: \"kubernetes.io/projected/c1186989-a7ab-41d3-a493-18c002acf360-kube-api-access-5z5d5\") pod \"prometheus-k8s-0\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.544547 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.544511 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:35.912771 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:35.912715 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 14 10:50:35.915643 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:50:35.915617 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1186989_a7ab_41d3_a493_18c002acf360.slice/crio-2f0a535e2c721e2263123395f2e9e3e7fc37c1036175aab3f220ec2cc919dcdb WatchSource:0}: Error finding container 2f0a535e2c721e2263123395f2e9e3e7fc37c1036175aab3f220ec2cc919dcdb: Status 404 returned error can't find the container with id 2f0a535e2c721e2263123395f2e9e3e7fc37c1036175aab3f220ec2cc919dcdb Sep 14 10:50:36.439827 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:36.439793 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84b587f84f-bbcq9" event={"ID":"0ae46dae-1601-4782-b36e-3f824d74c904","Type":"ContainerStarted","Data":"888484300e95c22b1a7b9d545d98cc45cb12787c9993d06175eef23d5ee63419"} Sep 14 10:50:36.440292 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:36.439907 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84b587f84f-bbcq9" event={"ID":"0ae46dae-1601-4782-b36e-3f824d74c904","Type":"ContainerStarted","Data":"c1156f6a38e9233ecd127a9a01b49b4710cd0a7745f0b3c3e2c631e5678da209"} Sep 14 10:50:36.440292 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:36.439939 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84b587f84f-bbcq9" event={"ID":"0ae46dae-1601-4782-b36e-3f824d74c904","Type":"ContainerStarted","Data":"dfc6fa0cb6539b6a519b5857aef1a4f3ef229fe718fa91faa02470c7e83b044b"} Sep 14 10:50:36.440292 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:36.440093 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-84b587f84f-bbcq9" Sep 14 10:50:36.442260 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:36.441786 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7d4578988b-8clwn" event={"ID":"191cacad-284e-4312-9ef5-ede80f847126","Type":"ContainerStarted","Data":"0a949076e34eb82fdc3f60f708083d5b831967d48529ce998c543c3cff9d663d"} Sep 14 10:50:36.443045 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:36.443022 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c1186989-a7ab-41d3-a493-18c002acf360","Type":"ContainerStarted","Data":"2f0a535e2c721e2263123395f2e9e3e7fc37c1036175aab3f220ec2cc919dcdb"} Sep 14 10:50:36.475671 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:36.475621 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-84b587f84f-bbcq9" podStartSLOduration=2.285941786 podStartE2EDuration="5.475604282s" totalImagesPullingTime="3.189662496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:50:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:50:36.473431837 +0000 UTC m=+110.215833816" watchObservedRunningTime="2026-09-14 10:50:36.475604282 +0000 UTC m=+110.218006251" Sep 14 10:50:36.494918 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:36.494868 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7d4578988b-8clwn" podStartSLOduration=2.906109728 podStartE2EDuration="4.494848873s" totalImagesPullingTime="1.588739145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:50:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:50:36.494439777 +0000 UTC m=+110.236841751" watchObservedRunningTime="2026-09-14 10:50:36.494848873 +0000 UTC m=+110.237250843" Sep 14 10:50:37.448654 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:37.448615 2581 generic.go:356] "Generic (PLEG): container finished" podID="c1186989-a7ab-41d3-a493-18c002acf360" containerID="91fb7a69bf7ea8b141b677868ee84a6e3234a225bdfb83abf0d6088b12c7dda9" exitCode=0 Sep 14 10:50:37.449029 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:37.448701 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c1186989-a7ab-41d3-a493-18c002acf360","Type":"ContainerDied","Data":"91fb7a69bf7ea8b141b677868ee84a6e3234a225bdfb83abf0d6088b12c7dda9"} Sep 14 10:50:38.395438 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:38.395405 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zp5bd" Sep 14 10:50:41.469590 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:41.469554 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c1186989-a7ab-41d3-a493-18c002acf360","Type":"ContainerStarted","Data":"f9a420104c3674ce52568ea522f22dbce0d5f29494a557955d56d873065a65b9"} Sep 14 10:50:41.469956 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:41.469650 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c1186989-a7ab-41d3-a493-18c002acf360","Type":"ContainerStarted","Data":"f3a4aabed274beaf711ff1aab4318c3d345e31710874f6b84aa7e074da9ad3ce"} Sep 14 10:50:41.469956 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:41.469677 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c1186989-a7ab-41d3-a493-18c002acf360","Type":"ContainerStarted","Data":"d91ff7ddc615b61d8363358d90a1cce12ce0b77355400e89f829c9466a206ebe"} Sep 14 10:50:41.469956 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:41.469700 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c1186989-a7ab-41d3-a493-18c002acf360","Type":"ContainerStarted","Data":"6b657c13974836c58ffc031dce2db91fcc787e497c48c7689ea2765f6aea41b6"} Sep 14 10:50:41.469956 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:41.469724 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c1186989-a7ab-41d3-a493-18c002acf360","Type":"ContainerStarted","Data":"253698bbec0d8718a82c9543c4f1046eaf2f9625d9d102e59bc8c6c1eeba9460"} Sep 14 10:50:41.469956 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:41.469746 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c1186989-a7ab-41d3-a493-18c002acf360","Type":"ContainerStarted","Data":"2b285eb46aab9b302ad90680b595ba86b581c15d488dbe14859e3b0c9a914e0b"} Sep 14 10:50:41.505627 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:41.505575 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.450976241 podStartE2EDuration="6.505556664s" totalImagesPullingTime="4.034636479s" totalInitContainerRuntime="19.943944ms" isStatefulPod=false podCreationTimestamp="2026-09-14 10:50:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:50:41.502960145 +0000 UTC m=+115.245362140" watchObservedRunningTime="2026-09-14 10:50:41.505556664 +0000 UTC m=+115.247958634" Sep 14 10:50:42.455394 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:42.455367 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-84b587f84f-bbcq9" Sep 14 10:50:44.498720 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:44.498683 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5799fbfd5d-272jz" Sep 14 10:50:44.499190 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:44.498773 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5799fbfd5d-272jz" Sep 14 10:50:44.504495 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:44.504470 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5799fbfd5d-272jz" Sep 14 10:50:45.489862 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:45.489833 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5799fbfd5d-272jz" Sep 14 10:50:45.540136 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:45.540101 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fdd647dcf-88ncj"] Sep 14 10:50:45.544839 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:45.544809 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:50:53.371893 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:53.371860 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7d4578988b-8clwn" Sep 14 10:50:53.372391 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:53.371924 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7d4578988b-8clwn" Sep 14 10:50:56.653690 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:56.653634 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04ff0710-e09c-48e9-9769-7ffce3645b66-metrics-certs\") pod \"network-metrics-daemon-29jrs\" (UID: \"04ff0710-e09c-48e9-9769-7ffce3645b66\") " pod="openshift-multus/network-metrics-daemon-29jrs" Sep 14 10:50:56.656187 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:56.656162 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04ff0710-e09c-48e9-9769-7ffce3645b66-metrics-certs\") pod \"network-metrics-daemon-29jrs\" (UID: \"04ff0710-e09c-48e9-9769-7ffce3645b66\") " pod="openshift-multus/network-metrics-daemon-29jrs" Sep 14 10:50:56.751100 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:56.751070 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bckh8\"" Sep 14 10:50:56.758675 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:56.758648 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-29jrs" Sep 14 10:50:57.051167 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:57.051140 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-29jrs"] Sep 14 10:50:57.052943 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:50:57.052920 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04ff0710_e09c_48e9_9769_7ffce3645b66.slice/crio-7d7c2304da4039720434895b8cccfb25ccd4e3fa557ecc9cf566671175ce9a03 WatchSource:0}: Error finding container 7d7c2304da4039720434895b8cccfb25ccd4e3fa557ecc9cf566671175ce9a03: Status 404 returned error can't find the container with id 7d7c2304da4039720434895b8cccfb25ccd4e3fa557ecc9cf566671175ce9a03 Sep 14 10:50:57.533839 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:57.533807 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-29jrs" event={"ID":"04ff0710-e09c-48e9-9769-7ffce3645b66","Type":"ContainerStarted","Data":"7d7c2304da4039720434895b8cccfb25ccd4e3fa557ecc9cf566671175ce9a03"} Sep 14 10:50:59.542722 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:59.542675 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-29jrs" event={"ID":"04ff0710-e09c-48e9-9769-7ffce3645b66","Type":"ContainerStarted","Data":"8f136634959d4c33c02f7e8044aaae309543be5d29d94f9a70194496db091ab8"} Sep 14 10:50:59.543114 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:59.542746 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-29jrs" event={"ID":"04ff0710-e09c-48e9-9769-7ffce3645b66","Type":"ContainerStarted","Data":"bc36b44a568931a9bc9a6c409878a5ebb6da9d134b4147dac262c23dd85a5321"} Sep 14 10:50:59.680117 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:50:59.680025 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-29jrs" podStartSLOduration=131.27066427 podStartE2EDuration="2m12.680009038s" totalImagesPullingTime="1.409344768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:48:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:50:59.679605594 +0000 UTC m=+133.422007563" watchObservedRunningTime="2026-09-14 10:50:59.680009038 +0000 UTC m=+133.422411009" Sep 14 10:51:03.561693 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:03.561656 2581 generic.go:356] "Generic (PLEG): container finished" podID="e2424929-e528-4e66-bba3-04fe380324af" containerID="dd8e321eede23339c1f6bf93195914f66e4a39195bea2cc3a1cb727d962a994d" exitCode=0 Sep 14 10:51:03.561693 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:03.561698 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-9tvnj" event={"ID":"e2424929-e528-4e66-bba3-04fe380324af","Type":"ContainerDied","Data":"dd8e321eede23339c1f6bf93195914f66e4a39195bea2cc3a1cb727d962a994d"} Sep 14 10:51:03.562252 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:03.562109 2581 scope.go:117] "RemoveContainer" containerID="dd8e321eede23339c1f6bf93195914f66e4a39195bea2cc3a1cb727d962a994d" Sep 14 10:51:04.568687 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:04.568654 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-9tvnj" event={"ID":"e2424929-e528-4e66-bba3-04fe380324af","Type":"ContainerStarted","Data":"681bc8e783ea126ce49fb996e8017f0117ebd36f7cfd9a0922207c41bc6c82c9"} Sep 14 10:51:10.561207 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:10.561148 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7fdd647dcf-88ncj" podUID="bac0f1cd-ab44-4309-8587-f6aca897c7f3" containerName="console" containerID="cri-o://9e691c9ed21a86961aa672375dc1af124103118dd6a675d1474760b2404146a1" gracePeriod=15 Sep 14 10:51:10.810680 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:10.810655 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fdd647dcf-88ncj" Sep 14 10:51:10.989248 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:10.989156 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bac0f1cd-ab44-4309-8587-f6aca897c7f3-console-serving-cert\") pod \"bac0f1cd-ab44-4309-8587-f6aca897c7f3\" (UID: \"bac0f1cd-ab44-4309-8587-f6aca897c7f3\") " Sep 14 10:51:10.989248 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:10.989199 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bac0f1cd-ab44-4309-8587-f6aca897c7f3-oauth-serving-cert\") pod \"bac0f1cd-ab44-4309-8587-f6aca897c7f3\" (UID: \"bac0f1cd-ab44-4309-8587-f6aca897c7f3\") " Sep 14 10:51:10.989248 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:10.989225 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bac0f1cd-ab44-4309-8587-f6aca897c7f3-console-config\") pod \"bac0f1cd-ab44-4309-8587-f6aca897c7f3\" (UID: \"bac0f1cd-ab44-4309-8587-f6aca897c7f3\") " Sep 14 10:51:10.989562 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:10.989265 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bac0f1cd-ab44-4309-8587-f6aca897c7f3-service-ca\") pod \"bac0f1cd-ab44-4309-8587-f6aca897c7f3\" (UID: \"bac0f1cd-ab44-4309-8587-f6aca897c7f3\") " Sep 14 10:51:10.989562 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:10.989407 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbztg\" (UniqueName: \"kubernetes.io/projected/bac0f1cd-ab44-4309-8587-f6aca897c7f3-kube-api-access-zbztg\") pod \"bac0f1cd-ab44-4309-8587-f6aca897c7f3\" (UID: \"bac0f1cd-ab44-4309-8587-f6aca897c7f3\") " Sep 14 10:51:10.989562 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:10.989464 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bac0f1cd-ab44-4309-8587-f6aca897c7f3-console-oauth-config\") pod \"bac0f1cd-ab44-4309-8587-f6aca897c7f3\" (UID: \"bac0f1cd-ab44-4309-8587-f6aca897c7f3\") " Sep 14 10:51:10.989705 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:10.989613 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bac0f1cd-ab44-4309-8587-f6aca897c7f3-service-ca" (OuterVolumeSpecName: "service-ca") pod "bac0f1cd-ab44-4309-8587-f6aca897c7f3" (UID: "bac0f1cd-ab44-4309-8587-f6aca897c7f3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:51:10.989705 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:10.989654 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bac0f1cd-ab44-4309-8587-f6aca897c7f3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bac0f1cd-ab44-4309-8587-f6aca897c7f3" (UID: "bac0f1cd-ab44-4309-8587-f6aca897c7f3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:51:10.989796 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:10.989743 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bac0f1cd-ab44-4309-8587-f6aca897c7f3-console-config" (OuterVolumeSpecName: "console-config") pod "bac0f1cd-ab44-4309-8587-f6aca897c7f3" (UID: "bac0f1cd-ab44-4309-8587-f6aca897c7f3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:51:10.989796 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:10.989761 2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bac0f1cd-ab44-4309-8587-f6aca897c7f3-oauth-serving-cert\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:51:10.989796 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:10.989782 2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bac0f1cd-ab44-4309-8587-f6aca897c7f3-service-ca\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:51:10.991720 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:10.991693 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bac0f1cd-ab44-4309-8587-f6aca897c7f3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bac0f1cd-ab44-4309-8587-f6aca897c7f3" (UID: "bac0f1cd-ab44-4309-8587-f6aca897c7f3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 10:51:10.991720 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:10.991704 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bac0f1cd-ab44-4309-8587-f6aca897c7f3-kube-api-access-zbztg" (OuterVolumeSpecName: "kube-api-access-zbztg") pod "bac0f1cd-ab44-4309-8587-f6aca897c7f3" (UID: "bac0f1cd-ab44-4309-8587-f6aca897c7f3"). InnerVolumeSpecName "kube-api-access-zbztg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:51:10.991861 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:10.991713 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bac0f1cd-ab44-4309-8587-f6aca897c7f3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bac0f1cd-ab44-4309-8587-f6aca897c7f3" (UID: "bac0f1cd-ab44-4309-8587-f6aca897c7f3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 10:51:11.090217 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:11.090178 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbztg\" (UniqueName: \"kubernetes.io/projected/bac0f1cd-ab44-4309-8587-f6aca897c7f3-kube-api-access-zbztg\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:51:11.090217 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:11.090210 2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bac0f1cd-ab44-4309-8587-f6aca897c7f3-console-oauth-config\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:51:11.090217 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:11.090219 2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bac0f1cd-ab44-4309-8587-f6aca897c7f3-console-serving-cert\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:51:11.090217 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:11.090228 2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bac0f1cd-ab44-4309-8587-f6aca897c7f3-console-config\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:51:11.597806 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:11.597767 2581 generic.go:356] "Generic (PLEG): container finished" podID="bac0f1cd-ab44-4309-8587-f6aca897c7f3" containerID="9e691c9ed21a86961aa672375dc1af124103118dd6a675d1474760b2404146a1" exitCode=0 Sep 14 10:51:11.597806 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:11.597808 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fdd647dcf-88ncj" Sep 14 10:51:11.598283 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:11.597848 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fdd647dcf-88ncj" event={"ID":"bac0f1cd-ab44-4309-8587-f6aca897c7f3","Type":"ContainerDied","Data":"9e691c9ed21a86961aa672375dc1af124103118dd6a675d1474760b2404146a1"} Sep 14 10:51:11.598283 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:11.597899 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fdd647dcf-88ncj" event={"ID":"bac0f1cd-ab44-4309-8587-f6aca897c7f3","Type":"ContainerDied","Data":"2c3d333f418a2f34de8d96f7b33dd8d10bf698be614136f12822e242388f5bb1"} Sep 14 10:51:11.598283 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:11.597920 2581 scope.go:117] "RemoveContainer" containerID="9e691c9ed21a86961aa672375dc1af124103118dd6a675d1474760b2404146a1" Sep 14 10:51:11.610619 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:11.610600 2581 scope.go:117] "RemoveContainer" containerID="9e691c9ed21a86961aa672375dc1af124103118dd6a675d1474760b2404146a1" Sep 14 10:51:11.610954 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:51:11.610933 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e691c9ed21a86961aa672375dc1af124103118dd6a675d1474760b2404146a1\": container with ID starting with 9e691c9ed21a86961aa672375dc1af124103118dd6a675d1474760b2404146a1 not found: ID does not exist" containerID="9e691c9ed21a86961aa672375dc1af124103118dd6a675d1474760b2404146a1" Sep 14 10:51:11.611006 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:11.610963 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e691c9ed21a86961aa672375dc1af124103118dd6a675d1474760b2404146a1"} err="failed to get container status \"9e691c9ed21a86961aa672375dc1af124103118dd6a675d1474760b2404146a1\": rpc error: code = NotFound desc = could not find container \"9e691c9ed21a86961aa672375dc1af124103118dd6a675d1474760b2404146a1\": container with ID starting with 9e691c9ed21a86961aa672375dc1af124103118dd6a675d1474760b2404146a1 not found: ID does not exist" Sep 14 10:51:11.634066 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:11.634026 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fdd647dcf-88ncj"] Sep 14 10:51:11.636397 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:11.636371 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7fdd647dcf-88ncj"] Sep 14 10:51:12.832298 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:12.832261 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bac0f1cd-ab44-4309-8587-f6aca897c7f3" path="/var/lib/kubelet/pods/bac0f1cd-ab44-4309-8587-f6aca897c7f3/volumes" Sep 14 10:51:13.376630 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:13.376602 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7d4578988b-8clwn" Sep 14 10:51:13.380430 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:13.380405 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7d4578988b-8clwn" Sep 14 10:51:13.608463 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:13.608426 2581 generic.go:356] "Generic (PLEG): container finished" podID="0cedfa94-2214-4710-86cb-448821993687" containerID="d970e3e67c1d7ad067f3f777a882da071a5ef58cf489e7dd12010e025df44863" exitCode=0 Sep 14 10:51:13.608632 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:13.608502 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-g76j6" event={"ID":"0cedfa94-2214-4710-86cb-448821993687","Type":"ContainerDied","Data":"d970e3e67c1d7ad067f3f777a882da071a5ef58cf489e7dd12010e025df44863"} Sep 14 10:51:13.608939 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:13.608924 2581 scope.go:117] "RemoveContainer" containerID="d970e3e67c1d7ad067f3f777a882da071a5ef58cf489e7dd12010e025df44863" Sep 14 10:51:14.614659 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:14.614627 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-g76j6" event={"ID":"0cedfa94-2214-4710-86cb-448821993687","Type":"ContainerStarted","Data":"c6338fc441d55b9bdcb0f4afcb29b37248b815f427106f3eb7ee251726bd3fa1"} Sep 14 10:51:35.545708 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:35.545670 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:35.564738 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:35.564709 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:35.713661 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:35.713633 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:53.631154 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.631123 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 14 10:51:53.632063 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.632029 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c1186989-a7ab-41d3-a493-18c002acf360" containerName="prometheus" containerID="cri-o://2b285eb46aab9b302ad90680b595ba86b581c15d488dbe14859e3b0c9a914e0b" gracePeriod=600 Sep 14 10:51:53.632175 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.632120 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c1186989-a7ab-41d3-a493-18c002acf360" containerName="thanos-sidecar" containerID="cri-o://6b657c13974836c58ffc031dce2db91fcc787e497c48c7689ea2765f6aea41b6" gracePeriod=600 Sep 14 10:51:53.632242 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.632044 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c1186989-a7ab-41d3-a493-18c002acf360" containerName="kube-rbac-proxy-thanos" containerID="cri-o://f9a420104c3674ce52568ea522f22dbce0d5f29494a557955d56d873065a65b9" gracePeriod=600 Sep 14 10:51:53.632358 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.632098 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c1186989-a7ab-41d3-a493-18c002acf360" containerName="kube-rbac-proxy" containerID="cri-o://f3a4aabed274beaf711ff1aab4318c3d345e31710874f6b84aa7e074da9ad3ce" gracePeriod=600 Sep 14 10:51:53.632455 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.632100 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c1186989-a7ab-41d3-a493-18c002acf360" containerName="kube-rbac-proxy-web" containerID="cri-o://d91ff7ddc615b61d8363358d90a1cce12ce0b77355400e89f829c9466a206ebe" gracePeriod=600 Sep 14 10:51:53.632523 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.632096 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c1186989-a7ab-41d3-a493-18c002acf360" containerName="config-reloader" containerID="cri-o://253698bbec0d8718a82c9543c4f1046eaf2f9625d9d102e59bc8c6c1eeba9460" gracePeriod=600 Sep 14 10:51:53.773102 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.773070 2581 generic.go:356] "Generic (PLEG): container finished" podID="c1186989-a7ab-41d3-a493-18c002acf360" containerID="f9a420104c3674ce52568ea522f22dbce0d5f29494a557955d56d873065a65b9" exitCode=0 Sep 14 10:51:53.773102 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.773096 2581 generic.go:356] "Generic (PLEG): container finished" podID="c1186989-a7ab-41d3-a493-18c002acf360" containerID="f3a4aabed274beaf711ff1aab4318c3d345e31710874f6b84aa7e074da9ad3ce" exitCode=0 Sep 14 10:51:53.773102 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.773103 2581 generic.go:356] "Generic (PLEG): container finished" podID="c1186989-a7ab-41d3-a493-18c002acf360" containerID="d91ff7ddc615b61d8363358d90a1cce12ce0b77355400e89f829c9466a206ebe" exitCode=0 Sep 14 10:51:53.773102 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.773111 2581 generic.go:356] "Generic (PLEG): container finished" podID="c1186989-a7ab-41d3-a493-18c002acf360" containerID="6b657c13974836c58ffc031dce2db91fcc787e497c48c7689ea2765f6aea41b6" exitCode=0 Sep 14 10:51:53.773350 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.773116 2581 generic.go:356] "Generic (PLEG): container finished" podID="c1186989-a7ab-41d3-a493-18c002acf360" containerID="253698bbec0d8718a82c9543c4f1046eaf2f9625d9d102e59bc8c6c1eeba9460" exitCode=0 Sep 14 10:51:53.773350 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.773123 2581 generic.go:356] "Generic (PLEG): container finished" podID="c1186989-a7ab-41d3-a493-18c002acf360" containerID="2b285eb46aab9b302ad90680b595ba86b581c15d488dbe14859e3b0c9a914e0b" exitCode=0 Sep 14 10:51:53.773350 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.773130 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c1186989-a7ab-41d3-a493-18c002acf360","Type":"ContainerDied","Data":"f9a420104c3674ce52568ea522f22dbce0d5f29494a557955d56d873065a65b9"} Sep 14 10:51:53.773350 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.773239 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c1186989-a7ab-41d3-a493-18c002acf360","Type":"ContainerDied","Data":"f3a4aabed274beaf711ff1aab4318c3d345e31710874f6b84aa7e074da9ad3ce"} Sep 14 10:51:53.773350 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.773292 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c1186989-a7ab-41d3-a493-18c002acf360","Type":"ContainerDied","Data":"d91ff7ddc615b61d8363358d90a1cce12ce0b77355400e89f829c9466a206ebe"} Sep 14 10:51:53.773506 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.773332 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c1186989-a7ab-41d3-a493-18c002acf360","Type":"ContainerDied","Data":"6b657c13974836c58ffc031dce2db91fcc787e497c48c7689ea2765f6aea41b6"} Sep 14 10:51:53.773506 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.773398 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c1186989-a7ab-41d3-a493-18c002acf360","Type":"ContainerDied","Data":"253698bbec0d8718a82c9543c4f1046eaf2f9625d9d102e59bc8c6c1eeba9460"} Sep 14 10:51:53.773506 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.773437 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c1186989-a7ab-41d3-a493-18c002acf360","Type":"ContainerDied","Data":"2b285eb46aab9b302ad90680b595ba86b581c15d488dbe14859e3b0c9a914e0b"} Sep 14 10:51:53.887552 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.887491 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:53.960424 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.960390 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-thanos-prometheus-http-client-file\") pod \"c1186989-a7ab-41d3-a493-18c002acf360\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " Sep 14 10:51:53.960424 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.960426 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1186989-a7ab-41d3-a493-18c002acf360-prometheus-trusted-ca-bundle\") pod \"c1186989-a7ab-41d3-a493-18c002acf360\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " Sep 14 10:51:53.960687 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.960462 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1186989-a7ab-41d3-a493-18c002acf360-configmap-kubelet-serving-ca-bundle\") pod \"c1186989-a7ab-41d3-a493-18c002acf360\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " Sep 14 10:51:53.960687 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.960586 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-web-config\") pod \"c1186989-a7ab-41d3-a493-18c002acf360\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " Sep 14 10:51:53.960687 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.960640 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c1186989-a7ab-41d3-a493-18c002acf360-tls-assets\") pod \"c1186989-a7ab-41d3-a493-18c002acf360\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " Sep 14 10:51:53.960687 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.960667 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1186989-a7ab-41d3-a493-18c002acf360-configmap-metrics-client-ca\") pod \"c1186989-a7ab-41d3-a493-18c002acf360\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " Sep 14 10:51:53.960879 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.960695 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-config\") pod \"c1186989-a7ab-41d3-a493-18c002acf360\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " Sep 14 10:51:53.960879 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.960721 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c1186989-a7ab-41d3-a493-18c002acf360-prometheus-k8s-rulefiles-0\") pod \"c1186989-a7ab-41d3-a493-18c002acf360\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " Sep 14 10:51:53.960879 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.960750 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c1186989-a7ab-41d3-a493-18c002acf360-config-out\") pod \"c1186989-a7ab-41d3-a493-18c002acf360\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " Sep 14 10:51:53.960879 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.960785 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1186989-a7ab-41d3-a493-18c002acf360-configmap-serving-certs-ca-bundle\") pod \"c1186989-a7ab-41d3-a493-18c002acf360\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " Sep 14 10:51:53.960879 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.960824 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1186989-a7ab-41d3-a493-18c002acf360-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "c1186989-a7ab-41d3-a493-18c002acf360" (UID: "c1186989-a7ab-41d3-a493-18c002acf360"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:51:53.960879 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.960836 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-secret-kube-rbac-proxy\") pod \"c1186989-a7ab-41d3-a493-18c002acf360\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " Sep 14 10:51:53.960879 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.960860 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1186989-a7ab-41d3-a493-18c002acf360-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "c1186989-a7ab-41d3-a493-18c002acf360" (UID: "c1186989-a7ab-41d3-a493-18c002acf360"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:51:53.961208 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.960884 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"c1186989-a7ab-41d3-a493-18c002acf360\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " Sep 14 10:51:53.961208 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.960925 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-secret-metrics-client-certs\") pod \"c1186989-a7ab-41d3-a493-18c002acf360\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " Sep 14 10:51:53.961208 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.960953 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c1186989-a7ab-41d3-a493-18c002acf360-prometheus-k8s-rulefiles-1\") pod \"c1186989-a7ab-41d3-a493-18c002acf360\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " Sep 14 10:51:53.961208 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.960983 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-secret-grpc-tls\") pod \"c1186989-a7ab-41d3-a493-18c002acf360\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " Sep 14 10:51:53.961417 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.961320 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1186989-a7ab-41d3-a493-18c002acf360-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "c1186989-a7ab-41d3-a493-18c002acf360" (UID: "c1186989-a7ab-41d3-a493-18c002acf360"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:51:53.961895 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.961765 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1186989-a7ab-41d3-a493-18c002acf360-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "c1186989-a7ab-41d3-a493-18c002acf360" (UID: "c1186989-a7ab-41d3-a493-18c002acf360"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:51:53.961895 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.961818 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-secret-prometheus-k8s-tls\") pod \"c1186989-a7ab-41d3-a493-18c002acf360\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " Sep 14 10:51:53.961895 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.961852 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"c1186989-a7ab-41d3-a493-18c002acf360\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " Sep 14 10:51:53.961895 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.961878 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5z5d5\" (UniqueName: \"kubernetes.io/projected/c1186989-a7ab-41d3-a493-18c002acf360-kube-api-access-5z5d5\") pod \"c1186989-a7ab-41d3-a493-18c002acf360\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " Sep 14 10:51:53.962156 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.961941 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c1186989-a7ab-41d3-a493-18c002acf360-prometheus-k8s-db\") pod \"c1186989-a7ab-41d3-a493-18c002acf360\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " Sep 14 10:51:53.962156 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.961984 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c1186989-a7ab-41d3-a493-18c002acf360-prometheus-k8s-rulefiles-2\") pod \"c1186989-a7ab-41d3-a493-18c002acf360\" (UID: \"c1186989-a7ab-41d3-a493-18c002acf360\") " Sep 14 10:51:53.962264 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.962247 2581 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1186989-a7ab-41d3-a493-18c002acf360-configmap-metrics-client-ca\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:51:53.962318 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.962265 2581 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1186989-a7ab-41d3-a493-18c002acf360-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:51:53.962318 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.962281 2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1186989-a7ab-41d3-a493-18c002acf360-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:51:53.962318 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.962296 2581 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1186989-a7ab-41d3-a493-18c002acf360-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:51:53.962989 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.962704 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1186989-a7ab-41d3-a493-18c002acf360-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "c1186989-a7ab-41d3-a493-18c002acf360" (UID: "c1186989-a7ab-41d3-a493-18c002acf360"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:51:53.962989 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.962732 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1186989-a7ab-41d3-a493-18c002acf360-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "c1186989-a7ab-41d3-a493-18c002acf360" (UID: "c1186989-a7ab-41d3-a493-18c002acf360"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:51:53.964192 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.963905 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "c1186989-a7ab-41d3-a493-18c002acf360" (UID: "c1186989-a7ab-41d3-a493-18c002acf360"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 10:51:53.964619 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.964590 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1186989-a7ab-41d3-a493-18c002acf360-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "c1186989-a7ab-41d3-a493-18c002acf360" (UID: "c1186989-a7ab-41d3-a493-18c002acf360"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 10:51:53.964730 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.964654 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "c1186989-a7ab-41d3-a493-18c002acf360" (UID: "c1186989-a7ab-41d3-a493-18c002acf360"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 10:51:53.964979 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.964957 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1186989-a7ab-41d3-a493-18c002acf360-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "c1186989-a7ab-41d3-a493-18c002acf360" (UID: "c1186989-a7ab-41d3-a493-18c002acf360"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:51:53.965578 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.965550 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "c1186989-a7ab-41d3-a493-18c002acf360" (UID: "c1186989-a7ab-41d3-a493-18c002acf360"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 10:51:53.966020 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.965987 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1186989-a7ab-41d3-a493-18c002acf360-config-out" (OuterVolumeSpecName: "config-out") pod "c1186989-a7ab-41d3-a493-18c002acf360" (UID: "c1186989-a7ab-41d3-a493-18c002acf360"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 10:51:53.966566 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.966543 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1186989-a7ab-41d3-a493-18c002acf360-kube-api-access-5z5d5" (OuterVolumeSpecName: "kube-api-access-5z5d5") pod "c1186989-a7ab-41d3-a493-18c002acf360" (UID: "c1186989-a7ab-41d3-a493-18c002acf360"). InnerVolumeSpecName "kube-api-access-5z5d5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:51:53.966924 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.966895 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "c1186989-a7ab-41d3-a493-18c002acf360" (UID: "c1186989-a7ab-41d3-a493-18c002acf360"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 10:51:53.967016 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.966931 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "c1186989-a7ab-41d3-a493-18c002acf360" (UID: "c1186989-a7ab-41d3-a493-18c002acf360"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 10:51:53.967016 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.966934 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "c1186989-a7ab-41d3-a493-18c002acf360" (UID: "c1186989-a7ab-41d3-a493-18c002acf360"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 10:51:53.967454 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.967325 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1186989-a7ab-41d3-a493-18c002acf360-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "c1186989-a7ab-41d3-a493-18c002acf360" (UID: "c1186989-a7ab-41d3-a493-18c002acf360"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:51:53.967454 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.967449 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-config" (OuterVolumeSpecName: "config") pod "c1186989-a7ab-41d3-a493-18c002acf360" (UID: "c1186989-a7ab-41d3-a493-18c002acf360"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 10:51:53.967914 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.967890 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "c1186989-a7ab-41d3-a493-18c002acf360" (UID: "c1186989-a7ab-41d3-a493-18c002acf360"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 10:51:53.981838 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:53.981786 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-web-config" (OuterVolumeSpecName: "web-config") pod "c1186989-a7ab-41d3-a493-18c002acf360" (UID: "c1186989-a7ab-41d3-a493-18c002acf360"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 10:51:54.063378 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.063316 2581 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c1186989-a7ab-41d3-a493-18c002acf360-tls-assets\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:51:54.063378 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.063371 2581 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-config\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:51:54.063378 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.063382 2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c1186989-a7ab-41d3-a493-18c002acf360-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:51:54.063624 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.063392 2581 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c1186989-a7ab-41d3-a493-18c002acf360-config-out\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:51:54.063624 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.063402 2581 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-secret-kube-rbac-proxy\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:51:54.063624 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.063413 2581 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:51:54.063624 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.063422 2581 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-secret-metrics-client-certs\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:51:54.063624 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.063430 2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c1186989-a7ab-41d3-a493-18c002acf360-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:51:54.063624 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.063438 2581 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-secret-grpc-tls\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:51:54.063624 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.063446 2581 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:51:54.063624 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.063453 2581 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:51:54.063624 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.063461 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z5d5\" (UniqueName: \"kubernetes.io/projected/c1186989-a7ab-41d3-a493-18c002acf360-kube-api-access-5z5d5\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:51:54.063624 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.063469 2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c1186989-a7ab-41d3-a493-18c002acf360-prometheus-k8s-db\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:51:54.063624 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.063478 2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c1186989-a7ab-41d3-a493-18c002acf360-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:51:54.063624 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.063486 2581 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:51:54.063624 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.063496 2581 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c1186989-a7ab-41d3-a493-18c002acf360-web-config\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:51:54.779964 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.779925 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c1186989-a7ab-41d3-a493-18c002acf360","Type":"ContainerDied","Data":"2f0a535e2c721e2263123395f2e9e3e7fc37c1036175aab3f220ec2cc919dcdb"} Sep 14 10:51:54.780374 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.780019 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:54.780374 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.780032 2581 scope.go:117] "RemoveContainer" containerID="f9a420104c3674ce52568ea522f22dbce0d5f29494a557955d56d873065a65b9" Sep 14 10:51:54.788621 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.788596 2581 scope.go:117] "RemoveContainer" containerID="f3a4aabed274beaf711ff1aab4318c3d345e31710874f6b84aa7e074da9ad3ce" Sep 14 10:51:54.795720 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.795703 2581 scope.go:117] "RemoveContainer" containerID="d91ff7ddc615b61d8363358d90a1cce12ce0b77355400e89f829c9466a206ebe" Sep 14 10:51:54.804479 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.804448 2581 scope.go:117] "RemoveContainer" containerID="6b657c13974836c58ffc031dce2db91fcc787e497c48c7689ea2765f6aea41b6" Sep 14 10:51:54.808822 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.808800 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 14 10:51:54.811583 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.811560 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 14 10:51:54.812473 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.812454 2581 scope.go:117] "RemoveContainer" containerID="253698bbec0d8718a82c9543c4f1046eaf2f9625d9d102e59bc8c6c1eeba9460" Sep 14 10:51:54.819327 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.819297 2581 scope.go:117] "RemoveContainer" containerID="2b285eb46aab9b302ad90680b595ba86b581c15d488dbe14859e3b0c9a914e0b" Sep 14 10:51:54.826638 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.826620 2581 scope.go:117] "RemoveContainer" containerID="91fb7a69bf7ea8b141b677868ee84a6e3234a225bdfb83abf0d6088b12c7dda9" Sep 14 10:51:54.830223 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.830203 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1186989-a7ab-41d3-a493-18c002acf360" path="/var/lib/kubelet/pods/c1186989-a7ab-41d3-a493-18c002acf360/volumes" Sep 14 10:51:54.844739 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.844718 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 14 10:51:54.844987 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.844976 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1186989-a7ab-41d3-a493-18c002acf360" containerName="kube-rbac-proxy-web" Sep 14 10:51:54.845036 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.844988 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1186989-a7ab-41d3-a493-18c002acf360" containerName="kube-rbac-proxy-web" Sep 14 10:51:54.845036 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.844999 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1186989-a7ab-41d3-a493-18c002acf360" containerName="init-config-reloader" Sep 14 10:51:54.845036 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.845006 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1186989-a7ab-41d3-a493-18c002acf360" containerName="init-config-reloader" Sep 14 10:51:54.845036 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.845014 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bac0f1cd-ab44-4309-8587-f6aca897c7f3" containerName="console" Sep 14 10:51:54.845036 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.845019 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bac0f1cd-ab44-4309-8587-f6aca897c7f3" containerName="console" Sep 14 10:51:54.845036 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.845030 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1186989-a7ab-41d3-a493-18c002acf360" containerName="config-reloader" Sep 14 10:51:54.845036 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.845036 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1186989-a7ab-41d3-a493-18c002acf360" containerName="config-reloader" Sep 14 10:51:54.845249 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.845043 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1186989-a7ab-41d3-a493-18c002acf360" containerName="kube-rbac-proxy" Sep 14 10:51:54.845249 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.845049 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1186989-a7ab-41d3-a493-18c002acf360" containerName="kube-rbac-proxy" Sep 14 10:51:54.845249 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.845057 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1186989-a7ab-41d3-a493-18c002acf360" containerName="prometheus" Sep 14 10:51:54.845249 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.845061 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1186989-a7ab-41d3-a493-18c002acf360" containerName="prometheus" Sep 14 10:51:54.845249 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.845066 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1186989-a7ab-41d3-a493-18c002acf360" containerName="thanos-sidecar" Sep 14 10:51:54.845249 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.845070 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1186989-a7ab-41d3-a493-18c002acf360" containerName="thanos-sidecar" Sep 14 10:51:54.845249 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.845076 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1186989-a7ab-41d3-a493-18c002acf360" containerName="kube-rbac-proxy-thanos" Sep 14 10:51:54.845249 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.845081 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1186989-a7ab-41d3-a493-18c002acf360" containerName="kube-rbac-proxy-thanos" Sep 14 10:51:54.845249 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.845130 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1186989-a7ab-41d3-a493-18c002acf360" containerName="thanos-sidecar" Sep 14 10:51:54.845249 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.845139 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1186989-a7ab-41d3-a493-18c002acf360" containerName="kube-rbac-proxy" Sep 14 10:51:54.845249 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.845144 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bac0f1cd-ab44-4309-8587-f6aca897c7f3" containerName="console" Sep 14 10:51:54.845249 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.845150 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1186989-a7ab-41d3-a493-18c002acf360" containerName="prometheus" Sep 14 10:51:54.845249 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.845155 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1186989-a7ab-41d3-a493-18c002acf360" containerName="kube-rbac-proxy-web" Sep 14 10:51:54.845249 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.845162 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1186989-a7ab-41d3-a493-18c002acf360" containerName="kube-rbac-proxy-thanos" Sep 14 10:51:54.845249 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.845168 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1186989-a7ab-41d3-a493-18c002acf360" containerName="config-reloader" Sep 14 10:51:54.850461 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.850442 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:54.855662 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.855641 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 14 10:51:54.855829 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.855811 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 14 10:51:54.855922 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.855875 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 14 10:51:54.855922 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.855891 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 14 10:51:54.856035 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.855964 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 14 10:51:54.856091 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.856061 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 14 10:51:54.856141 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.856091 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 14 10:51:54.856196 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.856140 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 14 10:51:54.856196 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.856183 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 14 10:51:54.856454 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.856435 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 14 10:51:54.856654 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.856529 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lrckz\"" Sep 14 10:51:54.857355 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.857319 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 14 10:51:54.857445 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.857430 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5r9hs63c49ru6\"" Sep 14 10:51:54.858367 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.858329 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 14 10:51:54.860784 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.860651 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 14 10:51:54.864497 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.864480 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 14 10:51:54.871306 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.871283 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 14 10:51:54.970698 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.970603 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dedac73-f046-46df-88df-536b7db9ed6b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:54.970698 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.970642 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8dedac73-f046-46df-88df-536b7db9ed6b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:54.970698 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.970660 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8dedac73-f046-46df-88df-536b7db9ed6b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:54.970913 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.970740 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js2bd\" (UniqueName: \"kubernetes.io/projected/8dedac73-f046-46df-88df-536b7db9ed6b-kube-api-access-js2bd\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:54.970913 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.970793 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8dedac73-f046-46df-88df-536b7db9ed6b-config-out\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:54.970913 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.970842 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8dedac73-f046-46df-88df-536b7db9ed6b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:54.970913 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.970881 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8dedac73-f046-46df-88df-536b7db9ed6b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:54.971029 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.970986 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8dedac73-f046-46df-88df-536b7db9ed6b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:54.971074 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.971048 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dedac73-f046-46df-88df-536b7db9ed6b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:54.971106 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.971093 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8dedac73-f046-46df-88df-536b7db9ed6b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:54.971140 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.971125 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8dedac73-f046-46df-88df-536b7db9ed6b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:54.971175 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.971143 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8dedac73-f046-46df-88df-536b7db9ed6b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:54.971208 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.971169 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8dedac73-f046-46df-88df-536b7db9ed6b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:54.971271 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.971258 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8dedac73-f046-46df-88df-536b7db9ed6b-config\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:54.971305 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.971279 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8dedac73-f046-46df-88df-536b7db9ed6b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:54.971305 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.971295 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8dedac73-f046-46df-88df-536b7db9ed6b-web-config\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:54.971448 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.971437 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8dedac73-f046-46df-88df-536b7db9ed6b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:54.971483 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.971458 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8dedac73-f046-46df-88df-536b7db9ed6b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:54.971483 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.971475 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dedac73-f046-46df-88df-536b7db9ed6b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:54.971545 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:54.971497 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8dedac73-f046-46df-88df-536b7db9ed6b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:55.072524 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.072480 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8dedac73-f046-46df-88df-536b7db9ed6b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:55.072524 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.072525 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8dedac73-f046-46df-88df-536b7db9ed6b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:55.072751 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.072550 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dedac73-f046-46df-88df-536b7db9ed6b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:55.072751 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.072570 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8dedac73-f046-46df-88df-536b7db9ed6b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:55.072751 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.072596 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dedac73-f046-46df-88df-536b7db9ed6b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:55.072751 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.072622 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8dedac73-f046-46df-88df-536b7db9ed6b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:55.072751 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.072644 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8dedac73-f046-46df-88df-536b7db9ed6b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:55.072751 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.072673 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js2bd\" (UniqueName: \"kubernetes.io/projected/8dedac73-f046-46df-88df-536b7db9ed6b-kube-api-access-js2bd\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:55.072751 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.072698 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8dedac73-f046-46df-88df-536b7db9ed6b-config-out\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:55.072751 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.072726 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8dedac73-f046-46df-88df-536b7db9ed6b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:55.072751 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.072750 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8dedac73-f046-46df-88df-536b7db9ed6b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:55.073272 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.072773 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8dedac73-f046-46df-88df-536b7db9ed6b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:55.073272 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.072808 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dedac73-f046-46df-88df-536b7db9ed6b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:55.073272 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.072837 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8dedac73-f046-46df-88df-536b7db9ed6b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:55.073272 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.072864 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8dedac73-f046-46df-88df-536b7db9ed6b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:55.073272 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.072893 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8dedac73-f046-46df-88df-536b7db9ed6b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:55.073272 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.072922 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8dedac73-f046-46df-88df-536b7db9ed6b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:55.073272 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.072958 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8dedac73-f046-46df-88df-536b7db9ed6b-config\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:55.073272 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.072981 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8dedac73-f046-46df-88df-536b7db9ed6b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:55.073272 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.073003 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8dedac73-f046-46df-88df-536b7db9ed6b-web-config\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:55.073272 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.073093 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8dedac73-f046-46df-88df-536b7db9ed6b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:55.073968 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.073462 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dedac73-f046-46df-88df-536b7db9ed6b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:55.073968 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.073588 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dedac73-f046-46df-88df-536b7db9ed6b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:55.074391 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.074367 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dedac73-f046-46df-88df-536b7db9ed6b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:55.075950 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.075924 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8dedac73-f046-46df-88df-536b7db9ed6b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:55.076049 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.075998 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8dedac73-f046-46df-88df-536b7db9ed6b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:55.076277 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.076185 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8dedac73-f046-46df-88df-536b7db9ed6b-config-out\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:55.076603 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.076541 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8dedac73-f046-46df-88df-536b7db9ed6b-web-config\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:55.076803 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.076670 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8dedac73-f046-46df-88df-536b7db9ed6b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:55.077310 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.077064 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8dedac73-f046-46df-88df-536b7db9ed6b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:55.077310 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.077133 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8dedac73-f046-46df-88df-536b7db9ed6b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:55.077631 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.077557 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8dedac73-f046-46df-88df-536b7db9ed6b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:55.077714 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.077688 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8dedac73-f046-46df-88df-536b7db9ed6b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:55.078193 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.078155 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8dedac73-f046-46df-88df-536b7db9ed6b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:55.078193 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.078170 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8dedac73-f046-46df-88df-536b7db9ed6b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:55.078609 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.078587 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8dedac73-f046-46df-88df-536b7db9ed6b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:55.079104 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.079082 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8dedac73-f046-46df-88df-536b7db9ed6b-config\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:55.079167 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.079144 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8dedac73-f046-46df-88df-536b7db9ed6b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:55.079571 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.079548 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8dedac73-f046-46df-88df-536b7db9ed6b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:55.085588 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.085568 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-js2bd\" (UniqueName: \"kubernetes.io/projected/8dedac73-f046-46df-88df-536b7db9ed6b-kube-api-access-js2bd\") pod \"prometheus-k8s-0\" (UID: \"8dedac73-f046-46df-88df-536b7db9ed6b\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:55.160305 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.160270 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:51:55.446870 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.446837 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 14 10:51:55.449971 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:51:55.449946 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dedac73_f046_46df_88df_536b7db9ed6b.slice/crio-0b2bdb3f6219953230b78c8f41b762b713f7b188257ba1c25d9be3a0e8f17be9 WatchSource:0}: Error finding container 0b2bdb3f6219953230b78c8f41b762b713f7b188257ba1c25d9be3a0e8f17be9: Status 404 returned error can't find the container with id 0b2bdb3f6219953230b78c8f41b762b713f7b188257ba1c25d9be3a0e8f17be9 Sep 14 10:51:55.784853 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.784823 2581 generic.go:356] "Generic (PLEG): container finished" podID="8dedac73-f046-46df-88df-536b7db9ed6b" containerID="c4038becaed05392af6481808d4497570a44f2606f4e5e6ab51bc1f055772912" exitCode=0 Sep 14 10:51:55.785329 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.784910 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8dedac73-f046-46df-88df-536b7db9ed6b","Type":"ContainerDied","Data":"c4038becaed05392af6481808d4497570a44f2606f4e5e6ab51bc1f055772912"} Sep 14 10:51:55.785329 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:55.785004 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8dedac73-f046-46df-88df-536b7db9ed6b","Type":"ContainerStarted","Data":"0b2bdb3f6219953230b78c8f41b762b713f7b188257ba1c25d9be3a0e8f17be9"} Sep 14 10:51:56.794610 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:56.794570 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8dedac73-f046-46df-88df-536b7db9ed6b","Type":"ContainerStarted","Data":"81de4a2512902d2427c358f780b0bb3d5069cf3e944bfe5c4ea30002534435cb"} Sep 14 10:51:56.795085 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:56.794693 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8dedac73-f046-46df-88df-536b7db9ed6b","Type":"ContainerStarted","Data":"a14e5310375f8852285811184fce9d2c99aee3c6f4f41771b92ef193f03401d3"} Sep 14 10:51:56.795085 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:56.794740 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8dedac73-f046-46df-88df-536b7db9ed6b","Type":"ContainerStarted","Data":"d58857bcebc6b58ee0b7df1689f67ca31815b7d28b9f4421ea2487a1a1a0eb4f"} Sep 14 10:51:56.795085 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:56.794773 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8dedac73-f046-46df-88df-536b7db9ed6b","Type":"ContainerStarted","Data":"5c3f591fff578392022ac93b52a96a22b0abcd75991b02303340c422070bf681"} Sep 14 10:51:56.795085 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:56.794808 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8dedac73-f046-46df-88df-536b7db9ed6b","Type":"ContainerStarted","Data":"731e8e7df13e6e07f30dfd25abbe8a6615d17b1aef12947950d3b418fbbe8aa2"} Sep 14 10:51:56.795085 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:56.794847 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8dedac73-f046-46df-88df-536b7db9ed6b","Type":"ContainerStarted","Data":"94549ffd870e7304c3f1dbedcba7d3823bf631823f1e53c5d682749b2acb1cde"} Sep 14 10:51:56.827188 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:51:56.827135 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.806783926 podStartE2EDuration="2.827117825s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.333899ms" isStatefulPod=false podCreationTimestamp="2026-09-14 10:51:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:51:56.824594357 +0000 UTC m=+190.566996379" watchObservedRunningTime="2026-09-14 10:51:56.827117825 +0000 UTC m=+190.569520149" Sep 14 10:52:00.160890 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:52:00.160848 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:52:07.856328 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:52:07.856298 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5799fbfd5d-272jz"] Sep 14 10:52:32.875818 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:52:32.875759 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5799fbfd5d-272jz" podUID="8d42234d-ca6b-4e18-af83-1c6c0022cb09" containerName="console" containerID="cri-o://69b81209404157385f4ec28384284dad15ab3ebcac81aa5ca0cd708e77b11208" gracePeriod=15 Sep 14 10:52:33.128630 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:52:33.128564 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5799fbfd5d-272jz" Sep 14 10:52:33.213072 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:52:33.213030 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d42234d-ca6b-4e18-af83-1c6c0022cb09-oauth-serving-cert\") pod \"8d42234d-ca6b-4e18-af83-1c6c0022cb09\" (UID: \"8d42234d-ca6b-4e18-af83-1c6c0022cb09\") " Sep 14 10:52:33.213072 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:52:33.213080 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d42234d-ca6b-4e18-af83-1c6c0022cb09-console-serving-cert\") pod \"8d42234d-ca6b-4e18-af83-1c6c0022cb09\" (UID: \"8d42234d-ca6b-4e18-af83-1c6c0022cb09\") " Sep 14 10:52:33.213315 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:52:33.213150 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d42234d-ca6b-4e18-af83-1c6c0022cb09-service-ca\") pod \"8d42234d-ca6b-4e18-af83-1c6c0022cb09\" (UID: \"8d42234d-ca6b-4e18-af83-1c6c0022cb09\") " Sep 14 10:52:33.213315 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:52:33.213183 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d42234d-ca6b-4e18-af83-1c6c0022cb09-trusted-ca-bundle\") pod \"8d42234d-ca6b-4e18-af83-1c6c0022cb09\" (UID: \"8d42234d-ca6b-4e18-af83-1c6c0022cb09\") " Sep 14 10:52:33.213315 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:52:33.213201 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d42234d-ca6b-4e18-af83-1c6c0022cb09-console-config\") pod \"8d42234d-ca6b-4e18-af83-1c6c0022cb09\" (UID: \"8d42234d-ca6b-4e18-af83-1c6c0022cb09\") " Sep 14 10:52:33.213315 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:52:33.213278 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2l2d\" (UniqueName: \"kubernetes.io/projected/8d42234d-ca6b-4e18-af83-1c6c0022cb09-kube-api-access-l2l2d\") pod \"8d42234d-ca6b-4e18-af83-1c6c0022cb09\" (UID: \"8d42234d-ca6b-4e18-af83-1c6c0022cb09\") " Sep 14 10:52:33.213315 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:52:33.213309 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d42234d-ca6b-4e18-af83-1c6c0022cb09-console-oauth-config\") pod \"8d42234d-ca6b-4e18-af83-1c6c0022cb09\" (UID: \"8d42234d-ca6b-4e18-af83-1c6c0022cb09\") " Sep 14 10:52:33.213623 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:52:33.213490 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d42234d-ca6b-4e18-af83-1c6c0022cb09-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8d42234d-ca6b-4e18-af83-1c6c0022cb09" (UID: "8d42234d-ca6b-4e18-af83-1c6c0022cb09"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:52:33.213675 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:52:33.213612 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d42234d-ca6b-4e18-af83-1c6c0022cb09-service-ca" (OuterVolumeSpecName: "service-ca") pod "8d42234d-ca6b-4e18-af83-1c6c0022cb09" (UID: "8d42234d-ca6b-4e18-af83-1c6c0022cb09"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:52:33.213675 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:52:33.213643 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d42234d-ca6b-4e18-af83-1c6c0022cb09-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8d42234d-ca6b-4e18-af83-1c6c0022cb09" (UID: "8d42234d-ca6b-4e18-af83-1c6c0022cb09"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:52:33.213776 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:52:33.213722 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d42234d-ca6b-4e18-af83-1c6c0022cb09-console-config" (OuterVolumeSpecName: "console-config") pod "8d42234d-ca6b-4e18-af83-1c6c0022cb09" (UID: "8d42234d-ca6b-4e18-af83-1c6c0022cb09"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:52:33.213776 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:52:33.213757 2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8d42234d-ca6b-4e18-af83-1c6c0022cb09-oauth-serving-cert\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:52:33.213776 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:52:33.213774 2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8d42234d-ca6b-4e18-af83-1c6c0022cb09-service-ca\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:52:33.214101 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:52:33.213783 2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d42234d-ca6b-4e18-af83-1c6c0022cb09-trusted-ca-bundle\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:52:33.215563 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:52:33.215535 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d42234d-ca6b-4e18-af83-1c6c0022cb09-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8d42234d-ca6b-4e18-af83-1c6c0022cb09" (UID: "8d42234d-ca6b-4e18-af83-1c6c0022cb09"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 10:52:33.215687 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:52:33.215581 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d42234d-ca6b-4e18-af83-1c6c0022cb09-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8d42234d-ca6b-4e18-af83-1c6c0022cb09" (UID: "8d42234d-ca6b-4e18-af83-1c6c0022cb09"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 10:52:33.215687 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:52:33.215666 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d42234d-ca6b-4e18-af83-1c6c0022cb09-kube-api-access-l2l2d" (OuterVolumeSpecName: "kube-api-access-l2l2d") pod "8d42234d-ca6b-4e18-af83-1c6c0022cb09" (UID: "8d42234d-ca6b-4e18-af83-1c6c0022cb09"). InnerVolumeSpecName "kube-api-access-l2l2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:52:33.315014 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:52:33.314972 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2l2d\" (UniqueName: \"kubernetes.io/projected/8d42234d-ca6b-4e18-af83-1c6c0022cb09-kube-api-access-l2l2d\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:52:33.315014 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:52:33.315003 2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8d42234d-ca6b-4e18-af83-1c6c0022cb09-console-oauth-config\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:52:33.315014 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:52:33.315012 2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d42234d-ca6b-4e18-af83-1c6c0022cb09-console-serving-cert\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:52:33.315014 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:52:33.315022 2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8d42234d-ca6b-4e18-af83-1c6c0022cb09-console-config\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:52:33.946714 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:52:33.946677 2581 generic.go:356] "Generic (PLEG): container finished" podID="8d42234d-ca6b-4e18-af83-1c6c0022cb09" containerID="69b81209404157385f4ec28384284dad15ab3ebcac81aa5ca0cd708e77b11208" exitCode=0 Sep 14 10:52:33.947118 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:52:33.946719 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5799fbfd5d-272jz" Sep 14 10:52:33.947118 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:52:33.946718 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5799fbfd5d-272jz" event={"ID":"8d42234d-ca6b-4e18-af83-1c6c0022cb09","Type":"ContainerDied","Data":"69b81209404157385f4ec28384284dad15ab3ebcac81aa5ca0cd708e77b11208"} Sep 14 10:52:33.947118 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:52:33.946843 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5799fbfd5d-272jz" event={"ID":"8d42234d-ca6b-4e18-af83-1c6c0022cb09","Type":"ContainerDied","Data":"548558dc7c8545839a9006394fcda56e44db1554771a19a64c451247e714159b"} Sep 14 10:52:33.947118 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:52:33.946868 2581 scope.go:117] "RemoveContainer" containerID="69b81209404157385f4ec28384284dad15ab3ebcac81aa5ca0cd708e77b11208" Sep 14 10:52:33.955441 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:52:33.955420 2581 scope.go:117] "RemoveContainer" containerID="69b81209404157385f4ec28384284dad15ab3ebcac81aa5ca0cd708e77b11208" Sep 14 10:52:33.955706 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:52:33.955690 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69b81209404157385f4ec28384284dad15ab3ebcac81aa5ca0cd708e77b11208\": container with ID starting with 69b81209404157385f4ec28384284dad15ab3ebcac81aa5ca0cd708e77b11208 not found: ID does not exist" containerID="69b81209404157385f4ec28384284dad15ab3ebcac81aa5ca0cd708e77b11208" Sep 14 10:52:33.955750 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:52:33.955716 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69b81209404157385f4ec28384284dad15ab3ebcac81aa5ca0cd708e77b11208"} err="failed to get container status \"69b81209404157385f4ec28384284dad15ab3ebcac81aa5ca0cd708e77b11208\": rpc error: code = NotFound desc = could not find container \"69b81209404157385f4ec28384284dad15ab3ebcac81aa5ca0cd708e77b11208\": container with ID starting with 69b81209404157385f4ec28384284dad15ab3ebcac81aa5ca0cd708e77b11208 not found: ID does not exist" Sep 14 10:52:33.967547 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:52:33.967523 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5799fbfd5d-272jz"] Sep 14 10:52:33.971166 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:52:33.971143 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5799fbfd5d-272jz"] Sep 14 10:52:34.831232 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:52:34.831199 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d42234d-ca6b-4e18-af83-1c6c0022cb09" path="/var/lib/kubelet/pods/8d42234d-ca6b-4e18-af83-1c6c0022cb09/volumes" Sep 14 10:52:55.160672 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:52:55.160628 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:52:55.176598 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:52:55.176567 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:52:56.048276 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:52:56.048245 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 10:53:46.697416 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:53:46.697388 2581 kubelet.go:1648] "Image garbage collection succeeded" Sep 14 10:55:36.966641 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:36.966609 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-587bf9cb9-nss2w"] Sep 14 10:55:36.967066 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:36.966896 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d42234d-ca6b-4e18-af83-1c6c0022cb09" containerName="console" Sep 14 10:55:36.967066 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:36.966907 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d42234d-ca6b-4e18-af83-1c6c0022cb09" containerName="console" Sep 14 10:55:36.967066 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:36.966965 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d42234d-ca6b-4e18-af83-1c6c0022cb09" containerName="console" Sep 14 10:55:36.969881 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:36.969866 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-587bf9cb9-nss2w" Sep 14 10:55:36.976865 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:36.976845 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-tstsr\"" Sep 14 10:55:36.978074 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:36.978029 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 14 10:55:36.978305 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:36.978284 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 14 10:55:36.978437 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:36.978284 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 14 10:55:36.978437 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:36.978285 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 14 10:55:36.983595 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:36.983577 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 14 10:55:37.078272 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:37.078235 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fbd52720-5820-4d71-89b1-dbce23230c7e-manager-config\") pod \"lws-controller-manager-587bf9cb9-nss2w\" (UID: \"fbd52720-5820-4d71-89b1-dbce23230c7e\") " pod="openshift-lws-operator/lws-controller-manager-587bf9cb9-nss2w" Sep 14 10:55:37.078464 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:37.078278 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fbd52720-5820-4d71-89b1-dbce23230c7e-metrics-cert\") pod \"lws-controller-manager-587bf9cb9-nss2w\" (UID: \"fbd52720-5820-4d71-89b1-dbce23230c7e\") " pod="openshift-lws-operator/lws-controller-manager-587bf9cb9-nss2w" Sep 14 10:55:37.078464 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:37.078304 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fbd52720-5820-4d71-89b1-dbce23230c7e-cert\") pod \"lws-controller-manager-587bf9cb9-nss2w\" (UID: \"fbd52720-5820-4d71-89b1-dbce23230c7e\") " pod="openshift-lws-operator/lws-controller-manager-587bf9cb9-nss2w" Sep 14 10:55:37.078464 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:37.078449 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrfxv\" (UniqueName: \"kubernetes.io/projected/fbd52720-5820-4d71-89b1-dbce23230c7e-kube-api-access-vrfxv\") pod \"lws-controller-manager-587bf9cb9-nss2w\" (UID: \"fbd52720-5820-4d71-89b1-dbce23230c7e\") " pod="openshift-lws-operator/lws-controller-manager-587bf9cb9-nss2w" Sep 14 10:55:37.112825 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:37.112794 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-587bf9cb9-nss2w"] Sep 14 10:55:37.179084 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:37.179045 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrfxv\" (UniqueName: \"kubernetes.io/projected/fbd52720-5820-4d71-89b1-dbce23230c7e-kube-api-access-vrfxv\") pod \"lws-controller-manager-587bf9cb9-nss2w\" (UID: \"fbd52720-5820-4d71-89b1-dbce23230c7e\") " pod="openshift-lws-operator/lws-controller-manager-587bf9cb9-nss2w" Sep 14 10:55:37.179084 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:37.179088 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fbd52720-5820-4d71-89b1-dbce23230c7e-manager-config\") pod \"lws-controller-manager-587bf9cb9-nss2w\" (UID: \"fbd52720-5820-4d71-89b1-dbce23230c7e\") " pod="openshift-lws-operator/lws-controller-manager-587bf9cb9-nss2w" Sep 14 10:55:37.179322 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:37.179115 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fbd52720-5820-4d71-89b1-dbce23230c7e-metrics-cert\") pod \"lws-controller-manager-587bf9cb9-nss2w\" (UID: \"fbd52720-5820-4d71-89b1-dbce23230c7e\") " pod="openshift-lws-operator/lws-controller-manager-587bf9cb9-nss2w" Sep 14 10:55:37.179322 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:37.179137 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fbd52720-5820-4d71-89b1-dbce23230c7e-cert\") pod \"lws-controller-manager-587bf9cb9-nss2w\" (UID: \"fbd52720-5820-4d71-89b1-dbce23230c7e\") " pod="openshift-lws-operator/lws-controller-manager-587bf9cb9-nss2w" Sep 14 10:55:37.179828 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:37.179799 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/fbd52720-5820-4d71-89b1-dbce23230c7e-manager-config\") pod \"lws-controller-manager-587bf9cb9-nss2w\" (UID: \"fbd52720-5820-4d71-89b1-dbce23230c7e\") " pod="openshift-lws-operator/lws-controller-manager-587bf9cb9-nss2w" Sep 14 10:55:37.181868 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:37.181843 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fbd52720-5820-4d71-89b1-dbce23230c7e-cert\") pod \"lws-controller-manager-587bf9cb9-nss2w\" (UID: \"fbd52720-5820-4d71-89b1-dbce23230c7e\") " pod="openshift-lws-operator/lws-controller-manager-587bf9cb9-nss2w" Sep 14 10:55:37.181963 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:37.181891 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/fbd52720-5820-4d71-89b1-dbce23230c7e-metrics-cert\") pod \"lws-controller-manager-587bf9cb9-nss2w\" (UID: \"fbd52720-5820-4d71-89b1-dbce23230c7e\") " pod="openshift-lws-operator/lws-controller-manager-587bf9cb9-nss2w" Sep 14 10:55:37.191673 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:37.191639 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrfxv\" (UniqueName: \"kubernetes.io/projected/fbd52720-5820-4d71-89b1-dbce23230c7e-kube-api-access-vrfxv\") pod \"lws-controller-manager-587bf9cb9-nss2w\" (UID: \"fbd52720-5820-4d71-89b1-dbce23230c7e\") " pod="openshift-lws-operator/lws-controller-manager-587bf9cb9-nss2w" Sep 14 10:55:37.279298 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:37.279263 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-587bf9cb9-nss2w" Sep 14 10:55:37.592001 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:37.591916 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-587bf9cb9-nss2w"] Sep 14 10:55:37.595136 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:55:37.595087 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbd52720_5820_4d71_89b1_dbce23230c7e.slice/crio-53d722b497754cdd7c61568401f0f896c50579be77dbdc5a37ec88d244423cf3 WatchSource:0}: Error finding container 53d722b497754cdd7c61568401f0f896c50579be77dbdc5a37ec88d244423cf3: Status 404 returned error can't find the container with id 53d722b497754cdd7c61568401f0f896c50579be77dbdc5a37ec88d244423cf3 Sep 14 10:55:37.597140 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:37.597119 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 10:55:37.657823 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:37.657786 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-587bf9cb9-nss2w" event={"ID":"fbd52720-5820-4d71-89b1-dbce23230c7e","Type":"ContainerStarted","Data":"53d722b497754cdd7c61568401f0f896c50579be77dbdc5a37ec88d244423cf3"} Sep 14 10:55:37.866918 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:37.866830 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f6d976668-jh7ms"] Sep 14 10:55:37.871539 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:37.871517 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-jh7ms" Sep 14 10:55:37.879163 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:37.879144 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 14 10:55:37.879301 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:37.879286 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-rs8sp\"" Sep 14 10:55:37.880182 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:37.880169 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 14 10:55:37.880367 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:37.880355 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 14 10:55:37.883749 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:37.883728 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 14 10:55:37.896367 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:37.896330 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f6d976668-jh7ms"] Sep 14 10:55:37.984724 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:37.984689 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1c88dbf8-5536-496c-ac71-5b0de205f57d-manifests\") pod \"opendatahub-operator-controller-manager-f6d976668-jh7ms\" (UID: \"1c88dbf8-5536-496c-ac71-5b0de205f57d\") " pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-jh7ms" Sep 14 10:55:37.985136 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:37.984744 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c88dbf8-5536-496c-ac71-5b0de205f57d-webhook-cert\") pod \"opendatahub-operator-controller-manager-f6d976668-jh7ms\" (UID: \"1c88dbf8-5536-496c-ac71-5b0de205f57d\") " pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-jh7ms" Sep 14 10:55:37.985136 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:37.984776 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvmdt\" (UniqueName: \"kubernetes.io/projected/1c88dbf8-5536-496c-ac71-5b0de205f57d-kube-api-access-fvmdt\") pod \"opendatahub-operator-controller-manager-f6d976668-jh7ms\" (UID: \"1c88dbf8-5536-496c-ac71-5b0de205f57d\") " pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-jh7ms" Sep 14 10:55:37.985136 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:37.984833 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1c88dbf8-5536-496c-ac71-5b0de205f57d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f6d976668-jh7ms\" (UID: \"1c88dbf8-5536-496c-ac71-5b0de205f57d\") " pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-jh7ms" Sep 14 10:55:37.985136 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:37.984892 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c88dbf8-5536-496c-ac71-5b0de205f57d-tmp\") pod \"opendatahub-operator-controller-manager-f6d976668-jh7ms\" (UID: \"1c88dbf8-5536-496c-ac71-5b0de205f57d\") " pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-jh7ms" Sep 14 10:55:38.086145 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:38.086113 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c88dbf8-5536-496c-ac71-5b0de205f57d-tmp\") pod \"opendatahub-operator-controller-manager-f6d976668-jh7ms\" (UID: \"1c88dbf8-5536-496c-ac71-5b0de205f57d\") " pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-jh7ms" Sep 14 10:55:38.086327 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:38.086161 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1c88dbf8-5536-496c-ac71-5b0de205f57d-manifests\") pod \"opendatahub-operator-controller-manager-f6d976668-jh7ms\" (UID: \"1c88dbf8-5536-496c-ac71-5b0de205f57d\") " pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-jh7ms" Sep 14 10:55:38.086327 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:38.086198 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c88dbf8-5536-496c-ac71-5b0de205f57d-webhook-cert\") pod \"opendatahub-operator-controller-manager-f6d976668-jh7ms\" (UID: \"1c88dbf8-5536-496c-ac71-5b0de205f57d\") " pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-jh7ms" Sep 14 10:55:38.086327 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:38.086233 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvmdt\" (UniqueName: \"kubernetes.io/projected/1c88dbf8-5536-496c-ac71-5b0de205f57d-kube-api-access-fvmdt\") pod \"opendatahub-operator-controller-manager-f6d976668-jh7ms\" (UID: \"1c88dbf8-5536-496c-ac71-5b0de205f57d\") " pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-jh7ms" Sep 14 10:55:38.086327 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:38.086291 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1c88dbf8-5536-496c-ac71-5b0de205f57d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f6d976668-jh7ms\" (UID: \"1c88dbf8-5536-496c-ac71-5b0de205f57d\") " pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-jh7ms" Sep 14 10:55:38.086652 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:38.086633 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1c88dbf8-5536-496c-ac71-5b0de205f57d-manifests\") pod \"opendatahub-operator-controller-manager-f6d976668-jh7ms\" (UID: \"1c88dbf8-5536-496c-ac71-5b0de205f57d\") " pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-jh7ms" Sep 14 10:55:38.088662 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:38.088636 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c88dbf8-5536-496c-ac71-5b0de205f57d-tmp\") pod \"opendatahub-operator-controller-manager-f6d976668-jh7ms\" (UID: \"1c88dbf8-5536-496c-ac71-5b0de205f57d\") " pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-jh7ms" Sep 14 10:55:38.088767 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:38.088710 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1c88dbf8-5536-496c-ac71-5b0de205f57d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f6d976668-jh7ms\" (UID: \"1c88dbf8-5536-496c-ac71-5b0de205f57d\") " pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-jh7ms" Sep 14 10:55:38.088827 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:38.088785 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c88dbf8-5536-496c-ac71-5b0de205f57d-webhook-cert\") pod \"opendatahub-operator-controller-manager-f6d976668-jh7ms\" (UID: \"1c88dbf8-5536-496c-ac71-5b0de205f57d\") " pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-jh7ms" Sep 14 10:55:38.103069 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:38.103042 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvmdt\" (UniqueName: \"kubernetes.io/projected/1c88dbf8-5536-496c-ac71-5b0de205f57d-kube-api-access-fvmdt\") pod \"opendatahub-operator-controller-manager-f6d976668-jh7ms\" (UID: \"1c88dbf8-5536-496c-ac71-5b0de205f57d\") " pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-jh7ms" Sep 14 10:55:38.182216 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:38.182122 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-jh7ms" Sep 14 10:55:38.534322 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:55:38.534292 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c88dbf8_5536_496c_ac71_5b0de205f57d.slice/crio-ef88f62f2a192a0366f24828974161051ba12dd55bf3528b67bb34eeba2ebea3 WatchSource:0}: Error finding container ef88f62f2a192a0366f24828974161051ba12dd55bf3528b67bb34eeba2ebea3: Status 404 returned error can't find the container with id ef88f62f2a192a0366f24828974161051ba12dd55bf3528b67bb34eeba2ebea3 Sep 14 10:55:38.543383 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:38.543330 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f6d976668-jh7ms"] Sep 14 10:55:38.664927 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:38.664850 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-jh7ms" event={"ID":"1c88dbf8-5536-496c-ac71-5b0de205f57d","Type":"ContainerStarted","Data":"ef88f62f2a192a0366f24828974161051ba12dd55bf3528b67bb34eeba2ebea3"} Sep 14 10:55:42.685528 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:42.685490 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-jh7ms" event={"ID":"1c88dbf8-5536-496c-ac71-5b0de205f57d","Type":"ContainerStarted","Data":"ae2be35661e95e141f40cbbf497c6ad59ffdbce6395a91260624958b7870099c"} Sep 14 10:55:42.687038 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:42.687014 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-587bf9cb9-nss2w" event={"ID":"fbd52720-5820-4d71-89b1-dbce23230c7e","Type":"ContainerStarted","Data":"e62d1fab1fb82b095045d9578e2abcbb1e1046043b6f8cf4fc2a3205016abf00"} Sep 14 10:55:42.687148 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:42.687066 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-587bf9cb9-nss2w" Sep 14 10:55:42.763118 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:42.763066 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-587bf9cb9-nss2w" podStartSLOduration=2.305248445 podStartE2EDuration="6.763048918s" totalImagesPullingTime="4.457800473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:55:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:55:42.762240879 +0000 UTC m=+416.504642849" watchObservedRunningTime="2026-09-14 10:55:42.763048918 +0000 UTC m=+416.505450887" Sep 14 10:55:43.692888 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:43.692850 2581 generic.go:356] "Generic (PLEG): container finished" podID="1c88dbf8-5536-496c-ac71-5b0de205f57d" containerID="ae2be35661e95e141f40cbbf497c6ad59ffdbce6395a91260624958b7870099c" exitCode=0 Sep 14 10:55:43.693372 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:43.692933 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-jh7ms" event={"ID":"1c88dbf8-5536-496c-ac71-5b0de205f57d","Type":"ContainerDied","Data":"ae2be35661e95e141f40cbbf497c6ad59ffdbce6395a91260624958b7870099c"} Sep 14 10:55:44.699746 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:44.699713 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-jh7ms" event={"ID":"1c88dbf8-5536-496c-ac71-5b0de205f57d","Type":"ContainerStarted","Data":"d8d94490bdc0d2236b16f0df530396bcd8f80abfab9cf72457a904feccfc529f"} Sep 14 10:55:44.700131 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:44.699821 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-jh7ms" Sep 14 10:55:44.771660 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:44.771602 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-jh7ms" podStartSLOduration=2.759065582 podStartE2EDuration="7.771580775s" totalImagesPullingTime="3.963883837s" totalInitContainerRuntime="1.048631356s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:55:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:55:44.770765086 +0000 UTC m=+418.513167056" watchObservedRunningTime="2026-09-14 10:55:44.771580775 +0000 UTC m=+418.513982748" Sep 14 10:55:53.695079 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:53.695045 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-587bf9cb9-nss2w" Sep 14 10:55:55.706380 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:55.706325 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-f6d976668-jh7ms" Sep 14 10:55:58.934138 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:58.934101 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-848c99f856-g64c6"] Sep 14 10:55:59.014072 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:59.014032 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-848c99f856-g64c6"] Sep 14 10:55:59.014230 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:59.014164 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-848c99f856-g64c6" Sep 14 10:55:59.021167 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:59.021143 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-66wvz\"" Sep 14 10:55:59.022170 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:59.022153 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 14 10:55:59.022423 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:59.022408 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 14 10:55:59.024946 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:59.024930 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 14 10:55:59.027643 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:59.027630 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 14 10:55:59.071500 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:59.071470 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8c2cd16-075b-41b4-8c6c-332a716d1012-tls-certs\") pod \"kube-auth-proxy-848c99f856-g64c6\" (UID: \"b8c2cd16-075b-41b4-8c6c-332a716d1012\") " pod="openshift-ingress/kube-auth-proxy-848c99f856-g64c6" Sep 14 10:55:59.071610 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:59.071526 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8c2cd16-075b-41b4-8c6c-332a716d1012-tmp\") pod \"kube-auth-proxy-848c99f856-g64c6\" (UID: \"b8c2cd16-075b-41b4-8c6c-332a716d1012\") " pod="openshift-ingress/kube-auth-proxy-848c99f856-g64c6" Sep 14 10:55:59.071610 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:59.071555 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w2z6\" (UniqueName: \"kubernetes.io/projected/b8c2cd16-075b-41b4-8c6c-332a716d1012-kube-api-access-5w2z6\") pod \"kube-auth-proxy-848c99f856-g64c6\" (UID: \"b8c2cd16-075b-41b4-8c6c-332a716d1012\") " pod="openshift-ingress/kube-auth-proxy-848c99f856-g64c6" Sep 14 10:55:59.172594 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:59.172558 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8c2cd16-075b-41b4-8c6c-332a716d1012-tls-certs\") pod \"kube-auth-proxy-848c99f856-g64c6\" (UID: \"b8c2cd16-075b-41b4-8c6c-332a716d1012\") " pod="openshift-ingress/kube-auth-proxy-848c99f856-g64c6" Sep 14 10:55:59.172753 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:59.172628 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8c2cd16-075b-41b4-8c6c-332a716d1012-tmp\") pod \"kube-auth-proxy-848c99f856-g64c6\" (UID: \"b8c2cd16-075b-41b4-8c6c-332a716d1012\") " pod="openshift-ingress/kube-auth-proxy-848c99f856-g64c6" Sep 14 10:55:59.172753 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:59.172664 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w2z6\" (UniqueName: \"kubernetes.io/projected/b8c2cd16-075b-41b4-8c6c-332a716d1012-kube-api-access-5w2z6\") pod \"kube-auth-proxy-848c99f856-g64c6\" (UID: \"b8c2cd16-075b-41b4-8c6c-332a716d1012\") " pod="openshift-ingress/kube-auth-proxy-848c99f856-g64c6" Sep 14 10:55:59.175138 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:59.175099 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8c2cd16-075b-41b4-8c6c-332a716d1012-tmp\") pod \"kube-auth-proxy-848c99f856-g64c6\" (UID: \"b8c2cd16-075b-41b4-8c6c-332a716d1012\") " pod="openshift-ingress/kube-auth-proxy-848c99f856-g64c6" Sep 14 10:55:59.175382 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:59.175363 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8c2cd16-075b-41b4-8c6c-332a716d1012-tls-certs\") pod \"kube-auth-proxy-848c99f856-g64c6\" (UID: \"b8c2cd16-075b-41b4-8c6c-332a716d1012\") " pod="openshift-ingress/kube-auth-proxy-848c99f856-g64c6" Sep 14 10:55:59.181923 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:59.181897 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w2z6\" (UniqueName: \"kubernetes.io/projected/b8c2cd16-075b-41b4-8c6c-332a716d1012-kube-api-access-5w2z6\") pod \"kube-auth-proxy-848c99f856-g64c6\" (UID: \"b8c2cd16-075b-41b4-8c6c-332a716d1012\") " pod="openshift-ingress/kube-auth-proxy-848c99f856-g64c6" Sep 14 10:55:59.323969 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:59.323932 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-848c99f856-g64c6" Sep 14 10:55:59.714618 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:59.714543 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-848c99f856-g64c6"] Sep 14 10:55:59.722016 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:55:59.721990 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c2cd16_075b_41b4_8c6c_332a716d1012.slice/crio-b6698ff712db7e086d7bad26f9425f7b29160d2cfa85e8337876b6d442caee88 WatchSource:0}: Error finding container b6698ff712db7e086d7bad26f9425f7b29160d2cfa85e8337876b6d442caee88: Status 404 returned error can't find the container with id b6698ff712db7e086d7bad26f9425f7b29160d2cfa85e8337876b6d442caee88 Sep 14 10:55:59.768491 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:55:59.768454 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-848c99f856-g64c6" event={"ID":"b8c2cd16-075b-41b4-8c6c-332a716d1012","Type":"ContainerStarted","Data":"b6698ff712db7e086d7bad26f9425f7b29160d2cfa85e8337876b6d442caee88"} Sep 14 10:56:02.785424 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:56:02.785389 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-848c99f856-g64c6" event={"ID":"b8c2cd16-075b-41b4-8c6c-332a716d1012","Type":"ContainerStarted","Data":"74a7977ede027e8ad17d9c034a5f9595c25500061ff0cc00aa2c3ec73cd5eedf"} Sep 14 10:56:02.812956 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:56:02.812910 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-848c99f856-g64c6" podStartSLOduration=2.216385675 podStartE2EDuration="4.812893215s" totalImagesPullingTime="2.59650754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:55:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:56:02.812195386 +0000 UTC m=+436.554597359" watchObservedRunningTime="2026-09-14 10:56:02.812893215 +0000 UTC m=+436.555295185" Sep 14 10:57:57.922877 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:57:57.922847 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-4ws2n"] Sep 14 10:57:57.926045 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:57:57.926025 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-4ws2n" Sep 14 10:57:57.932570 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:57:57.932374 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Sep 14 10:57:57.932627 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:57:57.932586 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 14 10:57:57.932661 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:57:57.932618 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Sep 14 10:57:57.933490 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:57:57.933475 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-j858j\"" Sep 14 10:57:57.933555 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:57:57.933477 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 14 10:57:57.937797 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:57:57.937778 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-4ws2n"] Sep 14 10:57:57.940112 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:57:57.940090 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7htd\" (UniqueName: \"kubernetes.io/projected/81dca805-7c69-4e16-8cf8-bafb46cdc3ab-kube-api-access-z7htd\") pod \"kuadrant-console-plugin-fbdd4f445-4ws2n\" (UID: \"81dca805-7c69-4e16-8cf8-bafb46cdc3ab\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-4ws2n" Sep 14 10:57:57.940199 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:57:57.940164 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/81dca805-7c69-4e16-8cf8-bafb46cdc3ab-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-4ws2n\" (UID: \"81dca805-7c69-4e16-8cf8-bafb46cdc3ab\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-4ws2n" Sep 14 10:57:57.940199 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:57:57.940183 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/81dca805-7c69-4e16-8cf8-bafb46cdc3ab-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-4ws2n\" (UID: \"81dca805-7c69-4e16-8cf8-bafb46cdc3ab\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-4ws2n" Sep 14 10:57:58.041508 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:57:58.041472 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7htd\" (UniqueName: \"kubernetes.io/projected/81dca805-7c69-4e16-8cf8-bafb46cdc3ab-kube-api-access-z7htd\") pod \"kuadrant-console-plugin-fbdd4f445-4ws2n\" (UID: \"81dca805-7c69-4e16-8cf8-bafb46cdc3ab\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-4ws2n" Sep 14 10:57:58.041708 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:57:58.041528 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/81dca805-7c69-4e16-8cf8-bafb46cdc3ab-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-4ws2n\" (UID: \"81dca805-7c69-4e16-8cf8-bafb46cdc3ab\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-4ws2n" Sep 14 10:57:58.041708 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:57:58.041546 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/81dca805-7c69-4e16-8cf8-bafb46cdc3ab-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-4ws2n\" (UID: \"81dca805-7c69-4e16-8cf8-bafb46cdc3ab\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-4ws2n" Sep 14 10:57:58.041708 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:57:58.041631 2581 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Sep 14 10:57:58.041883 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:57:58.041711 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81dca805-7c69-4e16-8cf8-bafb46cdc3ab-plugin-serving-cert podName:81dca805-7c69-4e16-8cf8-bafb46cdc3ab nodeName:}" failed. No retries permitted until 2026-09-14 10:57:58.541690587 +0000 UTC m=+552.284092545 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/81dca805-7c69-4e16-8cf8-bafb46cdc3ab-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-4ws2n" (UID: "81dca805-7c69-4e16-8cf8-bafb46cdc3ab") : secret "plugin-serving-cert" not found Sep 14 10:57:58.042137 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:57:58.042120 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/81dca805-7c69-4e16-8cf8-bafb46cdc3ab-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-4ws2n\" (UID: \"81dca805-7c69-4e16-8cf8-bafb46cdc3ab\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-4ws2n" Sep 14 10:57:58.053208 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:57:58.053183 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7htd\" (UniqueName: \"kubernetes.io/projected/81dca805-7c69-4e16-8cf8-bafb46cdc3ab-kube-api-access-z7htd\") pod \"kuadrant-console-plugin-fbdd4f445-4ws2n\" (UID: \"81dca805-7c69-4e16-8cf8-bafb46cdc3ab\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-4ws2n" Sep 14 10:57:58.545461 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:57:58.545425 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/81dca805-7c69-4e16-8cf8-bafb46cdc3ab-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-4ws2n\" (UID: \"81dca805-7c69-4e16-8cf8-bafb46cdc3ab\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-4ws2n" Sep 14 10:57:58.547993 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:57:58.547967 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/81dca805-7c69-4e16-8cf8-bafb46cdc3ab-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-4ws2n\" (UID: \"81dca805-7c69-4e16-8cf8-bafb46cdc3ab\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-4ws2n" Sep 14 10:57:58.835218 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:57:58.835144 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-4ws2n" Sep 14 10:57:59.114129 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:57:59.114101 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-4ws2n"] Sep 14 10:57:59.116266 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:57:59.116241 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81dca805_7c69_4e16_8cf8_bafb46cdc3ab.slice/crio-e1b576eb4efc0569b6a4dc7772a6c57f2b919db4138bbfd8c15d07c527c6deab WatchSource:0}: Error finding container e1b576eb4efc0569b6a4dc7772a6c57f2b919db4138bbfd8c15d07c527c6deab: Status 404 returned error can't find the container with id e1b576eb4efc0569b6a4dc7772a6c57f2b919db4138bbfd8c15d07c527c6deab Sep 14 10:57:59.271538 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:57:59.271502 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-4ws2n" event={"ID":"81dca805-7c69-4e16-8cf8-bafb46cdc3ab","Type":"ContainerStarted","Data":"e1b576eb4efc0569b6a4dc7772a6c57f2b919db4138bbfd8c15d07c527c6deab"} Sep 14 10:58:04.298588 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:58:04.298543 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-4ws2n" event={"ID":"81dca805-7c69-4e16-8cf8-bafb46cdc3ab","Type":"ContainerStarted","Data":"572d297d830ef3cd32d4af0786a3246e10411dee1e706b6f0405150724859572"} Sep 14 10:58:04.342118 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:58:04.342062 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-4ws2n" podStartSLOduration=2.506965937 podStartE2EDuration="7.342042847s" totalImagesPullingTime="4.83507691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:57:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:58:04.338967257 +0000 UTC m=+558.081369227" watchObservedRunningTime="2026-09-14 10:58:04.342042847 +0000 UTC m=+558.084444817" Sep 14 10:58:57.634793 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:58:57.634760 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-kmsjf"] Sep 14 10:58:57.638291 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:58:57.638275 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-kmsjf" Sep 14 10:58:57.641077 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:58:57.641055 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 14 10:58:57.643078 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:58:57.643059 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knvt6\" (UniqueName: \"kubernetes.io/projected/8c677f05-d94a-4ed0-924e-e0b63efe53f5-kube-api-access-knvt6\") pod \"limitador-limitador-8588749977-kmsjf\" (UID: \"8c677f05-d94a-4ed0-924e-e0b63efe53f5\") " pod="kuadrant-system/limitador-limitador-8588749977-kmsjf" Sep 14 10:58:57.643181 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:58:57.643101 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8c677f05-d94a-4ed0-924e-e0b63efe53f5-config-file\") pod \"limitador-limitador-8588749977-kmsjf\" (UID: \"8c677f05-d94a-4ed0-924e-e0b63efe53f5\") " pod="kuadrant-system/limitador-limitador-8588749977-kmsjf" Sep 14 10:58:57.647925 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:58:57.647906 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-kmsjf"] Sep 14 10:58:57.734716 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:58:57.734685 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-kmsjf"] Sep 14 10:58:57.744567 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:58:57.744538 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knvt6\" (UniqueName: \"kubernetes.io/projected/8c677f05-d94a-4ed0-924e-e0b63efe53f5-kube-api-access-knvt6\") pod \"limitador-limitador-8588749977-kmsjf\" (UID: \"8c677f05-d94a-4ed0-924e-e0b63efe53f5\") " pod="kuadrant-system/limitador-limitador-8588749977-kmsjf" Sep 14 10:58:57.744695 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:58:57.744611 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8c677f05-d94a-4ed0-924e-e0b63efe53f5-config-file\") pod \"limitador-limitador-8588749977-kmsjf\" (UID: \"8c677f05-d94a-4ed0-924e-e0b63efe53f5\") " pod="kuadrant-system/limitador-limitador-8588749977-kmsjf" Sep 14 10:58:57.745256 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:58:57.745239 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8c677f05-d94a-4ed0-924e-e0b63efe53f5-config-file\") pod \"limitador-limitador-8588749977-kmsjf\" (UID: \"8c677f05-d94a-4ed0-924e-e0b63efe53f5\") " pod="kuadrant-system/limitador-limitador-8588749977-kmsjf" Sep 14 10:58:57.754676 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:58:57.754650 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knvt6\" (UniqueName: \"kubernetes.io/projected/8c677f05-d94a-4ed0-924e-e0b63efe53f5-kube-api-access-knvt6\") pod \"limitador-limitador-8588749977-kmsjf\" (UID: \"8c677f05-d94a-4ed0-924e-e0b63efe53f5\") " pod="kuadrant-system/limitador-limitador-8588749977-kmsjf" Sep 14 10:58:57.948269 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:58:57.948177 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-kmsjf" Sep 14 10:58:58.220185 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:58:58.220163 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-kmsjf"] Sep 14 10:58:58.222709 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:58:58.222683 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c677f05_d94a_4ed0_924e_e0b63efe53f5.slice/crio-7a0e44a1f35e6a158e6fbef3381f1342ce69ca12766b9b033ea920becfe0a2a6 WatchSource:0}: Error finding container 7a0e44a1f35e6a158e6fbef3381f1342ce69ca12766b9b033ea920becfe0a2a6: Status 404 returned error can't find the container with id 7a0e44a1f35e6a158e6fbef3381f1342ce69ca12766b9b033ea920becfe0a2a6 Sep 14 10:58:58.533608 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:58:58.533575 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-kmsjf" event={"ID":"8c677f05-d94a-4ed0-924e-e0b63efe53f5","Type":"ContainerStarted","Data":"7a0e44a1f35e6a158e6fbef3381f1342ce69ca12766b9b033ea920becfe0a2a6"} Sep 14 10:58:58.767539 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:58:58.767504 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-82mbj"] Sep 14 10:58:58.772245 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:58:58.772227 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-82mbj" Sep 14 10:58:58.775721 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:58:58.775701 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wvnh6\"" Sep 14 10:58:58.786555 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:58:58.786482 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-82mbj"] Sep 14 10:58:58.852316 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:58:58.852282 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nrkd\" (UniqueName: \"kubernetes.io/projected/67a35ec5-3b2e-4c8c-825b-48fee1aa18ce-kube-api-access-9nrkd\") pod \"authorino-59c974bd48-82mbj\" (UID: \"67a35ec5-3b2e-4c8c-825b-48fee1aa18ce\") " pod="kuadrant-system/authorino-59c974bd48-82mbj" Sep 14 10:58:58.953445 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:58:58.953410 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nrkd\" (UniqueName: \"kubernetes.io/projected/67a35ec5-3b2e-4c8c-825b-48fee1aa18ce-kube-api-access-9nrkd\") pod \"authorino-59c974bd48-82mbj\" (UID: \"67a35ec5-3b2e-4c8c-825b-48fee1aa18ce\") " pod="kuadrant-system/authorino-59c974bd48-82mbj" Sep 14 10:58:58.964943 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:58:58.964921 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nrkd\" (UniqueName: \"kubernetes.io/projected/67a35ec5-3b2e-4c8c-825b-48fee1aa18ce-kube-api-access-9nrkd\") pod \"authorino-59c974bd48-82mbj\" (UID: \"67a35ec5-3b2e-4c8c-825b-48fee1aa18ce\") " pod="kuadrant-system/authorino-59c974bd48-82mbj" Sep 14 10:58:59.081849 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:58:59.081756 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-82mbj" Sep 14 10:58:59.353938 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:58:59.353905 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-82mbj"] Sep 14 10:58:59.356490 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:58:59.356461 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67a35ec5_3b2e_4c8c_825b_48fee1aa18ce.slice/crio-ca34f027fb331c8eb7602b41b776019cb17782036b24c4f90a52c64649be2210 WatchSource:0}: Error finding container ca34f027fb331c8eb7602b41b776019cb17782036b24c4f90a52c64649be2210: Status 404 returned error can't find the container with id ca34f027fb331c8eb7602b41b776019cb17782036b24c4f90a52c64649be2210 Sep 14 10:58:59.543421 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:58:59.543382 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-82mbj" event={"ID":"67a35ec5-3b2e-4c8c-825b-48fee1aa18ce","Type":"ContainerStarted","Data":"ca34f027fb331c8eb7602b41b776019cb17782036b24c4f90a52c64649be2210"} Sep 14 10:59:01.558995 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:01.558955 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-kmsjf" event={"ID":"8c677f05-d94a-4ed0-924e-e0b63efe53f5","Type":"ContainerStarted","Data":"3566251b9843116c68a4b409b51df2f66eca1c6f8de764df409daf0ac70de295"} Sep 14 10:59:01.559487 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:01.559067 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-kmsjf" Sep 14 10:59:01.584579 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:01.584516 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-kmsjf" podStartSLOduration=2.078275973 podStartE2EDuration="4.584496746s" totalImagesPullingTime="2.506220773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:58:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:59:01.582702623 +0000 UTC m=+615.325104597" watchObservedRunningTime="2026-09-14 10:59:01.584496746 +0000 UTC m=+615.326898715" Sep 14 10:59:02.564469 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:02.564430 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-82mbj" event={"ID":"67a35ec5-3b2e-4c8c-825b-48fee1aa18ce","Type":"ContainerStarted","Data":"64f773393f064e8b332e360d0fbe4bf4b0ba2cccd606890a082ac315d5af44e6"} Sep 14 10:59:02.581099 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:02.581047 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-82mbj" podStartSLOduration=2.256294994 podStartE2EDuration="4.581031386s" totalImagesPullingTime="2.324736392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:58:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:59:02.580435026 +0000 UTC m=+616.322836997" watchObservedRunningTime="2026-09-14 10:59:02.581031386 +0000 UTC m=+616.323433355" Sep 14 10:59:11.895180 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:11.895086 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-kmsjf"] Sep 14 10:59:11.895672 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:11.895334 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-kmsjf" podUID="8c677f05-d94a-4ed0-924e-e0b63efe53f5" containerName="limitador" containerID="cri-o://3566251b9843116c68a4b409b51df2f66eca1c6f8de764df409daf0ac70de295" gracePeriod=30 Sep 14 10:59:11.895986 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:11.895946 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-kmsjf" Sep 14 10:59:12.454572 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:12.454509 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-kmsjf" Sep 14 10:59:12.576555 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:12.576518 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8c677f05-d94a-4ed0-924e-e0b63efe53f5-config-file\") pod \"8c677f05-d94a-4ed0-924e-e0b63efe53f5\" (UID: \"8c677f05-d94a-4ed0-924e-e0b63efe53f5\") " Sep 14 10:59:12.576754 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:12.576639 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knvt6\" (UniqueName: \"kubernetes.io/projected/8c677f05-d94a-4ed0-924e-e0b63efe53f5-kube-api-access-knvt6\") pod \"8c677f05-d94a-4ed0-924e-e0b63efe53f5\" (UID: \"8c677f05-d94a-4ed0-924e-e0b63efe53f5\") " Sep 14 10:59:12.576946 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:12.576919 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c677f05-d94a-4ed0-924e-e0b63efe53f5-config-file" (OuterVolumeSpecName: "config-file") pod "8c677f05-d94a-4ed0-924e-e0b63efe53f5" (UID: "8c677f05-d94a-4ed0-924e-e0b63efe53f5"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 10:59:12.579078 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:12.579050 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c677f05-d94a-4ed0-924e-e0b63efe53f5-kube-api-access-knvt6" (OuterVolumeSpecName: "kube-api-access-knvt6") pod "8c677f05-d94a-4ed0-924e-e0b63efe53f5" (UID: "8c677f05-d94a-4ed0-924e-e0b63efe53f5"). InnerVolumeSpecName "kube-api-access-knvt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:59:12.611026 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:12.610992 2581 generic.go:356] "Generic (PLEG): container finished" podID="8c677f05-d94a-4ed0-924e-e0b63efe53f5" containerID="3566251b9843116c68a4b409b51df2f66eca1c6f8de764df409daf0ac70de295" exitCode=0 Sep 14 10:59:12.611185 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:12.611076 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-kmsjf" Sep 14 10:59:12.611185 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:12.611084 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-kmsjf" event={"ID":"8c677f05-d94a-4ed0-924e-e0b63efe53f5","Type":"ContainerDied","Data":"3566251b9843116c68a4b409b51df2f66eca1c6f8de764df409daf0ac70de295"} Sep 14 10:59:12.611185 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:12.611173 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-kmsjf" event={"ID":"8c677f05-d94a-4ed0-924e-e0b63efe53f5","Type":"ContainerDied","Data":"7a0e44a1f35e6a158e6fbef3381f1342ce69ca12766b9b033ea920becfe0a2a6"} Sep 14 10:59:12.611414 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:12.611201 2581 scope.go:117] "RemoveContainer" containerID="3566251b9843116c68a4b409b51df2f66eca1c6f8de764df409daf0ac70de295" Sep 14 10:59:12.620456 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:12.620432 2581 scope.go:117] "RemoveContainer" containerID="3566251b9843116c68a4b409b51df2f66eca1c6f8de764df409daf0ac70de295" Sep 14 10:59:12.620759 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:59:12.620736 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3566251b9843116c68a4b409b51df2f66eca1c6f8de764df409daf0ac70de295\": container with ID starting with 3566251b9843116c68a4b409b51df2f66eca1c6f8de764df409daf0ac70de295 not found: ID does not exist" containerID="3566251b9843116c68a4b409b51df2f66eca1c6f8de764df409daf0ac70de295" Sep 14 10:59:12.620800 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:12.620772 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3566251b9843116c68a4b409b51df2f66eca1c6f8de764df409daf0ac70de295"} err="failed to get container status \"3566251b9843116c68a4b409b51df2f66eca1c6f8de764df409daf0ac70de295\": rpc error: code = NotFound desc = could not find container \"3566251b9843116c68a4b409b51df2f66eca1c6f8de764df409daf0ac70de295\": container with ID starting with 3566251b9843116c68a4b409b51df2f66eca1c6f8de764df409daf0ac70de295 not found: ID does not exist" Sep 14 10:59:12.636568 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:12.636544 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-kmsjf"] Sep 14 10:59:12.641628 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:12.641604 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-kmsjf"] Sep 14 10:59:12.677597 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:12.677559 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knvt6\" (UniqueName: \"kubernetes.io/projected/8c677f05-d94a-4ed0-924e-e0b63efe53f5-kube-api-access-knvt6\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:59:12.677597 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:12.677593 2581 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8c677f05-d94a-4ed0-924e-e0b63efe53f5-config-file\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:59:12.832303 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:12.832274 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c677f05-d94a-4ed0-924e-e0b63efe53f5" path="/var/lib/kubelet/pods/8c677f05-d94a-4ed0-924e-e0b63efe53f5/volumes" Sep 14 10:59:17.879235 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:17.879200 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-s29t2"] Sep 14 10:59:17.879617 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:17.879587 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c677f05-d94a-4ed0-924e-e0b63efe53f5" containerName="limitador" Sep 14 10:59:17.879617 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:17.879600 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c677f05-d94a-4ed0-924e-e0b63efe53f5" containerName="limitador" Sep 14 10:59:17.879685 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:17.879672 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c677f05-d94a-4ed0-924e-e0b63efe53f5" containerName="limitador" Sep 14 10:59:17.883991 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:17.883974 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-s29t2" Sep 14 10:59:17.887124 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:17.887105 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-22pkb\"" Sep 14 10:59:17.887416 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:17.887397 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 10:59:17.888112 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:17.888096 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 14 10:59:17.888326 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:17.888313 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 14 10:59:17.897554 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:17.897534 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-s29t2"] Sep 14 10:59:18.024235 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:18.024207 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5a2ce312-ab2f-4f61-86d1-730e597f3b63-data\") pod \"postgres-69fb4594cf-s29t2\" (UID: \"5a2ce312-ab2f-4f61-86d1-730e597f3b63\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-s29t2" Sep 14 10:59:18.024438 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:18.024270 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75kgj\" (UniqueName: \"kubernetes.io/projected/5a2ce312-ab2f-4f61-86d1-730e597f3b63-kube-api-access-75kgj\") pod \"postgres-69fb4594cf-s29t2\" (UID: \"5a2ce312-ab2f-4f61-86d1-730e597f3b63\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-s29t2" Sep 14 10:59:18.125729 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:18.125695 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5a2ce312-ab2f-4f61-86d1-730e597f3b63-data\") pod \"postgres-69fb4594cf-s29t2\" (UID: \"5a2ce312-ab2f-4f61-86d1-730e597f3b63\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-s29t2" Sep 14 10:59:18.125815 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:18.125786 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75kgj\" (UniqueName: \"kubernetes.io/projected/5a2ce312-ab2f-4f61-86d1-730e597f3b63-kube-api-access-75kgj\") pod \"postgres-69fb4594cf-s29t2\" (UID: \"5a2ce312-ab2f-4f61-86d1-730e597f3b63\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-s29t2" Sep 14 10:59:18.126236 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:18.126209 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5a2ce312-ab2f-4f61-86d1-730e597f3b63-data\") pod \"postgres-69fb4594cf-s29t2\" (UID: \"5a2ce312-ab2f-4f61-86d1-730e597f3b63\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-s29t2" Sep 14 10:59:18.135248 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:18.135185 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75kgj\" (UniqueName: \"kubernetes.io/projected/5a2ce312-ab2f-4f61-86d1-730e597f3b63-kube-api-access-75kgj\") pod \"postgres-69fb4594cf-s29t2\" (UID: \"5a2ce312-ab2f-4f61-86d1-730e597f3b63\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-s29t2" Sep 14 10:59:18.194241 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:18.194215 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-s29t2" Sep 14 10:59:18.469694 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:18.469670 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-s29t2"] Sep 14 10:59:18.471995 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:59:18.471967 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a2ce312_ab2f_4f61_86d1_730e597f3b63.slice/crio-ccbf8c3b0199318e016a666aa2bc8d0477f742dbc207743a769d4cd20d42de23 WatchSource:0}: Error finding container ccbf8c3b0199318e016a666aa2bc8d0477f742dbc207743a769d4cd20d42de23: Status 404 returned error can't find the container with id ccbf8c3b0199318e016a666aa2bc8d0477f742dbc207743a769d4cd20d42de23 Sep 14 10:59:18.638960 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:18.638922 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-s29t2" event={"ID":"5a2ce312-ab2f-4f61-86d1-730e597f3b63","Type":"ContainerStarted","Data":"ccbf8c3b0199318e016a666aa2bc8d0477f742dbc207743a769d4cd20d42de23"} Sep 14 10:59:20.650421 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:20.650385 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-s29t2" event={"ID":"5a2ce312-ab2f-4f61-86d1-730e597f3b63","Type":"ContainerStarted","Data":"cf89ce8ad89505060dd8840b801a406b4c0917906a3684550a46db2f32daa7db"} Sep 14 10:59:20.650786 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:20.650462 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-s29t2" Sep 14 10:59:20.690359 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:20.690287 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-s29t2" podStartSLOduration=1.6246389479999999 podStartE2EDuration="3.690270032s" totalImagesPullingTime="2.065631084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:59:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:59:20.688757437 +0000 UTC m=+634.431159406" watchObservedRunningTime="2026-09-14 10:59:20.690270032 +0000 UTC m=+634.432672001" Sep 14 10:59:26.684367 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:26.684308 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-s29t2" Sep 14 10:59:27.219682 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:27.219646 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-wqg5m"] Sep 14 10:59:27.222843 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:27.222828 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-wqg5m" Sep 14 10:59:27.230270 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:27.230247 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-wqg5m"] Sep 14 10:59:27.317288 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:27.317253 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69l96\" (UniqueName: \"kubernetes.io/projected/1f6d94a5-2f21-438e-8a7f-5f45cb19d611-kube-api-access-69l96\") pod \"authorino-557f876f69-wqg5m\" (UID: \"1f6d94a5-2f21-438e-8a7f-5f45cb19d611\") " pod="kuadrant-system/authorino-557f876f69-wqg5m" Sep 14 10:59:27.361754 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:27.361721 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-wqg5m"] Sep 14 10:59:27.362005 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:59:27.361987 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-69l96], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-wqg5m" podUID="1f6d94a5-2f21-438e-8a7f-5f45cb19d611" Sep 14 10:59:27.395128 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:27.395094 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-nr4wk"] Sep 14 10:59:27.398455 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:27.398438 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-nr4wk" Sep 14 10:59:27.400989 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:27.400967 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 14 10:59:27.409525 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:27.409501 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-nr4wk"] Sep 14 10:59:27.417941 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:27.417920 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69l96\" (UniqueName: \"kubernetes.io/projected/1f6d94a5-2f21-438e-8a7f-5f45cb19d611-kube-api-access-69l96\") pod \"authorino-557f876f69-wqg5m\" (UID: \"1f6d94a5-2f21-438e-8a7f-5f45cb19d611\") " pod="kuadrant-system/authorino-557f876f69-wqg5m" Sep 14 10:59:27.428839 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:27.428805 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69l96\" (UniqueName: \"kubernetes.io/projected/1f6d94a5-2f21-438e-8a7f-5f45cb19d611-kube-api-access-69l96\") pod \"authorino-557f876f69-wqg5m\" (UID: \"1f6d94a5-2f21-438e-8a7f-5f45cb19d611\") " pod="kuadrant-system/authorino-557f876f69-wqg5m" Sep 14 10:59:27.469743 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:27.469650 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-nr4wk"] Sep 14 10:59:27.469941 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:59:27.469922 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-pqh4b tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-nr4wk" podUID="22e750c4-d96d-450e-ba2a-04ff7a08806d" Sep 14 10:59:27.498708 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:27.498675 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7c78cff58-m5xmd"] Sep 14 10:59:27.501923 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:27.501906 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c78cff58-m5xmd" Sep 14 10:59:27.510933 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:27.510908 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c78cff58-m5xmd"] Sep 14 10:59:27.518944 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:27.518919 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/22e750c4-d96d-450e-ba2a-04ff7a08806d-tls-cert\") pod \"authorino-6d4dfc649f-nr4wk\" (UID: \"22e750c4-d96d-450e-ba2a-04ff7a08806d\") " pod="kuadrant-system/authorino-6d4dfc649f-nr4wk" Sep 14 10:59:27.519072 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:27.519003 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqh4b\" (UniqueName: \"kubernetes.io/projected/22e750c4-d96d-450e-ba2a-04ff7a08806d-kube-api-access-pqh4b\") pod \"authorino-6d4dfc649f-nr4wk\" (UID: \"22e750c4-d96d-450e-ba2a-04ff7a08806d\") " pod="kuadrant-system/authorino-6d4dfc649f-nr4wk" Sep 14 10:59:27.619870 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:27.619832 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqh4b\" (UniqueName: \"kubernetes.io/projected/22e750c4-d96d-450e-ba2a-04ff7a08806d-kube-api-access-pqh4b\") pod \"authorino-6d4dfc649f-nr4wk\" (UID: \"22e750c4-d96d-450e-ba2a-04ff7a08806d\") " pod="kuadrant-system/authorino-6d4dfc649f-nr4wk" Sep 14 10:59:27.620051 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:27.619873 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5q7b\" (UniqueName: \"kubernetes.io/projected/cdaea662-41c0-4645-ad92-308cfd280049-kube-api-access-m5q7b\") pod \"authorino-7c78cff58-m5xmd\" (UID: \"cdaea662-41c0-4645-ad92-308cfd280049\") " pod="kuadrant-system/authorino-7c78cff58-m5xmd" Sep 14 10:59:27.620051 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:27.619909 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cdaea662-41c0-4645-ad92-308cfd280049-tls-cert\") pod \"authorino-7c78cff58-m5xmd\" (UID: \"cdaea662-41c0-4645-ad92-308cfd280049\") " pod="kuadrant-system/authorino-7c78cff58-m5xmd" Sep 14 10:59:27.620051 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:27.619977 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/22e750c4-d96d-450e-ba2a-04ff7a08806d-tls-cert\") pod \"authorino-6d4dfc649f-nr4wk\" (UID: \"22e750c4-d96d-450e-ba2a-04ff7a08806d\") " pod="kuadrant-system/authorino-6d4dfc649f-nr4wk" Sep 14 10:59:27.622583 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:27.622565 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/22e750c4-d96d-450e-ba2a-04ff7a08806d-tls-cert\") pod \"authorino-6d4dfc649f-nr4wk\" (UID: \"22e750c4-d96d-450e-ba2a-04ff7a08806d\") " pod="kuadrant-system/authorino-6d4dfc649f-nr4wk" Sep 14 10:59:27.631066 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:27.631043 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqh4b\" (UniqueName: \"kubernetes.io/projected/22e750c4-d96d-450e-ba2a-04ff7a08806d-kube-api-access-pqh4b\") pod \"authorino-6d4dfc649f-nr4wk\" (UID: \"22e750c4-d96d-450e-ba2a-04ff7a08806d\") " pod="kuadrant-system/authorino-6d4dfc649f-nr4wk" Sep 14 10:59:27.680977 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:27.680942 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-wqg5m" Sep 14 10:59:27.681176 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:27.680942 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-nr4wk" Sep 14 10:59:27.688063 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:27.688036 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-wqg5m" Sep 14 10:59:27.691056 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:27.691037 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-nr4wk" Sep 14 10:59:27.720880 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:27.720800 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5q7b\" (UniqueName: \"kubernetes.io/projected/cdaea662-41c0-4645-ad92-308cfd280049-kube-api-access-m5q7b\") pod \"authorino-7c78cff58-m5xmd\" (UID: \"cdaea662-41c0-4645-ad92-308cfd280049\") " pod="kuadrant-system/authorino-7c78cff58-m5xmd" Sep 14 10:59:27.720880 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:27.720831 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cdaea662-41c0-4645-ad92-308cfd280049-tls-cert\") pod \"authorino-7c78cff58-m5xmd\" (UID: \"cdaea662-41c0-4645-ad92-308cfd280049\") " pod="kuadrant-system/authorino-7c78cff58-m5xmd" Sep 14 10:59:27.723313 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:27.723292 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cdaea662-41c0-4645-ad92-308cfd280049-tls-cert\") pod \"authorino-7c78cff58-m5xmd\" (UID: \"cdaea662-41c0-4645-ad92-308cfd280049\") " pod="kuadrant-system/authorino-7c78cff58-m5xmd" Sep 14 10:59:27.729572 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:27.729530 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5q7b\" (UniqueName: \"kubernetes.io/projected/cdaea662-41c0-4645-ad92-308cfd280049-kube-api-access-m5q7b\") pod \"authorino-7c78cff58-m5xmd\" (UID: \"cdaea662-41c0-4645-ad92-308cfd280049\") " pod="kuadrant-system/authorino-7c78cff58-m5xmd" Sep 14 10:59:27.811758 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:27.811716 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c78cff58-m5xmd" Sep 14 10:59:27.821979 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:27.821952 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/22e750c4-d96d-450e-ba2a-04ff7a08806d-tls-cert\") pod \"22e750c4-d96d-450e-ba2a-04ff7a08806d\" (UID: \"22e750c4-d96d-450e-ba2a-04ff7a08806d\") " Sep 14 10:59:27.822093 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:27.822015 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqh4b\" (UniqueName: \"kubernetes.io/projected/22e750c4-d96d-450e-ba2a-04ff7a08806d-kube-api-access-pqh4b\") pod \"22e750c4-d96d-450e-ba2a-04ff7a08806d\" (UID: \"22e750c4-d96d-450e-ba2a-04ff7a08806d\") " Sep 14 10:59:27.822093 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:27.822065 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69l96\" (UniqueName: \"kubernetes.io/projected/1f6d94a5-2f21-438e-8a7f-5f45cb19d611-kube-api-access-69l96\") pod \"1f6d94a5-2f21-438e-8a7f-5f45cb19d611\" (UID: \"1f6d94a5-2f21-438e-8a7f-5f45cb19d611\") " Sep 14 10:59:27.824225 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:27.824199 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f6d94a5-2f21-438e-8a7f-5f45cb19d611-kube-api-access-69l96" (OuterVolumeSpecName: "kube-api-access-69l96") pod "1f6d94a5-2f21-438e-8a7f-5f45cb19d611" (UID: "1f6d94a5-2f21-438e-8a7f-5f45cb19d611"). InnerVolumeSpecName "kube-api-access-69l96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:59:27.824225 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:27.824199 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22e750c4-d96d-450e-ba2a-04ff7a08806d-kube-api-access-pqh4b" (OuterVolumeSpecName: "kube-api-access-pqh4b") pod "22e750c4-d96d-450e-ba2a-04ff7a08806d" (UID: "22e750c4-d96d-450e-ba2a-04ff7a08806d"). InnerVolumeSpecName "kube-api-access-pqh4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:59:27.824383 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:27.824228 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22e750c4-d96d-450e-ba2a-04ff7a08806d-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "22e750c4-d96d-450e-ba2a-04ff7a08806d" (UID: "22e750c4-d96d-450e-ba2a-04ff7a08806d"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 10:59:27.923176 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:27.923139 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69l96\" (UniqueName: \"kubernetes.io/projected/1f6d94a5-2f21-438e-8a7f-5f45cb19d611-kube-api-access-69l96\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:59:27.923176 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:27.923169 2581 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/22e750c4-d96d-450e-ba2a-04ff7a08806d-tls-cert\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:59:27.923176 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:27.923184 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqh4b\" (UniqueName: \"kubernetes.io/projected/22e750c4-d96d-450e-ba2a-04ff7a08806d-kube-api-access-pqh4b\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:59:28.082165 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:28.082131 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c78cff58-m5xmd"] Sep 14 10:59:28.085820 ip-10-0-137-53 kubenswrapper[2581]: W0914 10:59:28.085781 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcdaea662_41c0_4645_ad92_308cfd280049.slice/crio-2cc4430504612355520791a69724fbbbf2a5527043df0ce2369e0e890669bba6 WatchSource:0}: Error finding container 2cc4430504612355520791a69724fbbbf2a5527043df0ce2369e0e890669bba6: Status 404 returned error can't find the container with id 2cc4430504612355520791a69724fbbbf2a5527043df0ce2369e0e890669bba6 Sep 14 10:59:28.695204 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:28.695097 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c78cff58-m5xmd" event={"ID":"cdaea662-41c0-4645-ad92-308cfd280049","Type":"ContainerStarted","Data":"7291896432ca5de9cb5d060ef9fb411dd2ee86d0c7588802afeb943f31b8cfb6"} Sep 14 10:59:28.695204 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:28.695154 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c78cff58-m5xmd" event={"ID":"cdaea662-41c0-4645-ad92-308cfd280049","Type":"ContainerStarted","Data":"2cc4430504612355520791a69724fbbbf2a5527043df0ce2369e0e890669bba6"} Sep 14 10:59:28.695735 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:28.695258 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-wqg5m" Sep 14 10:59:28.695735 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:28.695479 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-nr4wk" Sep 14 10:59:28.720033 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:28.719977 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7c78cff58-m5xmd" podStartSLOduration=1.271323886 podStartE2EDuration="1.719960052s" totalImagesPullingTime="448.636166ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 10:59:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 10:59:28.719169293 +0000 UTC m=+642.461571263" watchObservedRunningTime="2026-09-14 10:59:28.719960052 +0000 UTC m=+642.462362019" Sep 14 10:59:28.772956 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:28.772923 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-82mbj"] Sep 14 10:59:28.773133 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:28.773104 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-82mbj" podUID="67a35ec5-3b2e-4c8c-825b-48fee1aa18ce" containerName="authorino" containerID="cri-o://64f773393f064e8b332e360d0fbe4bf4b0ba2cccd606890a082ac315d5af44e6" gracePeriod=30 Sep 14 10:59:28.775672 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:28.775644 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-wqg5m"] Sep 14 10:59:28.787330 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:28.787300 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-wqg5m"] Sep 14 10:59:28.818317 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:28.818287 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-nr4wk"] Sep 14 10:59:28.820480 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:28.820455 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-nr4wk"] Sep 14 10:59:28.831485 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:28.831458 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f6d94a5-2f21-438e-8a7f-5f45cb19d611" path="/var/lib/kubelet/pods/1f6d94a5-2f21-438e-8a7f-5f45cb19d611/volumes" Sep 14 10:59:28.831742 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:28.831729 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22e750c4-d96d-450e-ba2a-04ff7a08806d" path="/var/lib/kubelet/pods/22e750c4-d96d-450e-ba2a-04ff7a08806d/volumes" Sep 14 10:59:29.002631 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:29.002606 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-82mbj" Sep 14 10:59:29.133964 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:29.133927 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9nrkd\" (UniqueName: \"kubernetes.io/projected/67a35ec5-3b2e-4c8c-825b-48fee1aa18ce-kube-api-access-9nrkd\") pod \"67a35ec5-3b2e-4c8c-825b-48fee1aa18ce\" (UID: \"67a35ec5-3b2e-4c8c-825b-48fee1aa18ce\") " Sep 14 10:59:29.136175 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:29.136152 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67a35ec5-3b2e-4c8c-825b-48fee1aa18ce-kube-api-access-9nrkd" (OuterVolumeSpecName: "kube-api-access-9nrkd") pod "67a35ec5-3b2e-4c8c-825b-48fee1aa18ce" (UID: "67a35ec5-3b2e-4c8c-825b-48fee1aa18ce"). InnerVolumeSpecName "kube-api-access-9nrkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 10:59:29.235594 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:29.235553 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nrkd\" (UniqueName: \"kubernetes.io/projected/67a35ec5-3b2e-4c8c-825b-48fee1aa18ce-kube-api-access-9nrkd\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 10:59:29.701159 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:29.701071 2581 generic.go:356] "Generic (PLEG): container finished" podID="67a35ec5-3b2e-4c8c-825b-48fee1aa18ce" containerID="64f773393f064e8b332e360d0fbe4bf4b0ba2cccd606890a082ac315d5af44e6" exitCode=0 Sep 14 10:59:29.701159 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:29.701115 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-82mbj" Sep 14 10:59:29.701652 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:29.701156 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-82mbj" event={"ID":"67a35ec5-3b2e-4c8c-825b-48fee1aa18ce","Type":"ContainerDied","Data":"64f773393f064e8b332e360d0fbe4bf4b0ba2cccd606890a082ac315d5af44e6"} Sep 14 10:59:29.701652 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:29.701202 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-82mbj" event={"ID":"67a35ec5-3b2e-4c8c-825b-48fee1aa18ce","Type":"ContainerDied","Data":"ca34f027fb331c8eb7602b41b776019cb17782036b24c4f90a52c64649be2210"} Sep 14 10:59:29.701652 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:29.701223 2581 scope.go:117] "RemoveContainer" containerID="64f773393f064e8b332e360d0fbe4bf4b0ba2cccd606890a082ac315d5af44e6" Sep 14 10:59:29.710280 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:29.710263 2581 scope.go:117] "RemoveContainer" containerID="64f773393f064e8b332e360d0fbe4bf4b0ba2cccd606890a082ac315d5af44e6" Sep 14 10:59:29.710582 ip-10-0-137-53 kubenswrapper[2581]: E0914 10:59:29.710565 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64f773393f064e8b332e360d0fbe4bf4b0ba2cccd606890a082ac315d5af44e6\": container with ID starting with 64f773393f064e8b332e360d0fbe4bf4b0ba2cccd606890a082ac315d5af44e6 not found: ID does not exist" containerID="64f773393f064e8b332e360d0fbe4bf4b0ba2cccd606890a082ac315d5af44e6" Sep 14 10:59:29.710640 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:29.710590 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64f773393f064e8b332e360d0fbe4bf4b0ba2cccd606890a082ac315d5af44e6"} err="failed to get container status \"64f773393f064e8b332e360d0fbe4bf4b0ba2cccd606890a082ac315d5af44e6\": rpc error: code = NotFound desc = could not find container \"64f773393f064e8b332e360d0fbe4bf4b0ba2cccd606890a082ac315d5af44e6\": container with ID starting with 64f773393f064e8b332e360d0fbe4bf4b0ba2cccd606890a082ac315d5af44e6 not found: ID does not exist" Sep 14 10:59:29.722821 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:29.722799 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-82mbj"] Sep 14 10:59:29.730366 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:29.730330 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-82mbj"] Sep 14 10:59:30.831371 ip-10-0-137-53 kubenswrapper[2581]: I0914 10:59:30.831321 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67a35ec5-3b2e-4c8c-825b-48fee1aa18ce" path="/var/lib/kubelet/pods/67a35ec5-3b2e-4c8c-825b-48fee1aa18ce/volumes" Sep 14 11:00:12.106069 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:12.106028 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-xkx49"] Sep 14 11:00:12.106597 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:12.106422 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67a35ec5-3b2e-4c8c-825b-48fee1aa18ce" containerName="authorino" Sep 14 11:00:12.106597 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:12.106436 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="67a35ec5-3b2e-4c8c-825b-48fee1aa18ce" containerName="authorino" Sep 14 11:00:12.106597 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:12.106515 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="67a35ec5-3b2e-4c8c-825b-48fee1aa18ce" containerName="authorino" Sep 14 11:00:12.109564 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:12.109547 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xkx49" Sep 14 11:00:12.113926 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:12.113900 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 14 11:00:12.117719 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:12.117698 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-tj27z\"" Sep 14 11:00:12.125849 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:12.125816 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-xkx49"] Sep 14 11:00:12.222719 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:12.222671 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjsn4\" (UniqueName: \"kubernetes.io/projected/b03e91e7-a889-45d6-81d9-6937b92a85aa-kube-api-access-cjsn4\") pod \"payload-pre-processing-59fbc47bf8-xkx49\" (UID: \"b03e91e7-a889-45d6-81d9-6937b92a85aa\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xkx49" Sep 14 11:00:12.222896 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:12.222826 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b03e91e7-a889-45d6-81d9-6937b92a85aa-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-xkx49\" (UID: \"b03e91e7-a889-45d6-81d9-6937b92a85aa\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xkx49" Sep 14 11:00:12.323737 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:12.323700 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b03e91e7-a889-45d6-81d9-6937b92a85aa-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-xkx49\" (UID: \"b03e91e7-a889-45d6-81d9-6937b92a85aa\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xkx49" Sep 14 11:00:12.323922 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:12.323761 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjsn4\" (UniqueName: \"kubernetes.io/projected/b03e91e7-a889-45d6-81d9-6937b92a85aa-kube-api-access-cjsn4\") pod \"payload-pre-processing-59fbc47bf8-xkx49\" (UID: \"b03e91e7-a889-45d6-81d9-6937b92a85aa\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xkx49" Sep 14 11:00:12.324525 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:12.324500 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b03e91e7-a889-45d6-81d9-6937b92a85aa-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-xkx49\" (UID: \"b03e91e7-a889-45d6-81d9-6937b92a85aa\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xkx49" Sep 14 11:00:12.342540 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:12.342507 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjsn4\" (UniqueName: \"kubernetes.io/projected/b03e91e7-a889-45d6-81d9-6937b92a85aa-kube-api-access-cjsn4\") pod \"payload-pre-processing-59fbc47bf8-xkx49\" (UID: \"b03e91e7-a889-45d6-81d9-6937b92a85aa\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xkx49" Sep 14 11:00:12.429760 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:12.429667 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xkx49" Sep 14 11:00:12.720880 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:12.720856 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-xkx49"] Sep 14 11:00:12.723435 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:00:12.723407 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb03e91e7_a889_45d6_81d9_6937b92a85aa.slice/crio-4a5c4adb0f77d9b197480aa38d87e5e6de713b105cb07976c10ef49a2be52626 WatchSource:0}: Error finding container 4a5c4adb0f77d9b197480aa38d87e5e6de713b105cb07976c10ef49a2be52626: Status 404 returned error can't find the container with id 4a5c4adb0f77d9b197480aa38d87e5e6de713b105cb07976c10ef49a2be52626 Sep 14 11:00:12.888727 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:12.888682 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xkx49" event={"ID":"b03e91e7-a889-45d6-81d9-6937b92a85aa","Type":"ContainerStarted","Data":"4a5c4adb0f77d9b197480aa38d87e5e6de713b105cb07976c10ef49a2be52626"} Sep 14 11:00:15.905083 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:15.905049 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xkx49" event={"ID":"b03e91e7-a889-45d6-81d9-6937b92a85aa","Type":"ContainerStarted","Data":"fadb0091b644898ec9ef3c8f12695e0936ab4d3508c4daabfb0613bc44fbdc1a"} Sep 14 11:00:15.905508 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:15.905118 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xkx49" Sep 14 11:00:15.925291 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:15.925242 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xkx49" podStartSLOduration=1.675755032 podStartE2EDuration="3.925225432s" totalImagesPullingTime="2.2494704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:00:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:00:15.92451592 +0000 UTC m=+689.666917892" watchObservedRunningTime="2026-09-14 11:00:15.925225432 +0000 UTC m=+689.667627403" Sep 14 11:00:26.911238 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:26.911210 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xkx49" Sep 14 11:00:33.833988 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:33.833952 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-wvxgt"] Sep 14 11:00:33.837678 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:33.837657 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wvxgt" Sep 14 11:00:33.842356 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:33.842319 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 14 11:00:33.842513 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:33.842385 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-7mlsq\"" Sep 14 11:00:33.843497 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:33.843477 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 14 11:00:33.843599 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:33.843478 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 14 11:00:33.854568 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:33.854542 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-wvxgt"] Sep 14 11:00:33.919081 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:33.919046 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c6eb772-a931-41ca-9ef1-c3e3feaaedd3-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wvxgt\" (UID: \"7c6eb772-a931-41ca-9ef1-c3e3feaaedd3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wvxgt" Sep 14 11:00:33.919243 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:33.919093 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxptl\" (UniqueName: \"kubernetes.io/projected/7c6eb772-a931-41ca-9ef1-c3e3feaaedd3-kube-api-access-nxptl\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wvxgt\" (UID: \"7c6eb772-a931-41ca-9ef1-c3e3feaaedd3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wvxgt" Sep 14 11:00:33.919243 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:33.919114 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c6eb772-a931-41ca-9ef1-c3e3feaaedd3-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wvxgt\" (UID: \"7c6eb772-a931-41ca-9ef1-c3e3feaaedd3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wvxgt" Sep 14 11:00:33.919243 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:33.919178 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c6eb772-a931-41ca-9ef1-c3e3feaaedd3-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wvxgt\" (UID: \"7c6eb772-a931-41ca-9ef1-c3e3feaaedd3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wvxgt" Sep 14 11:00:33.919243 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:33.919205 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c6eb772-a931-41ca-9ef1-c3e3feaaedd3-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wvxgt\" (UID: \"7c6eb772-a931-41ca-9ef1-c3e3feaaedd3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wvxgt" Sep 14 11:00:33.919404 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:33.919254 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c6eb772-a931-41ca-9ef1-c3e3feaaedd3-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wvxgt\" (UID: \"7c6eb772-a931-41ca-9ef1-c3e3feaaedd3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wvxgt" Sep 14 11:00:34.019678 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:34.019645 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c6eb772-a931-41ca-9ef1-c3e3feaaedd3-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wvxgt\" (UID: \"7c6eb772-a931-41ca-9ef1-c3e3feaaedd3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wvxgt" Sep 14 11:00:34.019678 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:34.019689 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c6eb772-a931-41ca-9ef1-c3e3feaaedd3-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wvxgt\" (UID: \"7c6eb772-a931-41ca-9ef1-c3e3feaaedd3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wvxgt" Sep 14 11:00:34.019936 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:34.019707 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c6eb772-a931-41ca-9ef1-c3e3feaaedd3-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wvxgt\" (UID: \"7c6eb772-a931-41ca-9ef1-c3e3feaaedd3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wvxgt" Sep 14 11:00:34.019936 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:34.019740 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c6eb772-a931-41ca-9ef1-c3e3feaaedd3-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wvxgt\" (UID: \"7c6eb772-a931-41ca-9ef1-c3e3feaaedd3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wvxgt" Sep 14 11:00:34.019936 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:34.019778 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c6eb772-a931-41ca-9ef1-c3e3feaaedd3-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wvxgt\" (UID: \"7c6eb772-a931-41ca-9ef1-c3e3feaaedd3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wvxgt" Sep 14 11:00:34.019936 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:34.019810 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxptl\" (UniqueName: \"kubernetes.io/projected/7c6eb772-a931-41ca-9ef1-c3e3feaaedd3-kube-api-access-nxptl\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wvxgt\" (UID: \"7c6eb772-a931-41ca-9ef1-c3e3feaaedd3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wvxgt" Sep 14 11:00:34.020145 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:34.020102 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c6eb772-a931-41ca-9ef1-c3e3feaaedd3-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wvxgt\" (UID: \"7c6eb772-a931-41ca-9ef1-c3e3feaaedd3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wvxgt" Sep 14 11:00:34.020197 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:34.020155 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c6eb772-a931-41ca-9ef1-c3e3feaaedd3-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wvxgt\" (UID: \"7c6eb772-a931-41ca-9ef1-c3e3feaaedd3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wvxgt" Sep 14 11:00:34.020236 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:34.020193 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c6eb772-a931-41ca-9ef1-c3e3feaaedd3-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wvxgt\" (UID: \"7c6eb772-a931-41ca-9ef1-c3e3feaaedd3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wvxgt" Sep 14 11:00:34.022264 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:34.022244 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c6eb772-a931-41ca-9ef1-c3e3feaaedd3-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wvxgt\" (UID: \"7c6eb772-a931-41ca-9ef1-c3e3feaaedd3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wvxgt" Sep 14 11:00:34.022403 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:34.022385 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c6eb772-a931-41ca-9ef1-c3e3feaaedd3-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wvxgt\" (UID: \"7c6eb772-a931-41ca-9ef1-c3e3feaaedd3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wvxgt" Sep 14 11:00:34.028125 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:34.028100 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxptl\" (UniqueName: \"kubernetes.io/projected/7c6eb772-a931-41ca-9ef1-c3e3feaaedd3-kube-api-access-nxptl\") pod \"e2e-distinct-simulated-kserve-59975d69f9-wvxgt\" (UID: \"7c6eb772-a931-41ca-9ef1-c3e3feaaedd3\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wvxgt" Sep 14 11:00:34.148177 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:34.148090 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wvxgt" Sep 14 11:00:34.422492 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:34.422469 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-wvxgt"] Sep 14 11:00:34.425016 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:00:34.424990 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c6eb772_a931_41ca_9ef1_c3e3feaaedd3.slice/crio-86c936f7bb924d5ac00ceea92669d2458c0b37a3fb93e8f068f69406ca266f66 WatchSource:0}: Error finding container 86c936f7bb924d5ac00ceea92669d2458c0b37a3fb93e8f068f69406ca266f66: Status 404 returned error can't find the container with id 86c936f7bb924d5ac00ceea92669d2458c0b37a3fb93e8f068f69406ca266f66 Sep 14 11:00:34.993320 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:34.993287 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wvxgt" event={"ID":"7c6eb772-a931-41ca-9ef1-c3e3feaaedd3","Type":"ContainerStarted","Data":"86c936f7bb924d5ac00ceea92669d2458c0b37a3fb93e8f068f69406ca266f66"} Sep 14 11:00:35.999542 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:35.999505 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wvxgt" event={"ID":"7c6eb772-a931-41ca-9ef1-c3e3feaaedd3","Type":"ContainerStarted","Data":"8e317b2ad55da217d334ee967195f4a4eb7fe9c10df16975199398e32a861832"} Sep 14 11:00:36.018663 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:36.018614 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wvxgt" podStartSLOduration=1.628085573 podStartE2EDuration="3.018597669s" totalImagesPullingTime="1.390512096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:00:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:00:36.017022299 +0000 UTC m=+709.759424282" watchObservedRunningTime="2026-09-14 11:00:36.018597669 +0000 UTC m=+709.760999638" Sep 14 11:00:44.148813 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:44.148723 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wvxgt" Sep 14 11:00:44.148813 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:44.148799 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wvxgt" Sep 14 11:00:44.161289 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:44.161264 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wvxgt" Sep 14 11:00:45.049441 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:00:45.049409 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-wvxgt" Sep 14 11:01:12.738721 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:12.738676 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-szb5g"] Sep 14 11:01:12.742224 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:12.742208 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szb5g" Sep 14 11:01:12.745031 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:12.745006 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 14 11:01:12.761936 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:12.761912 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-szb5g"] Sep 14 11:01:12.766469 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:12.766443 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3312c8da-f19a-4279-9e7f-d1799c31c8d5-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-szb5g\" (UID: \"3312c8da-f19a-4279-9e7f-d1799c31c8d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szb5g" Sep 14 11:01:12.766600 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:12.766482 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3312c8da-f19a-4279-9e7f-d1799c31c8d5-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-szb5g\" (UID: \"3312c8da-f19a-4279-9e7f-d1799c31c8d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szb5g" Sep 14 11:01:12.766600 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:12.766502 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m8rd\" (UniqueName: \"kubernetes.io/projected/3312c8da-f19a-4279-9e7f-d1799c31c8d5-kube-api-access-4m8rd\") pod \"e2e-trlp-test-simulated-kserve-598466548d-szb5g\" (UID: \"3312c8da-f19a-4279-9e7f-d1799c31c8d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szb5g" Sep 14 11:01:12.766709 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:12.766666 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3312c8da-f19a-4279-9e7f-d1799c31c8d5-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-szb5g\" (UID: \"3312c8da-f19a-4279-9e7f-d1799c31c8d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szb5g" Sep 14 11:01:12.766769 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:12.766704 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3312c8da-f19a-4279-9e7f-d1799c31c8d5-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-szb5g\" (UID: \"3312c8da-f19a-4279-9e7f-d1799c31c8d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szb5g" Sep 14 11:01:12.766769 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:12.766734 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3312c8da-f19a-4279-9e7f-d1799c31c8d5-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-szb5g\" (UID: \"3312c8da-f19a-4279-9e7f-d1799c31c8d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szb5g" Sep 14 11:01:12.867578 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:12.867538 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3312c8da-f19a-4279-9e7f-d1799c31c8d5-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-szb5g\" (UID: \"3312c8da-f19a-4279-9e7f-d1799c31c8d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szb5g" Sep 14 11:01:12.867578 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:12.867577 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3312c8da-f19a-4279-9e7f-d1799c31c8d5-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-szb5g\" (UID: \"3312c8da-f19a-4279-9e7f-d1799c31c8d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szb5g" Sep 14 11:01:12.867819 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:12.867612 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3312c8da-f19a-4279-9e7f-d1799c31c8d5-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-szb5g\" (UID: \"3312c8da-f19a-4279-9e7f-d1799c31c8d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szb5g" Sep 14 11:01:12.867819 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:12.867637 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3312c8da-f19a-4279-9e7f-d1799c31c8d5-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-szb5g\" (UID: \"3312c8da-f19a-4279-9e7f-d1799c31c8d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szb5g" Sep 14 11:01:12.867819 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:12.867658 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m8rd\" (UniqueName: \"kubernetes.io/projected/3312c8da-f19a-4279-9e7f-d1799c31c8d5-kube-api-access-4m8rd\") pod \"e2e-trlp-test-simulated-kserve-598466548d-szb5g\" (UID: \"3312c8da-f19a-4279-9e7f-d1799c31c8d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szb5g" Sep 14 11:01:12.867819 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:12.867754 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3312c8da-f19a-4279-9e7f-d1799c31c8d5-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-szb5g\" (UID: \"3312c8da-f19a-4279-9e7f-d1799c31c8d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szb5g" Sep 14 11:01:12.868083 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:12.868060 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3312c8da-f19a-4279-9e7f-d1799c31c8d5-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-szb5g\" (UID: \"3312c8da-f19a-4279-9e7f-d1799c31c8d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szb5g" Sep 14 11:01:12.868206 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:12.868105 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3312c8da-f19a-4279-9e7f-d1799c31c8d5-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-szb5g\" (UID: \"3312c8da-f19a-4279-9e7f-d1799c31c8d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szb5g" Sep 14 11:01:12.868206 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:12.868114 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3312c8da-f19a-4279-9e7f-d1799c31c8d5-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-szb5g\" (UID: \"3312c8da-f19a-4279-9e7f-d1799c31c8d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szb5g" Sep 14 11:01:12.870500 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:12.870482 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3312c8da-f19a-4279-9e7f-d1799c31c8d5-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-szb5g\" (UID: \"3312c8da-f19a-4279-9e7f-d1799c31c8d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szb5g" Sep 14 11:01:12.870805 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:12.870789 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3312c8da-f19a-4279-9e7f-d1799c31c8d5-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-szb5g\" (UID: \"3312c8da-f19a-4279-9e7f-d1799c31c8d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szb5g" Sep 14 11:01:12.876558 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:12.876538 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m8rd\" (UniqueName: \"kubernetes.io/projected/3312c8da-f19a-4279-9e7f-d1799c31c8d5-kube-api-access-4m8rd\") pod \"e2e-trlp-test-simulated-kserve-598466548d-szb5g\" (UID: \"3312c8da-f19a-4279-9e7f-d1799c31c8d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szb5g" Sep 14 11:01:13.052416 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:13.052381 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szb5g" Sep 14 11:01:13.327380 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:13.327315 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-szb5g"] Sep 14 11:01:13.329810 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:01:13.329774 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3312c8da_f19a_4279_9e7f_d1799c31c8d5.slice/crio-b8b3a2c04418cd4a81da0b48793244e5c0b799da2dd4bc380474c64d95aaabc7 WatchSource:0}: Error finding container b8b3a2c04418cd4a81da0b48793244e5c0b799da2dd4bc380474c64d95aaabc7: Status 404 returned error can't find the container with id b8b3a2c04418cd4a81da0b48793244e5c0b799da2dd4bc380474c64d95aaabc7 Sep 14 11:01:13.331326 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:13.331303 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 11:01:14.169912 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:14.169863 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szb5g" event={"ID":"3312c8da-f19a-4279-9e7f-d1799c31c8d5","Type":"ContainerStarted","Data":"ea5ba858d15b0efd9635e3dd1de87f5afc377d14be33fe079f2fecf7dccd1f0f"} Sep 14 11:01:14.170451 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:14.169969 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szb5g" event={"ID":"3312c8da-f19a-4279-9e7f-d1799c31c8d5","Type":"ContainerStarted","Data":"b8b3a2c04418cd4a81da0b48793244e5c0b799da2dd4bc380474c64d95aaabc7"} Sep 14 11:01:14.196146 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:14.196083 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szb5g" podStartSLOduration=2.006128171 podStartE2EDuration="2.196061903s" totalImagesPullingTime="189.933732ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:01:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:01:14.195647465 +0000 UTC m=+747.938049438" watchObservedRunningTime="2026-09-14 11:01:14.196061903 +0000 UTC m=+747.938463872" Sep 14 11:01:20.729908 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:20.729847 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf"] Sep 14 11:01:20.733693 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:20.733674 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf" Sep 14 11:01:20.736360 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:20.736317 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Sep 14 11:01:20.750689 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:20.750662 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf"] Sep 14 11:01:20.835830 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:20.835783 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f79f3b5-6765-4f84-9325-d766b9baf87c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf\" (UID: \"5f79f3b5-6765-4f84-9325-d766b9baf87c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf" Sep 14 11:01:20.835830 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:20.835820 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f79f3b5-6765-4f84-9325-d766b9baf87c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf\" (UID: \"5f79f3b5-6765-4f84-9325-d766b9baf87c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf" Sep 14 11:01:20.836048 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:20.835856 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f79f3b5-6765-4f84-9325-d766b9baf87c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf\" (UID: \"5f79f3b5-6765-4f84-9325-d766b9baf87c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf" Sep 14 11:01:20.836048 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:20.835910 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g26d9\" (UniqueName: \"kubernetes.io/projected/5f79f3b5-6765-4f84-9325-d766b9baf87c-kube-api-access-g26d9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf\" (UID: \"5f79f3b5-6765-4f84-9325-d766b9baf87c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf" Sep 14 11:01:20.836048 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:20.835984 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f79f3b5-6765-4f84-9325-d766b9baf87c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf\" (UID: \"5f79f3b5-6765-4f84-9325-d766b9baf87c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf" Sep 14 11:01:20.836048 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:20.836014 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f79f3b5-6765-4f84-9325-d766b9baf87c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf\" (UID: \"5f79f3b5-6765-4f84-9325-d766b9baf87c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf" Sep 14 11:01:20.936587 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:20.936547 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f79f3b5-6765-4f84-9325-d766b9baf87c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf\" (UID: \"5f79f3b5-6765-4f84-9325-d766b9baf87c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf" Sep 14 11:01:20.936587 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:20.936581 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f79f3b5-6765-4f84-9325-d766b9baf87c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf\" (UID: \"5f79f3b5-6765-4f84-9325-d766b9baf87c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf" Sep 14 11:01:20.936587 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:20.936597 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f79f3b5-6765-4f84-9325-d766b9baf87c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf\" (UID: \"5f79f3b5-6765-4f84-9325-d766b9baf87c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf" Sep 14 11:01:20.936883 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:20.936717 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g26d9\" (UniqueName: \"kubernetes.io/projected/5f79f3b5-6765-4f84-9325-d766b9baf87c-kube-api-access-g26d9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf\" (UID: \"5f79f3b5-6765-4f84-9325-d766b9baf87c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf" Sep 14 11:01:20.936883 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:20.936812 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f79f3b5-6765-4f84-9325-d766b9baf87c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf\" (UID: \"5f79f3b5-6765-4f84-9325-d766b9baf87c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf" Sep 14 11:01:20.936883 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:20.936851 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f79f3b5-6765-4f84-9325-d766b9baf87c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf\" (UID: \"5f79f3b5-6765-4f84-9325-d766b9baf87c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf" Sep 14 11:01:20.937105 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:20.937031 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f79f3b5-6765-4f84-9325-d766b9baf87c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf\" (UID: \"5f79f3b5-6765-4f84-9325-d766b9baf87c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf" Sep 14 11:01:20.937166 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:20.937146 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f79f3b5-6765-4f84-9325-d766b9baf87c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf\" (UID: \"5f79f3b5-6765-4f84-9325-d766b9baf87c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf" Sep 14 11:01:20.937266 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:20.937246 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f79f3b5-6765-4f84-9325-d766b9baf87c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf\" (UID: \"5f79f3b5-6765-4f84-9325-d766b9baf87c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf" Sep 14 11:01:20.939211 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:20.939191 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f79f3b5-6765-4f84-9325-d766b9baf87c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf\" (UID: \"5f79f3b5-6765-4f84-9325-d766b9baf87c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf" Sep 14 11:01:20.939465 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:20.939446 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f79f3b5-6765-4f84-9325-d766b9baf87c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf\" (UID: \"5f79f3b5-6765-4f84-9325-d766b9baf87c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf" Sep 14 11:01:20.948140 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:20.948120 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g26d9\" (UniqueName: \"kubernetes.io/projected/5f79f3b5-6765-4f84-9325-d766b9baf87c-kube-api-access-g26d9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf\" (UID: \"5f79f3b5-6765-4f84-9325-d766b9baf87c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf" Sep 14 11:01:21.044711 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:21.044672 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf" Sep 14 11:01:21.322828 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:21.322800 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf"] Sep 14 11:01:21.325071 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:01:21.325045 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f79f3b5_6765_4f84_9325_d766b9baf87c.slice/crio-d53b0c67e813ce192c2bf084339fd573d518efc4add3037a802c488a2f6a169f WatchSource:0}: Error finding container d53b0c67e813ce192c2bf084339fd573d518efc4add3037a802c488a2f6a169f: Status 404 returned error can't find the container with id d53b0c67e813ce192c2bf084339fd573d518efc4add3037a802c488a2f6a169f Sep 14 11:01:22.208532 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:22.208499 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf" event={"ID":"5f79f3b5-6765-4f84-9325-d766b9baf87c","Type":"ContainerStarted","Data":"7c8d9dc5e724f543e829a946da0a7eab23f752d8c63f1901f68ff53930f47e9b"} Sep 14 11:01:22.209021 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:22.208557 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf" event={"ID":"5f79f3b5-6765-4f84-9325-d766b9baf87c","Type":"ContainerStarted","Data":"d53b0c67e813ce192c2bf084339fd573d518efc4add3037a802c488a2f6a169f"} Sep 14 11:01:22.230681 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:22.230616 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf" podStartSLOduration=1.985378036 podStartE2EDuration="2.230592824s" totalImagesPullingTime="245.214788ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:01:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:01:22.227929818 +0000 UTC m=+755.970331805" watchObservedRunningTime="2026-09-14 11:01:22.230592824 +0000 UTC m=+755.972994795" Sep 14 11:01:23.053206 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:23.053168 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szb5g" Sep 14 11:01:23.053423 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:23.053237 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szb5g" Sep 14 11:01:23.065865 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:23.065836 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szb5g" Sep 14 11:01:23.224539 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:23.224511 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-szb5g" Sep 14 11:01:31.045512 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:31.045474 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf" Sep 14 11:01:31.045973 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:31.045546 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf" Sep 14 11:01:31.058782 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:31.058751 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf" Sep 14 11:01:31.261409 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:31.261381 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf" Sep 14 11:01:39.446964 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:39.446879 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh"] Sep 14 11:01:39.450413 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:39.450390 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh" Sep 14 11:01:39.454681 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:39.454660 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Sep 14 11:01:39.464018 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:39.463991 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh"] Sep 14 11:01:39.517719 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:39.517687 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/501ae912-66e1-478f-987c-218f6151677f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh\" (UID: \"501ae912-66e1-478f-987c-218f6151677f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh" Sep 14 11:01:39.517901 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:39.517734 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/501ae912-66e1-478f-987c-218f6151677f-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh\" (UID: \"501ae912-66e1-478f-987c-218f6151677f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh" Sep 14 11:01:39.517901 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:39.517760 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/501ae912-66e1-478f-987c-218f6151677f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh\" (UID: \"501ae912-66e1-478f-987c-218f6151677f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh" Sep 14 11:01:39.517901 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:39.517845 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/501ae912-66e1-478f-987c-218f6151677f-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh\" (UID: \"501ae912-66e1-478f-987c-218f6151677f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh" Sep 14 11:01:39.517901 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:39.517863 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/501ae912-66e1-478f-987c-218f6151677f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh\" (UID: \"501ae912-66e1-478f-987c-218f6151677f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh" Sep 14 11:01:39.517901 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:39.517880 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dskdw\" (UniqueName: \"kubernetes.io/projected/501ae912-66e1-478f-987c-218f6151677f-kube-api-access-dskdw\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh\" (UID: \"501ae912-66e1-478f-987c-218f6151677f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh" Sep 14 11:01:39.618735 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:39.618693 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/501ae912-66e1-478f-987c-218f6151677f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh\" (UID: \"501ae912-66e1-478f-987c-218f6151677f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh" Sep 14 11:01:39.618735 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:39.618735 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dskdw\" (UniqueName: \"kubernetes.io/projected/501ae912-66e1-478f-987c-218f6151677f-kube-api-access-dskdw\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh\" (UID: \"501ae912-66e1-478f-987c-218f6151677f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh" Sep 14 11:01:39.618976 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:39.618774 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/501ae912-66e1-478f-987c-218f6151677f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh\" (UID: \"501ae912-66e1-478f-987c-218f6151677f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh" Sep 14 11:01:39.618976 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:39.618818 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/501ae912-66e1-478f-987c-218f6151677f-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh\" (UID: \"501ae912-66e1-478f-987c-218f6151677f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh" Sep 14 11:01:39.618976 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:39.618857 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/501ae912-66e1-478f-987c-218f6151677f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh\" (UID: \"501ae912-66e1-478f-987c-218f6151677f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh" Sep 14 11:01:39.618976 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:39.618965 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/501ae912-66e1-478f-987c-218f6151677f-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh\" (UID: \"501ae912-66e1-478f-987c-218f6151677f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh" Sep 14 11:01:39.619209 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:39.619092 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/501ae912-66e1-478f-987c-218f6151677f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh\" (UID: \"501ae912-66e1-478f-987c-218f6151677f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh" Sep 14 11:01:39.619293 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:39.619271 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/501ae912-66e1-478f-987c-218f6151677f-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh\" (UID: \"501ae912-66e1-478f-987c-218f6151677f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh" Sep 14 11:01:39.619434 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:39.619419 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/501ae912-66e1-478f-987c-218f6151677f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh\" (UID: \"501ae912-66e1-478f-987c-218f6151677f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh" Sep 14 11:01:39.621277 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:39.621254 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/501ae912-66e1-478f-987c-218f6151677f-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh\" (UID: \"501ae912-66e1-478f-987c-218f6151677f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh" Sep 14 11:01:39.621406 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:39.621389 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/501ae912-66e1-478f-987c-218f6151677f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh\" (UID: \"501ae912-66e1-478f-987c-218f6151677f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh" Sep 14 11:01:39.627427 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:39.627404 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dskdw\" (UniqueName: \"kubernetes.io/projected/501ae912-66e1-478f-987c-218f6151677f-kube-api-access-dskdw\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh\" (UID: \"501ae912-66e1-478f-987c-218f6151677f\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh" Sep 14 11:01:39.759564 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:39.759477 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh" Sep 14 11:01:40.031389 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:40.031361 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh"] Sep 14 11:01:40.033034 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:01:40.033011 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod501ae912_66e1_478f_987c_218f6151677f.slice/crio-c87e50b05032bfc781cfe5e114b7f1f891c741c19e65b23ec841ffd6572285b2 WatchSource:0}: Error finding container c87e50b05032bfc781cfe5e114b7f1f891c741c19e65b23ec841ffd6572285b2: Status 404 returned error can't find the container with id c87e50b05032bfc781cfe5e114b7f1f891c741c19e65b23ec841ffd6572285b2 Sep 14 11:01:40.294606 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:40.294525 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh" event={"ID":"501ae912-66e1-478f-987c-218f6151677f","Type":"ContainerStarted","Data":"15f0107bd44e1071cf7af1d6a8da979773f9e527ba65f243050876852f24e8fb"} Sep 14 11:01:40.294606 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:40.294570 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh" event={"ID":"501ae912-66e1-478f-987c-218f6151677f","Type":"ContainerStarted","Data":"c87e50b05032bfc781cfe5e114b7f1f891c741c19e65b23ec841ffd6572285b2"} Sep 14 11:01:40.317280 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:40.317225 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh" podStartSLOduration=1.125447798 podStartE2EDuration="1.317205791s" totalImagesPullingTime="191.757993ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:01:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:01:40.315801683 +0000 UTC m=+774.058203665" watchObservedRunningTime="2026-09-14 11:01:40.317205791 +0000 UTC m=+774.059607775" Sep 14 11:01:49.759847 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:49.759807 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh" Sep 14 11:01:49.760465 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:49.759865 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh" Sep 14 11:01:49.772901 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:49.772877 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh" Sep 14 11:01:50.350693 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:01:50.350658 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh" Sep 14 11:03:21.236201 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:21.236163 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7c78cff58-m5xmd"] Sep 14 11:03:21.236726 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:21.236451 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7c78cff58-m5xmd" podUID="cdaea662-41c0-4645-ad92-308cfd280049" containerName="authorino" containerID="cri-o://7291896432ca5de9cb5d060ef9fb411dd2ee86d0c7588802afeb943f31b8cfb6" gracePeriod=30 Sep 14 11:03:21.479271 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:21.479247 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c78cff58-m5xmd" Sep 14 11:03:21.588627 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:21.588596 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5q7b\" (UniqueName: \"kubernetes.io/projected/cdaea662-41c0-4645-ad92-308cfd280049-kube-api-access-m5q7b\") pod \"cdaea662-41c0-4645-ad92-308cfd280049\" (UID: \"cdaea662-41c0-4645-ad92-308cfd280049\") " Sep 14 11:03:21.588812 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:21.588667 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cdaea662-41c0-4645-ad92-308cfd280049-tls-cert\") pod \"cdaea662-41c0-4645-ad92-308cfd280049\" (UID: \"cdaea662-41c0-4645-ad92-308cfd280049\") " Sep 14 11:03:21.590879 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:21.590846 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdaea662-41c0-4645-ad92-308cfd280049-kube-api-access-m5q7b" (OuterVolumeSpecName: "kube-api-access-m5q7b") pod "cdaea662-41c0-4645-ad92-308cfd280049" (UID: "cdaea662-41c0-4645-ad92-308cfd280049"). InnerVolumeSpecName "kube-api-access-m5q7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:03:21.598840 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:21.598815 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdaea662-41c0-4645-ad92-308cfd280049-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "cdaea662-41c0-4645-ad92-308cfd280049" (UID: "cdaea662-41c0-4645-ad92-308cfd280049"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:03:21.690082 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:21.690050 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5q7b\" (UniqueName: \"kubernetes.io/projected/cdaea662-41c0-4645-ad92-308cfd280049-kube-api-access-m5q7b\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:03:21.690082 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:21.690079 2581 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cdaea662-41c0-4645-ad92-308cfd280049-tls-cert\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:03:21.773015 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:21.772975 2581 generic.go:356] "Generic (PLEG): container finished" podID="cdaea662-41c0-4645-ad92-308cfd280049" containerID="7291896432ca5de9cb5d060ef9fb411dd2ee86d0c7588802afeb943f31b8cfb6" exitCode=0 Sep 14 11:03:21.773195 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:21.773026 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c78cff58-m5xmd" Sep 14 11:03:21.773195 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:21.773098 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c78cff58-m5xmd" event={"ID":"cdaea662-41c0-4645-ad92-308cfd280049","Type":"ContainerDied","Data":"7291896432ca5de9cb5d060ef9fb411dd2ee86d0c7588802afeb943f31b8cfb6"} Sep 14 11:03:21.773195 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:21.773147 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c78cff58-m5xmd" event={"ID":"cdaea662-41c0-4645-ad92-308cfd280049","Type":"ContainerDied","Data":"2cc4430504612355520791a69724fbbbf2a5527043df0ce2369e0e890669bba6"} Sep 14 11:03:21.773195 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:21.773169 2581 scope.go:117] "RemoveContainer" containerID="7291896432ca5de9cb5d060ef9fb411dd2ee86d0c7588802afeb943f31b8cfb6" Sep 14 11:03:21.782172 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:21.782154 2581 scope.go:117] "RemoveContainer" containerID="7291896432ca5de9cb5d060ef9fb411dd2ee86d0c7588802afeb943f31b8cfb6" Sep 14 11:03:21.782521 ip-10-0-137-53 kubenswrapper[2581]: E0914 11:03:21.782500 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7291896432ca5de9cb5d060ef9fb411dd2ee86d0c7588802afeb943f31b8cfb6\": container with ID starting with 7291896432ca5de9cb5d060ef9fb411dd2ee86d0c7588802afeb943f31b8cfb6 not found: ID does not exist" containerID="7291896432ca5de9cb5d060ef9fb411dd2ee86d0c7588802afeb943f31b8cfb6" Sep 14 11:03:21.782572 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:21.782530 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7291896432ca5de9cb5d060ef9fb411dd2ee86d0c7588802afeb943f31b8cfb6"} err="failed to get container status \"7291896432ca5de9cb5d060ef9fb411dd2ee86d0c7588802afeb943f31b8cfb6\": rpc error: code = NotFound desc = could not find container \"7291896432ca5de9cb5d060ef9fb411dd2ee86d0c7588802afeb943f31b8cfb6\": container with ID starting with 7291896432ca5de9cb5d060ef9fb411dd2ee86d0c7588802afeb943f31b8cfb6 not found: ID does not exist" Sep 14 11:03:21.801182 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:21.801155 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7c78cff58-m5xmd"] Sep 14 11:03:21.812376 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:21.812329 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7c78cff58-m5xmd"] Sep 14 11:03:22.832520 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:22.832476 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdaea662-41c0-4645-ad92-308cfd280049" path="/var/lib/kubelet/pods/cdaea662-41c0-4645-ad92-308cfd280049/volumes" Sep 14 11:03:44.575151 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:44.575069 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4ebf73-5567b59786-mzx5m"] Sep 14 11:03:44.575661 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:44.575464 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdaea662-41c0-4645-ad92-308cfd280049" containerName="authorino" Sep 14 11:03:44.575661 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:44.575477 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdaea662-41c0-4645-ad92-308cfd280049" containerName="authorino" Sep 14 11:03:44.575661 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:44.575560 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdaea662-41c0-4645-ad92-308cfd280049" containerName="authorino" Sep 14 11:03:44.578608 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:44.578588 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ebf73-5567b59786-mzx5m" Sep 14 11:03:44.581147 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:44.581126 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-4ebf73\"" Sep 14 11:03:44.581214 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:44.581147 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-4ebf73-dockercfg-694m7\"" Sep 14 11:03:44.598864 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:44.598841 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4ebf73-5567b59786-mzx5m"] Sep 14 11:03:44.603775 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:44.603749 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e4b7137-fa3a-43d9-aeed-fffd3fda99a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-4ebf73-5567b59786-mzx5m\" (UID: \"3e4b7137-fa3a-43d9-aeed-fffd3fda99a3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ebf73-5567b59786-mzx5m" Sep 14 11:03:44.603907 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:44.603798 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzvd7\" (UniqueName: \"kubernetes.io/projected/3e4b7137-fa3a-43d9-aeed-fffd3fda99a3-kube-api-access-zzvd7\") pod \"payload-pre-processing-e2e-worker-w4-4ebf73-5567b59786-mzx5m\" (UID: \"3e4b7137-fa3a-43d9-aeed-fffd3fda99a3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ebf73-5567b59786-mzx5m" Sep 14 11:03:44.685627 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:44.685595 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4ebf73-9687f9869-zpnm6"] Sep 14 11:03:44.689058 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:44.689037 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-4ebf73-9687f9869-zpnm6" Sep 14 11:03:44.704565 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:44.704534 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9017a38-f02d-4997-98ef-078dd6c49977-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-4ebf73-9687f9869-zpnm6\" (UID: \"c9017a38-f02d-4997-98ef-078dd6c49977\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4ebf73-9687f9869-zpnm6" Sep 14 11:03:44.704724 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:44.704597 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e4b7137-fa3a-43d9-aeed-fffd3fda99a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-4ebf73-5567b59786-mzx5m\" (UID: \"3e4b7137-fa3a-43d9-aeed-fffd3fda99a3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ebf73-5567b59786-mzx5m" Sep 14 11:03:44.704724 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:44.704627 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b42x\" (UniqueName: \"kubernetes.io/projected/c9017a38-f02d-4997-98ef-078dd6c49977-kube-api-access-8b42x\") pod \"payload-processing-e2e-worker-w4-4ebf73-9687f9869-zpnm6\" (UID: \"c9017a38-f02d-4997-98ef-078dd6c49977\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4ebf73-9687f9869-zpnm6" Sep 14 11:03:44.704724 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:44.704672 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzvd7\" (UniqueName: \"kubernetes.io/projected/3e4b7137-fa3a-43d9-aeed-fffd3fda99a3-kube-api-access-zzvd7\") pod \"payload-pre-processing-e2e-worker-w4-4ebf73-5567b59786-mzx5m\" (UID: \"3e4b7137-fa3a-43d9-aeed-fffd3fda99a3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ebf73-5567b59786-mzx5m" Sep 14 11:03:44.705459 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:44.705440 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e4b7137-fa3a-43d9-aeed-fffd3fda99a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-4ebf73-5567b59786-mzx5m\" (UID: \"3e4b7137-fa3a-43d9-aeed-fffd3fda99a3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ebf73-5567b59786-mzx5m" Sep 14 11:03:44.705641 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:44.705623 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4ebf73-9687f9869-zpnm6"] Sep 14 11:03:44.721482 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:44.721452 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzvd7\" (UniqueName: \"kubernetes.io/projected/3e4b7137-fa3a-43d9-aeed-fffd3fda99a3-kube-api-access-zzvd7\") pod \"payload-pre-processing-e2e-worker-w4-4ebf73-5567b59786-mzx5m\" (UID: \"3e4b7137-fa3a-43d9-aeed-fffd3fda99a3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ebf73-5567b59786-mzx5m" Sep 14 11:03:44.805647 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:44.805613 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b42x\" (UniqueName: \"kubernetes.io/projected/c9017a38-f02d-4997-98ef-078dd6c49977-kube-api-access-8b42x\") pod \"payload-processing-e2e-worker-w4-4ebf73-9687f9869-zpnm6\" (UID: \"c9017a38-f02d-4997-98ef-078dd6c49977\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4ebf73-9687f9869-zpnm6" Sep 14 11:03:44.805834 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:44.805756 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9017a38-f02d-4997-98ef-078dd6c49977-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-4ebf73-9687f9869-zpnm6\" (UID: \"c9017a38-f02d-4997-98ef-078dd6c49977\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4ebf73-9687f9869-zpnm6" Sep 14 11:03:44.806509 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:44.806483 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9017a38-f02d-4997-98ef-078dd6c49977-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-4ebf73-9687f9869-zpnm6\" (UID: \"c9017a38-f02d-4997-98ef-078dd6c49977\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4ebf73-9687f9869-zpnm6" Sep 14 11:03:44.818621 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:44.818596 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b42x\" (UniqueName: \"kubernetes.io/projected/c9017a38-f02d-4997-98ef-078dd6c49977-kube-api-access-8b42x\") pod \"payload-processing-e2e-worker-w4-4ebf73-9687f9869-zpnm6\" (UID: \"c9017a38-f02d-4997-98ef-078dd6c49977\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4ebf73-9687f9869-zpnm6" Sep 14 11:03:44.916600 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:44.916517 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ebf73-5567b59786-mzx5m" Sep 14 11:03:44.999676 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:44.999641 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-4ebf73-9687f9869-zpnm6" Sep 14 11:03:45.189923 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:45.189896 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4ebf73-5567b59786-mzx5m"] Sep 14 11:03:45.191414 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:03:45.191386 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e4b7137_fa3a_43d9_aeed_fffd3fda99a3.slice/crio-20e8fba5688c081e4d6874635c8595de6fb66e2799289d9cc99d7a921081a196 WatchSource:0}: Error finding container 20e8fba5688c081e4d6874635c8595de6fb66e2799289d9cc99d7a921081a196: Status 404 returned error can't find the container with id 20e8fba5688c081e4d6874635c8595de6fb66e2799289d9cc99d7a921081a196 Sep 14 11:03:45.280040 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:45.280004 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4ebf73-9687f9869-zpnm6"] Sep 14 11:03:45.285107 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:03:45.285080 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9017a38_f02d_4997_98ef_078dd6c49977.slice/crio-e3feacdb50851f744bd07525e1ca574d71700d32f2204a596bc9340a8dedd58c WatchSource:0}: Error finding container e3feacdb50851f744bd07525e1ca574d71700d32f2204a596bc9340a8dedd58c: Status 404 returned error can't find the container with id e3feacdb50851f744bd07525e1ca574d71700d32f2204a596bc9340a8dedd58c Sep 14 11:03:45.886488 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:45.886449 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ebf73-5567b59786-mzx5m" event={"ID":"3e4b7137-fa3a-43d9-aeed-fffd3fda99a3","Type":"ContainerStarted","Data":"24eac3aa3b20b2c94401a26fbe18f3d5bf5c0dbd2722cdcac67aad16a0d1b3f3"} Sep 14 11:03:45.886948 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:45.886518 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ebf73-5567b59786-mzx5m" Sep 14 11:03:45.886948 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:45.886543 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ebf73-5567b59786-mzx5m" event={"ID":"3e4b7137-fa3a-43d9-aeed-fffd3fda99a3","Type":"ContainerStarted","Data":"20e8fba5688c081e4d6874635c8595de6fb66e2799289d9cc99d7a921081a196"} Sep 14 11:03:45.888122 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:45.888081 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-4ebf73-9687f9869-zpnm6" event={"ID":"c9017a38-f02d-4997-98ef-078dd6c49977","Type":"ContainerStarted","Data":"78c11751d331f022030a16bce3e7d4a42096acc784207baaa24bf211803ac241"} Sep 14 11:03:45.888235 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:45.888132 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-4ebf73-9687f9869-zpnm6" event={"ID":"c9017a38-f02d-4997-98ef-078dd6c49977","Type":"ContainerStarted","Data":"e3feacdb50851f744bd07525e1ca574d71700d32f2204a596bc9340a8dedd58c"} Sep 14 11:03:45.888235 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:45.888153 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-4ebf73-9687f9869-zpnm6" Sep 14 11:03:45.914446 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:45.914400 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ebf73-5567b59786-mzx5m" podStartSLOduration=1.9143815690000001 podStartE2EDuration="1.914381569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:03:45.913071658 +0000 UTC m=+899.655473629" watchObservedRunningTime="2026-09-14 11:03:45.914381569 +0000 UTC m=+899.656783540" Sep 14 11:03:46.050424 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:46.050370 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-4ebf73-9687f9869-zpnm6" podStartSLOduration=2.050349472 podStartE2EDuration="2.050349472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:03:45.944481964 +0000 UTC m=+899.686883934" watchObservedRunningTime="2026-09-14 11:03:46.050349472 +0000 UTC m=+899.792751459" Sep 14 11:03:46.050712 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:46.050695 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7984b1-6999cf6c6b-8vbcv"] Sep 14 11:03:46.054022 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:46.054001 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7984b1-6999cf6c6b-8vbcv" Sep 14 11:03:46.057280 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:46.057257 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-7984b1\"" Sep 14 11:03:46.057444 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:46.057258 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-7984b1-dockercfg-xdqhz\"" Sep 14 11:03:46.065711 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:46.065689 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7984b1-6999cf6c6b-8vbcv"] Sep 14 11:03:46.117874 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:46.117839 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c3e8eac-9ead-40d3-a619-2021c3a203de-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-7984b1-6999cf6c6b-8vbcv\" (UID: \"5c3e8eac-9ead-40d3-a619-2021c3a203de\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7984b1-6999cf6c6b-8vbcv" Sep 14 11:03:46.118028 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:46.117882 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vblw\" (UniqueName: \"kubernetes.io/projected/5c3e8eac-9ead-40d3-a619-2021c3a203de-kube-api-access-9vblw\") pod \"payload-pre-processing-e2e-worker-w0-7984b1-6999cf6c6b-8vbcv\" (UID: \"5c3e8eac-9ead-40d3-a619-2021c3a203de\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7984b1-6999cf6c6b-8vbcv" Sep 14 11:03:46.136766 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:46.136696 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7984b1-56cdcd4d45-bfgfv"] Sep 14 11:03:46.140211 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:46.140193 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-7984b1-56cdcd4d45-bfgfv" Sep 14 11:03:46.162641 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:46.162613 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7984b1-56cdcd4d45-bfgfv"] Sep 14 11:03:46.218602 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:46.218567 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znw97\" (UniqueName: \"kubernetes.io/projected/a7e74eea-f3ea-492e-9dab-fdf852dad5c0-kube-api-access-znw97\") pod \"payload-processing-e2e-worker-w0-7984b1-56cdcd4d45-bfgfv\" (UID: \"a7e74eea-f3ea-492e-9dab-fdf852dad5c0\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7984b1-56cdcd4d45-bfgfv" Sep 14 11:03:46.218771 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:46.218611 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7e74eea-f3ea-492e-9dab-fdf852dad5c0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-7984b1-56cdcd4d45-bfgfv\" (UID: \"a7e74eea-f3ea-492e-9dab-fdf852dad5c0\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7984b1-56cdcd4d45-bfgfv" Sep 14 11:03:46.218771 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:46.218709 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c3e8eac-9ead-40d3-a619-2021c3a203de-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-7984b1-6999cf6c6b-8vbcv\" (UID: \"5c3e8eac-9ead-40d3-a619-2021c3a203de\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7984b1-6999cf6c6b-8vbcv" Sep 14 11:03:46.218771 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:46.218754 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vblw\" (UniqueName: \"kubernetes.io/projected/5c3e8eac-9ead-40d3-a619-2021c3a203de-kube-api-access-9vblw\") pod \"payload-pre-processing-e2e-worker-w0-7984b1-6999cf6c6b-8vbcv\" (UID: \"5c3e8eac-9ead-40d3-a619-2021c3a203de\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7984b1-6999cf6c6b-8vbcv" Sep 14 11:03:46.219411 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:46.219393 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c3e8eac-9ead-40d3-a619-2021c3a203de-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-7984b1-6999cf6c6b-8vbcv\" (UID: \"5c3e8eac-9ead-40d3-a619-2021c3a203de\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7984b1-6999cf6c6b-8vbcv" Sep 14 11:03:46.229447 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:46.229423 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vblw\" (UniqueName: \"kubernetes.io/projected/5c3e8eac-9ead-40d3-a619-2021c3a203de-kube-api-access-9vblw\") pod \"payload-pre-processing-e2e-worker-w0-7984b1-6999cf6c6b-8vbcv\" (UID: \"5c3e8eac-9ead-40d3-a619-2021c3a203de\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7984b1-6999cf6c6b-8vbcv" Sep 14 11:03:46.319637 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:46.319588 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znw97\" (UniqueName: \"kubernetes.io/projected/a7e74eea-f3ea-492e-9dab-fdf852dad5c0-kube-api-access-znw97\") pod \"payload-processing-e2e-worker-w0-7984b1-56cdcd4d45-bfgfv\" (UID: \"a7e74eea-f3ea-492e-9dab-fdf852dad5c0\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7984b1-56cdcd4d45-bfgfv" Sep 14 11:03:46.319897 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:46.319791 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7e74eea-f3ea-492e-9dab-fdf852dad5c0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-7984b1-56cdcd4d45-bfgfv\" (UID: \"a7e74eea-f3ea-492e-9dab-fdf852dad5c0\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7984b1-56cdcd4d45-bfgfv" Sep 14 11:03:46.321062 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:46.321032 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7e74eea-f3ea-492e-9dab-fdf852dad5c0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-7984b1-56cdcd4d45-bfgfv\" (UID: \"a7e74eea-f3ea-492e-9dab-fdf852dad5c0\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7984b1-56cdcd4d45-bfgfv" Sep 14 11:03:46.339538 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:46.339511 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znw97\" (UniqueName: \"kubernetes.io/projected/a7e74eea-f3ea-492e-9dab-fdf852dad5c0-kube-api-access-znw97\") pod \"payload-processing-e2e-worker-w0-7984b1-56cdcd4d45-bfgfv\" (UID: \"a7e74eea-f3ea-492e-9dab-fdf852dad5c0\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7984b1-56cdcd4d45-bfgfv" Sep 14 11:03:46.365008 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:46.364980 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7984b1-6999cf6c6b-8vbcv" Sep 14 11:03:46.450530 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:46.450490 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-7984b1-56cdcd4d45-bfgfv" Sep 14 11:03:46.651401 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:46.651367 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7984b1-6999cf6c6b-8vbcv"] Sep 14 11:03:46.655254 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:03:46.655227 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c3e8eac_9ead_40d3_a619_2021c3a203de.slice/crio-ef44d569095ddc5d7b50729fcc7635ddcae7c5150fa4c7b04c80560c6297cc9f WatchSource:0}: Error finding container ef44d569095ddc5d7b50729fcc7635ddcae7c5150fa4c7b04c80560c6297cc9f: Status 404 returned error can't find the container with id ef44d569095ddc5d7b50729fcc7635ddcae7c5150fa4c7b04c80560c6297cc9f Sep 14 11:03:46.733858 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:46.733537 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7984b1-56cdcd4d45-bfgfv"] Sep 14 11:03:46.736938 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:03:46.736906 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7e74eea_f3ea_492e_9dab_fdf852dad5c0.slice/crio-493d4bbcca1d06b222a3becfcf8626c455d41857c479989ef3337cb40bc09b46 WatchSource:0}: Error finding container 493d4bbcca1d06b222a3becfcf8626c455d41857c479989ef3337cb40bc09b46: Status 404 returned error can't find the container with id 493d4bbcca1d06b222a3becfcf8626c455d41857c479989ef3337cb40bc09b46 Sep 14 11:03:46.894775 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:46.894731 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7984b1-6999cf6c6b-8vbcv" event={"ID":"5c3e8eac-9ead-40d3-a619-2021c3a203de","Type":"ContainerStarted","Data":"d8937e07a02f4804ff9ea0b7da1661c955132a9cb6c0ae8515c5c5a873a0b213"} Sep 14 11:03:46.895213 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:46.894798 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7984b1-6999cf6c6b-8vbcv" event={"ID":"5c3e8eac-9ead-40d3-a619-2021c3a203de","Type":"ContainerStarted","Data":"ef44d569095ddc5d7b50729fcc7635ddcae7c5150fa4c7b04c80560c6297cc9f"} Sep 14 11:03:46.895213 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:46.895023 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7984b1-6999cf6c6b-8vbcv" Sep 14 11:03:46.896601 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:46.896576 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-7984b1-56cdcd4d45-bfgfv" event={"ID":"a7e74eea-f3ea-492e-9dab-fdf852dad5c0","Type":"ContainerStarted","Data":"3037e1017bb01e8faf3b3713d471839746e14045eb2e8dd97237daa7f9a5e2e8"} Sep 14 11:03:46.896756 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:46.896620 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-7984b1-56cdcd4d45-bfgfv" event={"ID":"a7e74eea-f3ea-492e-9dab-fdf852dad5c0","Type":"ContainerStarted","Data":"493d4bbcca1d06b222a3becfcf8626c455d41857c479989ef3337cb40bc09b46"} Sep 14 11:03:46.896756 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:46.896647 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-7984b1-56cdcd4d45-bfgfv" Sep 14 11:03:46.920272 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:46.920176 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7984b1-6999cf6c6b-8vbcv" podStartSLOduration=0.920157291 podStartE2EDuration="920.157291ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:03:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:03:46.917476369 +0000 UTC m=+900.659878341" watchObservedRunningTime="2026-09-14 11:03:46.920157291 +0000 UTC m=+900.662559265" Sep 14 11:03:46.951872 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:46.951771 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-7984b1-56cdcd4d45-bfgfv" podStartSLOduration=0.95175176 podStartE2EDuration="951.75176ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:03:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:03:46.948692139 +0000 UTC m=+900.691094111" watchObservedRunningTime="2026-09-14 11:03:46.95175176 +0000 UTC m=+900.694153926" Sep 14 11:03:47.638183 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:47.638150 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-04195c-5d85fcdd86-dlzdz"] Sep 14 11:03:47.642035 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:47.642010 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04195c-5d85fcdd86-dlzdz" Sep 14 11:03:47.647910 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:47.647889 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-04195c-dockercfg-nvnb8\"" Sep 14 11:03:47.648380 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:47.648360 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-04195c\"" Sep 14 11:03:47.682948 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:47.682915 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-04195c-5d85fcdd86-dlzdz"] Sep 14 11:03:47.740613 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:47.740571 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj8ft\" (UniqueName: \"kubernetes.io/projected/899c3f91-4d36-4800-923b-cbab2e572a6c-kube-api-access-pj8ft\") pod \"payload-pre-processing-e2e-inf-a-04195c-5d85fcdd86-dlzdz\" (UID: \"899c3f91-4d36-4800-923b-cbab2e572a6c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04195c-5d85fcdd86-dlzdz" Sep 14 11:03:47.740816 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:47.740630 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/899c3f91-4d36-4800-923b-cbab2e572a6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-04195c-5d85fcdd86-dlzdz\" (UID: \"899c3f91-4d36-4800-923b-cbab2e572a6c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04195c-5d85fcdd86-dlzdz" Sep 14 11:03:47.746319 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:47.746290 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-04195c-559d584fc-6gqvc"] Sep 14 11:03:47.750427 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:47.750407 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-04195c-559d584fc-6gqvc" Sep 14 11:03:47.770721 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:47.770695 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-04195c-559d584fc-6gqvc"] Sep 14 11:03:47.841915 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:47.841881 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj8ft\" (UniqueName: \"kubernetes.io/projected/899c3f91-4d36-4800-923b-cbab2e572a6c-kube-api-access-pj8ft\") pod \"payload-pre-processing-e2e-inf-a-04195c-5d85fcdd86-dlzdz\" (UID: \"899c3f91-4d36-4800-923b-cbab2e572a6c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04195c-5d85fcdd86-dlzdz" Sep 14 11:03:47.842098 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:47.841936 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/899c3f91-4d36-4800-923b-cbab2e572a6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-04195c-5d85fcdd86-dlzdz\" (UID: \"899c3f91-4d36-4800-923b-cbab2e572a6c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04195c-5d85fcdd86-dlzdz" Sep 14 11:03:47.842158 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:47.842122 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jwbh\" (UniqueName: \"kubernetes.io/projected/b870f894-e235-49c4-a502-c9dd63500b2e-kube-api-access-8jwbh\") pod \"payload-processing-e2e-inf-a-04195c-559d584fc-6gqvc\" (UID: \"b870f894-e235-49c4-a502-c9dd63500b2e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-04195c-559d584fc-6gqvc" Sep 14 11:03:47.842211 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:47.842174 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b870f894-e235-49c4-a502-c9dd63500b2e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-04195c-559d584fc-6gqvc\" (UID: \"b870f894-e235-49c4-a502-c9dd63500b2e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-04195c-559d584fc-6gqvc" Sep 14 11:03:47.842757 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:47.842734 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/899c3f91-4d36-4800-923b-cbab2e572a6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-04195c-5d85fcdd86-dlzdz\" (UID: \"899c3f91-4d36-4800-923b-cbab2e572a6c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04195c-5d85fcdd86-dlzdz" Sep 14 11:03:47.852131 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:47.852095 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj8ft\" (UniqueName: \"kubernetes.io/projected/899c3f91-4d36-4800-923b-cbab2e572a6c-kube-api-access-pj8ft\") pod \"payload-pre-processing-e2e-inf-a-04195c-5d85fcdd86-dlzdz\" (UID: \"899c3f91-4d36-4800-923b-cbab2e572a6c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04195c-5d85fcdd86-dlzdz" Sep 14 11:03:47.943400 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:47.943283 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jwbh\" (UniqueName: \"kubernetes.io/projected/b870f894-e235-49c4-a502-c9dd63500b2e-kube-api-access-8jwbh\") pod \"payload-processing-e2e-inf-a-04195c-559d584fc-6gqvc\" (UID: \"b870f894-e235-49c4-a502-c9dd63500b2e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-04195c-559d584fc-6gqvc" Sep 14 11:03:47.943907 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:47.943398 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b870f894-e235-49c4-a502-c9dd63500b2e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-04195c-559d584fc-6gqvc\" (UID: \"b870f894-e235-49c4-a502-c9dd63500b2e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-04195c-559d584fc-6gqvc" Sep 14 11:03:47.944473 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:47.944452 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b870f894-e235-49c4-a502-c9dd63500b2e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-04195c-559d584fc-6gqvc\" (UID: \"b870f894-e235-49c4-a502-c9dd63500b2e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-04195c-559d584fc-6gqvc" Sep 14 11:03:47.953054 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:47.953024 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jwbh\" (UniqueName: \"kubernetes.io/projected/b870f894-e235-49c4-a502-c9dd63500b2e-kube-api-access-8jwbh\") pod \"payload-processing-e2e-inf-a-04195c-559d584fc-6gqvc\" (UID: \"b870f894-e235-49c4-a502-c9dd63500b2e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-04195c-559d584fc-6gqvc" Sep 14 11:03:47.955458 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:47.955391 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04195c-5d85fcdd86-dlzdz" Sep 14 11:03:48.062465 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:48.062398 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-04195c-559d584fc-6gqvc" Sep 14 11:03:48.237040 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:48.237012 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-04195c-5d85fcdd86-dlzdz"] Sep 14 11:03:48.240715 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:03:48.240692 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod899c3f91_4d36_4800_923b_cbab2e572a6c.slice/crio-f77b9a2201a508510b9785edfae3c3c3971f10afa2dbcc9c46056c56882690a7 WatchSource:0}: Error finding container f77b9a2201a508510b9785edfae3c3c3971f10afa2dbcc9c46056c56882690a7: Status 404 returned error can't find the container with id f77b9a2201a508510b9785edfae3c3c3971f10afa2dbcc9c46056c56882690a7 Sep 14 11:03:48.353898 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:48.353866 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-04195c-559d584fc-6gqvc"] Sep 14 11:03:48.355903 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:03:48.355863 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb870f894_e235_49c4_a502_c9dd63500b2e.slice/crio-2a9de38b3beddb4fec9fcc7837aeaf034516679d48b3080c6c67eed425eb246d WatchSource:0}: Error finding container 2a9de38b3beddb4fec9fcc7837aeaf034516679d48b3080c6c67eed425eb246d: Status 404 returned error can't find the container with id 2a9de38b3beddb4fec9fcc7837aeaf034516679d48b3080c6c67eed425eb246d Sep 14 11:03:48.908944 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:48.908905 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04195c-5d85fcdd86-dlzdz" event={"ID":"899c3f91-4d36-4800-923b-cbab2e572a6c","Type":"ContainerStarted","Data":"f610ae859aec96839769a00f462e811402e21efb4489964a8be08c5e733c8a89"} Sep 14 11:03:48.909140 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:48.908966 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04195c-5d85fcdd86-dlzdz" event={"ID":"899c3f91-4d36-4800-923b-cbab2e572a6c","Type":"ContainerStarted","Data":"f77b9a2201a508510b9785edfae3c3c3971f10afa2dbcc9c46056c56882690a7"} Sep 14 11:03:48.909140 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:48.909014 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04195c-5d85fcdd86-dlzdz" Sep 14 11:03:48.910608 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:48.910585 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-04195c-559d584fc-6gqvc" event={"ID":"b870f894-e235-49c4-a502-c9dd63500b2e","Type":"ContainerStarted","Data":"865a9fecda1604bc101b128e4acb2fef63e562011399fa3d6974df1fcd753a04"} Sep 14 11:03:48.910719 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:48.910618 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-04195c-559d584fc-6gqvc" event={"ID":"b870f894-e235-49c4-a502-c9dd63500b2e","Type":"ContainerStarted","Data":"2a9de38b3beddb4fec9fcc7837aeaf034516679d48b3080c6c67eed425eb246d"} Sep 14 11:03:48.910767 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:48.910714 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-04195c-559d584fc-6gqvc" Sep 14 11:03:48.947244 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:48.947184 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04195c-5d85fcdd86-dlzdz" podStartSLOduration=1.947171747 podStartE2EDuration="1.947171747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:03:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:03:48.945080848 +0000 UTC m=+902.687482831" watchObservedRunningTime="2026-09-14 11:03:48.947171747 +0000 UTC m=+902.689573716" Sep 14 11:03:49.000116 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:49.000060 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-04195c-559d584fc-6gqvc" podStartSLOduration=2.000038537 podStartE2EDuration="2.000038537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:03:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:03:48.998283541 +0000 UTC m=+902.740685511" watchObservedRunningTime="2026-09-14 11:03:49.000038537 +0000 UTC m=+902.742440532" Sep 14 11:03:53.018664 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:53.018624 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b275d0-8dd84d5db-6tsg4"] Sep 14 11:03:53.022379 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:53.022332 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-b275d0-8dd84d5db-6tsg4" Sep 14 11:03:53.032644 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:53.032623 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-b275d0\"" Sep 14 11:03:53.096753 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:53.096715 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73791610-34cd-4a9e-8aa5-1ea9ab8ebaac-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-b275d0-8dd84d5db-6tsg4\" (UID: \"73791610-34cd-4a9e-8aa5-1ea9ab8ebaac\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b275d0-8dd84d5db-6tsg4" Sep 14 11:03:53.096913 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:53.096843 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spk46\" (UniqueName: \"kubernetes.io/projected/73791610-34cd-4a9e-8aa5-1ea9ab8ebaac-kube-api-access-spk46\") pod \"payload-pre-processing-e2e-autores-b275d0-8dd84d5db-6tsg4\" (UID: \"73791610-34cd-4a9e-8aa5-1ea9ab8ebaac\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b275d0-8dd84d5db-6tsg4" Sep 14 11:03:53.138798 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:53.138761 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b275d0-8dd84d5db-6tsg4"] Sep 14 11:03:53.198491 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:53.198448 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73791610-34cd-4a9e-8aa5-1ea9ab8ebaac-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-b275d0-8dd84d5db-6tsg4\" (UID: \"73791610-34cd-4a9e-8aa5-1ea9ab8ebaac\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b275d0-8dd84d5db-6tsg4" Sep 14 11:03:53.198688 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:53.198541 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spk46\" (UniqueName: \"kubernetes.io/projected/73791610-34cd-4a9e-8aa5-1ea9ab8ebaac-kube-api-access-spk46\") pod \"payload-pre-processing-e2e-autores-b275d0-8dd84d5db-6tsg4\" (UID: \"73791610-34cd-4a9e-8aa5-1ea9ab8ebaac\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b275d0-8dd84d5db-6tsg4" Sep 14 11:03:53.199168 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:53.199147 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73791610-34cd-4a9e-8aa5-1ea9ab8ebaac-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-b275d0-8dd84d5db-6tsg4\" (UID: \"73791610-34cd-4a9e-8aa5-1ea9ab8ebaac\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b275d0-8dd84d5db-6tsg4" Sep 14 11:03:53.225975 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:53.225941 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spk46\" (UniqueName: \"kubernetes.io/projected/73791610-34cd-4a9e-8aa5-1ea9ab8ebaac-kube-api-access-spk46\") pod \"payload-pre-processing-e2e-autores-b275d0-8dd84d5db-6tsg4\" (UID: \"73791610-34cd-4a9e-8aa5-1ea9ab8ebaac\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b275d0-8dd84d5db-6tsg4" Sep 14 11:03:53.332944 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:53.332910 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-b275d0-8dd84d5db-6tsg4" Sep 14 11:03:53.628463 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:53.628435 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b275d0-8dd84d5db-6tsg4"] Sep 14 11:03:53.631172 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:03:53.631140 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73791610_34cd_4a9e_8aa5_1ea9ab8ebaac.slice/crio-fbd030a29005b3a7af033de70d686e8f6defaab1ee74bf1dcf2499aad212fa08 WatchSource:0}: Error finding container fbd030a29005b3a7af033de70d686e8f6defaab1ee74bf1dcf2499aad212fa08: Status 404 returned error can't find the container with id fbd030a29005b3a7af033de70d686e8f6defaab1ee74bf1dcf2499aad212fa08 Sep 14 11:03:53.944038 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:53.943949 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-b275d0-8dd84d5db-6tsg4" event={"ID":"73791610-34cd-4a9e-8aa5-1ea9ab8ebaac","Type":"ContainerStarted","Data":"cd1da84e4a1d132afb0e82ec59e8857a0158c314a56546026eee31f3abb46308"} Sep 14 11:03:53.944038 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:53.944002 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-b275d0-8dd84d5db-6tsg4" event={"ID":"73791610-34cd-4a9e-8aa5-1ea9ab8ebaac","Type":"ContainerStarted","Data":"fbd030a29005b3a7af033de70d686e8f6defaab1ee74bf1dcf2499aad212fa08"} Sep 14 11:03:53.944242 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:53.944098 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-b275d0-8dd84d5db-6tsg4" Sep 14 11:03:53.994059 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:53.994005 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-b275d0-8dd84d5db-6tsg4" podStartSLOduration=1.993987927 podStartE2EDuration="1.993987927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:03:53.991533253 +0000 UTC m=+907.733935224" watchObservedRunningTime="2026-09-14 11:03:53.993987927 +0000 UTC m=+907.736389896" Sep 14 11:03:56.899121 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:56.899086 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-4ebf73-9687f9869-zpnm6" Sep 14 11:03:56.899531 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:56.899155 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ebf73-5567b59786-mzx5m" Sep 14 11:03:57.902966 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:57.902931 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-7984b1-56cdcd4d45-bfgfv" Sep 14 11:03:57.903580 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:57.903187 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7984b1-6999cf6c6b-8vbcv" Sep 14 11:03:58.883769 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:58.883727 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6706857a-6cd6dbdc54-rn5g5"] Sep 14 11:03:58.887588 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:58.887571 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6706857a-6cd6dbdc54-rn5g5" Sep 14 11:03:58.891404 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:58.891377 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-6706857a\"" Sep 14 11:03:58.891512 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:58.891409 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-6706857a-dockercfg-7hrc2\"" Sep 14 11:03:58.908142 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:58.908115 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6706857a-6cd6dbdc54-rn5g5"] Sep 14 11:03:58.953771 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:58.953734 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llgqj\" (UniqueName: \"kubernetes.io/projected/e7c48a04-49fe-4143-ac3f-9104a74c143c-kube-api-access-llgqj\") pod \"payload-pre-processing-e2e-ait-6706857a-6cd6dbdc54-rn5g5\" (UID: \"e7c48a04-49fe-4143-ac3f-9104a74c143c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6706857a-6cd6dbdc54-rn5g5" Sep 14 11:03:58.953973 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:58.953778 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7c48a04-49fe-4143-ac3f-9104a74c143c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6706857a-6cd6dbdc54-rn5g5\" (UID: \"e7c48a04-49fe-4143-ac3f-9104a74c143c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6706857a-6cd6dbdc54-rn5g5" Sep 14 11:03:58.998411 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:58.998374 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6706857a-798454d459-tm86r"] Sep 14 11:03:59.002057 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:59.002033 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6706857a-798454d459-tm86r" Sep 14 11:03:59.028725 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:59.028694 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6706857a-798454d459-tm86r"] Sep 14 11:03:59.054874 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:59.054841 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmvm7\" (UniqueName: \"kubernetes.io/projected/b3973ad2-4fdc-4e2b-a465-3841543ed092-kube-api-access-fmvm7\") pod \"payload-processing-e2e-ait-6706857a-798454d459-tm86r\" (UID: \"b3973ad2-4fdc-4e2b-a465-3841543ed092\") " pod="openshift-ingress/payload-processing-e2e-ait-6706857a-798454d459-tm86r" Sep 14 11:03:59.055052 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:59.054892 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llgqj\" (UniqueName: \"kubernetes.io/projected/e7c48a04-49fe-4143-ac3f-9104a74c143c-kube-api-access-llgqj\") pod \"payload-pre-processing-e2e-ait-6706857a-6cd6dbdc54-rn5g5\" (UID: \"e7c48a04-49fe-4143-ac3f-9104a74c143c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6706857a-6cd6dbdc54-rn5g5" Sep 14 11:03:59.055052 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:59.054959 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7c48a04-49fe-4143-ac3f-9104a74c143c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6706857a-6cd6dbdc54-rn5g5\" (UID: \"e7c48a04-49fe-4143-ac3f-9104a74c143c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6706857a-6cd6dbdc54-rn5g5" Sep 14 11:03:59.055052 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:59.055049 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3973ad2-4fdc-4e2b-a465-3841543ed092-plugins-config-volume\") pod \"payload-processing-e2e-ait-6706857a-798454d459-tm86r\" (UID: \"b3973ad2-4fdc-4e2b-a465-3841543ed092\") " pod="openshift-ingress/payload-processing-e2e-ait-6706857a-798454d459-tm86r" Sep 14 11:03:59.055718 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:59.055698 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7c48a04-49fe-4143-ac3f-9104a74c143c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6706857a-6cd6dbdc54-rn5g5\" (UID: \"e7c48a04-49fe-4143-ac3f-9104a74c143c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6706857a-6cd6dbdc54-rn5g5" Sep 14 11:03:59.067878 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:59.067855 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llgqj\" (UniqueName: \"kubernetes.io/projected/e7c48a04-49fe-4143-ac3f-9104a74c143c-kube-api-access-llgqj\") pod \"payload-pre-processing-e2e-ait-6706857a-6cd6dbdc54-rn5g5\" (UID: \"e7c48a04-49fe-4143-ac3f-9104a74c143c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6706857a-6cd6dbdc54-rn5g5" Sep 14 11:03:59.156015 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:59.155917 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3973ad2-4fdc-4e2b-a465-3841543ed092-plugins-config-volume\") pod \"payload-processing-e2e-ait-6706857a-798454d459-tm86r\" (UID: \"b3973ad2-4fdc-4e2b-a465-3841543ed092\") " pod="openshift-ingress/payload-processing-e2e-ait-6706857a-798454d459-tm86r" Sep 14 11:03:59.156015 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:59.155977 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmvm7\" (UniqueName: \"kubernetes.io/projected/b3973ad2-4fdc-4e2b-a465-3841543ed092-kube-api-access-fmvm7\") pod \"payload-processing-e2e-ait-6706857a-798454d459-tm86r\" (UID: \"b3973ad2-4fdc-4e2b-a465-3841543ed092\") " pod="openshift-ingress/payload-processing-e2e-ait-6706857a-798454d459-tm86r" Sep 14 11:03:59.156666 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:59.156648 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3973ad2-4fdc-4e2b-a465-3841543ed092-plugins-config-volume\") pod \"payload-processing-e2e-ait-6706857a-798454d459-tm86r\" (UID: \"b3973ad2-4fdc-4e2b-a465-3841543ed092\") " pod="openshift-ingress/payload-processing-e2e-ait-6706857a-798454d459-tm86r" Sep 14 11:03:59.172063 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:59.172031 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmvm7\" (UniqueName: \"kubernetes.io/projected/b3973ad2-4fdc-4e2b-a465-3841543ed092-kube-api-access-fmvm7\") pod \"payload-processing-e2e-ait-6706857a-798454d459-tm86r\" (UID: \"b3973ad2-4fdc-4e2b-a465-3841543ed092\") " pod="openshift-ingress/payload-processing-e2e-ait-6706857a-798454d459-tm86r" Sep 14 11:03:59.199532 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:59.199492 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6706857a-6cd6dbdc54-rn5g5" Sep 14 11:03:59.313251 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:59.313218 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6706857a-798454d459-tm86r" Sep 14 11:03:59.485576 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:59.485549 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6706857a-6cd6dbdc54-rn5g5"] Sep 14 11:03:59.487198 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:03:59.487174 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7c48a04_49fe_4143_ac3f_9104a74c143c.slice/crio-8c53edaf30a387841c0c7bda014832e5c53931c623eeb832d7b487fbda2cb543 WatchSource:0}: Error finding container 8c53edaf30a387841c0c7bda014832e5c53931c623eeb832d7b487fbda2cb543: Status 404 returned error can't find the container with id 8c53edaf30a387841c0c7bda014832e5c53931c623eeb832d7b487fbda2cb543 Sep 14 11:03:59.615937 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:59.615790 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6706857a-798454d459-tm86r"] Sep 14 11:03:59.618207 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:03:59.618181 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3973ad2_4fdc_4e2b_a465_3841543ed092.slice/crio-1662ca13daea809a3a88ef5c206808d9c43516c7003a181d6cf84fd3313621be WatchSource:0}: Error finding container 1662ca13daea809a3a88ef5c206808d9c43516c7003a181d6cf84fd3313621be: Status 404 returned error can't find the container with id 1662ca13daea809a3a88ef5c206808d9c43516c7003a181d6cf84fd3313621be Sep 14 11:03:59.920547 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:59.920509 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04195c-5d85fcdd86-dlzdz" Sep 14 11:03:59.921034 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:59.920889 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-04195c-559d584fc-6gqvc" Sep 14 11:03:59.974964 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:59.974927 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6706857a-798454d459-tm86r" event={"ID":"b3973ad2-4fdc-4e2b-a465-3841543ed092","Type":"ContainerStarted","Data":"c4c68a0e3f91e43cd114fbb423f60097f745fdb03188561eb1d4ff9dcdd095d2"} Sep 14 11:03:59.975177 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:59.974994 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6706857a-798454d459-tm86r" event={"ID":"b3973ad2-4fdc-4e2b-a465-3841543ed092","Type":"ContainerStarted","Data":"1662ca13daea809a3a88ef5c206808d9c43516c7003a181d6cf84fd3313621be"} Sep 14 11:03:59.975177 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:59.975055 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-6706857a-798454d459-tm86r" Sep 14 11:03:59.977256 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:59.977217 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6706857a-6cd6dbdc54-rn5g5" event={"ID":"e7c48a04-49fe-4143-ac3f-9104a74c143c","Type":"ContainerStarted","Data":"1022c6bc11f97e52d75ea4e2b79b6b596e283a071792d6b9c49c50d2e0b68488"} Sep 14 11:03:59.977418 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:59.977272 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6706857a-6cd6dbdc54-rn5g5" event={"ID":"e7c48a04-49fe-4143-ac3f-9104a74c143c","Type":"ContainerStarted","Data":"8c53edaf30a387841c0c7bda014832e5c53931c623eeb832d7b487fbda2cb543"} Sep 14 11:03:59.977543 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:03:59.977520 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6706857a-6cd6dbdc54-rn5g5" Sep 14 11:04:00.018673 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:00.018618 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-6706857a-6cd6dbdc54-rn5g5" podStartSLOduration=2.018596339 podStartE2EDuration="2.018596339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:03:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:04:00.016252158 +0000 UTC m=+913.758654128" watchObservedRunningTime="2026-09-14 11:04:00.018596339 +0000 UTC m=+913.760998311" Sep 14 11:04:00.047074 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:00.047014 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-6706857a-798454d459-tm86r" podStartSLOduration=2.046994031 podStartE2EDuration="2.046994031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:03:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:04:00.043311273 +0000 UTC m=+913.785713239" watchObservedRunningTime="2026-09-14 11:04:00.046994031 +0000 UTC m=+913.789396007" Sep 14 11:04:04.951680 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:04.951652 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-b275d0-8dd84d5db-6tsg4" Sep 14 11:04:10.985026 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:10.984996 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6706857a-6cd6dbdc54-rn5g5" Sep 14 11:04:10.985525 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:10.985063 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-6706857a-798454d459-tm86r" Sep 14 11:04:11.992324 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:11.992285 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0777c1-75869b6f54-t2s9l"] Sep 14 11:04:11.995752 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:11.995733 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0777c1-75869b6f54-t2s9l" Sep 14 11:04:11.999078 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:11.999041 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-0777c1\"" Sep 14 11:04:12.001472 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:12.001440 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0777c1-75869b6f54-t2s9l"] Sep 14 11:04:12.014827 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:12.014798 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0777c1-6c5445689c-wqd5f"] Sep 14 11:04:12.018569 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:12.018540 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0777c1-6c5445689c-wqd5f" Sep 14 11:04:12.021941 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:12.021921 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-0777c1-dockercfg-j7b56\"" Sep 14 11:04:12.031595 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:12.031569 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0777c1-6c5445689c-wqd5f"] Sep 14 11:04:12.084731 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:12.084690 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c41213ee-43d5-4d4d-a368-fd363e0644fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0777c1-75869b6f54-t2s9l\" (UID: \"c41213ee-43d5-4d4d-a368-fd363e0644fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0777c1-75869b6f54-t2s9l" Sep 14 11:04:12.084908 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:12.084766 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lthl\" (UniqueName: \"kubernetes.io/projected/c41213ee-43d5-4d4d-a368-fd363e0644fd-kube-api-access-4lthl\") pod \"payload-pre-processing-e2e-inf-b-0777c1-75869b6f54-t2s9l\" (UID: \"c41213ee-43d5-4d4d-a368-fd363e0644fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0777c1-75869b6f54-t2s9l" Sep 14 11:04:12.185962 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:12.185916 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/873c3405-3775-43ba-838f-52dccccc4d59-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0777c1-6c5445689c-wqd5f\" (UID: \"873c3405-3775-43ba-838f-52dccccc4d59\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0777c1-6c5445689c-wqd5f" Sep 14 11:04:12.186227 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:12.186008 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lthl\" (UniqueName: \"kubernetes.io/projected/c41213ee-43d5-4d4d-a368-fd363e0644fd-kube-api-access-4lthl\") pod \"payload-pre-processing-e2e-inf-b-0777c1-75869b6f54-t2s9l\" (UID: \"c41213ee-43d5-4d4d-a368-fd363e0644fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0777c1-75869b6f54-t2s9l" Sep 14 11:04:12.186227 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:12.186053 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnzwj\" (UniqueName: \"kubernetes.io/projected/873c3405-3775-43ba-838f-52dccccc4d59-kube-api-access-dnzwj\") pod \"payload-processing-e2e-inf-b-0777c1-6c5445689c-wqd5f\" (UID: \"873c3405-3775-43ba-838f-52dccccc4d59\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0777c1-6c5445689c-wqd5f" Sep 14 11:04:12.186227 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:12.186139 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c41213ee-43d5-4d4d-a368-fd363e0644fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0777c1-75869b6f54-t2s9l\" (UID: \"c41213ee-43d5-4d4d-a368-fd363e0644fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0777c1-75869b6f54-t2s9l" Sep 14 11:04:12.186883 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:12.186860 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c41213ee-43d5-4d4d-a368-fd363e0644fd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0777c1-75869b6f54-t2s9l\" (UID: \"c41213ee-43d5-4d4d-a368-fd363e0644fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0777c1-75869b6f54-t2s9l" Sep 14 11:04:12.194716 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:12.194696 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lthl\" (UniqueName: \"kubernetes.io/projected/c41213ee-43d5-4d4d-a368-fd363e0644fd-kube-api-access-4lthl\") pod \"payload-pre-processing-e2e-inf-b-0777c1-75869b6f54-t2s9l\" (UID: \"c41213ee-43d5-4d4d-a368-fd363e0644fd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0777c1-75869b6f54-t2s9l" Sep 14 11:04:12.286881 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:12.286840 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnzwj\" (UniqueName: \"kubernetes.io/projected/873c3405-3775-43ba-838f-52dccccc4d59-kube-api-access-dnzwj\") pod \"payload-processing-e2e-inf-b-0777c1-6c5445689c-wqd5f\" (UID: \"873c3405-3775-43ba-838f-52dccccc4d59\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0777c1-6c5445689c-wqd5f" Sep 14 11:04:12.287083 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:12.286958 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/873c3405-3775-43ba-838f-52dccccc4d59-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0777c1-6c5445689c-wqd5f\" (UID: \"873c3405-3775-43ba-838f-52dccccc4d59\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0777c1-6c5445689c-wqd5f" Sep 14 11:04:12.287707 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:12.287689 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/873c3405-3775-43ba-838f-52dccccc4d59-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0777c1-6c5445689c-wqd5f\" (UID: \"873c3405-3775-43ba-838f-52dccccc4d59\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0777c1-6c5445689c-wqd5f" Sep 14 11:04:12.295381 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:12.295333 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnzwj\" (UniqueName: \"kubernetes.io/projected/873c3405-3775-43ba-838f-52dccccc4d59-kube-api-access-dnzwj\") pod \"payload-processing-e2e-inf-b-0777c1-6c5445689c-wqd5f\" (UID: \"873c3405-3775-43ba-838f-52dccccc4d59\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0777c1-6c5445689c-wqd5f" Sep 14 11:04:12.309237 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:12.309212 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0777c1-75869b6f54-t2s9l" Sep 14 11:04:12.328698 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:12.328672 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0777c1-6c5445689c-wqd5f" Sep 14 11:04:12.594058 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:12.594030 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0777c1-75869b6f54-t2s9l"] Sep 14 11:04:12.640292 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:12.640138 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0777c1-6c5445689c-wqd5f"] Sep 14 11:04:12.643777 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:04:12.643744 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod873c3405_3775_43ba_838f_52dccccc4d59.slice/crio-97c2e5a29bc07fdcd213a9f5d0ce40dbda845d109c15852e1f355e5f7b78fe60 WatchSource:0}: Error finding container 97c2e5a29bc07fdcd213a9f5d0ce40dbda845d109c15852e1f355e5f7b78fe60: Status 404 returned error can't find the container with id 97c2e5a29bc07fdcd213a9f5d0ce40dbda845d109c15852e1f355e5f7b78fe60 Sep 14 11:04:13.048114 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:13.048062 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0777c1-75869b6f54-t2s9l" event={"ID":"c41213ee-43d5-4d4d-a368-fd363e0644fd","Type":"ContainerStarted","Data":"7831680f2bb7659c5541f316dec1a6d4958fb27bf3b946dc774ec0c2809dd5f5"} Sep 14 11:04:13.048668 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:13.048130 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0777c1-75869b6f54-t2s9l" event={"ID":"c41213ee-43d5-4d4d-a368-fd363e0644fd","Type":"ContainerStarted","Data":"a113ee74c215e79855a15636f89d09fb21ccb8e3727ed0807889cf06fce7d573"} Sep 14 11:04:13.048668 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:13.048156 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0777c1-75869b6f54-t2s9l" Sep 14 11:04:13.050045 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:13.050020 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0777c1-6c5445689c-wqd5f" event={"ID":"873c3405-3775-43ba-838f-52dccccc4d59","Type":"ContainerStarted","Data":"983ad1c3c689327a4c6e12bedc93dd331fe171d013096a669888a70d3cc78f39"} Sep 14 11:04:13.050153 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:13.050060 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0777c1-6c5445689c-wqd5f" event={"ID":"873c3405-3775-43ba-838f-52dccccc4d59","Type":"ContainerStarted","Data":"97c2e5a29bc07fdcd213a9f5d0ce40dbda845d109c15852e1f355e5f7b78fe60"} Sep 14 11:04:13.050214 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:13.050163 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0777c1-6c5445689c-wqd5f" Sep 14 11:04:13.066213 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:13.066154 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0777c1-75869b6f54-t2s9l" podStartSLOduration=2.066132738 podStartE2EDuration="2.066132738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:04:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:04:13.064511619 +0000 UTC m=+926.806913627" watchObservedRunningTime="2026-09-14 11:04:13.066132738 +0000 UTC m=+926.808534711" Sep 14 11:04:13.086425 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:13.086374 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-0777c1-6c5445689c-wqd5f" podStartSLOduration=2.086356567 podStartE2EDuration="2.086356567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:04:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:04:13.083970877 +0000 UTC m=+926.826372851" watchObservedRunningTime="2026-09-14 11:04:13.086356567 +0000 UTC m=+926.828758534" Sep 14 11:04:13.314296 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:13.314211 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6706857a-jjmld"] Sep 14 11:04:13.318537 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:13.318520 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6706857a-jjmld" Sep 14 11:04:13.321745 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:13.321720 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pv9g8\"" Sep 14 11:04:13.326027 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:13.325998 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6706857a-jjmld"] Sep 14 11:04:13.397276 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:13.397244 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d93957d-e182-444b-b9d6-84d8ce687cb0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6706857a-jjmld\" (UID: \"0d93957d-e182-444b-b9d6-84d8ce687cb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6706857a-jjmld" Sep 14 11:04:13.498638 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:13.498602 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d93957d-e182-444b-b9d6-84d8ce687cb0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6706857a-jjmld\" (UID: \"0d93957d-e182-444b-b9d6-84d8ce687cb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6706857a-jjmld" Sep 14 11:04:13.499250 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:13.499230 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d93957d-e182-444b-b9d6-84d8ce687cb0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6706857a-jjmld\" (UID: \"0d93957d-e182-444b-b9d6-84d8ce687cb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6706857a-jjmld" Sep 14 11:04:13.633624 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:13.633535 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6706857a-jjmld" Sep 14 11:04:13.907759 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:13.907735 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6706857a-jjmld"] Sep 14 11:04:13.910399 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:04:13.910371 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d93957d_e182_444b_b9d6_84d8ce687cb0.slice/crio-a5d9df5f5e9627751fd6e6130e456688d5c438f8c2adb603ce4e7c5b8cce8016 WatchSource:0}: Error finding container a5d9df5f5e9627751fd6e6130e456688d5c438f8c2adb603ce4e7c5b8cce8016: Status 404 returned error can't find the container with id a5d9df5f5e9627751fd6e6130e456688d5c438f8c2adb603ce4e7c5b8cce8016 Sep 14 11:04:14.056278 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:14.056242 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6706857a-jjmld" event={"ID":"0d93957d-e182-444b-b9d6-84d8ce687cb0","Type":"ContainerStarted","Data":"a5d9df5f5e9627751fd6e6130e456688d5c438f8c2adb603ce4e7c5b8cce8016"} Sep 14 11:04:24.058745 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:24.058712 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0777c1-75869b6f54-t2s9l" Sep 14 11:04:24.059399 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:24.058797 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0777c1-6c5445689c-wqd5f" Sep 14 11:04:30.146869 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:30.146837 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6706857a-jjmld" event={"ID":"0d93957d-e182-444b-b9d6-84d8ce687cb0","Type":"ContainerStarted","Data":"dd8aa786fc1bc681cfae1f5a1ee61f5677b2fbbd446729ec73053e25f4e988f5"} Sep 14 11:04:30.165686 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:30.165638 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6706857a-jjmld" podStartSLOduration=1.3890643919999999 podStartE2EDuration="17.165623023s" totalImagesPullingTime="15.776558631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:04:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:04:30.164467512 +0000 UTC m=+943.906869483" watchObservedRunningTime="2026-09-14 11:04:30.165623023 +0000 UTC m=+943.908024991" Sep 14 11:04:31.154066 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:31.154034 2581 generic.go:356] "Generic (PLEG): container finished" podID="0d93957d-e182-444b-b9d6-84d8ce687cb0" containerID="dd8aa786fc1bc681cfae1f5a1ee61f5677b2fbbd446729ec73053e25f4e988f5" exitCode=0 Sep 14 11:04:31.154484 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:31.154125 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6706857a-jjmld" event={"ID":"0d93957d-e182-444b-b9d6-84d8ce687cb0","Type":"ContainerDied","Data":"dd8aa786fc1bc681cfae1f5a1ee61f5677b2fbbd446729ec73053e25f4e988f5"} Sep 14 11:04:32.294737 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:32.294708 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6706857a-jjmld" Sep 14 11:04:32.384350 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:32.384315 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d93957d-e182-444b-b9d6-84d8ce687cb0-maas-api-service-ca\") pod \"0d93957d-e182-444b-b9d6-84d8ce687cb0\" (UID: \"0d93957d-e182-444b-b9d6-84d8ce687cb0\") " Sep 14 11:04:32.384665 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:32.384643 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d93957d-e182-444b-b9d6-84d8ce687cb0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0d93957d-e182-444b-b9d6-84d8ce687cb0" (UID: "0d93957d-e182-444b-b9d6-84d8ce687cb0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:04:32.485958 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:32.485867 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d93957d-e182-444b-b9d6-84d8ce687cb0-maas-api-service-ca\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:04:33.166287 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:33.166201 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6706857a-jjmld" event={"ID":"0d93957d-e182-444b-b9d6-84d8ce687cb0","Type":"ContainerDied","Data":"a5d9df5f5e9627751fd6e6130e456688d5c438f8c2adb603ce4e7c5b8cce8016"} Sep 14 11:04:33.166287 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:33.166220 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6706857a-jjmld" Sep 14 11:04:33.166287 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:33.166249 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5d9df5f5e9627751fd6e6130e456688d5c438f8c2adb603ce4e7c5b8cce8016" Sep 14 11:04:33.755103 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:33.755061 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6706857a-798454d459-tm86r"] Sep 14 11:04:33.755514 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:33.755399 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-6706857a-798454d459-tm86r" podUID="b3973ad2-4fdc-4e2b-a465-3841543ed092" containerName="payload-processing" containerID="cri-o://c4c68a0e3f91e43cd114fbb423f60097f745fdb03188561eb1d4ff9dcdd095d2" gracePeriod=30 Sep 14 11:04:33.767928 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:33.767895 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6706857a-6cd6dbdc54-rn5g5"] Sep 14 11:04:33.768195 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:33.768146 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-6706857a-6cd6dbdc54-rn5g5" podUID="e7c48a04-49fe-4143-ac3f-9104a74c143c" containerName="payload-pre-processing" containerID="cri-o://1022c6bc11f97e52d75ea4e2b79b6b596e283a071792d6b9c49c50d2e0b68488" gracePeriod=30 Sep 14 11:04:34.056181 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:34.056159 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6706857a-798454d459-tm86r" Sep 14 11:04:34.062479 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:34.062450 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6706857a-6cd6dbdc54-rn5g5" Sep 14 11:04:34.101183 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:34.101149 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-llgqj\" (UniqueName: \"kubernetes.io/projected/e7c48a04-49fe-4143-ac3f-9104a74c143c-kube-api-access-llgqj\") pod \"e7c48a04-49fe-4143-ac3f-9104a74c143c\" (UID: \"e7c48a04-49fe-4143-ac3f-9104a74c143c\") " Sep 14 11:04:34.101411 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:34.101226 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3973ad2-4fdc-4e2b-a465-3841543ed092-plugins-config-volume\") pod \"b3973ad2-4fdc-4e2b-a465-3841543ed092\" (UID: \"b3973ad2-4fdc-4e2b-a465-3841543ed092\") " Sep 14 11:04:34.101411 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:34.101378 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmvm7\" (UniqueName: \"kubernetes.io/projected/b3973ad2-4fdc-4e2b-a465-3841543ed092-kube-api-access-fmvm7\") pod \"b3973ad2-4fdc-4e2b-a465-3841543ed092\" (UID: \"b3973ad2-4fdc-4e2b-a465-3841543ed092\") " Sep 14 11:04:34.101549 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:34.101441 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7c48a04-49fe-4143-ac3f-9104a74c143c-plugins-config-volume\") pod \"e7c48a04-49fe-4143-ac3f-9104a74c143c\" (UID: \"e7c48a04-49fe-4143-ac3f-9104a74c143c\") " Sep 14 11:04:34.102169 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:34.102142 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3973ad2-4fdc-4e2b-a465-3841543ed092-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b3973ad2-4fdc-4e2b-a465-3841543ed092" (UID: "b3973ad2-4fdc-4e2b-a465-3841543ed092"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:04:34.102383 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:34.102142 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7c48a04-49fe-4143-ac3f-9104a74c143c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e7c48a04-49fe-4143-ac3f-9104a74c143c" (UID: "e7c48a04-49fe-4143-ac3f-9104a74c143c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:04:34.105213 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:34.105189 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3973ad2-4fdc-4e2b-a465-3841543ed092-kube-api-access-fmvm7" (OuterVolumeSpecName: "kube-api-access-fmvm7") pod "b3973ad2-4fdc-4e2b-a465-3841543ed092" (UID: "b3973ad2-4fdc-4e2b-a465-3841543ed092"). InnerVolumeSpecName "kube-api-access-fmvm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:04:34.105330 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:34.105221 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7c48a04-49fe-4143-ac3f-9104a74c143c-kube-api-access-llgqj" (OuterVolumeSpecName: "kube-api-access-llgqj") pod "e7c48a04-49fe-4143-ac3f-9104a74c143c" (UID: "e7c48a04-49fe-4143-ac3f-9104a74c143c"). InnerVolumeSpecName "kube-api-access-llgqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:04:34.113436 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:34.113410 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6706857a-jjmld"] Sep 14 11:04:34.116631 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:34.116603 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6706857a-jjmld"] Sep 14 11:04:34.173032 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:34.172995 2581 generic.go:356] "Generic (PLEG): container finished" podID="e7c48a04-49fe-4143-ac3f-9104a74c143c" containerID="1022c6bc11f97e52d75ea4e2b79b6b596e283a071792d6b9c49c50d2e0b68488" exitCode=0 Sep 14 11:04:34.173224 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:34.173047 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6706857a-6cd6dbdc54-rn5g5" Sep 14 11:04:34.173224 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:34.173084 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6706857a-6cd6dbdc54-rn5g5" event={"ID":"e7c48a04-49fe-4143-ac3f-9104a74c143c","Type":"ContainerDied","Data":"1022c6bc11f97e52d75ea4e2b79b6b596e283a071792d6b9c49c50d2e0b68488"} Sep 14 11:04:34.173224 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:34.173134 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6706857a-6cd6dbdc54-rn5g5" event={"ID":"e7c48a04-49fe-4143-ac3f-9104a74c143c","Type":"ContainerDied","Data":"8c53edaf30a387841c0c7bda014832e5c53931c623eeb832d7b487fbda2cb543"} Sep 14 11:04:34.173224 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:34.173161 2581 scope.go:117] "RemoveContainer" containerID="1022c6bc11f97e52d75ea4e2b79b6b596e283a071792d6b9c49c50d2e0b68488" Sep 14 11:04:34.174819 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:34.174632 2581 generic.go:356] "Generic (PLEG): container finished" podID="b3973ad2-4fdc-4e2b-a465-3841543ed092" containerID="c4c68a0e3f91e43cd114fbb423f60097f745fdb03188561eb1d4ff9dcdd095d2" exitCode=0 Sep 14 11:04:34.174819 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:34.174691 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6706857a-798454d459-tm86r" Sep 14 11:04:34.174819 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:34.174692 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6706857a-798454d459-tm86r" event={"ID":"b3973ad2-4fdc-4e2b-a465-3841543ed092","Type":"ContainerDied","Data":"c4c68a0e3f91e43cd114fbb423f60097f745fdb03188561eb1d4ff9dcdd095d2"} Sep 14 11:04:34.174819 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:34.174791 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6706857a-798454d459-tm86r" event={"ID":"b3973ad2-4fdc-4e2b-a465-3841543ed092","Type":"ContainerDied","Data":"1662ca13daea809a3a88ef5c206808d9c43516c7003a181d6cf84fd3313621be"} Sep 14 11:04:34.183242 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:34.183223 2581 scope.go:117] "RemoveContainer" containerID="1022c6bc11f97e52d75ea4e2b79b6b596e283a071792d6b9c49c50d2e0b68488" Sep 14 11:04:34.183581 ip-10-0-137-53 kubenswrapper[2581]: E0914 11:04:34.183559 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1022c6bc11f97e52d75ea4e2b79b6b596e283a071792d6b9c49c50d2e0b68488\": container with ID starting with 1022c6bc11f97e52d75ea4e2b79b6b596e283a071792d6b9c49c50d2e0b68488 not found: ID does not exist" containerID="1022c6bc11f97e52d75ea4e2b79b6b596e283a071792d6b9c49c50d2e0b68488" Sep 14 11:04:34.183659 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:34.183589 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1022c6bc11f97e52d75ea4e2b79b6b596e283a071792d6b9c49c50d2e0b68488"} err="failed to get container status \"1022c6bc11f97e52d75ea4e2b79b6b596e283a071792d6b9c49c50d2e0b68488\": rpc error: code = NotFound desc = could not find container \"1022c6bc11f97e52d75ea4e2b79b6b596e283a071792d6b9c49c50d2e0b68488\": container with ID starting with 1022c6bc11f97e52d75ea4e2b79b6b596e283a071792d6b9c49c50d2e0b68488 not found: ID does not exist" Sep 14 11:04:34.183659 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:34.183606 2581 scope.go:117] "RemoveContainer" containerID="c4c68a0e3f91e43cd114fbb423f60097f745fdb03188561eb1d4ff9dcdd095d2" Sep 14 11:04:34.192893 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:34.192874 2581 scope.go:117] "RemoveContainer" containerID="c4c68a0e3f91e43cd114fbb423f60097f745fdb03188561eb1d4ff9dcdd095d2" Sep 14 11:04:34.193215 ip-10-0-137-53 kubenswrapper[2581]: E0914 11:04:34.193192 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4c68a0e3f91e43cd114fbb423f60097f745fdb03188561eb1d4ff9dcdd095d2\": container with ID starting with c4c68a0e3f91e43cd114fbb423f60097f745fdb03188561eb1d4ff9dcdd095d2 not found: ID does not exist" containerID="c4c68a0e3f91e43cd114fbb423f60097f745fdb03188561eb1d4ff9dcdd095d2" Sep 14 11:04:34.193262 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:34.193227 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4c68a0e3f91e43cd114fbb423f60097f745fdb03188561eb1d4ff9dcdd095d2"} err="failed to get container status \"c4c68a0e3f91e43cd114fbb423f60097f745fdb03188561eb1d4ff9dcdd095d2\": rpc error: code = NotFound desc = could not find container \"c4c68a0e3f91e43cd114fbb423f60097f745fdb03188561eb1d4ff9dcdd095d2\": container with ID starting with c4c68a0e3f91e43cd114fbb423f60097f745fdb03188561eb1d4ff9dcdd095d2 not found: ID does not exist" Sep 14 11:04:34.203140 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:34.203114 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmvm7\" (UniqueName: \"kubernetes.io/projected/b3973ad2-4fdc-4e2b-a465-3841543ed092-kube-api-access-fmvm7\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:04:34.203140 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:34.203139 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7c48a04-49fe-4143-ac3f-9104a74c143c-plugins-config-volume\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:04:34.203261 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:34.203149 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llgqj\" (UniqueName: \"kubernetes.io/projected/e7c48a04-49fe-4143-ac3f-9104a74c143c-kube-api-access-llgqj\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:04:34.203261 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:34.203160 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3973ad2-4fdc-4e2b-a465-3841543ed092-plugins-config-volume\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:04:34.214374 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:34.214327 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6706857a-798454d459-tm86r"] Sep 14 11:04:34.234016 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:34.233991 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6706857a-798454d459-tm86r"] Sep 14 11:04:34.254503 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:34.254470 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6706857a-6cd6dbdc54-rn5g5"] Sep 14 11:04:34.262358 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:34.262310 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6706857a-6cd6dbdc54-rn5g5"] Sep 14 11:04:34.832040 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:34.832008 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d93957d-e182-444b-b9d6-84d8ce687cb0" path="/var/lib/kubelet/pods/0d93957d-e182-444b-b9d6-84d8ce687cb0/volumes" Sep 14 11:04:34.832447 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:34.832382 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3973ad2-4fdc-4e2b-a465-3841543ed092" path="/var/lib/kubelet/pods/b3973ad2-4fdc-4e2b-a465-3841543ed092/volumes" Sep 14 11:04:34.832697 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:34.832683 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7c48a04-49fe-4143-ac3f-9104a74c143c" path="/var/lib/kubelet/pods/e7c48a04-49fe-4143-ac3f-9104a74c143c/volumes" Sep 14 11:04:41.969852 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:41.969815 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-73290f4b-59d955fd4b-fnrpb"] Sep 14 11:04:41.970253 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:41.970240 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7c48a04-49fe-4143-ac3f-9104a74c143c" containerName="payload-pre-processing" Sep 14 11:04:41.970301 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:41.970254 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7c48a04-49fe-4143-ac3f-9104a74c143c" containerName="payload-pre-processing" Sep 14 11:04:41.970301 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:41.970266 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3973ad2-4fdc-4e2b-a465-3841543ed092" containerName="payload-processing" Sep 14 11:04:41.970301 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:41.970272 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3973ad2-4fdc-4e2b-a465-3841543ed092" containerName="payload-processing" Sep 14 11:04:41.970301 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:41.970283 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d93957d-e182-444b-b9d6-84d8ce687cb0" containerName="revoke-keys" Sep 14 11:04:41.970301 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:41.970289 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d93957d-e182-444b-b9d6-84d8ce687cb0" containerName="revoke-keys" Sep 14 11:04:41.970486 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:41.970380 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7c48a04-49fe-4143-ac3f-9104a74c143c" containerName="payload-pre-processing" Sep 14 11:04:41.970486 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:41.970396 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3973ad2-4fdc-4e2b-a465-3841543ed092" containerName="payload-processing" Sep 14 11:04:41.970486 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:41.970405 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d93957d-e182-444b-b9d6-84d8ce687cb0" containerName="revoke-keys" Sep 14 11:04:41.973570 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:41.973555 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-73290f4b-59d955fd4b-fnrpb" Sep 14 11:04:41.976152 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:41.976127 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-73290f4b\"" Sep 14 11:04:41.976264 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:41.976167 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-73290f4b-dockercfg-z4pm7\"" Sep 14 11:04:41.983534 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:41.983508 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-73290f4b-59d955fd4b-fnrpb"] Sep 14 11:04:42.082225 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:42.082187 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmwtn\" (UniqueName: \"kubernetes.io/projected/d655bfff-98d2-459f-908f-a67592854f94-kube-api-access-zmwtn\") pod \"payload-processing-e2e-migrate-73290f4b-59d955fd4b-fnrpb\" (UID: \"d655bfff-98d2-459f-908f-a67592854f94\") " pod="openshift-ingress/payload-processing-e2e-migrate-73290f4b-59d955fd4b-fnrpb" Sep 14 11:04:42.082437 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:42.082233 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d655bfff-98d2-459f-908f-a67592854f94-plugins-config-volume\") pod \"payload-processing-e2e-migrate-73290f4b-59d955fd4b-fnrpb\" (UID: \"d655bfff-98d2-459f-908f-a67592854f94\") " pod="openshift-ingress/payload-processing-e2e-migrate-73290f4b-59d955fd4b-fnrpb" Sep 14 11:04:42.182924 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:42.182882 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmwtn\" (UniqueName: \"kubernetes.io/projected/d655bfff-98d2-459f-908f-a67592854f94-kube-api-access-zmwtn\") pod \"payload-processing-e2e-migrate-73290f4b-59d955fd4b-fnrpb\" (UID: \"d655bfff-98d2-459f-908f-a67592854f94\") " pod="openshift-ingress/payload-processing-e2e-migrate-73290f4b-59d955fd4b-fnrpb" Sep 14 11:04:42.183145 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:42.182942 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d655bfff-98d2-459f-908f-a67592854f94-plugins-config-volume\") pod \"payload-processing-e2e-migrate-73290f4b-59d955fd4b-fnrpb\" (UID: \"d655bfff-98d2-459f-908f-a67592854f94\") " pod="openshift-ingress/payload-processing-e2e-migrate-73290f4b-59d955fd4b-fnrpb" Sep 14 11:04:42.183698 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:42.183679 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d655bfff-98d2-459f-908f-a67592854f94-plugins-config-volume\") pod \"payload-processing-e2e-migrate-73290f4b-59d955fd4b-fnrpb\" (UID: \"d655bfff-98d2-459f-908f-a67592854f94\") " pod="openshift-ingress/payload-processing-e2e-migrate-73290f4b-59d955fd4b-fnrpb" Sep 14 11:04:42.192165 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:42.192129 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmwtn\" (UniqueName: \"kubernetes.io/projected/d655bfff-98d2-459f-908f-a67592854f94-kube-api-access-zmwtn\") pod \"payload-processing-e2e-migrate-73290f4b-59d955fd4b-fnrpb\" (UID: \"d655bfff-98d2-459f-908f-a67592854f94\") " pod="openshift-ingress/payload-processing-e2e-migrate-73290f4b-59d955fd4b-fnrpb" Sep 14 11:04:42.285712 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:42.285681 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-73290f4b-59d955fd4b-fnrpb" Sep 14 11:04:42.561699 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:42.561669 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-73290f4b-59d955fd4b-fnrpb"] Sep 14 11:04:42.563469 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:04:42.563443 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd655bfff_98d2_459f_908f_a67592854f94.slice/crio-7162fdb54aa437c93e9c72ac68fee28270762f0bd88ae2dc515258cb8252f697 WatchSource:0}: Error finding container 7162fdb54aa437c93e9c72ac68fee28270762f0bd88ae2dc515258cb8252f697: Status 404 returned error can't find the container with id 7162fdb54aa437c93e9c72ac68fee28270762f0bd88ae2dc515258cb8252f697 Sep 14 11:04:43.223833 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:43.223794 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-73290f4b-59d955fd4b-fnrpb" event={"ID":"d655bfff-98d2-459f-908f-a67592854f94","Type":"ContainerStarted","Data":"9b3c80c6dcc75715164a7102b289d4aaf4cc70e343ea30047748c66371ae3353"} Sep 14 11:04:43.224244 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:43.223862 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-73290f4b-59d955fd4b-fnrpb" event={"ID":"d655bfff-98d2-459f-908f-a67592854f94","Type":"ContainerStarted","Data":"7162fdb54aa437c93e9c72ac68fee28270762f0bd88ae2dc515258cb8252f697"} Sep 14 11:04:43.224244 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:43.223895 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-73290f4b-59d955fd4b-fnrpb" Sep 14 11:04:43.242993 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:43.242936 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-73290f4b-59d955fd4b-fnrpb" podStartSLOduration=2.242919927 podStartE2EDuration="2.242919927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:04:43.240711652 +0000 UTC m=+956.983113622" watchObservedRunningTime="2026-09-14 11:04:43.242919927 +0000 UTC m=+956.985321896" Sep 14 11:04:54.231972 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:04:54.231939 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-73290f4b-59d955fd4b-fnrpb" Sep 14 11:05:08.149756 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:08.149676 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73290f4b-c8tvm"] Sep 14 11:05:08.158214 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:08.158194 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73290f4b-c8tvm" Sep 14 11:05:08.161168 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:08.161138 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pv9g8\"" Sep 14 11:05:08.161959 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:08.161930 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73290f4b-c8tvm"] Sep 14 11:05:08.244724 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:08.244689 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ccda2d5-0cc7-4e95-874c-eafdd7eaf175-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-73290f4b-c8tvm\" (UID: \"0ccda2d5-0cc7-4e95-874c-eafdd7eaf175\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73290f4b-c8tvm" Sep 14 11:05:08.345819 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:08.345775 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ccda2d5-0cc7-4e95-874c-eafdd7eaf175-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-73290f4b-c8tvm\" (UID: \"0ccda2d5-0cc7-4e95-874c-eafdd7eaf175\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73290f4b-c8tvm" Sep 14 11:05:08.346407 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:08.346388 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ccda2d5-0cc7-4e95-874c-eafdd7eaf175-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-73290f4b-c8tvm\" (UID: \"0ccda2d5-0cc7-4e95-874c-eafdd7eaf175\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73290f4b-c8tvm" Sep 14 11:05:08.469723 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:08.469635 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73290f4b-c8tvm" Sep 14 11:05:08.789478 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:08.789449 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73290f4b-c8tvm"] Sep 14 11:05:08.792322 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:05:08.792293 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ccda2d5_0cc7_4e95_874c_eafdd7eaf175.slice/crio-332ce38f97de26204622cb76e5dd1d1e17deb6e3e0dfede72eeaabce29c0e533 WatchSource:0}: Error finding container 332ce38f97de26204622cb76e5dd1d1e17deb6e3e0dfede72eeaabce29c0e533: Status 404 returned error can't find the container with id 332ce38f97de26204622cb76e5dd1d1e17deb6e3e0dfede72eeaabce29c0e533 Sep 14 11:05:09.357556 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:09.357453 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73290f4b-c8tvm" event={"ID":"0ccda2d5-0cc7-4e95-874c-eafdd7eaf175","Type":"ContainerStarted","Data":"b6a0395114b6deb8d95eaae3f60df62375f24d7e422e4b6d73ede821f2664674"} Sep 14 11:05:09.357556 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:09.357536 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73290f4b-c8tvm" event={"ID":"0ccda2d5-0cc7-4e95-874c-eafdd7eaf175","Type":"ContainerStarted","Data":"332ce38f97de26204622cb76e5dd1d1e17deb6e3e0dfede72eeaabce29c0e533"} Sep 14 11:05:09.376828 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:09.376775 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73290f4b-c8tvm" podStartSLOduration=1.3767603 podStartE2EDuration="1.3767603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:05:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:05:09.375238732 +0000 UTC m=+983.117640701" watchObservedRunningTime="2026-09-14 11:05:09.3767603 +0000 UTC m=+983.119162270" Sep 14 11:05:10.364537 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:10.364497 2581 generic.go:356] "Generic (PLEG): container finished" podID="0ccda2d5-0cc7-4e95-874c-eafdd7eaf175" containerID="b6a0395114b6deb8d95eaae3f60df62375f24d7e422e4b6d73ede821f2664674" exitCode=0 Sep 14 11:05:10.364948 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:10.364585 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73290f4b-c8tvm" event={"ID":"0ccda2d5-0cc7-4e95-874c-eafdd7eaf175","Type":"ContainerDied","Data":"b6a0395114b6deb8d95eaae3f60df62375f24d7e422e4b6d73ede821f2664674"} Sep 14 11:05:11.505066 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:11.505041 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73290f4b-c8tvm" Sep 14 11:05:11.581309 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:11.581276 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ccda2d5-0cc7-4e95-874c-eafdd7eaf175-maas-api-service-ca\") pod \"0ccda2d5-0cc7-4e95-874c-eafdd7eaf175\" (UID: \"0ccda2d5-0cc7-4e95-874c-eafdd7eaf175\") " Sep 14 11:05:11.581677 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:11.581655 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ccda2d5-0cc7-4e95-874c-eafdd7eaf175-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0ccda2d5-0cc7-4e95-874c-eafdd7eaf175" (UID: "0ccda2d5-0cc7-4e95-874c-eafdd7eaf175"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:05:11.683051 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:11.682966 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ccda2d5-0cc7-4e95-874c-eafdd7eaf175-maas-api-service-ca\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:05:12.375894 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:12.375862 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73290f4b-c8tvm" Sep 14 11:05:12.375894 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:12.375876 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73290f4b-c8tvm" event={"ID":"0ccda2d5-0cc7-4e95-874c-eafdd7eaf175","Type":"ContainerDied","Data":"332ce38f97de26204622cb76e5dd1d1e17deb6e3e0dfede72eeaabce29c0e533"} Sep 14 11:05:12.376074 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:12.375923 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="332ce38f97de26204622cb76e5dd1d1e17deb6e3e0dfede72eeaabce29c0e533" Sep 14 11:05:20.043994 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.043953 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7984b1/simulator-w0-7984b1-kserve-77b7979974-n6lvt"] Sep 14 11:05:20.044718 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.044697 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ccda2d5-0cc7-4e95-874c-eafdd7eaf175" containerName="revoke-keys" Sep 14 11:05:20.044767 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.044723 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ccda2d5-0cc7-4e95-874c-eafdd7eaf175" containerName="revoke-keys" Sep 14 11:05:20.044915 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.044900 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ccda2d5-0cc7-4e95-874c-eafdd7eaf175" containerName="revoke-keys" Sep 14 11:05:20.048829 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.048805 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7984b1/simulator-w0-7984b1-kserve-77b7979974-n6lvt" Sep 14 11:05:20.051575 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.051554 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7984b1\"/\"simulator-w0-7984b1-kserve-self-signed-certs\"" Sep 14 11:05:20.052580 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.052554 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7984b1\"/\"kube-root-ca.crt\"" Sep 14 11:05:20.052685 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.052581 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7984b1\"/\"openshift-service-ca.crt\"" Sep 14 11:05:20.053033 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.053001 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7984b1\"/\"default-dockercfg-ff6n2\"" Sep 14 11:05:20.063859 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.063817 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7984b1/simulator-w0-7984b1-kserve-77b7979974-n6lvt"] Sep 14 11:05:20.161648 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.161605 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-73290f4b-59d955fd4b-fnrpb"] Sep 14 11:05:20.161939 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.161884 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-73290f4b-59d955fd4b-fnrpb" podUID="d655bfff-98d2-459f-908f-a67592854f94" containerName="payload-processing" containerID="cri-o://9b3c80c6dcc75715164a7102b289d4aaf4cc70e343ea30047748c66371ae3353" gracePeriod=30 Sep 14 11:05:20.165724 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.165689 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57bc673d-5288-43bc-8712-45dc88500013-home\") pod \"simulator-w0-7984b1-kserve-77b7979974-n6lvt\" (UID: \"57bc673d-5288-43bc-8712-45dc88500013\") " pod="e2e-models-e2e-worker-w0-7984b1/simulator-w0-7984b1-kserve-77b7979974-n6lvt" Sep 14 11:05:20.165886 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.165814 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57bc673d-5288-43bc-8712-45dc88500013-dshm\") pod \"simulator-w0-7984b1-kserve-77b7979974-n6lvt\" (UID: \"57bc673d-5288-43bc-8712-45dc88500013\") " pod="e2e-models-e2e-worker-w0-7984b1/simulator-w0-7984b1-kserve-77b7979974-n6lvt" Sep 14 11:05:20.165886 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.165850 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57bc673d-5288-43bc-8712-45dc88500013-model-cache\") pod \"simulator-w0-7984b1-kserve-77b7979974-n6lvt\" (UID: \"57bc673d-5288-43bc-8712-45dc88500013\") " pod="e2e-models-e2e-worker-w0-7984b1/simulator-w0-7984b1-kserve-77b7979974-n6lvt" Sep 14 11:05:20.166075 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.166055 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57bc673d-5288-43bc-8712-45dc88500013-tmp-dir\") pod \"simulator-w0-7984b1-kserve-77b7979974-n6lvt\" (UID: \"57bc673d-5288-43bc-8712-45dc88500013\") " pod="e2e-models-e2e-worker-w0-7984b1/simulator-w0-7984b1-kserve-77b7979974-n6lvt" Sep 14 11:05:20.166240 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.166219 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57bc673d-5288-43bc-8712-45dc88500013-tls-certs\") pod \"simulator-w0-7984b1-kserve-77b7979974-n6lvt\" (UID: \"57bc673d-5288-43bc-8712-45dc88500013\") " pod="e2e-models-e2e-worker-w0-7984b1/simulator-w0-7984b1-kserve-77b7979974-n6lvt" Sep 14 11:05:20.166433 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.166411 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6trj\" (UniqueName: \"kubernetes.io/projected/57bc673d-5288-43bc-8712-45dc88500013-kube-api-access-z6trj\") pod \"simulator-w0-7984b1-kserve-77b7979974-n6lvt\" (UID: \"57bc673d-5288-43bc-8712-45dc88500013\") " pod="e2e-models-e2e-worker-w0-7984b1/simulator-w0-7984b1-kserve-77b7979974-n6lvt" Sep 14 11:05:20.267403 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.267371 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57bc673d-5288-43bc-8712-45dc88500013-home\") pod \"simulator-w0-7984b1-kserve-77b7979974-n6lvt\" (UID: \"57bc673d-5288-43bc-8712-45dc88500013\") " pod="e2e-models-e2e-worker-w0-7984b1/simulator-w0-7984b1-kserve-77b7979974-n6lvt" Sep 14 11:05:20.267594 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.267430 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57bc673d-5288-43bc-8712-45dc88500013-dshm\") pod \"simulator-w0-7984b1-kserve-77b7979974-n6lvt\" (UID: \"57bc673d-5288-43bc-8712-45dc88500013\") " pod="e2e-models-e2e-worker-w0-7984b1/simulator-w0-7984b1-kserve-77b7979974-n6lvt" Sep 14 11:05:20.267594 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.267447 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57bc673d-5288-43bc-8712-45dc88500013-model-cache\") pod \"simulator-w0-7984b1-kserve-77b7979974-n6lvt\" (UID: \"57bc673d-5288-43bc-8712-45dc88500013\") " pod="e2e-models-e2e-worker-w0-7984b1/simulator-w0-7984b1-kserve-77b7979974-n6lvt" Sep 14 11:05:20.267594 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.267482 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57bc673d-5288-43bc-8712-45dc88500013-tmp-dir\") pod \"simulator-w0-7984b1-kserve-77b7979974-n6lvt\" (UID: \"57bc673d-5288-43bc-8712-45dc88500013\") " pod="e2e-models-e2e-worker-w0-7984b1/simulator-w0-7984b1-kserve-77b7979974-n6lvt" Sep 14 11:05:20.267594 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.267532 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57bc673d-5288-43bc-8712-45dc88500013-tls-certs\") pod \"simulator-w0-7984b1-kserve-77b7979974-n6lvt\" (UID: \"57bc673d-5288-43bc-8712-45dc88500013\") " pod="e2e-models-e2e-worker-w0-7984b1/simulator-w0-7984b1-kserve-77b7979974-n6lvt" Sep 14 11:05:20.267853 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.267670 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6trj\" (UniqueName: \"kubernetes.io/projected/57bc673d-5288-43bc-8712-45dc88500013-kube-api-access-z6trj\") pod \"simulator-w0-7984b1-kserve-77b7979974-n6lvt\" (UID: \"57bc673d-5288-43bc-8712-45dc88500013\") " pod="e2e-models-e2e-worker-w0-7984b1/simulator-w0-7984b1-kserve-77b7979974-n6lvt" Sep 14 11:05:20.267911 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.267866 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57bc673d-5288-43bc-8712-45dc88500013-home\") pod \"simulator-w0-7984b1-kserve-77b7979974-n6lvt\" (UID: \"57bc673d-5288-43bc-8712-45dc88500013\") " pod="e2e-models-e2e-worker-w0-7984b1/simulator-w0-7984b1-kserve-77b7979974-n6lvt" Sep 14 11:05:20.268012 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.267993 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57bc673d-5288-43bc-8712-45dc88500013-tmp-dir\") pod \"simulator-w0-7984b1-kserve-77b7979974-n6lvt\" (UID: \"57bc673d-5288-43bc-8712-45dc88500013\") " pod="e2e-models-e2e-worker-w0-7984b1/simulator-w0-7984b1-kserve-77b7979974-n6lvt" Sep 14 11:05:20.268074 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.267990 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57bc673d-5288-43bc-8712-45dc88500013-model-cache\") pod \"simulator-w0-7984b1-kserve-77b7979974-n6lvt\" (UID: \"57bc673d-5288-43bc-8712-45dc88500013\") " pod="e2e-models-e2e-worker-w0-7984b1/simulator-w0-7984b1-kserve-77b7979974-n6lvt" Sep 14 11:05:20.269779 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.269750 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57bc673d-5288-43bc-8712-45dc88500013-dshm\") pod \"simulator-w0-7984b1-kserve-77b7979974-n6lvt\" (UID: \"57bc673d-5288-43bc-8712-45dc88500013\") " pod="e2e-models-e2e-worker-w0-7984b1/simulator-w0-7984b1-kserve-77b7979974-n6lvt" Sep 14 11:05:20.270521 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.270499 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57bc673d-5288-43bc-8712-45dc88500013-tls-certs\") pod \"simulator-w0-7984b1-kserve-77b7979974-n6lvt\" (UID: \"57bc673d-5288-43bc-8712-45dc88500013\") " pod="e2e-models-e2e-worker-w0-7984b1/simulator-w0-7984b1-kserve-77b7979974-n6lvt" Sep 14 11:05:20.278801 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.278775 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6trj\" (UniqueName: \"kubernetes.io/projected/57bc673d-5288-43bc-8712-45dc88500013-kube-api-access-z6trj\") pod \"simulator-w0-7984b1-kserve-77b7979974-n6lvt\" (UID: \"57bc673d-5288-43bc-8712-45dc88500013\") " pod="e2e-models-e2e-worker-w0-7984b1/simulator-w0-7984b1-kserve-77b7979974-n6lvt" Sep 14 11:05:20.361072 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.361030 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7984b1/simulator-w0-7984b1-kserve-77b7979974-n6lvt" Sep 14 11:05:20.404731 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.404699 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-73290f4b-59d955fd4b-fnrpb" Sep 14 11:05:20.421142 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.420749 2581 generic.go:356] "Generic (PLEG): container finished" podID="d655bfff-98d2-459f-908f-a67592854f94" containerID="9b3c80c6dcc75715164a7102b289d4aaf4cc70e343ea30047748c66371ae3353" exitCode=0 Sep 14 11:05:20.421142 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.420804 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-73290f4b-59d955fd4b-fnrpb" Sep 14 11:05:20.421142 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.420978 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-73290f4b-59d955fd4b-fnrpb" event={"ID":"d655bfff-98d2-459f-908f-a67592854f94","Type":"ContainerDied","Data":"9b3c80c6dcc75715164a7102b289d4aaf4cc70e343ea30047748c66371ae3353"} Sep 14 11:05:20.421142 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.421049 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-73290f4b-59d955fd4b-fnrpb" event={"ID":"d655bfff-98d2-459f-908f-a67592854f94","Type":"ContainerDied","Data":"7162fdb54aa437c93e9c72ac68fee28270762f0bd88ae2dc515258cb8252f697"} Sep 14 11:05:20.421142 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.421084 2581 scope.go:117] "RemoveContainer" containerID="9b3c80c6dcc75715164a7102b289d4aaf4cc70e343ea30047748c66371ae3353" Sep 14 11:05:20.434668 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.434543 2581 scope.go:117] "RemoveContainer" containerID="9b3c80c6dcc75715164a7102b289d4aaf4cc70e343ea30047748c66371ae3353" Sep 14 11:05:20.434924 ip-10-0-137-53 kubenswrapper[2581]: E0914 11:05:20.434900 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b3c80c6dcc75715164a7102b289d4aaf4cc70e343ea30047748c66371ae3353\": container with ID starting with 9b3c80c6dcc75715164a7102b289d4aaf4cc70e343ea30047748c66371ae3353 not found: ID does not exist" containerID="9b3c80c6dcc75715164a7102b289d4aaf4cc70e343ea30047748c66371ae3353" Sep 14 11:05:20.435000 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.434934 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b3c80c6dcc75715164a7102b289d4aaf4cc70e343ea30047748c66371ae3353"} err="failed to get container status \"9b3c80c6dcc75715164a7102b289d4aaf4cc70e343ea30047748c66371ae3353\": rpc error: code = NotFound desc = could not find container \"9b3c80c6dcc75715164a7102b289d4aaf4cc70e343ea30047748c66371ae3353\": container with ID starting with 9b3c80c6dcc75715164a7102b289d4aaf4cc70e343ea30047748c66371ae3353 not found: ID does not exist" Sep 14 11:05:20.469236 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.469175 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmwtn\" (UniqueName: \"kubernetes.io/projected/d655bfff-98d2-459f-908f-a67592854f94-kube-api-access-zmwtn\") pod \"d655bfff-98d2-459f-908f-a67592854f94\" (UID: \"d655bfff-98d2-459f-908f-a67592854f94\") " Sep 14 11:05:20.469445 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.469321 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d655bfff-98d2-459f-908f-a67592854f94-plugins-config-volume\") pod \"d655bfff-98d2-459f-908f-a67592854f94\" (UID: \"d655bfff-98d2-459f-908f-a67592854f94\") " Sep 14 11:05:20.469723 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.469701 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d655bfff-98d2-459f-908f-a67592854f94-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d655bfff-98d2-459f-908f-a67592854f94" (UID: "d655bfff-98d2-459f-908f-a67592854f94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:05:20.471580 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.471556 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d655bfff-98d2-459f-908f-a67592854f94-kube-api-access-zmwtn" (OuterVolumeSpecName: "kube-api-access-zmwtn") pod "d655bfff-98d2-459f-908f-a67592854f94" (UID: "d655bfff-98d2-459f-908f-a67592854f94"). InnerVolumeSpecName "kube-api-access-zmwtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:05:20.474759 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.474736 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73290f4b-c8tvm"] Sep 14 11:05:20.481199 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.481172 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-73290f4b-c8tvm"] Sep 14 11:05:20.570930 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.570897 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmwtn\" (UniqueName: \"kubernetes.io/projected/d655bfff-98d2-459f-908f-a67592854f94-kube-api-access-zmwtn\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:05:20.570930 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.570930 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d655bfff-98d2-459f-908f-a67592854f94-plugins-config-volume\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:05:20.656972 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.656946 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7984b1/simulator-w0-7984b1-kserve-77b7979974-n6lvt"] Sep 14 11:05:20.657763 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:05:20.657735 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57bc673d_5288_43bc_8712_45dc88500013.slice/crio-f6e0357e369936b07036f3ed4c7b55ce4736545988c21535966864c58e86e3f7 WatchSource:0}: Error finding container f6e0357e369936b07036f3ed4c7b55ce4736545988c21535966864c58e86e3f7: Status 404 returned error can't find the container with id f6e0357e369936b07036f3ed4c7b55ce4736545988c21535966864c58e86e3f7 Sep 14 11:05:20.749958 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.749919 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-73290f4b-59d955fd4b-fnrpb"] Sep 14 11:05:20.756640 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.756611 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-73290f4b-59d955fd4b-fnrpb"] Sep 14 11:05:20.832595 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.832555 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ccda2d5-0cc7-4e95-874c-eafdd7eaf175" path="/var/lib/kubelet/pods/0ccda2d5-0cc7-4e95-874c-eafdd7eaf175/volumes" Sep 14 11:05:20.833042 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:20.833021 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d655bfff-98d2-459f-908f-a67592854f94" path="/var/lib/kubelet/pods/d655bfff-98d2-459f-908f-a67592854f94/volumes" Sep 14 11:05:21.428909 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:21.428874 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7984b1/simulator-w0-7984b1-kserve-77b7979974-n6lvt" event={"ID":"57bc673d-5288-43bc-8712-45dc88500013","Type":"ContainerStarted","Data":"8b72d79d244473e9414cdf680b76e8545707d2a05d348821b421b144c8a1ee1a"} Sep 14 11:05:21.429380 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:21.428918 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7984b1/simulator-w0-7984b1-kserve-77b7979974-n6lvt" event={"ID":"57bc673d-5288-43bc-8712-45dc88500013","Type":"ContainerStarted","Data":"f6e0357e369936b07036f3ed4c7b55ce4736545988c21535966864c58e86e3f7"} Sep 14 11:05:21.498231 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:21.498170 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7984b1/simulator-w0-7984b1-kserve-77b7979974-n6lvt" podStartSLOduration=1.498154797 podStartE2EDuration="1.498154797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:05:21.495835955 +0000 UTC m=+995.238237938" watchObservedRunningTime="2026-09-14 11:05:21.498154797 +0000 UTC m=+995.240556767" Sep 14 11:05:28.716690 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:28.716659 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1d16dc05-566f77cf98-x8hpc"] Sep 14 11:05:28.717081 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:28.717074 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d655bfff-98d2-459f-908f-a67592854f94" containerName="payload-processing" Sep 14 11:05:28.717123 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:28.717087 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d655bfff-98d2-459f-908f-a67592854f94" containerName="payload-processing" Sep 14 11:05:28.717177 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:28.717167 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d655bfff-98d2-459f-908f-a67592854f94" containerName="payload-processing" Sep 14 11:05:28.721446 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:28.721428 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d16dc05-566f77cf98-x8hpc" Sep 14 11:05:28.725745 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:28.725721 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1d16dc05\"" Sep 14 11:05:28.725847 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:28.725809 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1d16dc05-dockercfg-6kzs9\"" Sep 14 11:05:28.740585 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:28.740546 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1d16dc05-566f77cf98-x8hpc"] Sep 14 11:05:28.763291 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:28.763261 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/353a5313-d18b-4145-9143-2f3ef14b0b97-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1d16dc05-566f77cf98-x8hpc\" (UID: \"353a5313-d18b-4145-9143-2f3ef14b0b97\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1d16dc05-566f77cf98-x8hpc" Sep 14 11:05:28.763451 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:28.763322 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lczfm\" (UniqueName: \"kubernetes.io/projected/353a5313-d18b-4145-9143-2f3ef14b0b97-kube-api-access-lczfm\") pod \"payload-pre-processing-e2e-ait-1d16dc05-566f77cf98-x8hpc\" (UID: \"353a5313-d18b-4145-9143-2f3ef14b0b97\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1d16dc05-566f77cf98-x8hpc" Sep 14 11:05:28.864101 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:28.864070 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/353a5313-d18b-4145-9143-2f3ef14b0b97-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1d16dc05-566f77cf98-x8hpc\" (UID: \"353a5313-d18b-4145-9143-2f3ef14b0b97\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1d16dc05-566f77cf98-x8hpc" Sep 14 11:05:28.864266 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:28.864140 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lczfm\" (UniqueName: \"kubernetes.io/projected/353a5313-d18b-4145-9143-2f3ef14b0b97-kube-api-access-lczfm\") pod \"payload-pre-processing-e2e-ait-1d16dc05-566f77cf98-x8hpc\" (UID: \"353a5313-d18b-4145-9143-2f3ef14b0b97\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1d16dc05-566f77cf98-x8hpc" Sep 14 11:05:28.864829 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:28.864810 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/353a5313-d18b-4145-9143-2f3ef14b0b97-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1d16dc05-566f77cf98-x8hpc\" (UID: \"353a5313-d18b-4145-9143-2f3ef14b0b97\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1d16dc05-566f77cf98-x8hpc" Sep 14 11:05:28.874590 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:28.874557 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lczfm\" (UniqueName: \"kubernetes.io/projected/353a5313-d18b-4145-9143-2f3ef14b0b97-kube-api-access-lczfm\") pod \"payload-pre-processing-e2e-ait-1d16dc05-566f77cf98-x8hpc\" (UID: \"353a5313-d18b-4145-9143-2f3ef14b0b97\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1d16dc05-566f77cf98-x8hpc" Sep 14 11:05:29.031081 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:29.031043 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d16dc05-566f77cf98-x8hpc" Sep 14 11:05:29.314861 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:29.314833 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1d16dc05-566f77cf98-x8hpc"] Sep 14 11:05:29.316308 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:05:29.316283 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod353a5313_d18b_4145_9143_2f3ef14b0b97.slice/crio-b67c64430421de14fa163ea171e45d1a1cab100245a97e3d582b1ffab02ed2ea WatchSource:0}: Error finding container b67c64430421de14fa163ea171e45d1a1cab100245a97e3d582b1ffab02ed2ea: Status 404 returned error can't find the container with id b67c64430421de14fa163ea171e45d1a1cab100245a97e3d582b1ffab02ed2ea Sep 14 11:05:29.474507 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:29.474455 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d16dc05-566f77cf98-x8hpc" event={"ID":"353a5313-d18b-4145-9143-2f3ef14b0b97","Type":"ContainerStarted","Data":"a2797b838a4b86e2b904967a7c19c515ee71dd5aaf58ef6c8ba8e18482f9a9c3"} Sep 14 11:05:29.474693 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:29.474550 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d16dc05-566f77cf98-x8hpc" event={"ID":"353a5313-d18b-4145-9143-2f3ef14b0b97","Type":"ContainerStarted","Data":"b67c64430421de14fa163ea171e45d1a1cab100245a97e3d582b1ffab02ed2ea"} Sep 14 11:05:29.474693 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:29.474597 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d16dc05-566f77cf98-x8hpc" Sep 14 11:05:29.502081 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:29.502029 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d16dc05-566f77cf98-x8hpc" podStartSLOduration=1.5020134 podStartE2EDuration="1.5020134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:05:29.500005939 +0000 UTC m=+1003.242407909" watchObservedRunningTime="2026-09-14 11:05:29.5020134 +0000 UTC m=+1003.244415397" Sep 14 11:05:30.361479 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:30.361444 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7984b1/simulator-w0-7984b1-kserve-77b7979974-n6lvt" Sep 14 11:05:30.361979 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:30.361501 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7984b1/simulator-w0-7984b1-kserve-77b7979974-n6lvt" Sep 14 11:05:30.374678 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:30.374649 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7984b1/simulator-w0-7984b1-kserve-77b7979974-n6lvt" Sep 14 11:05:30.491742 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:30.491712 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7984b1/simulator-w0-7984b1-kserve-77b7979974-n6lvt" Sep 14 11:05:40.483048 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:40.483016 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d16dc05-566f77cf98-x8hpc" Sep 14 11:05:55.054534 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:55.054498 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d16dc05-s29pk"] Sep 14 11:05:55.059449 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:55.059427 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d16dc05-s29pk" Sep 14 11:05:55.062219 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:55.062201 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pv9g8\"" Sep 14 11:05:55.081272 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:55.081235 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d16dc05-s29pk"] Sep 14 11:05:55.231193 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:55.231152 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c31878eb-73f6-4840-939e-29fb77803d77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1d16dc05-s29pk\" (UID: \"c31878eb-73f6-4840-939e-29fb77803d77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d16dc05-s29pk" Sep 14 11:05:55.332443 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:55.332318 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c31878eb-73f6-4840-939e-29fb77803d77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1d16dc05-s29pk\" (UID: \"c31878eb-73f6-4840-939e-29fb77803d77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d16dc05-s29pk" Sep 14 11:05:55.332974 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:55.332950 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c31878eb-73f6-4840-939e-29fb77803d77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1d16dc05-s29pk\" (UID: \"c31878eb-73f6-4840-939e-29fb77803d77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d16dc05-s29pk" Sep 14 11:05:55.370303 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:55.370263 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d16dc05-s29pk" Sep 14 11:05:55.647065 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:55.647043 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d16dc05-s29pk"] Sep 14 11:05:55.648396 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:05:55.648364 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc31878eb_73f6_4840_939e_29fb77803d77.slice/crio-d81cc883ed3fb2eb407a9000834c6949f0dfa4099c739f58231aec1d7d644463 WatchSource:0}: Error finding container d81cc883ed3fb2eb407a9000834c6949f0dfa4099c739f58231aec1d7d644463: Status 404 returned error can't find the container with id d81cc883ed3fb2eb407a9000834c6949f0dfa4099c739f58231aec1d7d644463 Sep 14 11:05:56.623348 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:56.623244 2581 generic.go:356] "Generic (PLEG): container finished" podID="c31878eb-73f6-4840-939e-29fb77803d77" containerID="e4bf9ab2502da1272f15047302a28908e59cf1cd525c55199552d705e1019f9f" exitCode=0 Sep 14 11:05:56.623348 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:56.623291 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d16dc05-s29pk" event={"ID":"c31878eb-73f6-4840-939e-29fb77803d77","Type":"ContainerDied","Data":"e4bf9ab2502da1272f15047302a28908e59cf1cd525c55199552d705e1019f9f"} Sep 14 11:05:56.623348 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:56.623348 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d16dc05-s29pk" event={"ID":"c31878eb-73f6-4840-939e-29fb77803d77","Type":"ContainerStarted","Data":"d81cc883ed3fb2eb407a9000834c6949f0dfa4099c739f58231aec1d7d644463"} Sep 14 11:05:57.773523 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:57.773496 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d16dc05-s29pk" Sep 14 11:05:57.856068 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:57.856035 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c31878eb-73f6-4840-939e-29fb77803d77-maas-api-service-ca\") pod \"c31878eb-73f6-4840-939e-29fb77803d77\" (UID: \"c31878eb-73f6-4840-939e-29fb77803d77\") " Sep 14 11:05:57.856423 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:57.856398 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c31878eb-73f6-4840-939e-29fb77803d77-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c31878eb-73f6-4840-939e-29fb77803d77" (UID: "c31878eb-73f6-4840-939e-29fb77803d77"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:05:57.957268 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:57.957178 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c31878eb-73f6-4840-939e-29fb77803d77-maas-api-service-ca\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:05:58.642097 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:58.642061 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d16dc05-s29pk" event={"ID":"c31878eb-73f6-4840-939e-29fb77803d77","Type":"ContainerDied","Data":"d81cc883ed3fb2eb407a9000834c6949f0dfa4099c739f58231aec1d7d644463"} Sep 14 11:05:58.642097 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:58.642094 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d16dc05-s29pk" Sep 14 11:05:58.642310 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:05:58.642117 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d81cc883ed3fb2eb407a9000834c6949f0dfa4099c739f58231aec1d7d644463" Sep 14 11:06:04.557974 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:04.557940 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-04195c/test-model-04195c-kserve-54bf6676f-9rn9r"] Sep 14 11:06:04.558408 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:04.558395 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c31878eb-73f6-4840-939e-29fb77803d77" containerName="revoke-keys" Sep 14 11:06:04.558408 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:04.558409 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c31878eb-73f6-4840-939e-29fb77803d77" containerName="revoke-keys" Sep 14 11:06:04.558493 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:04.558477 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c31878eb-73f6-4840-939e-29fb77803d77" containerName="revoke-keys" Sep 14 11:06:04.561905 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:04.561887 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-04195c/test-model-04195c-kserve-54bf6676f-9rn9r" Sep 14 11:06:04.576222 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:04.576199 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-04195c\"/\"kube-root-ca.crt\"" Sep 14 11:06:04.576397 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:04.576266 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-04195c\"/\"test-model-04195c-kserve-self-signed-certs\"" Sep 14 11:06:04.576397 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:04.576271 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-04195c\"/\"openshift-service-ca.crt\"" Sep 14 11:06:04.576397 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:04.576291 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-04195c\"/\"default-dockercfg-vql9r\"" Sep 14 11:06:04.663829 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:04.663794 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-04195c/test-model-04195c-kserve-54bf6676f-9rn9r"] Sep 14 11:06:04.726853 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:04.726808 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb7ef955-452f-477c-bb40-432847f1f672-dshm\") pod \"test-model-04195c-kserve-54bf6676f-9rn9r\" (UID: \"bb7ef955-452f-477c-bb40-432847f1f672\") " pod="ai-tenant-e2e-inf-a-04195c/test-model-04195c-kserve-54bf6676f-9rn9r" Sep 14 11:06:04.727060 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:04.726866 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-265tv\" (UniqueName: \"kubernetes.io/projected/bb7ef955-452f-477c-bb40-432847f1f672-kube-api-access-265tv\") pod \"test-model-04195c-kserve-54bf6676f-9rn9r\" (UID: \"bb7ef955-452f-477c-bb40-432847f1f672\") " pod="ai-tenant-e2e-inf-a-04195c/test-model-04195c-kserve-54bf6676f-9rn9r" Sep 14 11:06:04.727060 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:04.726977 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb7ef955-452f-477c-bb40-432847f1f672-tmp-dir\") pod \"test-model-04195c-kserve-54bf6676f-9rn9r\" (UID: \"bb7ef955-452f-477c-bb40-432847f1f672\") " pod="ai-tenant-e2e-inf-a-04195c/test-model-04195c-kserve-54bf6676f-9rn9r" Sep 14 11:06:04.727060 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:04.727026 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb7ef955-452f-477c-bb40-432847f1f672-model-cache\") pod \"test-model-04195c-kserve-54bf6676f-9rn9r\" (UID: \"bb7ef955-452f-477c-bb40-432847f1f672\") " pod="ai-tenant-e2e-inf-a-04195c/test-model-04195c-kserve-54bf6676f-9rn9r" Sep 14 11:06:04.727223 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:04.727065 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb7ef955-452f-477c-bb40-432847f1f672-tls-certs\") pod \"test-model-04195c-kserve-54bf6676f-9rn9r\" (UID: \"bb7ef955-452f-477c-bb40-432847f1f672\") " pod="ai-tenant-e2e-inf-a-04195c/test-model-04195c-kserve-54bf6676f-9rn9r" Sep 14 11:06:04.727223 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:04.727157 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb7ef955-452f-477c-bb40-432847f1f672-home\") pod \"test-model-04195c-kserve-54bf6676f-9rn9r\" (UID: \"bb7ef955-452f-477c-bb40-432847f1f672\") " pod="ai-tenant-e2e-inf-a-04195c/test-model-04195c-kserve-54bf6676f-9rn9r" Sep 14 11:06:04.828446 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:04.828351 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb7ef955-452f-477c-bb40-432847f1f672-home\") pod \"test-model-04195c-kserve-54bf6676f-9rn9r\" (UID: \"bb7ef955-452f-477c-bb40-432847f1f672\") " pod="ai-tenant-e2e-inf-a-04195c/test-model-04195c-kserve-54bf6676f-9rn9r" Sep 14 11:06:04.828446 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:04.828431 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb7ef955-452f-477c-bb40-432847f1f672-dshm\") pod \"test-model-04195c-kserve-54bf6676f-9rn9r\" (UID: \"bb7ef955-452f-477c-bb40-432847f1f672\") " pod="ai-tenant-e2e-inf-a-04195c/test-model-04195c-kserve-54bf6676f-9rn9r" Sep 14 11:06:04.828680 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:04.828469 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-265tv\" (UniqueName: \"kubernetes.io/projected/bb7ef955-452f-477c-bb40-432847f1f672-kube-api-access-265tv\") pod \"test-model-04195c-kserve-54bf6676f-9rn9r\" (UID: \"bb7ef955-452f-477c-bb40-432847f1f672\") " pod="ai-tenant-e2e-inf-a-04195c/test-model-04195c-kserve-54bf6676f-9rn9r" Sep 14 11:06:04.828680 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:04.828630 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb7ef955-452f-477c-bb40-432847f1f672-tmp-dir\") pod \"test-model-04195c-kserve-54bf6676f-9rn9r\" (UID: \"bb7ef955-452f-477c-bb40-432847f1f672\") " pod="ai-tenant-e2e-inf-a-04195c/test-model-04195c-kserve-54bf6676f-9rn9r" Sep 14 11:06:04.828802 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:04.828696 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb7ef955-452f-477c-bb40-432847f1f672-model-cache\") pod \"test-model-04195c-kserve-54bf6676f-9rn9r\" (UID: \"bb7ef955-452f-477c-bb40-432847f1f672\") " pod="ai-tenant-e2e-inf-a-04195c/test-model-04195c-kserve-54bf6676f-9rn9r" Sep 14 11:06:04.828802 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:04.828725 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb7ef955-452f-477c-bb40-432847f1f672-tls-certs\") pod \"test-model-04195c-kserve-54bf6676f-9rn9r\" (UID: \"bb7ef955-452f-477c-bb40-432847f1f672\") " pod="ai-tenant-e2e-inf-a-04195c/test-model-04195c-kserve-54bf6676f-9rn9r" Sep 14 11:06:04.828802 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:04.828755 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb7ef955-452f-477c-bb40-432847f1f672-home\") pod \"test-model-04195c-kserve-54bf6676f-9rn9r\" (UID: \"bb7ef955-452f-477c-bb40-432847f1f672\") " pod="ai-tenant-e2e-inf-a-04195c/test-model-04195c-kserve-54bf6676f-9rn9r" Sep 14 11:06:04.829069 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:04.829046 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb7ef955-452f-477c-bb40-432847f1f672-tmp-dir\") pod \"test-model-04195c-kserve-54bf6676f-9rn9r\" (UID: \"bb7ef955-452f-477c-bb40-432847f1f672\") " pod="ai-tenant-e2e-inf-a-04195c/test-model-04195c-kserve-54bf6676f-9rn9r" Sep 14 11:06:04.829133 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:04.829070 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb7ef955-452f-477c-bb40-432847f1f672-model-cache\") pod \"test-model-04195c-kserve-54bf6676f-9rn9r\" (UID: \"bb7ef955-452f-477c-bb40-432847f1f672\") " pod="ai-tenant-e2e-inf-a-04195c/test-model-04195c-kserve-54bf6676f-9rn9r" Sep 14 11:06:04.830915 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:04.830891 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb7ef955-452f-477c-bb40-432847f1f672-dshm\") pod \"test-model-04195c-kserve-54bf6676f-9rn9r\" (UID: \"bb7ef955-452f-477c-bb40-432847f1f672\") " pod="ai-tenant-e2e-inf-a-04195c/test-model-04195c-kserve-54bf6676f-9rn9r" Sep 14 11:06:04.831539 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:04.831522 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb7ef955-452f-477c-bb40-432847f1f672-tls-certs\") pod \"test-model-04195c-kserve-54bf6676f-9rn9r\" (UID: \"bb7ef955-452f-477c-bb40-432847f1f672\") " pod="ai-tenant-e2e-inf-a-04195c/test-model-04195c-kserve-54bf6676f-9rn9r" Sep 14 11:06:04.849936 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:04.849907 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-265tv\" (UniqueName: \"kubernetes.io/projected/bb7ef955-452f-477c-bb40-432847f1f672-kube-api-access-265tv\") pod \"test-model-04195c-kserve-54bf6676f-9rn9r\" (UID: \"bb7ef955-452f-477c-bb40-432847f1f672\") " pod="ai-tenant-e2e-inf-a-04195c/test-model-04195c-kserve-54bf6676f-9rn9r" Sep 14 11:06:04.870858 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:04.870829 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-04195c/test-model-04195c-kserve-54bf6676f-9rn9r" Sep 14 11:06:05.151291 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:05.151266 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-04195c/test-model-04195c-kserve-54bf6676f-9rn9r"] Sep 14 11:06:05.153431 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:06:05.153410 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb7ef955_452f_477c_bb40_432847f1f672.slice/crio-feef36de15a973a6243bd1e738f5024d4608949022a13836a3ecee30eeab8dff WatchSource:0}: Error finding container feef36de15a973a6243bd1e738f5024d4608949022a13836a3ecee30eeab8dff: Status 404 returned error can't find the container with id feef36de15a973a6243bd1e738f5024d4608949022a13836a3ecee30eeab8dff Sep 14 11:06:05.680506 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:05.680474 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-04195c/test-model-04195c-kserve-54bf6676f-9rn9r" event={"ID":"bb7ef955-452f-477c-bb40-432847f1f672","Type":"ContainerStarted","Data":"5ae18d530150cf2a3c8363760bed718091a789aef584f992f9b3883cd3d1452f"} Sep 14 11:06:05.680989 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:05.680516 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-04195c/test-model-04195c-kserve-54bf6676f-9rn9r" event={"ID":"bb7ef955-452f-477c-bb40-432847f1f672","Type":"ContainerStarted","Data":"feef36de15a973a6243bd1e738f5024d4608949022a13836a3ecee30eeab8dff"} Sep 14 11:06:05.704096 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:05.704046 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-04195c/test-model-04195c-kserve-54bf6676f-9rn9r" podStartSLOduration=1.704029184 podStartE2EDuration="1.704029184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:06:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:06:05.701059324 +0000 UTC m=+1039.443461296" watchObservedRunningTime="2026-09-14 11:06:05.704029184 +0000 UTC m=+1039.446431153" Sep 14 11:06:07.045680 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:07.045642 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1d16dc05-566f77cf98-x8hpc"] Sep 14 11:06:07.046105 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:07.045917 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d16dc05-566f77cf98-x8hpc" podUID="353a5313-d18b-4145-9143-2f3ef14b0b97" containerName="payload-pre-processing" containerID="cri-o://a2797b838a4b86e2b904967a7c19c515ee71dd5aaf58ef6c8ba8e18482f9a9c3" gracePeriod=30 Sep 14 11:06:07.289142 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:07.289106 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d16dc05-s29pk"] Sep 14 11:06:07.291288 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:07.291268 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1d16dc05-s29pk"] Sep 14 11:06:07.303986 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:07.303931 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d16dc05-566f77cf98-x8hpc" Sep 14 11:06:07.353302 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:07.353266 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/353a5313-d18b-4145-9143-2f3ef14b0b97-plugins-config-volume\") pod \"353a5313-d18b-4145-9143-2f3ef14b0b97\" (UID: \"353a5313-d18b-4145-9143-2f3ef14b0b97\") " Sep 14 11:06:07.353302 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:07.353303 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lczfm\" (UniqueName: \"kubernetes.io/projected/353a5313-d18b-4145-9143-2f3ef14b0b97-kube-api-access-lczfm\") pod \"353a5313-d18b-4145-9143-2f3ef14b0b97\" (UID: \"353a5313-d18b-4145-9143-2f3ef14b0b97\") " Sep 14 11:06:07.353732 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:07.353710 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/353a5313-d18b-4145-9143-2f3ef14b0b97-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "353a5313-d18b-4145-9143-2f3ef14b0b97" (UID: "353a5313-d18b-4145-9143-2f3ef14b0b97"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:06:07.355637 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:07.355611 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/353a5313-d18b-4145-9143-2f3ef14b0b97-kube-api-access-lczfm" (OuterVolumeSpecName: "kube-api-access-lczfm") pod "353a5313-d18b-4145-9143-2f3ef14b0b97" (UID: "353a5313-d18b-4145-9143-2f3ef14b0b97"). InnerVolumeSpecName "kube-api-access-lczfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:06:07.454090 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:07.454060 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/353a5313-d18b-4145-9143-2f3ef14b0b97-plugins-config-volume\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:06:07.454090 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:07.454090 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lczfm\" (UniqueName: \"kubernetes.io/projected/353a5313-d18b-4145-9143-2f3ef14b0b97-kube-api-access-lczfm\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:06:07.696918 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:07.696821 2581 generic.go:356] "Generic (PLEG): container finished" podID="353a5313-d18b-4145-9143-2f3ef14b0b97" containerID="a2797b838a4b86e2b904967a7c19c515ee71dd5aaf58ef6c8ba8e18482f9a9c3" exitCode=0 Sep 14 11:06:07.696918 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:07.696865 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d16dc05-566f77cf98-x8hpc" Sep 14 11:06:07.697152 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:07.696916 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d16dc05-566f77cf98-x8hpc" event={"ID":"353a5313-d18b-4145-9143-2f3ef14b0b97","Type":"ContainerDied","Data":"a2797b838a4b86e2b904967a7c19c515ee71dd5aaf58ef6c8ba8e18482f9a9c3"} Sep 14 11:06:07.697152 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:07.696973 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1d16dc05-566f77cf98-x8hpc" event={"ID":"353a5313-d18b-4145-9143-2f3ef14b0b97","Type":"ContainerDied","Data":"b67c64430421de14fa163ea171e45d1a1cab100245a97e3d582b1ffab02ed2ea"} Sep 14 11:06:07.697152 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:07.697003 2581 scope.go:117] "RemoveContainer" containerID="a2797b838a4b86e2b904967a7c19c515ee71dd5aaf58ef6c8ba8e18482f9a9c3" Sep 14 11:06:07.706373 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:07.706333 2581 scope.go:117] "RemoveContainer" containerID="a2797b838a4b86e2b904967a7c19c515ee71dd5aaf58ef6c8ba8e18482f9a9c3" Sep 14 11:06:07.706748 ip-10-0-137-53 kubenswrapper[2581]: E0914 11:06:07.706730 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2797b838a4b86e2b904967a7c19c515ee71dd5aaf58ef6c8ba8e18482f9a9c3\": container with ID starting with a2797b838a4b86e2b904967a7c19c515ee71dd5aaf58ef6c8ba8e18482f9a9c3 not found: ID does not exist" containerID="a2797b838a4b86e2b904967a7c19c515ee71dd5aaf58ef6c8ba8e18482f9a9c3" Sep 14 11:06:07.706828 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:07.706758 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2797b838a4b86e2b904967a7c19c515ee71dd5aaf58ef6c8ba8e18482f9a9c3"} err="failed to get container status \"a2797b838a4b86e2b904967a7c19c515ee71dd5aaf58ef6c8ba8e18482f9a9c3\": rpc error: code = NotFound desc = could not find container \"a2797b838a4b86e2b904967a7c19c515ee71dd5aaf58ef6c8ba8e18482f9a9c3\": container with ID starting with a2797b838a4b86e2b904967a7c19c515ee71dd5aaf58ef6c8ba8e18482f9a9c3 not found: ID does not exist" Sep 14 11:06:07.723794 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:07.723759 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1d16dc05-566f77cf98-x8hpc"] Sep 14 11:06:07.729060 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:07.729034 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1d16dc05-566f77cf98-x8hpc"] Sep 14 11:06:08.832461 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:08.832427 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="353a5313-d18b-4145-9143-2f3ef14b0b97" path="/var/lib/kubelet/pods/353a5313-d18b-4145-9143-2f3ef14b0b97/volumes" Sep 14 11:06:08.832841 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:08.832787 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c31878eb-73f6-4840-939e-29fb77803d77" path="/var/lib/kubelet/pods/c31878eb-73f6-4840-939e-29fb77803d77/volumes" Sep 14 11:06:09.097864 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:09.097773 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b275d0-2zc77"] Sep 14 11:06:09.098214 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:09.098202 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="353a5313-d18b-4145-9143-2f3ef14b0b97" containerName="payload-pre-processing" Sep 14 11:06:09.098256 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:09.098216 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="353a5313-d18b-4145-9143-2f3ef14b0b97" containerName="payload-pre-processing" Sep 14 11:06:09.098303 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:09.098294 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="353a5313-d18b-4145-9143-2f3ef14b0b97" containerName="payload-pre-processing" Sep 14 11:06:09.102511 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:09.102494 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b275d0-2zc77" Sep 14 11:06:09.107094 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:09.107079 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pv9g8\"" Sep 14 11:06:09.123661 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:09.123583 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b275d0-2zc77"] Sep 14 11:06:09.172827 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:09.172791 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12cfa524-810d-4980-b2a2-65aefc829045-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-b275d0-2zc77\" (UID: \"12cfa524-810d-4980-b2a2-65aefc829045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b275d0-2zc77" Sep 14 11:06:09.273744 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:09.273712 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12cfa524-810d-4980-b2a2-65aefc829045-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-b275d0-2zc77\" (UID: \"12cfa524-810d-4980-b2a2-65aefc829045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b275d0-2zc77" Sep 14 11:06:09.274418 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:09.274392 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12cfa524-810d-4980-b2a2-65aefc829045-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-b275d0-2zc77\" (UID: \"12cfa524-810d-4980-b2a2-65aefc829045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b275d0-2zc77" Sep 14 11:06:09.413134 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:09.413043 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b275d0-2zc77" Sep 14 11:06:09.702949 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:09.702858 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b275d0-2zc77"] Sep 14 11:06:09.713736 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:06:09.706931 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12cfa524_810d_4980_b2a2_65aefc829045.slice/crio-e20eb19b21f9d4f0e25997efbb9e9de5b50a49888d0558271efc10661fb7d836 WatchSource:0}: Error finding container e20eb19b21f9d4f0e25997efbb9e9de5b50a49888d0558271efc10661fb7d836: Status 404 returned error can't find the container with id e20eb19b21f9d4f0e25997efbb9e9de5b50a49888d0558271efc10661fb7d836 Sep 14 11:06:10.719293 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:10.719202 2581 generic.go:356] "Generic (PLEG): container finished" podID="12cfa524-810d-4980-b2a2-65aefc829045" containerID="caeda90321ee9230cf29a52cff9a136a3a1c6dd689a2bd98f5c77ef170fc4dc5" exitCode=0 Sep 14 11:06:10.719682 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:10.719298 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b275d0-2zc77" event={"ID":"12cfa524-810d-4980-b2a2-65aefc829045","Type":"ContainerDied","Data":"caeda90321ee9230cf29a52cff9a136a3a1c6dd689a2bd98f5c77ef170fc4dc5"} Sep 14 11:06:10.719682 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:10.719365 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b275d0-2zc77" event={"ID":"12cfa524-810d-4980-b2a2-65aefc829045","Type":"ContainerStarted","Data":"e20eb19b21f9d4f0e25997efbb9e9de5b50a49888d0558271efc10661fb7d836"} Sep 14 11:06:11.865295 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:11.865272 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b275d0-2zc77" Sep 14 11:06:11.901221 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:11.901192 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12cfa524-810d-4980-b2a2-65aefc829045-maas-api-service-ca\") pod \"12cfa524-810d-4980-b2a2-65aefc829045\" (UID: \"12cfa524-810d-4980-b2a2-65aefc829045\") " Sep 14 11:06:11.901652 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:11.901623 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12cfa524-810d-4980-b2a2-65aefc829045-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "12cfa524-810d-4980-b2a2-65aefc829045" (UID: "12cfa524-810d-4980-b2a2-65aefc829045"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:06:12.002678 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:12.002588 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12cfa524-810d-4980-b2a2-65aefc829045-maas-api-service-ca\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:06:12.731352 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:12.731305 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b275d0-2zc77" Sep 14 11:06:12.731545 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:12.731369 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b275d0-2zc77" event={"ID":"12cfa524-810d-4980-b2a2-65aefc829045","Type":"ContainerDied","Data":"e20eb19b21f9d4f0e25997efbb9e9de5b50a49888d0558271efc10661fb7d836"} Sep 14 11:06:12.731545 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:12.731462 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e20eb19b21f9d4f0e25997efbb9e9de5b50a49888d0558271efc10661fb7d836" Sep 14 11:06:14.871431 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:14.871400 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-04195c/test-model-04195c-kserve-54bf6676f-9rn9r" Sep 14 11:06:14.871899 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:14.871452 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-04195c/test-model-04195c-kserve-54bf6676f-9rn9r" Sep 14 11:06:14.884491 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:14.884467 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-04195c/test-model-04195c-kserve-54bf6676f-9rn9r" Sep 14 11:06:15.760401 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:15.760374 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-04195c/test-model-04195c-kserve-54bf6676f-9rn9r" Sep 14 11:06:20.384630 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:20.384597 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b275d0-8dd84d5db-6tsg4"] Sep 14 11:06:20.385031 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:20.384807 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-b275d0-8dd84d5db-6tsg4" podUID="73791610-34cd-4a9e-8aa5-1ea9ab8ebaac" containerName="payload-pre-processing" containerID="cri-o://cd1da84e4a1d132afb0e82ec59e8857a0158c314a56546026eee31f3abb46308" gracePeriod=30 Sep 14 11:06:20.651202 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:20.651128 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-b275d0-8dd84d5db-6tsg4" Sep 14 11:06:20.730808 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:20.730767 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b275d0-2zc77"] Sep 14 11:06:20.744278 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:20.744247 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b275d0-2zc77"] Sep 14 11:06:20.779609 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:20.779581 2581 generic.go:356] "Generic (PLEG): container finished" podID="73791610-34cd-4a9e-8aa5-1ea9ab8ebaac" containerID="cd1da84e4a1d132afb0e82ec59e8857a0158c314a56546026eee31f3abb46308" exitCode=0 Sep 14 11:06:20.779781 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:20.779632 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-b275d0-8dd84d5db-6tsg4" Sep 14 11:06:20.779781 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:20.779668 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-b275d0-8dd84d5db-6tsg4" event={"ID":"73791610-34cd-4a9e-8aa5-1ea9ab8ebaac","Type":"ContainerDied","Data":"cd1da84e4a1d132afb0e82ec59e8857a0158c314a56546026eee31f3abb46308"} Sep 14 11:06:20.779781 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:20.779725 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-b275d0-8dd84d5db-6tsg4" event={"ID":"73791610-34cd-4a9e-8aa5-1ea9ab8ebaac","Type":"ContainerDied","Data":"fbd030a29005b3a7af033de70d686e8f6defaab1ee74bf1dcf2499aad212fa08"} Sep 14 11:06:20.779781 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:20.779748 2581 scope.go:117] "RemoveContainer" containerID="cd1da84e4a1d132afb0e82ec59e8857a0158c314a56546026eee31f3abb46308" Sep 14 11:06:20.789072 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:20.789047 2581 scope.go:117] "RemoveContainer" containerID="cd1da84e4a1d132afb0e82ec59e8857a0158c314a56546026eee31f3abb46308" Sep 14 11:06:20.789460 ip-10-0-137-53 kubenswrapper[2581]: E0914 11:06:20.789439 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd1da84e4a1d132afb0e82ec59e8857a0158c314a56546026eee31f3abb46308\": container with ID starting with cd1da84e4a1d132afb0e82ec59e8857a0158c314a56546026eee31f3abb46308 not found: ID does not exist" containerID="cd1da84e4a1d132afb0e82ec59e8857a0158c314a56546026eee31f3abb46308" Sep 14 11:06:20.789529 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:20.789472 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd1da84e4a1d132afb0e82ec59e8857a0158c314a56546026eee31f3abb46308"} err="failed to get container status \"cd1da84e4a1d132afb0e82ec59e8857a0158c314a56546026eee31f3abb46308\": rpc error: code = NotFound desc = could not find container \"cd1da84e4a1d132afb0e82ec59e8857a0158c314a56546026eee31f3abb46308\": container with ID starting with cd1da84e4a1d132afb0e82ec59e8857a0158c314a56546026eee31f3abb46308 not found: ID does not exist" Sep 14 11:06:20.795653 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:20.795636 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-spk46\" (UniqueName: \"kubernetes.io/projected/73791610-34cd-4a9e-8aa5-1ea9ab8ebaac-kube-api-access-spk46\") pod \"73791610-34cd-4a9e-8aa5-1ea9ab8ebaac\" (UID: \"73791610-34cd-4a9e-8aa5-1ea9ab8ebaac\") " Sep 14 11:06:20.795781 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:20.795762 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73791610-34cd-4a9e-8aa5-1ea9ab8ebaac-plugins-config-volume\") pod \"73791610-34cd-4a9e-8aa5-1ea9ab8ebaac\" (UID: \"73791610-34cd-4a9e-8aa5-1ea9ab8ebaac\") " Sep 14 11:06:20.796088 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:20.796067 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73791610-34cd-4a9e-8aa5-1ea9ab8ebaac-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "73791610-34cd-4a9e-8aa5-1ea9ab8ebaac" (UID: "73791610-34cd-4a9e-8aa5-1ea9ab8ebaac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:06:20.797776 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:20.797751 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73791610-34cd-4a9e-8aa5-1ea9ab8ebaac-kube-api-access-spk46" (OuterVolumeSpecName: "kube-api-access-spk46") pod "73791610-34cd-4a9e-8aa5-1ea9ab8ebaac" (UID: "73791610-34cd-4a9e-8aa5-1ea9ab8ebaac"). InnerVolumeSpecName "kube-api-access-spk46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:06:20.832156 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:20.832116 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12cfa524-810d-4980-b2a2-65aefc829045" path="/var/lib/kubelet/pods/12cfa524-810d-4980-b2a2-65aefc829045/volumes" Sep 14 11:06:20.896853 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:20.896818 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spk46\" (UniqueName: \"kubernetes.io/projected/73791610-34cd-4a9e-8aa5-1ea9ab8ebaac-kube-api-access-spk46\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:06:20.896853 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:20.896853 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73791610-34cd-4a9e-8aa5-1ea9ab8ebaac-plugins-config-volume\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:06:21.113408 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:21.113374 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b275d0-8dd84d5db-6tsg4"] Sep 14 11:06:21.116791 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:21.116766 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b275d0-8dd84d5db-6tsg4"] Sep 14 11:06:22.832512 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:22.832482 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73791610-34cd-4a9e-8aa5-1ea9ab8ebaac" path="/var/lib/kubelet/pods/73791610-34cd-4a9e-8aa5-1ea9ab8ebaac/volumes" Sep 14 11:06:31.265193 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:31.265159 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bcc15f82-br5q7"] Sep 14 11:06:31.265623 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:31.265599 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73791610-34cd-4a9e-8aa5-1ea9ab8ebaac" containerName="payload-pre-processing" Sep 14 11:06:31.265623 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:31.265611 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="73791610-34cd-4a9e-8aa5-1ea9ab8ebaac" containerName="payload-pre-processing" Sep 14 11:06:31.265695 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:31.265629 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12cfa524-810d-4980-b2a2-65aefc829045" containerName="revoke-keys" Sep 14 11:06:31.265695 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:31.265635 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="12cfa524-810d-4980-b2a2-65aefc829045" containerName="revoke-keys" Sep 14 11:06:31.265761 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:31.265701 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="12cfa524-810d-4980-b2a2-65aefc829045" containerName="revoke-keys" Sep 14 11:06:31.265761 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:31.265713 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="73791610-34cd-4a9e-8aa5-1ea9ab8ebaac" containerName="payload-pre-processing" Sep 14 11:06:31.270052 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:31.270033 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bcc15f82-br5q7" Sep 14 11:06:31.273997 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:31.273975 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pv9g8\"" Sep 14 11:06:31.282521 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:31.282497 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bcc15f82-br5q7"] Sep 14 11:06:31.406047 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:31.406007 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68a6379a-3899-4d7e-b102-b1e078c5565d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bcc15f82-br5q7\" (UID: \"68a6379a-3899-4d7e-b102-b1e078c5565d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bcc15f82-br5q7" Sep 14 11:06:31.506920 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:31.506829 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68a6379a-3899-4d7e-b102-b1e078c5565d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bcc15f82-br5q7\" (UID: \"68a6379a-3899-4d7e-b102-b1e078c5565d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bcc15f82-br5q7" Sep 14 11:06:31.507493 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:31.507473 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68a6379a-3899-4d7e-b102-b1e078c5565d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-bcc15f82-br5q7\" (UID: \"68a6379a-3899-4d7e-b102-b1e078c5565d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bcc15f82-br5q7" Sep 14 11:06:31.581317 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:31.581278 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bcc15f82-br5q7" Sep 14 11:06:31.858048 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:31.858004 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bcc15f82-br5q7"] Sep 14 11:06:31.860445 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:06:31.860401 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68a6379a_3899_4d7e_b102_b1e078c5565d.slice/crio-bed87ff5c7a309f9f95905d8287536add2df783229c71384784173849c7b6367 WatchSource:0}: Error finding container bed87ff5c7a309f9f95905d8287536add2df783229c71384784173849c7b6367: Status 404 returned error can't find the container with id bed87ff5c7a309f9f95905d8287536add2df783229c71384784173849c7b6367 Sep 14 11:06:32.844991 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:32.844956 2581 generic.go:356] "Generic (PLEG): container finished" podID="68a6379a-3899-4d7e-b102-b1e078c5565d" containerID="fba0b43155baae8e2a31d3d1ca2b214cafe853ffe3154cc5a6b92433db06689d" exitCode=0 Sep 14 11:06:32.845405 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:32.845042 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bcc15f82-br5q7" event={"ID":"68a6379a-3899-4d7e-b102-b1e078c5565d","Type":"ContainerDied","Data":"fba0b43155baae8e2a31d3d1ca2b214cafe853ffe3154cc5a6b92433db06689d"} Sep 14 11:06:32.845405 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:32.845100 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bcc15f82-br5q7" event={"ID":"68a6379a-3899-4d7e-b102-b1e078c5565d","Type":"ContainerStarted","Data":"bed87ff5c7a309f9f95905d8287536add2df783229c71384784173849c7b6367"} Sep 14 11:06:33.995125 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:33.995102 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bcc15f82-br5q7" Sep 14 11:06:34.136525 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:34.136445 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68a6379a-3899-4d7e-b102-b1e078c5565d-maas-api-service-ca\") pod \"68a6379a-3899-4d7e-b102-b1e078c5565d\" (UID: \"68a6379a-3899-4d7e-b102-b1e078c5565d\") " Sep 14 11:06:34.136734 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:34.136714 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68a6379a-3899-4d7e-b102-b1e078c5565d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "68a6379a-3899-4d7e-b102-b1e078c5565d" (UID: "68a6379a-3899-4d7e-b102-b1e078c5565d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:06:34.238366 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:34.238310 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68a6379a-3899-4d7e-b102-b1e078c5565d-maas-api-service-ca\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:06:34.856769 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:34.856735 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bcc15f82-br5q7" event={"ID":"68a6379a-3899-4d7e-b102-b1e078c5565d","Type":"ContainerDied","Data":"bed87ff5c7a309f9f95905d8287536add2df783229c71384784173849c7b6367"} Sep 14 11:06:34.856769 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:34.856762 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bcc15f82-br5q7" Sep 14 11:06:34.856989 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:34.856777 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bed87ff5c7a309f9f95905d8287536add2df783229c71384784173849c7b6367" Sep 14 11:06:42.880882 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:42.880799 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bcc15f82-br5q7"] Sep 14 11:06:42.899391 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:42.899361 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bcc15f82-br5q7"] Sep 14 11:06:44.832406 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:06:44.832364 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68a6379a-3899-4d7e-b102-b1e078c5565d" path="/var/lib/kubelet/pods/68a6379a-3899-4d7e-b102-b1e078c5565d/volumes" Sep 14 11:07:46.344226 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:07:46.344194 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm"] Sep 14 11:07:46.344918 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:07:46.344849 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68a6379a-3899-4d7e-b102-b1e078c5565d" containerName="revoke-keys" Sep 14 11:07:46.344918 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:07:46.344863 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="68a6379a-3899-4d7e-b102-b1e078c5565d" containerName="revoke-keys" Sep 14 11:07:46.345048 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:07:46.344951 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="68a6379a-3899-4d7e-b102-b1e078c5565d" containerName="revoke-keys" Sep 14 11:07:46.348182 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:07:46.348160 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm" Sep 14 11:07:46.351458 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:07:46.351439 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4ebf73\"/\"kube-root-ca.crt\"" Sep 14 11:07:46.352374 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:07:46.352352 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4ebf73\"/\"default-dockercfg-8975n\"" Sep 14 11:07:46.352479 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:07:46.352391 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4ebf73\"/\"openshift-service-ca.crt\"" Sep 14 11:07:46.352540 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:07:46.352491 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4ebf73\"/\"e2e-conflict-a-3c60bd48-kserve-self-signed-certs\"" Sep 14 11:07:46.367837 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:07:46.367812 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm"] Sep 14 11:07:46.473382 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:07:46.473322 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8650c87-2cf5-4198-a02c-82c1807e3e6e-model-cache\") pod \"e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm\" (UID: \"c8650c87-2cf5-4198-a02c-82c1807e3e6e\") " pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm" Sep 14 11:07:46.473556 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:07:46.473413 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8650c87-2cf5-4198-a02c-82c1807e3e6e-tmp-dir\") pod \"e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm\" (UID: \"c8650c87-2cf5-4198-a02c-82c1807e3e6e\") " pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm" Sep 14 11:07:46.473556 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:07:46.473459 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8650c87-2cf5-4198-a02c-82c1807e3e6e-dshm\") pod \"e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm\" (UID: \"c8650c87-2cf5-4198-a02c-82c1807e3e6e\") " pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm" Sep 14 11:07:46.473556 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:07:46.473551 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br76s\" (UniqueName: \"kubernetes.io/projected/c8650c87-2cf5-4198-a02c-82c1807e3e6e-kube-api-access-br76s\") pod \"e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm\" (UID: \"c8650c87-2cf5-4198-a02c-82c1807e3e6e\") " pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm" Sep 14 11:07:46.473683 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:07:46.473580 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8650c87-2cf5-4198-a02c-82c1807e3e6e-tls-certs\") pod \"e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm\" (UID: \"c8650c87-2cf5-4198-a02c-82c1807e3e6e\") " pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm" Sep 14 11:07:46.473683 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:07:46.473634 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8650c87-2cf5-4198-a02c-82c1807e3e6e-home\") pod \"e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm\" (UID: \"c8650c87-2cf5-4198-a02c-82c1807e3e6e\") " pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm" Sep 14 11:07:46.574634 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:07:46.574602 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br76s\" (UniqueName: \"kubernetes.io/projected/c8650c87-2cf5-4198-a02c-82c1807e3e6e-kube-api-access-br76s\") pod \"e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm\" (UID: \"c8650c87-2cf5-4198-a02c-82c1807e3e6e\") " pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm" Sep 14 11:07:46.574837 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:07:46.574662 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8650c87-2cf5-4198-a02c-82c1807e3e6e-tls-certs\") pod \"e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm\" (UID: \"c8650c87-2cf5-4198-a02c-82c1807e3e6e\") " pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm" Sep 14 11:07:46.574837 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:07:46.574697 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8650c87-2cf5-4198-a02c-82c1807e3e6e-home\") pod \"e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm\" (UID: \"c8650c87-2cf5-4198-a02c-82c1807e3e6e\") " pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm" Sep 14 11:07:46.574837 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:07:46.574724 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8650c87-2cf5-4198-a02c-82c1807e3e6e-model-cache\") pod \"e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm\" (UID: \"c8650c87-2cf5-4198-a02c-82c1807e3e6e\") " pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm" Sep 14 11:07:46.574837 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:07:46.574778 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8650c87-2cf5-4198-a02c-82c1807e3e6e-tmp-dir\") pod \"e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm\" (UID: \"c8650c87-2cf5-4198-a02c-82c1807e3e6e\") " pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm" Sep 14 11:07:46.574837 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:07:46.574836 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8650c87-2cf5-4198-a02c-82c1807e3e6e-dshm\") pod \"e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm\" (UID: \"c8650c87-2cf5-4198-a02c-82c1807e3e6e\") " pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm" Sep 14 11:07:46.575396 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:07:46.575325 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8650c87-2cf5-4198-a02c-82c1807e3e6e-model-cache\") pod \"e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm\" (UID: \"c8650c87-2cf5-4198-a02c-82c1807e3e6e\") " pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm" Sep 14 11:07:46.575505 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:07:46.575403 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8650c87-2cf5-4198-a02c-82c1807e3e6e-home\") pod \"e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm\" (UID: \"c8650c87-2cf5-4198-a02c-82c1807e3e6e\") " pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm" Sep 14 11:07:46.575579 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:07:46.575552 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8650c87-2cf5-4198-a02c-82c1807e3e6e-tmp-dir\") pod \"e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm\" (UID: \"c8650c87-2cf5-4198-a02c-82c1807e3e6e\") " pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm" Sep 14 11:07:46.577749 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:07:46.577721 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8650c87-2cf5-4198-a02c-82c1807e3e6e-dshm\") pod \"e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm\" (UID: \"c8650c87-2cf5-4198-a02c-82c1807e3e6e\") " pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm" Sep 14 11:07:46.577918 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:07:46.577897 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8650c87-2cf5-4198-a02c-82c1807e3e6e-tls-certs\") pod \"e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm\" (UID: \"c8650c87-2cf5-4198-a02c-82c1807e3e6e\") " pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm" Sep 14 11:07:46.586791 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:07:46.586767 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-br76s\" (UniqueName: \"kubernetes.io/projected/c8650c87-2cf5-4198-a02c-82c1807e3e6e-kube-api-access-br76s\") pod \"e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm\" (UID: \"c8650c87-2cf5-4198-a02c-82c1807e3e6e\") " pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm" Sep 14 11:07:46.658555 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:07:46.658455 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm" Sep 14 11:07:46.948191 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:07:46.948160 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm"] Sep 14 11:07:46.950684 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:07:46.950654 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8650c87_2cf5_4198_a02c_82c1807e3e6e.slice/crio-601416ba36d14432755ab0b19661dcbbe9990b05189b4bec30c20b6a1b1d056e WatchSource:0}: Error finding container 601416ba36d14432755ab0b19661dcbbe9990b05189b4bec30c20b6a1b1d056e: Status 404 returned error can't find the container with id 601416ba36d14432755ab0b19661dcbbe9990b05189b4bec30c20b6a1b1d056e Sep 14 11:07:47.254449 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:07:47.254324 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm" event={"ID":"c8650c87-2cf5-4198-a02c-82c1807e3e6e","Type":"ContainerStarted","Data":"5310b6e703a71a17df510d0875c173846f814f9f34bda126dec55799a9cb4f64"} Sep 14 11:07:47.254449 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:07:47.254397 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm" event={"ID":"c8650c87-2cf5-4198-a02c-82c1807e3e6e","Type":"ContainerStarted","Data":"601416ba36d14432755ab0b19661dcbbe9990b05189b4bec30c20b6a1b1d056e"} Sep 14 11:07:47.277416 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:07:47.276189 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm" podStartSLOduration=1.2761641049999999 podStartE2EDuration="1.276164105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:07:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:07:47.27494367 +0000 UTC m=+1141.017345640" watchObservedRunningTime="2026-09-14 11:07:47.276164105 +0000 UTC m=+1141.018566076" Sep 14 11:07:56.659410 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:07:56.659378 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm" Sep 14 11:07:56.659410 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:07:56.659421 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm" Sep 14 11:07:56.672045 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:07:56.672017 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm" Sep 14 11:07:57.326257 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:07:57.326224 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm" Sep 14 11:08:33.690112 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:08:33.690076 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cc14fe-hpstv"] Sep 14 11:08:33.693999 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:08:33.693979 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cc14fe-hpstv" Sep 14 11:08:33.696570 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:08:33.696548 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pv9g8\"" Sep 14 11:08:33.701979 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:08:33.701954 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cc14fe-hpstv"] Sep 14 11:08:33.763649 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:08:33.763611 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9264809-4d1f-438a-b60a-dc0b011f3459-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-cc14fe-hpstv\" (UID: \"e9264809-4d1f-438a-b60a-dc0b011f3459\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cc14fe-hpstv" Sep 14 11:08:33.864716 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:08:33.864669 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9264809-4d1f-438a-b60a-dc0b011f3459-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-cc14fe-hpstv\" (UID: \"e9264809-4d1f-438a-b60a-dc0b011f3459\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cc14fe-hpstv" Sep 14 11:08:33.865304 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:08:33.865283 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9264809-4d1f-438a-b60a-dc0b011f3459-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-cc14fe-hpstv\" (UID: \"e9264809-4d1f-438a-b60a-dc0b011f3459\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cc14fe-hpstv" Sep 14 11:08:34.006405 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:08:34.006292 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cc14fe-hpstv" Sep 14 11:08:34.286997 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:08:34.286965 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cc14fe-hpstv"] Sep 14 11:08:34.288975 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:08:34.288948 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9264809_4d1f_438a_b60a_dc0b011f3459.slice/crio-961d5d43f31ddf4200cd8e1915d6fc1c14e822650f653d97ce2d634e29f20a9c WatchSource:0}: Error finding container 961d5d43f31ddf4200cd8e1915d6fc1c14e822650f653d97ce2d634e29f20a9c: Status 404 returned error can't find the container with id 961d5d43f31ddf4200cd8e1915d6fc1c14e822650f653d97ce2d634e29f20a9c Sep 14 11:08:34.517233 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:08:34.517201 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cc14fe-hpstv" event={"ID":"e9264809-4d1f-438a-b60a-dc0b011f3459","Type":"ContainerStarted","Data":"961d5d43f31ddf4200cd8e1915d6fc1c14e822650f653d97ce2d634e29f20a9c"} Sep 14 11:08:35.523941 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:08:35.523907 2581 generic.go:356] "Generic (PLEG): container finished" podID="e9264809-4d1f-438a-b60a-dc0b011f3459" containerID="2c79ce167dc78e047da9ff4524dc1f4cffe4326c678b5acb2006235dda3db5ce" exitCode=0 Sep 14 11:08:35.524409 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:08:35.523969 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cc14fe-hpstv" event={"ID":"e9264809-4d1f-438a-b60a-dc0b011f3459","Type":"ContainerDied","Data":"2c79ce167dc78e047da9ff4524dc1f4cffe4326c678b5acb2006235dda3db5ce"} Sep 14 11:08:36.685696 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:08:36.685671 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cc14fe-hpstv" Sep 14 11:08:36.795661 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:08:36.795576 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9264809-4d1f-438a-b60a-dc0b011f3459-maas-api-service-ca\") pod \"e9264809-4d1f-438a-b60a-dc0b011f3459\" (UID: \"e9264809-4d1f-438a-b60a-dc0b011f3459\") " Sep 14 11:08:36.795967 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:08:36.795938 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9264809-4d1f-438a-b60a-dc0b011f3459-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e9264809-4d1f-438a-b60a-dc0b011f3459" (UID: "e9264809-4d1f-438a-b60a-dc0b011f3459"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:08:36.898035 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:08:36.897934 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9264809-4d1f-438a-b60a-dc0b011f3459-maas-api-service-ca\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:08:37.537260 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:08:37.537230 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cc14fe-hpstv" Sep 14 11:08:37.537260 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:08:37.537241 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cc14fe-hpstv" event={"ID":"e9264809-4d1f-438a-b60a-dc0b011f3459","Type":"ContainerDied","Data":"961d5d43f31ddf4200cd8e1915d6fc1c14e822650f653d97ce2d634e29f20a9c"} Sep 14 11:08:37.537507 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:08:37.537293 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="961d5d43f31ddf4200cd8e1915d6fc1c14e822650f653d97ce2d634e29f20a9c" Sep 14 11:08:46.714248 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:08:46.714215 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cc14fe-hpstv"] Sep 14 11:08:46.749392 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:08:46.747608 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cc14fe-hpstv"] Sep 14 11:08:46.832325 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:08:46.832294 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9264809-4d1f-438a-b60a-dc0b011f3459" path="/var/lib/kubelet/pods/e9264809-4d1f-438a-b60a-dc0b011f3459/volumes" Sep 14 11:09:07.323453 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:07.323414 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7984b1/distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk"] Sep 14 11:09:07.324095 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:07.324076 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9264809-4d1f-438a-b60a-dc0b011f3459" containerName="revoke-keys" Sep 14 11:09:07.324149 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:07.324102 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9264809-4d1f-438a-b60a-dc0b011f3459" containerName="revoke-keys" Sep 14 11:09:07.324252 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:07.324240 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9264809-4d1f-438a-b60a-dc0b011f3459" containerName="revoke-keys" Sep 14 11:09:07.329167 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:07.329149 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7984b1/distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk" Sep 14 11:09:07.332355 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:07.332320 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7984b1\"/\"distinct-w0-7984b1-kserve-self-signed-certs\"" Sep 14 11:09:07.340089 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:07.340065 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7984b1/distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk"] Sep 14 11:09:07.402131 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:07.402096 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/762c6088-d668-4d97-9700-d68b50d3277f-home\") pod \"distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk\" (UID: \"762c6088-d668-4d97-9700-d68b50d3277f\") " pod="e2e-models-e2e-worker-w0-7984b1/distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk" Sep 14 11:09:07.402131 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:07.402132 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/762c6088-d668-4d97-9700-d68b50d3277f-dshm\") pod \"distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk\" (UID: \"762c6088-d668-4d97-9700-d68b50d3277f\") " pod="e2e-models-e2e-worker-w0-7984b1/distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk" Sep 14 11:09:07.402480 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:07.402247 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/762c6088-d668-4d97-9700-d68b50d3277f-tmp-dir\") pod \"distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk\" (UID: \"762c6088-d668-4d97-9700-d68b50d3277f\") " pod="e2e-models-e2e-worker-w0-7984b1/distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk" Sep 14 11:09:07.402480 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:07.402286 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/762c6088-d668-4d97-9700-d68b50d3277f-model-cache\") pod \"distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk\" (UID: \"762c6088-d668-4d97-9700-d68b50d3277f\") " pod="e2e-models-e2e-worker-w0-7984b1/distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk" Sep 14 11:09:07.402480 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:07.402386 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/762c6088-d668-4d97-9700-d68b50d3277f-tls-certs\") pod \"distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk\" (UID: \"762c6088-d668-4d97-9700-d68b50d3277f\") " pod="e2e-models-e2e-worker-w0-7984b1/distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk" Sep 14 11:09:07.402480 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:07.402445 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l8qg\" (UniqueName: \"kubernetes.io/projected/762c6088-d668-4d97-9700-d68b50d3277f-kube-api-access-7l8qg\") pod \"distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk\" (UID: \"762c6088-d668-4d97-9700-d68b50d3277f\") " pod="e2e-models-e2e-worker-w0-7984b1/distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk" Sep 14 11:09:07.503573 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:07.503538 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/762c6088-d668-4d97-9700-d68b50d3277f-tmp-dir\") pod \"distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk\" (UID: \"762c6088-d668-4d97-9700-d68b50d3277f\") " pod="e2e-models-e2e-worker-w0-7984b1/distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk" Sep 14 11:09:07.503573 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:07.503573 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/762c6088-d668-4d97-9700-d68b50d3277f-model-cache\") pod \"distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk\" (UID: \"762c6088-d668-4d97-9700-d68b50d3277f\") " pod="e2e-models-e2e-worker-w0-7984b1/distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk" Sep 14 11:09:07.503810 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:07.503630 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/762c6088-d668-4d97-9700-d68b50d3277f-tls-certs\") pod \"distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk\" (UID: \"762c6088-d668-4d97-9700-d68b50d3277f\") " pod="e2e-models-e2e-worker-w0-7984b1/distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk" Sep 14 11:09:07.503810 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:07.503667 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l8qg\" (UniqueName: \"kubernetes.io/projected/762c6088-d668-4d97-9700-d68b50d3277f-kube-api-access-7l8qg\") pod \"distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk\" (UID: \"762c6088-d668-4d97-9700-d68b50d3277f\") " pod="e2e-models-e2e-worker-w0-7984b1/distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk" Sep 14 11:09:07.503810 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:07.503718 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/762c6088-d668-4d97-9700-d68b50d3277f-home\") pod \"distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk\" (UID: \"762c6088-d668-4d97-9700-d68b50d3277f\") " pod="e2e-models-e2e-worker-w0-7984b1/distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk" Sep 14 11:09:07.503956 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:07.503848 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/762c6088-d668-4d97-9700-d68b50d3277f-dshm\") pod \"distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk\" (UID: \"762c6088-d668-4d97-9700-d68b50d3277f\") " pod="e2e-models-e2e-worker-w0-7984b1/distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk" Sep 14 11:09:07.504000 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:07.503986 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/762c6088-d668-4d97-9700-d68b50d3277f-tmp-dir\") pod \"distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk\" (UID: \"762c6088-d668-4d97-9700-d68b50d3277f\") " pod="e2e-models-e2e-worker-w0-7984b1/distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk" Sep 14 11:09:07.504059 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:07.504032 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/762c6088-d668-4d97-9700-d68b50d3277f-home\") pod \"distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk\" (UID: \"762c6088-d668-4d97-9700-d68b50d3277f\") " pod="e2e-models-e2e-worker-w0-7984b1/distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk" Sep 14 11:09:07.504102 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:07.504071 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/762c6088-d668-4d97-9700-d68b50d3277f-model-cache\") pod \"distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk\" (UID: \"762c6088-d668-4d97-9700-d68b50d3277f\") " pod="e2e-models-e2e-worker-w0-7984b1/distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk" Sep 14 11:09:07.506105 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:07.506077 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/762c6088-d668-4d97-9700-d68b50d3277f-dshm\") pod \"distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk\" (UID: \"762c6088-d668-4d97-9700-d68b50d3277f\") " pod="e2e-models-e2e-worker-w0-7984b1/distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk" Sep 14 11:09:07.506239 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:07.506223 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/762c6088-d668-4d97-9700-d68b50d3277f-tls-certs\") pod \"distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk\" (UID: \"762c6088-d668-4d97-9700-d68b50d3277f\") " pod="e2e-models-e2e-worker-w0-7984b1/distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk" Sep 14 11:09:07.515724 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:07.515702 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l8qg\" (UniqueName: \"kubernetes.io/projected/762c6088-d668-4d97-9700-d68b50d3277f-kube-api-access-7l8qg\") pod \"distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk\" (UID: \"762c6088-d668-4d97-9700-d68b50d3277f\") " pod="e2e-models-e2e-worker-w0-7984b1/distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk" Sep 14 11:09:07.641830 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:07.641727 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7984b1/distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk" Sep 14 11:09:07.918323 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:07.918245 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7984b1/distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk"] Sep 14 11:09:07.920756 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:09:07.920732 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod762c6088_d668_4d97_9700_d68b50d3277f.slice/crio-5c53c95e6ecd6fe8e8be550c7a1fb6f79f895c1aed31a7e04b983c28a9beb351 WatchSource:0}: Error finding container 5c53c95e6ecd6fe8e8be550c7a1fb6f79f895c1aed31a7e04b983c28a9beb351: Status 404 returned error can't find the container with id 5c53c95e6ecd6fe8e8be550c7a1fb6f79f895c1aed31a7e04b983c28a9beb351 Sep 14 11:09:08.710211 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:08.710168 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7984b1/distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk" event={"ID":"762c6088-d668-4d97-9700-d68b50d3277f","Type":"ContainerStarted","Data":"d29a0f2f7e2fcc922f9c702b4e1ec8c1cd080948fc9ae4793ed4a39c801c3ba1"} Sep 14 11:09:08.710211 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:08.710222 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7984b1/distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk" event={"ID":"762c6088-d668-4d97-9700-d68b50d3277f","Type":"ContainerStarted","Data":"5c53c95e6ecd6fe8e8be550c7a1fb6f79f895c1aed31a7e04b983c28a9beb351"} Sep 14 11:09:08.734641 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:08.734587 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7984b1/distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk" podStartSLOduration=1.734570214 podStartE2EDuration="1.734570214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:09:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:09:08.733446354 +0000 UTC m=+1222.475848327" watchObservedRunningTime="2026-09-14 11:09:08.734570214 +0000 UTC m=+1222.476972184" Sep 14 11:09:17.642533 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:17.642491 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7984b1/distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk" Sep 14 11:09:17.643018 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:17.642560 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7984b1/distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk" Sep 14 11:09:17.655509 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:17.655483 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7984b1/distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk" Sep 14 11:09:17.771315 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:17.771286 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7984b1/distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk" Sep 14 11:09:20.300863 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:20.300808 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e977dc37-6fjrn"] Sep 14 11:09:20.304883 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:20.304864 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e977dc37-6fjrn" Sep 14 11:09:20.310398 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:20.310370 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pv9g8\"" Sep 14 11:09:20.311793 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:20.311772 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e977dc37-6fjrn"] Sep 14 11:09:20.434692 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:20.434651 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f8c884-f19c-491e-95af-1228903be6ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e977dc37-6fjrn\" (UID: \"d4f8c884-f19c-491e-95af-1228903be6ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e977dc37-6fjrn" Sep 14 11:09:20.536293 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:20.536255 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f8c884-f19c-491e-95af-1228903be6ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e977dc37-6fjrn\" (UID: \"d4f8c884-f19c-491e-95af-1228903be6ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e977dc37-6fjrn" Sep 14 11:09:20.536896 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:20.536877 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f8c884-f19c-491e-95af-1228903be6ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e977dc37-6fjrn\" (UID: \"d4f8c884-f19c-491e-95af-1228903be6ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e977dc37-6fjrn" Sep 14 11:09:20.618542 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:20.618452 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e977dc37-6fjrn" Sep 14 11:09:20.904935 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:20.904913 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e977dc37-6fjrn"] Sep 14 11:09:20.907021 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:09:20.906989 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f8c884_f19c_491e_95af_1228903be6ea.slice/crio-a29040dbb7685669634c392a8614cd50b05d01faf9efb47a9d068c24605699c1 WatchSource:0}: Error finding container a29040dbb7685669634c392a8614cd50b05d01faf9efb47a9d068c24605699c1: Status 404 returned error can't find the container with id a29040dbb7685669634c392a8614cd50b05d01faf9efb47a9d068c24605699c1 Sep 14 11:09:21.784017 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:21.783977 2581 generic.go:356] "Generic (PLEG): container finished" podID="d4f8c884-f19c-491e-95af-1228903be6ea" containerID="ed78c94bf81180914f73798d534ad5f6e2ca7e0131f5afd525c64a5f5973fca6" exitCode=0 Sep 14 11:09:21.784489 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:21.784062 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e977dc37-6fjrn" event={"ID":"d4f8c884-f19c-491e-95af-1228903be6ea","Type":"ContainerDied","Data":"ed78c94bf81180914f73798d534ad5f6e2ca7e0131f5afd525c64a5f5973fca6"} Sep 14 11:09:21.784489 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:21.784118 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e977dc37-6fjrn" event={"ID":"d4f8c884-f19c-491e-95af-1228903be6ea","Type":"ContainerStarted","Data":"a29040dbb7685669634c392a8614cd50b05d01faf9efb47a9d068c24605699c1"} Sep 14 11:09:22.935977 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:22.935955 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e977dc37-6fjrn" Sep 14 11:09:23.064019 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:23.063932 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f8c884-f19c-491e-95af-1228903be6ea-maas-api-service-ca\") pod \"d4f8c884-f19c-491e-95af-1228903be6ea\" (UID: \"d4f8c884-f19c-491e-95af-1228903be6ea\") " Sep 14 11:09:23.064251 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:23.064230 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4f8c884-f19c-491e-95af-1228903be6ea-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d4f8c884-f19c-491e-95af-1228903be6ea" (UID: "d4f8c884-f19c-491e-95af-1228903be6ea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:09:23.165731 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:23.165690 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4f8c884-f19c-491e-95af-1228903be6ea-maas-api-service-ca\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:09:23.795615 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:23.795583 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e977dc37-6fjrn" Sep 14 11:09:23.795788 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:23.795633 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e977dc37-6fjrn" event={"ID":"d4f8c884-f19c-491e-95af-1228903be6ea","Type":"ContainerDied","Data":"a29040dbb7685669634c392a8614cd50b05d01faf9efb47a9d068c24605699c1"} Sep 14 11:09:23.795788 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:23.795680 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a29040dbb7685669634c392a8614cd50b05d01faf9efb47a9d068c24605699c1" Sep 14 11:09:27.228415 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:27.228383 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-04195c/test-model-04195c-kserve-54bf6676f-9rn9r"] Sep 14 11:09:31.112522 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:31.112489 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e977dc37-6fjrn"] Sep 14 11:09:31.119484 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:31.119451 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e977dc37-6fjrn"] Sep 14 11:09:32.832578 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:32.832534 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4f8c884-f19c-491e-95af-1228903be6ea" path="/var/lib/kubelet/pods/d4f8c884-f19c-491e-95af-1228903be6ea/volumes" Sep 14 11:09:42.248242 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:42.248129 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-04195c/test-model-04195c-kserve-54bf6676f-9rn9r" podUID="bb7ef955-452f-477c-bb40-432847f1f672" containerName="main" containerID="cri-o://5ae18d530150cf2a3c8363760bed718091a789aef584f992f9b3883cd3d1452f" gracePeriod=45 Sep 14 11:09:42.498109 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:42.498085 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-04195c/test-model-04195c-kserve-54bf6676f-9rn9r" Sep 14 11:09:42.574560 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:42.574530 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb7ef955-452f-477c-bb40-432847f1f672-tls-certs\") pod \"bb7ef955-452f-477c-bb40-432847f1f672\" (UID: \"bb7ef955-452f-477c-bb40-432847f1f672\") " Sep 14 11:09:42.574756 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:42.574567 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb7ef955-452f-477c-bb40-432847f1f672-dshm\") pod \"bb7ef955-452f-477c-bb40-432847f1f672\" (UID: \"bb7ef955-452f-477c-bb40-432847f1f672\") " Sep 14 11:09:42.574756 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:42.574647 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb7ef955-452f-477c-bb40-432847f1f672-tmp-dir\") pod \"bb7ef955-452f-477c-bb40-432847f1f672\" (UID: \"bb7ef955-452f-477c-bb40-432847f1f672\") " Sep 14 11:09:42.574756 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:42.574664 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb7ef955-452f-477c-bb40-432847f1f672-home\") pod \"bb7ef955-452f-477c-bb40-432847f1f672\" (UID: \"bb7ef955-452f-477c-bb40-432847f1f672\") " Sep 14 11:09:42.574756 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:42.574695 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-265tv\" (UniqueName: \"kubernetes.io/projected/bb7ef955-452f-477c-bb40-432847f1f672-kube-api-access-265tv\") pod \"bb7ef955-452f-477c-bb40-432847f1f672\" (UID: \"bb7ef955-452f-477c-bb40-432847f1f672\") " Sep 14 11:09:42.574756 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:42.574752 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb7ef955-452f-477c-bb40-432847f1f672-model-cache\") pod \"bb7ef955-452f-477c-bb40-432847f1f672\" (UID: \"bb7ef955-452f-477c-bb40-432847f1f672\") " Sep 14 11:09:42.575016 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:42.574927 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb7ef955-452f-477c-bb40-432847f1f672-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bb7ef955-452f-477c-bb40-432847f1f672" (UID: "bb7ef955-452f-477c-bb40-432847f1f672"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:09:42.575016 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:42.574942 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb7ef955-452f-477c-bb40-432847f1f672-home" (OuterVolumeSpecName: "home") pod "bb7ef955-452f-477c-bb40-432847f1f672" (UID: "bb7ef955-452f-477c-bb40-432847f1f672"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:09:42.575138 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:42.575063 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb7ef955-452f-477c-bb40-432847f1f672-model-cache" (OuterVolumeSpecName: "model-cache") pod "bb7ef955-452f-477c-bb40-432847f1f672" (UID: "bb7ef955-452f-477c-bb40-432847f1f672"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:09:42.575138 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:42.575121 2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb7ef955-452f-477c-bb40-432847f1f672-tmp-dir\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:09:42.575245 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:42.575139 2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb7ef955-452f-477c-bb40-432847f1f672-home\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:09:42.577030 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:42.577003 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb7ef955-452f-477c-bb40-432847f1f672-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bb7ef955-452f-477c-bb40-432847f1f672" (UID: "bb7ef955-452f-477c-bb40-432847f1f672"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:09:42.577030 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:42.577009 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb7ef955-452f-477c-bb40-432847f1f672-dshm" (OuterVolumeSpecName: "dshm") pod "bb7ef955-452f-477c-bb40-432847f1f672" (UID: "bb7ef955-452f-477c-bb40-432847f1f672"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:09:42.577178 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:42.577033 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb7ef955-452f-477c-bb40-432847f1f672-kube-api-access-265tv" (OuterVolumeSpecName: "kube-api-access-265tv") pod "bb7ef955-452f-477c-bb40-432847f1f672" (UID: "bb7ef955-452f-477c-bb40-432847f1f672"). InnerVolumeSpecName "kube-api-access-265tv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:09:42.676030 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:42.675995 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-265tv\" (UniqueName: \"kubernetes.io/projected/bb7ef955-452f-477c-bb40-432847f1f672-kube-api-access-265tv\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:09:42.676030 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:42.676024 2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb7ef955-452f-477c-bb40-432847f1f672-model-cache\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:09:42.676030 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:42.676034 2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb7ef955-452f-477c-bb40-432847f1f672-tls-certs\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:09:42.676030 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:42.676041 2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb7ef955-452f-477c-bb40-432847f1f672-dshm\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:09:42.911709 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:42.911679 2581 generic.go:356] "Generic (PLEG): container finished" podID="bb7ef955-452f-477c-bb40-432847f1f672" containerID="5ae18d530150cf2a3c8363760bed718091a789aef584f992f9b3883cd3d1452f" exitCode=0 Sep 14 11:09:42.911909 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:42.911735 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-04195c/test-model-04195c-kserve-54bf6676f-9rn9r" Sep 14 11:09:42.911909 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:42.911758 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-04195c/test-model-04195c-kserve-54bf6676f-9rn9r" event={"ID":"bb7ef955-452f-477c-bb40-432847f1f672","Type":"ContainerDied","Data":"5ae18d530150cf2a3c8363760bed718091a789aef584f992f9b3883cd3d1452f"} Sep 14 11:09:42.911909 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:42.911798 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-04195c/test-model-04195c-kserve-54bf6676f-9rn9r" event={"ID":"bb7ef955-452f-477c-bb40-432847f1f672","Type":"ContainerDied","Data":"feef36de15a973a6243bd1e738f5024d4608949022a13836a3ecee30eeab8dff"} Sep 14 11:09:42.911909 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:42.911814 2581 scope.go:117] "RemoveContainer" containerID="5ae18d530150cf2a3c8363760bed718091a789aef584f992f9b3883cd3d1452f" Sep 14 11:09:42.923256 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:42.923235 2581 scope.go:117] "RemoveContainer" containerID="5ae18d530150cf2a3c8363760bed718091a789aef584f992f9b3883cd3d1452f" Sep 14 11:09:42.923564 ip-10-0-137-53 kubenswrapper[2581]: E0914 11:09:42.923545 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ae18d530150cf2a3c8363760bed718091a789aef584f992f9b3883cd3d1452f\": container with ID starting with 5ae18d530150cf2a3c8363760bed718091a789aef584f992f9b3883cd3d1452f not found: ID does not exist" containerID="5ae18d530150cf2a3c8363760bed718091a789aef584f992f9b3883cd3d1452f" Sep 14 11:09:42.923654 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:42.923574 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ae18d530150cf2a3c8363760bed718091a789aef584f992f9b3883cd3d1452f"} err="failed to get container status \"5ae18d530150cf2a3c8363760bed718091a789aef584f992f9b3883cd3d1452f\": rpc error: code = NotFound desc = could not find container \"5ae18d530150cf2a3c8363760bed718091a789aef584f992f9b3883cd3d1452f\": container with ID starting with 5ae18d530150cf2a3c8363760bed718091a789aef584f992f9b3883cd3d1452f not found: ID does not exist" Sep 14 11:09:42.933101 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:42.933080 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-04195c/test-model-04195c-kserve-54bf6676f-9rn9r"] Sep 14 11:09:42.938001 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:42.937982 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-04195c/test-model-04195c-kserve-54bf6676f-9rn9r"] Sep 14 11:09:44.835165 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:44.835129 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb7ef955-452f-477c-bb40-432847f1f672" path="/var/lib/kubelet/pods/bb7ef955-452f-477c-bb40-432847f1f672/volumes" Sep 14 11:09:58.317760 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:58.317730 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04195c-tnf96"] Sep 14 11:09:58.318222 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:58.318186 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb7ef955-452f-477c-bb40-432847f1f672" containerName="main" Sep 14 11:09:58.318222 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:58.318198 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb7ef955-452f-477c-bb40-432847f1f672" containerName="main" Sep 14 11:09:58.318222 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:58.318209 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4f8c884-f19c-491e-95af-1228903be6ea" containerName="revoke-keys" Sep 14 11:09:58.318222 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:58.318214 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4f8c884-f19c-491e-95af-1228903be6ea" containerName="revoke-keys" Sep 14 11:09:58.318377 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:58.318295 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb7ef955-452f-477c-bb40-432847f1f672" containerName="main" Sep 14 11:09:58.318377 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:58.318303 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4f8c884-f19c-491e-95af-1228903be6ea" containerName="revoke-keys" Sep 14 11:09:58.322814 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:58.322795 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04195c-tnf96" Sep 14 11:09:58.329089 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:58.329069 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pv9g8\"" Sep 14 11:09:58.397892 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:58.397852 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04195c-tnf96"] Sep 14 11:09:58.435735 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:58.435699 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd9eb61-9907-4947-84e5-d2c38ff06e1b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-04195c-tnf96\" (UID: \"6bd9eb61-9907-4947-84e5-d2c38ff06e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04195c-tnf96" Sep 14 11:09:58.536494 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:58.536462 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd9eb61-9907-4947-84e5-d2c38ff06e1b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-04195c-tnf96\" (UID: \"6bd9eb61-9907-4947-84e5-d2c38ff06e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04195c-tnf96" Sep 14 11:09:58.537074 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:58.537052 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd9eb61-9907-4947-84e5-d2c38ff06e1b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-04195c-tnf96\" (UID: \"6bd9eb61-9907-4947-84e5-d2c38ff06e1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04195c-tnf96" Sep 14 11:09:58.632799 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:58.632711 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04195c-tnf96" Sep 14 11:09:59.031025 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:59.030998 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04195c-tnf96"] Sep 14 11:09:59.033353 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:09:59.033309 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bd9eb61_9907_4947_84e5_d2c38ff06e1b.slice/crio-c7afc60164795952a5c5306439152a041af1f4c3322a38114faad9d1d8259e19 WatchSource:0}: Error finding container c7afc60164795952a5c5306439152a041af1f4c3322a38114faad9d1d8259e19: Status 404 returned error can't find the container with id c7afc60164795952a5c5306439152a041af1f4c3322a38114faad9d1d8259e19 Sep 14 11:09:59.095369 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:09:59.095326 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm"] Sep 14 11:10:00.009418 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:00.009310 2581 generic.go:356] "Generic (PLEG): container finished" podID="6bd9eb61-9907-4947-84e5-d2c38ff06e1b" containerID="645c753bcb7d31fe353316c7fc845433d08c779c4a8e57201bf3c22d0fb0058f" exitCode=0 Sep 14 11:10:00.009418 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:00.009365 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04195c-tnf96" event={"ID":"6bd9eb61-9907-4947-84e5-d2c38ff06e1b","Type":"ContainerDied","Data":"645c753bcb7d31fe353316c7fc845433d08c779c4a8e57201bf3c22d0fb0058f"} Sep 14 11:10:00.009418 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:00.009413 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04195c-tnf96" event={"ID":"6bd9eb61-9907-4947-84e5-d2c38ff06e1b","Type":"ContainerStarted","Data":"c7afc60164795952a5c5306439152a041af1f4c3322a38114faad9d1d8259e19"} Sep 14 11:10:01.156655 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:01.156631 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04195c-tnf96" Sep 14 11:10:01.263869 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:01.263828 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd9eb61-9907-4947-84e5-d2c38ff06e1b-maas-api-service-ca\") pod \"6bd9eb61-9907-4947-84e5-d2c38ff06e1b\" (UID: \"6bd9eb61-9907-4947-84e5-d2c38ff06e1b\") " Sep 14 11:10:01.264215 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:01.264197 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bd9eb61-9907-4947-84e5-d2c38ff06e1b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6bd9eb61-9907-4947-84e5-d2c38ff06e1b" (UID: "6bd9eb61-9907-4947-84e5-d2c38ff06e1b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:10:01.364703 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:01.364671 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd9eb61-9907-4947-84e5-d2c38ff06e1b-maas-api-service-ca\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:10:02.021293 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:02.021254 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04195c-tnf96" event={"ID":"6bd9eb61-9907-4947-84e5-d2c38ff06e1b","Type":"ContainerDied","Data":"c7afc60164795952a5c5306439152a041af1f4c3322a38114faad9d1d8259e19"} Sep 14 11:10:02.021293 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:02.021297 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04195c-tnf96" Sep 14 11:10:02.021525 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:02.021308 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7afc60164795952a5c5306439152a041af1f4c3322a38114faad9d1d8259e19" Sep 14 11:10:09.300457 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:09.300414 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-04195c-559d584fc-6gqvc"] Sep 14 11:10:09.301053 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:09.300677 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-04195c-559d584fc-6gqvc" podUID="b870f894-e235-49c4-a502-c9dd63500b2e" containerName="payload-processing" containerID="cri-o://865a9fecda1604bc101b128e4acb2fef63e562011399fa3d6974df1fcd753a04" gracePeriod=30 Sep 14 11:10:09.303739 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:09.303707 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-04195c-5d85fcdd86-dlzdz"] Sep 14 11:10:09.303963 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:09.303918 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04195c-5d85fcdd86-dlzdz" podUID="899c3f91-4d36-4800-923b-cbab2e572a6c" containerName="payload-pre-processing" containerID="cri-o://f610ae859aec96839769a00f462e811402e21efb4489964a8be08c5e733c8a89" gracePeriod=30 Sep 14 11:10:09.552871 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:09.552789 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04195c-tnf96"] Sep 14 11:10:09.555604 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:09.555579 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04195c-tnf96"] Sep 14 11:10:09.586582 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:09.586562 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-04195c-559d584fc-6gqvc" Sep 14 11:10:09.589215 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:09.589198 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04195c-5d85fcdd86-dlzdz" Sep 14 11:10:09.749748 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:09.749715 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pj8ft\" (UniqueName: \"kubernetes.io/projected/899c3f91-4d36-4800-923b-cbab2e572a6c-kube-api-access-pj8ft\") pod \"899c3f91-4d36-4800-923b-cbab2e572a6c\" (UID: \"899c3f91-4d36-4800-923b-cbab2e572a6c\") " Sep 14 11:10:09.749935 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:09.749780 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jwbh\" (UniqueName: \"kubernetes.io/projected/b870f894-e235-49c4-a502-c9dd63500b2e-kube-api-access-8jwbh\") pod \"b870f894-e235-49c4-a502-c9dd63500b2e\" (UID: \"b870f894-e235-49c4-a502-c9dd63500b2e\") " Sep 14 11:10:09.749935 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:09.749835 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/899c3f91-4d36-4800-923b-cbab2e572a6c-plugins-config-volume\") pod \"899c3f91-4d36-4800-923b-cbab2e572a6c\" (UID: \"899c3f91-4d36-4800-923b-cbab2e572a6c\") " Sep 14 11:10:09.749935 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:09.749872 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b870f894-e235-49c4-a502-c9dd63500b2e-plugins-config-volume\") pod \"b870f894-e235-49c4-a502-c9dd63500b2e\" (UID: \"b870f894-e235-49c4-a502-c9dd63500b2e\") " Sep 14 11:10:09.750275 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:09.750240 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/899c3f91-4d36-4800-923b-cbab2e572a6c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "899c3f91-4d36-4800-923b-cbab2e572a6c" (UID: "899c3f91-4d36-4800-923b-cbab2e572a6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:10:09.750423 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:09.750294 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b870f894-e235-49c4-a502-c9dd63500b2e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b870f894-e235-49c4-a502-c9dd63500b2e" (UID: "b870f894-e235-49c4-a502-c9dd63500b2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:10:09.752045 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:09.752026 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b870f894-e235-49c4-a502-c9dd63500b2e-kube-api-access-8jwbh" (OuterVolumeSpecName: "kube-api-access-8jwbh") pod "b870f894-e235-49c4-a502-c9dd63500b2e" (UID: "b870f894-e235-49c4-a502-c9dd63500b2e"). InnerVolumeSpecName "kube-api-access-8jwbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:10:09.752136 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:09.752045 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/899c3f91-4d36-4800-923b-cbab2e572a6c-kube-api-access-pj8ft" (OuterVolumeSpecName: "kube-api-access-pj8ft") pod "899c3f91-4d36-4800-923b-cbab2e572a6c" (UID: "899c3f91-4d36-4800-923b-cbab2e572a6c"). InnerVolumeSpecName "kube-api-access-pj8ft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:10:09.851262 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:09.851166 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj8ft\" (UniqueName: \"kubernetes.io/projected/899c3f91-4d36-4800-923b-cbab2e572a6c-kube-api-access-pj8ft\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:10:09.851262 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:09.851192 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jwbh\" (UniqueName: \"kubernetes.io/projected/b870f894-e235-49c4-a502-c9dd63500b2e-kube-api-access-8jwbh\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:10:09.851262 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:09.851200 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/899c3f91-4d36-4800-923b-cbab2e572a6c-plugins-config-volume\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:10:09.851262 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:09.851209 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b870f894-e235-49c4-a502-c9dd63500b2e-plugins-config-volume\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:10:10.068754 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:10.068717 2581 generic.go:356] "Generic (PLEG): container finished" podID="899c3f91-4d36-4800-923b-cbab2e572a6c" containerID="f610ae859aec96839769a00f462e811402e21efb4489964a8be08c5e733c8a89" exitCode=0 Sep 14 11:10:10.068901 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:10.068767 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04195c-5d85fcdd86-dlzdz" Sep 14 11:10:10.068901 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:10.068809 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04195c-5d85fcdd86-dlzdz" event={"ID":"899c3f91-4d36-4800-923b-cbab2e572a6c","Type":"ContainerDied","Data":"f610ae859aec96839769a00f462e811402e21efb4489964a8be08c5e733c8a89"} Sep 14 11:10:10.068901 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:10.068864 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04195c-5d85fcdd86-dlzdz" event={"ID":"899c3f91-4d36-4800-923b-cbab2e572a6c","Type":"ContainerDied","Data":"f77b9a2201a508510b9785edfae3c3c3971f10afa2dbcc9c46056c56882690a7"} Sep 14 11:10:10.068901 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:10.068891 2581 scope.go:117] "RemoveContainer" containerID="f610ae859aec96839769a00f462e811402e21efb4489964a8be08c5e733c8a89" Sep 14 11:10:10.070154 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:10.070135 2581 generic.go:356] "Generic (PLEG): container finished" podID="b870f894-e235-49c4-a502-c9dd63500b2e" containerID="865a9fecda1604bc101b128e4acb2fef63e562011399fa3d6974df1fcd753a04" exitCode=0 Sep 14 11:10:10.070238 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:10.070184 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-04195c-559d584fc-6gqvc" Sep 14 11:10:10.070238 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:10.070186 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-04195c-559d584fc-6gqvc" event={"ID":"b870f894-e235-49c4-a502-c9dd63500b2e","Type":"ContainerDied","Data":"865a9fecda1604bc101b128e4acb2fef63e562011399fa3d6974df1fcd753a04"} Sep 14 11:10:10.070314 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:10.070243 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-04195c-559d584fc-6gqvc" event={"ID":"b870f894-e235-49c4-a502-c9dd63500b2e","Type":"ContainerDied","Data":"2a9de38b3beddb4fec9fcc7837aeaf034516679d48b3080c6c67eed425eb246d"} Sep 14 11:10:10.079271 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:10.079252 2581 scope.go:117] "RemoveContainer" containerID="f610ae859aec96839769a00f462e811402e21efb4489964a8be08c5e733c8a89" Sep 14 11:10:10.079565 ip-10-0-137-53 kubenswrapper[2581]: E0914 11:10:10.079545 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f610ae859aec96839769a00f462e811402e21efb4489964a8be08c5e733c8a89\": container with ID starting with f610ae859aec96839769a00f462e811402e21efb4489964a8be08c5e733c8a89 not found: ID does not exist" containerID="f610ae859aec96839769a00f462e811402e21efb4489964a8be08c5e733c8a89" Sep 14 11:10:10.079646 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:10.079579 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f610ae859aec96839769a00f462e811402e21efb4489964a8be08c5e733c8a89"} err="failed to get container status \"f610ae859aec96839769a00f462e811402e21efb4489964a8be08c5e733c8a89\": rpc error: code = NotFound desc = could not find container \"f610ae859aec96839769a00f462e811402e21efb4489964a8be08c5e733c8a89\": container with ID starting with f610ae859aec96839769a00f462e811402e21efb4489964a8be08c5e733c8a89 not found: ID does not exist" Sep 14 11:10:10.079646 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:10.079598 2581 scope.go:117] "RemoveContainer" containerID="865a9fecda1604bc101b128e4acb2fef63e562011399fa3d6974df1fcd753a04" Sep 14 11:10:10.088650 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:10.088635 2581 scope.go:117] "RemoveContainer" containerID="865a9fecda1604bc101b128e4acb2fef63e562011399fa3d6974df1fcd753a04" Sep 14 11:10:10.088886 ip-10-0-137-53 kubenswrapper[2581]: E0914 11:10:10.088871 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"865a9fecda1604bc101b128e4acb2fef63e562011399fa3d6974df1fcd753a04\": container with ID starting with 865a9fecda1604bc101b128e4acb2fef63e562011399fa3d6974df1fcd753a04 not found: ID does not exist" containerID="865a9fecda1604bc101b128e4acb2fef63e562011399fa3d6974df1fcd753a04" Sep 14 11:10:10.088935 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:10.088893 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"865a9fecda1604bc101b128e4acb2fef63e562011399fa3d6974df1fcd753a04"} err="failed to get container status \"865a9fecda1604bc101b128e4acb2fef63e562011399fa3d6974df1fcd753a04\": rpc error: code = NotFound desc = could not find container \"865a9fecda1604bc101b128e4acb2fef63e562011399fa3d6974df1fcd753a04\": container with ID starting with 865a9fecda1604bc101b128e4acb2fef63e562011399fa3d6974df1fcd753a04 not found: ID does not exist" Sep 14 11:10:10.135784 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:10.135699 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-04195c-5d85fcdd86-dlzdz"] Sep 14 11:10:10.147298 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:10.147268 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-04195c-5d85fcdd86-dlzdz"] Sep 14 11:10:10.168814 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:10.168779 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-04195c-559d584fc-6gqvc"] Sep 14 11:10:10.170978 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:10.170953 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-04195c-559d584fc-6gqvc"] Sep 14 11:10:10.832482 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:10.832443 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bd9eb61-9907-4947-84e5-d2c38ff06e1b" path="/var/lib/kubelet/pods/6bd9eb61-9907-4947-84e5-d2c38ff06e1b/volumes" Sep 14 11:10:10.832900 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:10.832807 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="899c3f91-4d36-4800-923b-cbab2e572a6c" path="/var/lib/kubelet/pods/899c3f91-4d36-4800-923b-cbab2e572a6c/volumes" Sep 14 11:10:10.833140 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:10.833126 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b870f894-e235-49c4-a502-c9dd63500b2e" path="/var/lib/kubelet/pods/b870f894-e235-49c4-a502-c9dd63500b2e/volumes" Sep 14 11:10:14.114669 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:14.114629 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm" podUID="c8650c87-2cf5-4198-a02c-82c1807e3e6e" containerName="main" containerID="cri-o://5310b6e703a71a17df510d0875c173846f814f9f34bda126dec55799a9cb4f64" gracePeriod=45 Sep 14 11:10:14.364791 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:14.364728 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm" Sep 14 11:10:14.390910 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:14.390879 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8650c87-2cf5-4198-a02c-82c1807e3e6e-dshm\") pod \"c8650c87-2cf5-4198-a02c-82c1807e3e6e\" (UID: \"c8650c87-2cf5-4198-a02c-82c1807e3e6e\") " Sep 14 11:10:14.391077 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:14.390930 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8650c87-2cf5-4198-a02c-82c1807e3e6e-home\") pod \"c8650c87-2cf5-4198-a02c-82c1807e3e6e\" (UID: \"c8650c87-2cf5-4198-a02c-82c1807e3e6e\") " Sep 14 11:10:14.391077 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:14.391053 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8650c87-2cf5-4198-a02c-82c1807e3e6e-tmp-dir\") pod \"c8650c87-2cf5-4198-a02c-82c1807e3e6e\" (UID: \"c8650c87-2cf5-4198-a02c-82c1807e3e6e\") " Sep 14 11:10:14.391186 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:14.391109 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8650c87-2cf5-4198-a02c-82c1807e3e6e-tls-certs\") pod \"c8650c87-2cf5-4198-a02c-82c1807e3e6e\" (UID: \"c8650c87-2cf5-4198-a02c-82c1807e3e6e\") " Sep 14 11:10:14.391186 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:14.391142 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8650c87-2cf5-4198-a02c-82c1807e3e6e-model-cache\") pod \"c8650c87-2cf5-4198-a02c-82c1807e3e6e\" (UID: \"c8650c87-2cf5-4198-a02c-82c1807e3e6e\") " Sep 14 11:10:14.391186 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:14.391170 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-br76s\" (UniqueName: \"kubernetes.io/projected/c8650c87-2cf5-4198-a02c-82c1807e3e6e-kube-api-access-br76s\") pod \"c8650c87-2cf5-4198-a02c-82c1807e3e6e\" (UID: \"c8650c87-2cf5-4198-a02c-82c1807e3e6e\") " Sep 14 11:10:14.391384 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:14.391260 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8650c87-2cf5-4198-a02c-82c1807e3e6e-home" (OuterVolumeSpecName: "home") pod "c8650c87-2cf5-4198-a02c-82c1807e3e6e" (UID: "c8650c87-2cf5-4198-a02c-82c1807e3e6e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:10:14.391715 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:14.391512 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8650c87-2cf5-4198-a02c-82c1807e3e6e-model-cache" (OuterVolumeSpecName: "model-cache") pod "c8650c87-2cf5-4198-a02c-82c1807e3e6e" (UID: "c8650c87-2cf5-4198-a02c-82c1807e3e6e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:10:14.391715 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:14.391536 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8650c87-2cf5-4198-a02c-82c1807e3e6e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c8650c87-2cf5-4198-a02c-82c1807e3e6e" (UID: "c8650c87-2cf5-4198-a02c-82c1807e3e6e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:10:14.391715 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:14.391640 2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8650c87-2cf5-4198-a02c-82c1807e3e6e-home\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:10:14.391715 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:14.391680 2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8650c87-2cf5-4198-a02c-82c1807e3e6e-tmp-dir\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:10:14.391715 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:14.391693 2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8650c87-2cf5-4198-a02c-82c1807e3e6e-model-cache\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:10:14.394089 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:14.394015 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8650c87-2cf5-4198-a02c-82c1807e3e6e-dshm" (OuterVolumeSpecName: "dshm") pod "c8650c87-2cf5-4198-a02c-82c1807e3e6e" (UID: "c8650c87-2cf5-4198-a02c-82c1807e3e6e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:10:14.394184 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:14.394081 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8650c87-2cf5-4198-a02c-82c1807e3e6e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c8650c87-2cf5-4198-a02c-82c1807e3e6e" (UID: "c8650c87-2cf5-4198-a02c-82c1807e3e6e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:10:14.394227 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:14.394195 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8650c87-2cf5-4198-a02c-82c1807e3e6e-kube-api-access-br76s" (OuterVolumeSpecName: "kube-api-access-br76s") pod "c8650c87-2cf5-4198-a02c-82c1807e3e6e" (UID: "c8650c87-2cf5-4198-a02c-82c1807e3e6e"). InnerVolumeSpecName "kube-api-access-br76s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:10:14.492478 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:14.492438 2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8650c87-2cf5-4198-a02c-82c1807e3e6e-tls-certs\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:10:14.492650 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:14.492491 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br76s\" (UniqueName: \"kubernetes.io/projected/c8650c87-2cf5-4198-a02c-82c1807e3e6e-kube-api-access-br76s\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:10:14.492650 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:14.492501 2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8650c87-2cf5-4198-a02c-82c1807e3e6e-dshm\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:10:15.102675 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:15.102584 2581 generic.go:356] "Generic (PLEG): container finished" podID="c8650c87-2cf5-4198-a02c-82c1807e3e6e" containerID="5310b6e703a71a17df510d0875c173846f814f9f34bda126dec55799a9cb4f64" exitCode=0 Sep 14 11:10:15.102675 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:15.102637 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm" Sep 14 11:10:15.102930 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:15.102670 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm" event={"ID":"c8650c87-2cf5-4198-a02c-82c1807e3e6e","Type":"ContainerDied","Data":"5310b6e703a71a17df510d0875c173846f814f9f34bda126dec55799a9cb4f64"} Sep 14 11:10:15.102930 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:15.102716 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm" event={"ID":"c8650c87-2cf5-4198-a02c-82c1807e3e6e","Type":"ContainerDied","Data":"601416ba36d14432755ab0b19661dcbbe9990b05189b4bec30c20b6a1b1d056e"} Sep 14 11:10:15.102930 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:15.102735 2581 scope.go:117] "RemoveContainer" containerID="5310b6e703a71a17df510d0875c173846f814f9f34bda126dec55799a9cb4f64" Sep 14 11:10:15.112044 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:15.112024 2581 scope.go:117] "RemoveContainer" containerID="5310b6e703a71a17df510d0875c173846f814f9f34bda126dec55799a9cb4f64" Sep 14 11:10:15.112325 ip-10-0-137-53 kubenswrapper[2581]: E0914 11:10:15.112306 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5310b6e703a71a17df510d0875c173846f814f9f34bda126dec55799a9cb4f64\": container with ID starting with 5310b6e703a71a17df510d0875c173846f814f9f34bda126dec55799a9cb4f64 not found: ID does not exist" containerID="5310b6e703a71a17df510d0875c173846f814f9f34bda126dec55799a9cb4f64" Sep 14 11:10:15.112405 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:15.112350 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5310b6e703a71a17df510d0875c173846f814f9f34bda126dec55799a9cb4f64"} err="failed to get container status \"5310b6e703a71a17df510d0875c173846f814f9f34bda126dec55799a9cb4f64\": rpc error: code = NotFound desc = could not find container \"5310b6e703a71a17df510d0875c173846f814f9f34bda126dec55799a9cb4f64\": container with ID starting with 5310b6e703a71a17df510d0875c173846f814f9f34bda126dec55799a9cb4f64 not found: ID does not exist" Sep 14 11:10:15.122664 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:15.122635 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm"] Sep 14 11:10:15.129424 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:15.129401 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4ebf73/e2e-conflict-a-3c60bd48-kserve-74cc89f45f-p2jcm"] Sep 14 11:10:16.832724 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:16.832694 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8650c87-2cf5-4198-a02c-82c1807e3e6e" path="/var/lib/kubelet/pods/c8650c87-2cf5-4198-a02c-82c1807e3e6e/volumes" Sep 14 11:10:17.703643 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:17.703606 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0777c1-h8ndl"] Sep 14 11:10:17.704222 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:17.704207 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b870f894-e235-49c4-a502-c9dd63500b2e" containerName="payload-processing" Sep 14 11:10:17.704267 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:17.704226 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b870f894-e235-49c4-a502-c9dd63500b2e" containerName="payload-processing" Sep 14 11:10:17.704267 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:17.704245 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bd9eb61-9907-4947-84e5-d2c38ff06e1b" containerName="revoke-keys" Sep 14 11:10:17.704267 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:17.704253 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bd9eb61-9907-4947-84e5-d2c38ff06e1b" containerName="revoke-keys" Sep 14 11:10:17.704392 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:17.704269 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8650c87-2cf5-4198-a02c-82c1807e3e6e" containerName="main" Sep 14 11:10:17.704392 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:17.704279 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8650c87-2cf5-4198-a02c-82c1807e3e6e" containerName="main" Sep 14 11:10:17.704392 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:17.704303 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="899c3f91-4d36-4800-923b-cbab2e572a6c" containerName="payload-pre-processing" Sep 14 11:10:17.704392 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:17.704311 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="899c3f91-4d36-4800-923b-cbab2e572a6c" containerName="payload-pre-processing" Sep 14 11:10:17.704516 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:17.704423 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b870f894-e235-49c4-a502-c9dd63500b2e" containerName="payload-processing" Sep 14 11:10:17.704516 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:17.704435 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8650c87-2cf5-4198-a02c-82c1807e3e6e" containerName="main" Sep 14 11:10:17.704516 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:17.704443 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bd9eb61-9907-4947-84e5-d2c38ff06e1b" containerName="revoke-keys" Sep 14 11:10:17.704516 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:17.704457 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="899c3f91-4d36-4800-923b-cbab2e572a6c" containerName="payload-pre-processing" Sep 14 11:10:17.709129 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:17.709112 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0777c1-h8ndl" Sep 14 11:10:17.711570 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:17.711550 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pv9g8\"" Sep 14 11:10:17.719013 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:17.718984 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0777c1-h8ndl"] Sep 14 11:10:17.725330 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:17.725300 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/953a2210-9632-4057-b1b3-29c3985e285a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0777c1-h8ndl\" (UID: \"953a2210-9632-4057-b1b3-29c3985e285a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0777c1-h8ndl" Sep 14 11:10:17.825981 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:17.825936 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/953a2210-9632-4057-b1b3-29c3985e285a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0777c1-h8ndl\" (UID: \"953a2210-9632-4057-b1b3-29c3985e285a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0777c1-h8ndl" Sep 14 11:10:17.826639 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:17.826614 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/953a2210-9632-4057-b1b3-29c3985e285a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0777c1-h8ndl\" (UID: \"953a2210-9632-4057-b1b3-29c3985e285a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0777c1-h8ndl" Sep 14 11:10:18.021530 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:18.021427 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0777c1-h8ndl" Sep 14 11:10:18.299908 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:18.299886 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0777c1-h8ndl"] Sep 14 11:10:18.301133 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:10:18.301109 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod953a2210_9632_4057_b1b3_29c3985e285a.slice/crio-b3fb47d31594d97da946e9785bece5b0ed95b5d91db25464cea12a45ab69f855 WatchSource:0}: Error finding container b3fb47d31594d97da946e9785bece5b0ed95b5d91db25464cea12a45ab69f855: Status 404 returned error can't find the container with id b3fb47d31594d97da946e9785bece5b0ed95b5d91db25464cea12a45ab69f855 Sep 14 11:10:19.128848 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:19.128805 2581 generic.go:356] "Generic (PLEG): container finished" podID="953a2210-9632-4057-b1b3-29c3985e285a" containerID="e964e4d87052e191a142d7052e8fe3c0093d76a5e9405e608aae1eb4098de7e8" exitCode=0 Sep 14 11:10:19.129228 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:19.128859 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0777c1-h8ndl" event={"ID":"953a2210-9632-4057-b1b3-29c3985e285a","Type":"ContainerDied","Data":"e964e4d87052e191a142d7052e8fe3c0093d76a5e9405e608aae1eb4098de7e8"} Sep 14 11:10:19.129228 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:19.128950 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0777c1-h8ndl" event={"ID":"953a2210-9632-4057-b1b3-29c3985e285a","Type":"ContainerStarted","Data":"b3fb47d31594d97da946e9785bece5b0ed95b5d91db25464cea12a45ab69f855"} Sep 14 11:10:20.270811 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:20.270780 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0777c1-h8ndl" Sep 14 11:10:20.349842 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:20.349806 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/953a2210-9632-4057-b1b3-29c3985e285a-maas-api-service-ca\") pod \"953a2210-9632-4057-b1b3-29c3985e285a\" (UID: \"953a2210-9632-4057-b1b3-29c3985e285a\") " Sep 14 11:10:20.350167 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:20.350144 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/953a2210-9632-4057-b1b3-29c3985e285a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "953a2210-9632-4057-b1b3-29c3985e285a" (UID: "953a2210-9632-4057-b1b3-29c3985e285a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:10:20.451125 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:20.451031 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/953a2210-9632-4057-b1b3-29c3985e285a-maas-api-service-ca\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:10:21.140979 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:21.140900 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0777c1-h8ndl" Sep 14 11:10:21.141138 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:21.140898 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0777c1-h8ndl" event={"ID":"953a2210-9632-4057-b1b3-29c3985e285a","Type":"ContainerDied","Data":"b3fb47d31594d97da946e9785bece5b0ed95b5d91db25464cea12a45ab69f855"} Sep 14 11:10:21.141138 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:21.141017 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3fb47d31594d97da946e9785bece5b0ed95b5d91db25464cea12a45ab69f855" Sep 14 11:10:25.817952 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:25.817915 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22b014-55ddd8cc54-m4lfm"] Sep 14 11:10:25.818392 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:25.818323 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="953a2210-9632-4057-b1b3-29c3985e285a" containerName="revoke-keys" Sep 14 11:10:25.818392 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:25.818347 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="953a2210-9632-4057-b1b3-29c3985e285a" containerName="revoke-keys" Sep 14 11:10:25.818492 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:25.818423 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="953a2210-9632-4057-b1b3-29c3985e285a" containerName="revoke-keys" Sep 14 11:10:25.821686 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:25.821670 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22b014-55ddd8cc54-m4lfm" Sep 14 11:10:25.827839 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:25.827816 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-22b014\"" Sep 14 11:10:25.827839 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:25.827828 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-22b014-dockercfg-z2jw2\"" Sep 14 11:10:25.859690 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:25.859659 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22b014-55ddd8cc54-m4lfm"] Sep 14 11:10:25.905521 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:25.905485 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00f56b03-0457-4ffd-87b9-110d2eba6e6d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-22b014-55ddd8cc54-m4lfm\" (UID: \"00f56b03-0457-4ffd-87b9-110d2eba6e6d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22b014-55ddd8cc54-m4lfm" Sep 14 11:10:25.905707 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:25.905613 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7zxw\" (UniqueName: \"kubernetes.io/projected/00f56b03-0457-4ffd-87b9-110d2eba6e6d-kube-api-access-x7zxw\") pod \"payload-pre-processing-e2e-shared-b-w2-22b014-55ddd8cc54-m4lfm\" (UID: \"00f56b03-0457-4ffd-87b9-110d2eba6e6d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22b014-55ddd8cc54-m4lfm" Sep 14 11:10:26.006771 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:26.006735 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7zxw\" (UniqueName: \"kubernetes.io/projected/00f56b03-0457-4ffd-87b9-110d2eba6e6d-kube-api-access-x7zxw\") pod \"payload-pre-processing-e2e-shared-b-w2-22b014-55ddd8cc54-m4lfm\" (UID: \"00f56b03-0457-4ffd-87b9-110d2eba6e6d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22b014-55ddd8cc54-m4lfm" Sep 14 11:10:26.006940 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:26.006815 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00f56b03-0457-4ffd-87b9-110d2eba6e6d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-22b014-55ddd8cc54-m4lfm\" (UID: \"00f56b03-0457-4ffd-87b9-110d2eba6e6d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22b014-55ddd8cc54-m4lfm" Sep 14 11:10:26.007472 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:26.007452 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00f56b03-0457-4ffd-87b9-110d2eba6e6d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-22b014-55ddd8cc54-m4lfm\" (UID: \"00f56b03-0457-4ffd-87b9-110d2eba6e6d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22b014-55ddd8cc54-m4lfm" Sep 14 11:10:26.026023 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:26.025996 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7zxw\" (UniqueName: \"kubernetes.io/projected/00f56b03-0457-4ffd-87b9-110d2eba6e6d-kube-api-access-x7zxw\") pod \"payload-pre-processing-e2e-shared-b-w2-22b014-55ddd8cc54-m4lfm\" (UID: \"00f56b03-0457-4ffd-87b9-110d2eba6e6d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22b014-55ddd8cc54-m4lfm" Sep 14 11:10:26.131877 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:26.131774 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22b014-55ddd8cc54-m4lfm" Sep 14 11:10:26.436505 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:26.436434 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22b014-55ddd8cc54-m4lfm"] Sep 14 11:10:26.438887 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:10:26.438857 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00f56b03_0457_4ffd_87b9_110d2eba6e6d.slice/crio-75411b97c3fa5b69157786d26e159c9bc18e1139246fd72e3a17255c41c4463f WatchSource:0}: Error finding container 75411b97c3fa5b69157786d26e159c9bc18e1139246fd72e3a17255c41c4463f: Status 404 returned error can't find the container with id 75411b97c3fa5b69157786d26e159c9bc18e1139246fd72e3a17255c41c4463f Sep 14 11:10:27.175686 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:27.175648 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22b014-55ddd8cc54-m4lfm" event={"ID":"00f56b03-0457-4ffd-87b9-110d2eba6e6d","Type":"ContainerStarted","Data":"b83e87c6281ef3a6437aef42799143b5c8f4556ef23b63df747cc396d71c2269"} Sep 14 11:10:27.176081 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:27.175707 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22b014-55ddd8cc54-m4lfm" event={"ID":"00f56b03-0457-4ffd-87b9-110d2eba6e6d","Type":"ContainerStarted","Data":"75411b97c3fa5b69157786d26e159c9bc18e1139246fd72e3a17255c41c4463f"} Sep 14 11:10:27.176081 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:27.175727 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22b014-55ddd8cc54-m4lfm" Sep 14 11:10:27.238101 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:27.238038 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22b014-55ddd8cc54-m4lfm" podStartSLOduration=2.238017087 podStartE2EDuration="2.238017087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:10:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:10:27.230028881 +0000 UTC m=+1300.972430842" watchObservedRunningTime="2026-09-14 11:10:27.238017087 +0000 UTC m=+1300.980419064" Sep 14 11:10:30.092774 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:30.092742 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0777c1-6c5445689c-wqd5f"] Sep 14 11:10:30.093222 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:30.092974 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-0777c1-6c5445689c-wqd5f" podUID="873c3405-3775-43ba-838f-52dccccc4d59" containerName="payload-processing" containerID="cri-o://983ad1c3c689327a4c6e12bedc93dd331fe171d013096a669888a70d3cc78f39" gracePeriod=30 Sep 14 11:10:30.109495 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:30.109458 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0777c1-75869b6f54-t2s9l"] Sep 14 11:10:30.109693 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:30.109648 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0777c1-75869b6f54-t2s9l" podUID="c41213ee-43d5-4d4d-a368-fd363e0644fd" containerName="payload-pre-processing" containerID="cri-o://7831680f2bb7659c5541f316dec1a6d4958fb27bf3b946dc774ec0c2809dd5f5" gracePeriod=30 Sep 14 11:10:30.396840 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:30.396815 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0777c1-6c5445689c-wqd5f" Sep 14 11:10:30.400023 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:30.400005 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0777c1-75869b6f54-t2s9l" Sep 14 11:10:30.451679 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:30.451645 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c41213ee-43d5-4d4d-a368-fd363e0644fd-plugins-config-volume\") pod \"c41213ee-43d5-4d4d-a368-fd363e0644fd\" (UID: \"c41213ee-43d5-4d4d-a368-fd363e0644fd\") " Sep 14 11:10:30.451959 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:30.451705 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lthl\" (UniqueName: \"kubernetes.io/projected/c41213ee-43d5-4d4d-a368-fd363e0644fd-kube-api-access-4lthl\") pod \"c41213ee-43d5-4d4d-a368-fd363e0644fd\" (UID: \"c41213ee-43d5-4d4d-a368-fd363e0644fd\") " Sep 14 11:10:30.451959 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:30.451768 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnzwj\" (UniqueName: \"kubernetes.io/projected/873c3405-3775-43ba-838f-52dccccc4d59-kube-api-access-dnzwj\") pod \"873c3405-3775-43ba-838f-52dccccc4d59\" (UID: \"873c3405-3775-43ba-838f-52dccccc4d59\") " Sep 14 11:10:30.451959 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:30.451817 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/873c3405-3775-43ba-838f-52dccccc4d59-plugins-config-volume\") pod \"873c3405-3775-43ba-838f-52dccccc4d59\" (UID: \"873c3405-3775-43ba-838f-52dccccc4d59\") " Sep 14 11:10:30.452142 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:30.452066 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c41213ee-43d5-4d4d-a368-fd363e0644fd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c41213ee-43d5-4d4d-a368-fd363e0644fd" (UID: "c41213ee-43d5-4d4d-a368-fd363e0644fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:10:30.452194 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:30.452162 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/873c3405-3775-43ba-838f-52dccccc4d59-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "873c3405-3775-43ba-838f-52dccccc4d59" (UID: "873c3405-3775-43ba-838f-52dccccc4d59"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:10:30.452259 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:30.452236 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c41213ee-43d5-4d4d-a368-fd363e0644fd-plugins-config-volume\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:10:30.454168 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:30.454137 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/873c3405-3775-43ba-838f-52dccccc4d59-kube-api-access-dnzwj" (OuterVolumeSpecName: "kube-api-access-dnzwj") pod "873c3405-3775-43ba-838f-52dccccc4d59" (UID: "873c3405-3775-43ba-838f-52dccccc4d59"). InnerVolumeSpecName "kube-api-access-dnzwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:10:30.454276 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:30.454200 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c41213ee-43d5-4d4d-a368-fd363e0644fd-kube-api-access-4lthl" (OuterVolumeSpecName: "kube-api-access-4lthl") pod "c41213ee-43d5-4d4d-a368-fd363e0644fd" (UID: "c41213ee-43d5-4d4d-a368-fd363e0644fd"). InnerVolumeSpecName "kube-api-access-4lthl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:10:30.470488 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:30.470461 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0777c1-h8ndl"] Sep 14 11:10:30.476967 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:30.476937 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0777c1-h8ndl"] Sep 14 11:10:30.553009 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:30.552966 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lthl\" (UniqueName: \"kubernetes.io/projected/c41213ee-43d5-4d4d-a368-fd363e0644fd-kube-api-access-4lthl\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:10:30.553009 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:30.552998 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnzwj\" (UniqueName: \"kubernetes.io/projected/873c3405-3775-43ba-838f-52dccccc4d59-kube-api-access-dnzwj\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:10:30.553009 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:30.553011 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/873c3405-3775-43ba-838f-52dccccc4d59-plugins-config-volume\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:10:30.832901 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:30.832857 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="953a2210-9632-4057-b1b3-29c3985e285a" path="/var/lib/kubelet/pods/953a2210-9632-4057-b1b3-29c3985e285a/volumes" Sep 14 11:10:31.198042 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:31.198006 2581 generic.go:356] "Generic (PLEG): container finished" podID="c41213ee-43d5-4d4d-a368-fd363e0644fd" containerID="7831680f2bb7659c5541f316dec1a6d4958fb27bf3b946dc774ec0c2809dd5f5" exitCode=0 Sep 14 11:10:31.198542 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:31.198055 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0777c1-75869b6f54-t2s9l" Sep 14 11:10:31.198542 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:31.198088 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0777c1-75869b6f54-t2s9l" event={"ID":"c41213ee-43d5-4d4d-a368-fd363e0644fd","Type":"ContainerDied","Data":"7831680f2bb7659c5541f316dec1a6d4958fb27bf3b946dc774ec0c2809dd5f5"} Sep 14 11:10:31.198542 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:31.198142 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0777c1-75869b6f54-t2s9l" event={"ID":"c41213ee-43d5-4d4d-a368-fd363e0644fd","Type":"ContainerDied","Data":"a113ee74c215e79855a15636f89d09fb21ccb8e3727ed0807889cf06fce7d573"} Sep 14 11:10:31.198542 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:31.198173 2581 scope.go:117] "RemoveContainer" containerID="7831680f2bb7659c5541f316dec1a6d4958fb27bf3b946dc774ec0c2809dd5f5" Sep 14 11:10:31.199692 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:31.199621 2581 generic.go:356] "Generic (PLEG): container finished" podID="873c3405-3775-43ba-838f-52dccccc4d59" containerID="983ad1c3c689327a4c6e12bedc93dd331fe171d013096a669888a70d3cc78f39" exitCode=0 Sep 14 11:10:31.199692 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:31.199654 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0777c1-6c5445689c-wqd5f" event={"ID":"873c3405-3775-43ba-838f-52dccccc4d59","Type":"ContainerDied","Data":"983ad1c3c689327a4c6e12bedc93dd331fe171d013096a669888a70d3cc78f39"} Sep 14 11:10:31.199692 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:31.199679 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0777c1-6c5445689c-wqd5f" Sep 14 11:10:31.199857 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:31.199704 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0777c1-6c5445689c-wqd5f" event={"ID":"873c3405-3775-43ba-838f-52dccccc4d59","Type":"ContainerDied","Data":"97c2e5a29bc07fdcd213a9f5d0ce40dbda845d109c15852e1f355e5f7b78fe60"} Sep 14 11:10:31.207747 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:31.207728 2581 scope.go:117] "RemoveContainer" containerID="7831680f2bb7659c5541f316dec1a6d4958fb27bf3b946dc774ec0c2809dd5f5" Sep 14 11:10:31.208055 ip-10-0-137-53 kubenswrapper[2581]: E0914 11:10:31.208037 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7831680f2bb7659c5541f316dec1a6d4958fb27bf3b946dc774ec0c2809dd5f5\": container with ID starting with 7831680f2bb7659c5541f316dec1a6d4958fb27bf3b946dc774ec0c2809dd5f5 not found: ID does not exist" containerID="7831680f2bb7659c5541f316dec1a6d4958fb27bf3b946dc774ec0c2809dd5f5" Sep 14 11:10:31.208107 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:31.208065 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7831680f2bb7659c5541f316dec1a6d4958fb27bf3b946dc774ec0c2809dd5f5"} err="failed to get container status \"7831680f2bb7659c5541f316dec1a6d4958fb27bf3b946dc774ec0c2809dd5f5\": rpc error: code = NotFound desc = could not find container \"7831680f2bb7659c5541f316dec1a6d4958fb27bf3b946dc774ec0c2809dd5f5\": container with ID starting with 7831680f2bb7659c5541f316dec1a6d4958fb27bf3b946dc774ec0c2809dd5f5 not found: ID does not exist" Sep 14 11:10:31.208107 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:31.208082 2581 scope.go:117] "RemoveContainer" containerID="983ad1c3c689327a4c6e12bedc93dd331fe171d013096a669888a70d3cc78f39" Sep 14 11:10:31.216939 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:31.216921 2581 scope.go:117] "RemoveContainer" containerID="983ad1c3c689327a4c6e12bedc93dd331fe171d013096a669888a70d3cc78f39" Sep 14 11:10:31.217227 ip-10-0-137-53 kubenswrapper[2581]: E0914 11:10:31.217205 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"983ad1c3c689327a4c6e12bedc93dd331fe171d013096a669888a70d3cc78f39\": container with ID starting with 983ad1c3c689327a4c6e12bedc93dd331fe171d013096a669888a70d3cc78f39 not found: ID does not exist" containerID="983ad1c3c689327a4c6e12bedc93dd331fe171d013096a669888a70d3cc78f39" Sep 14 11:10:31.217272 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:31.217237 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"983ad1c3c689327a4c6e12bedc93dd331fe171d013096a669888a70d3cc78f39"} err="failed to get container status \"983ad1c3c689327a4c6e12bedc93dd331fe171d013096a669888a70d3cc78f39\": rpc error: code = NotFound desc = could not find container \"983ad1c3c689327a4c6e12bedc93dd331fe171d013096a669888a70d3cc78f39\": container with ID starting with 983ad1c3c689327a4c6e12bedc93dd331fe171d013096a669888a70d3cc78f39 not found: ID does not exist" Sep 14 11:10:31.252055 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:31.252029 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0777c1-75869b6f54-t2s9l"] Sep 14 11:10:31.269987 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:31.269959 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0777c1-75869b6f54-t2s9l"] Sep 14 11:10:31.314991 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:31.314961 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0777c1-6c5445689c-wqd5f"] Sep 14 11:10:31.327884 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:31.327859 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0777c1-6c5445689c-wqd5f"] Sep 14 11:10:32.322825 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:32.322791 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4ebf73/distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks"] Sep 14 11:10:32.323209 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:32.323189 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="873c3405-3775-43ba-838f-52dccccc4d59" containerName="payload-processing" Sep 14 11:10:32.323209 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:32.323201 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="873c3405-3775-43ba-838f-52dccccc4d59" containerName="payload-processing" Sep 14 11:10:32.323209 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:32.323209 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c41213ee-43d5-4d4d-a368-fd363e0644fd" containerName="payload-pre-processing" Sep 14 11:10:32.323314 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:32.323214 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c41213ee-43d5-4d4d-a368-fd363e0644fd" containerName="payload-pre-processing" Sep 14 11:10:32.323314 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:32.323286 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c41213ee-43d5-4d4d-a368-fd363e0644fd" containerName="payload-pre-processing" Sep 14 11:10:32.323314 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:32.323294 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="873c3405-3775-43ba-838f-52dccccc4d59" containerName="payload-processing" Sep 14 11:10:32.327704 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:32.327689 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ebf73/distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks" Sep 14 11:10:32.331299 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:32.331278 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4ebf73\"/\"openshift-service-ca.crt\"" Sep 14 11:10:32.332205 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:32.332188 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4ebf73\"/\"distinct-w4-4ebf73-kserve-self-signed-certs\"" Sep 14 11:10:32.332459 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:32.332435 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4ebf73\"/\"default-dockercfg-8975n\"" Sep 14 11:10:32.332548 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:32.332480 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4ebf73\"/\"kube-root-ca.crt\"" Sep 14 11:10:32.348879 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:32.348856 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4ebf73/distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks"] Sep 14 11:10:32.472299 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:32.472268 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55d476ad-c696-4d47-b3c1-d3cd01d5a658-home\") pod \"distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks\" (UID: \"55d476ad-c696-4d47-b3c1-d3cd01d5a658\") " pod="e2e-models-e2e-worker-w4-4ebf73/distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks" Sep 14 11:10:32.472514 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:32.472318 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55d476ad-c696-4d47-b3c1-d3cd01d5a658-tmp-dir\") pod \"distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks\" (UID: \"55d476ad-c696-4d47-b3c1-d3cd01d5a658\") " pod="e2e-models-e2e-worker-w4-4ebf73/distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks" Sep 14 11:10:32.472514 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:32.472385 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgsl2\" (UniqueName: \"kubernetes.io/projected/55d476ad-c696-4d47-b3c1-d3cd01d5a658-kube-api-access-pgsl2\") pod \"distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks\" (UID: \"55d476ad-c696-4d47-b3c1-d3cd01d5a658\") " pod="e2e-models-e2e-worker-w4-4ebf73/distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks" Sep 14 11:10:32.472514 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:32.472481 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55d476ad-c696-4d47-b3c1-d3cd01d5a658-dshm\") pod \"distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks\" (UID: \"55d476ad-c696-4d47-b3c1-d3cd01d5a658\") " pod="e2e-models-e2e-worker-w4-4ebf73/distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks" Sep 14 11:10:32.472514 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:32.472510 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55d476ad-c696-4d47-b3c1-d3cd01d5a658-model-cache\") pod \"distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks\" (UID: \"55d476ad-c696-4d47-b3c1-d3cd01d5a658\") " pod="e2e-models-e2e-worker-w4-4ebf73/distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks" Sep 14 11:10:32.472658 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:32.472557 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55d476ad-c696-4d47-b3c1-d3cd01d5a658-tls-certs\") pod \"distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks\" (UID: \"55d476ad-c696-4d47-b3c1-d3cd01d5a658\") " pod="e2e-models-e2e-worker-w4-4ebf73/distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks" Sep 14 11:10:32.573855 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:32.573758 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55d476ad-c696-4d47-b3c1-d3cd01d5a658-dshm\") pod \"distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks\" (UID: \"55d476ad-c696-4d47-b3c1-d3cd01d5a658\") " pod="e2e-models-e2e-worker-w4-4ebf73/distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks" Sep 14 11:10:32.573855 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:32.573810 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55d476ad-c696-4d47-b3c1-d3cd01d5a658-model-cache\") pod \"distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks\" (UID: \"55d476ad-c696-4d47-b3c1-d3cd01d5a658\") " pod="e2e-models-e2e-worker-w4-4ebf73/distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks" Sep 14 11:10:32.573855 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:32.573847 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55d476ad-c696-4d47-b3c1-d3cd01d5a658-tls-certs\") pod \"distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks\" (UID: \"55d476ad-c696-4d47-b3c1-d3cd01d5a658\") " pod="e2e-models-e2e-worker-w4-4ebf73/distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks" Sep 14 11:10:32.574068 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:32.573899 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55d476ad-c696-4d47-b3c1-d3cd01d5a658-home\") pod \"distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks\" (UID: \"55d476ad-c696-4d47-b3c1-d3cd01d5a658\") " pod="e2e-models-e2e-worker-w4-4ebf73/distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks" Sep 14 11:10:32.574068 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:32.573948 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55d476ad-c696-4d47-b3c1-d3cd01d5a658-tmp-dir\") pod \"distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks\" (UID: \"55d476ad-c696-4d47-b3c1-d3cd01d5a658\") " pod="e2e-models-e2e-worker-w4-4ebf73/distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks" Sep 14 11:10:32.574068 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:32.573992 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgsl2\" (UniqueName: \"kubernetes.io/projected/55d476ad-c696-4d47-b3c1-d3cd01d5a658-kube-api-access-pgsl2\") pod \"distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks\" (UID: \"55d476ad-c696-4d47-b3c1-d3cd01d5a658\") " pod="e2e-models-e2e-worker-w4-4ebf73/distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks" Sep 14 11:10:32.574311 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:32.574288 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55d476ad-c696-4d47-b3c1-d3cd01d5a658-model-cache\") pod \"distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks\" (UID: \"55d476ad-c696-4d47-b3c1-d3cd01d5a658\") " pod="e2e-models-e2e-worker-w4-4ebf73/distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks" Sep 14 11:10:32.574418 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:32.574300 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55d476ad-c696-4d47-b3c1-d3cd01d5a658-home\") pod \"distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks\" (UID: \"55d476ad-c696-4d47-b3c1-d3cd01d5a658\") " pod="e2e-models-e2e-worker-w4-4ebf73/distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks" Sep 14 11:10:32.574470 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:32.574407 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55d476ad-c696-4d47-b3c1-d3cd01d5a658-tmp-dir\") pod \"distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks\" (UID: \"55d476ad-c696-4d47-b3c1-d3cd01d5a658\") " pod="e2e-models-e2e-worker-w4-4ebf73/distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks" Sep 14 11:10:32.576453 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:32.576433 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55d476ad-c696-4d47-b3c1-d3cd01d5a658-tls-certs\") pod \"distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks\" (UID: \"55d476ad-c696-4d47-b3c1-d3cd01d5a658\") " pod="e2e-models-e2e-worker-w4-4ebf73/distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks" Sep 14 11:10:32.576647 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:32.576451 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55d476ad-c696-4d47-b3c1-d3cd01d5a658-dshm\") pod \"distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks\" (UID: \"55d476ad-c696-4d47-b3c1-d3cd01d5a658\") " pod="e2e-models-e2e-worker-w4-4ebf73/distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks" Sep 14 11:10:32.584229 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:32.583785 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgsl2\" (UniqueName: \"kubernetes.io/projected/55d476ad-c696-4d47-b3c1-d3cd01d5a658-kube-api-access-pgsl2\") pod \"distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks\" (UID: \"55d476ad-c696-4d47-b3c1-d3cd01d5a658\") " pod="e2e-models-e2e-worker-w4-4ebf73/distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks" Sep 14 11:10:32.637344 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:32.637313 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ebf73/distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks" Sep 14 11:10:32.832595 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:32.832500 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="873c3405-3775-43ba-838f-52dccccc4d59" path="/var/lib/kubelet/pods/873c3405-3775-43ba-838f-52dccccc4d59/volumes" Sep 14 11:10:32.833000 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:32.832981 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c41213ee-43d5-4d4d-a368-fd363e0644fd" path="/var/lib/kubelet/pods/c41213ee-43d5-4d4d-a368-fd363e0644fd/volumes" Sep 14 11:10:32.917029 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:32.916981 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4ebf73/distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks"] Sep 14 11:10:32.919451 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:10:32.919422 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55d476ad_c696_4d47_b3c1_d3cd01d5a658.slice/crio-3f3b0a80c8768f736ac609e24be20b274ff8141fc3904d4c76f5e0d04047c568 WatchSource:0}: Error finding container 3f3b0a80c8768f736ac609e24be20b274ff8141fc3904d4c76f5e0d04047c568: Status 404 returned error can't find the container with id 3f3b0a80c8768f736ac609e24be20b274ff8141fc3904d4c76f5e0d04047c568 Sep 14 11:10:33.215264 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:33.215168 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ebf73/distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks" event={"ID":"55d476ad-c696-4d47-b3c1-d3cd01d5a658","Type":"ContainerStarted","Data":"23cc0d1518d40278a85e7f162e3f2ddb3607c2574d477bf8763a983a2cb8d8d2"} Sep 14 11:10:33.215264 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:33.215215 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ebf73/distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks" event={"ID":"55d476ad-c696-4d47-b3c1-d3cd01d5a658","Type":"ContainerStarted","Data":"3f3b0a80c8768f736ac609e24be20b274ff8141fc3904d4c76f5e0d04047c568"} Sep 14 11:10:33.238064 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:33.238016 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4ebf73/distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks" podStartSLOduration=1.23799939 podStartE2EDuration="1.23799939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:10:33.237234885 +0000 UTC m=+1306.979636857" watchObservedRunningTime="2026-09-14 11:10:33.23799939 +0000 UTC m=+1306.980401360" Sep 14 11:10:38.182711 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:38.182681 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22b014-55ddd8cc54-m4lfm" Sep 14 11:10:42.637930 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:42.637891 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ebf73/distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks" Sep 14 11:10:42.638354 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:42.637949 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4ebf73/distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks" Sep 14 11:10:42.650592 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:42.650564 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4ebf73/distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks" Sep 14 11:10:43.283016 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:43.282990 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ebf73/distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks" Sep 14 11:10:46.866947 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:46.866920 2581 scope.go:117] "RemoveContainer" containerID="dd8aa786fc1bc681cfae1f5a1ee61f5677b2fbbd446729ec73053e25f4e988f5" Sep 14 11:10:46.929962 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:46.929926 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-fdbc00-64d8fb85bf-5zgk5"] Sep 14 11:10:46.934644 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:46.934626 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fdbc00-64d8fb85bf-5zgk5" Sep 14 11:10:46.937459 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:46.937431 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-fdbc00\"" Sep 14 11:10:46.937809 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:46.937793 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-fdbc00-dockercfg-k2rsw\"" Sep 14 11:10:46.944835 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:46.944815 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-fdbc00-64d8fb85bf-5zgk5"] Sep 14 11:10:46.979532 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:46.979499 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdbc00-67944d958-zfr26"] Sep 14 11:10:46.983034 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:46.983015 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdbc00-67944d958-zfr26" Sep 14 11:10:46.991956 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:46.991929 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdbc00-67944d958-zfr26"] Sep 14 11:10:47.022429 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:47.022402 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbzvq\" (UniqueName: \"kubernetes.io/projected/ef8546d1-98b8-4f03-bf2c-90b013c279de-kube-api-access-qbzvq\") pod \"payload-processing-e2e-shared-a-w3-fdbc00-64d8fb85bf-5zgk5\" (UID: \"ef8546d1-98b8-4f03-bf2c-90b013c279de\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fdbc00-64d8fb85bf-5zgk5" Sep 14 11:10:47.022595 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:47.022450 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef8546d1-98b8-4f03-bf2c-90b013c279de-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-fdbc00-64d8fb85bf-5zgk5\" (UID: \"ef8546d1-98b8-4f03-bf2c-90b013c279de\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fdbc00-64d8fb85bf-5zgk5" Sep 14 11:10:47.123643 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:47.123554 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef8546d1-98b8-4f03-bf2c-90b013c279de-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-fdbc00-64d8fb85bf-5zgk5\" (UID: \"ef8546d1-98b8-4f03-bf2c-90b013c279de\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fdbc00-64d8fb85bf-5zgk5" Sep 14 11:10:47.123643 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:47.123606 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4d3967c-eeb2-4edd-af84-18df43e18c09-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-fdbc00-67944d958-zfr26\" (UID: \"d4d3967c-eeb2-4edd-af84-18df43e18c09\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdbc00-67944d958-zfr26" Sep 14 11:10:47.123880 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:47.123665 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz7xd\" (UniqueName: \"kubernetes.io/projected/d4d3967c-eeb2-4edd-af84-18df43e18c09-kube-api-access-bz7xd\") pod \"payload-pre-processing-e2e-shared-a-w3-fdbc00-67944d958-zfr26\" (UID: \"d4d3967c-eeb2-4edd-af84-18df43e18c09\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdbc00-67944d958-zfr26" Sep 14 11:10:47.123880 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:47.123833 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbzvq\" (UniqueName: \"kubernetes.io/projected/ef8546d1-98b8-4f03-bf2c-90b013c279de-kube-api-access-qbzvq\") pod \"payload-processing-e2e-shared-a-w3-fdbc00-64d8fb85bf-5zgk5\" (UID: \"ef8546d1-98b8-4f03-bf2c-90b013c279de\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fdbc00-64d8fb85bf-5zgk5" Sep 14 11:10:47.124390 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:47.124373 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef8546d1-98b8-4f03-bf2c-90b013c279de-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-fdbc00-64d8fb85bf-5zgk5\" (UID: \"ef8546d1-98b8-4f03-bf2c-90b013c279de\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fdbc00-64d8fb85bf-5zgk5" Sep 14 11:10:47.132969 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:47.132946 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbzvq\" (UniqueName: \"kubernetes.io/projected/ef8546d1-98b8-4f03-bf2c-90b013c279de-kube-api-access-qbzvq\") pod \"payload-processing-e2e-shared-a-w3-fdbc00-64d8fb85bf-5zgk5\" (UID: \"ef8546d1-98b8-4f03-bf2c-90b013c279de\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fdbc00-64d8fb85bf-5zgk5" Sep 14 11:10:47.224605 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:47.224566 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz7xd\" (UniqueName: \"kubernetes.io/projected/d4d3967c-eeb2-4edd-af84-18df43e18c09-kube-api-access-bz7xd\") pod \"payload-pre-processing-e2e-shared-a-w3-fdbc00-67944d958-zfr26\" (UID: \"d4d3967c-eeb2-4edd-af84-18df43e18c09\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdbc00-67944d958-zfr26" Sep 14 11:10:47.224809 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:47.224757 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4d3967c-eeb2-4edd-af84-18df43e18c09-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-fdbc00-67944d958-zfr26\" (UID: \"d4d3967c-eeb2-4edd-af84-18df43e18c09\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdbc00-67944d958-zfr26" Sep 14 11:10:47.225423 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:47.225406 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4d3967c-eeb2-4edd-af84-18df43e18c09-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-fdbc00-67944d958-zfr26\" (UID: \"d4d3967c-eeb2-4edd-af84-18df43e18c09\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdbc00-67944d958-zfr26" Sep 14 11:10:47.236023 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:47.236003 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz7xd\" (UniqueName: \"kubernetes.io/projected/d4d3967c-eeb2-4edd-af84-18df43e18c09-kube-api-access-bz7xd\") pod \"payload-pre-processing-e2e-shared-a-w3-fdbc00-67944d958-zfr26\" (UID: \"d4d3967c-eeb2-4edd-af84-18df43e18c09\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdbc00-67944d958-zfr26" Sep 14 11:10:47.245920 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:47.245894 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fdbc00-64d8fb85bf-5zgk5" Sep 14 11:10:47.294687 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:47.294649 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdbc00-67944d958-zfr26" Sep 14 11:10:47.523569 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:47.523544 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-fdbc00-64d8fb85bf-5zgk5"] Sep 14 11:10:47.524913 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:10:47.524886 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef8546d1_98b8_4f03_bf2c_90b013c279de.slice/crio-73333f4f5f41a8e351390923a26eaf3065975a24453aa68b0b2c749f603ae188 WatchSource:0}: Error finding container 73333f4f5f41a8e351390923a26eaf3065975a24453aa68b0b2c749f603ae188: Status 404 returned error can't find the container with id 73333f4f5f41a8e351390923a26eaf3065975a24453aa68b0b2c749f603ae188 Sep 14 11:10:47.590495 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:47.590196 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdbc00-67944d958-zfr26"] Sep 14 11:10:48.301525 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:48.301487 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdbc00-67944d958-zfr26" event={"ID":"d4d3967c-eeb2-4edd-af84-18df43e18c09","Type":"ContainerStarted","Data":"a6ab1f990a938445a70dbb698b1e4a2165919f3f9b056caa4e15eac40e19b5cf"} Sep 14 11:10:48.302005 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:48.301555 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdbc00-67944d958-zfr26" Sep 14 11:10:48.302005 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:48.301581 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdbc00-67944d958-zfr26" event={"ID":"d4d3967c-eeb2-4edd-af84-18df43e18c09","Type":"ContainerStarted","Data":"a7aa19a5768edb8e9c07f523e9c56c0f3fb5b98ec1eb49cb2582c961136ea7c8"} Sep 14 11:10:48.303195 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:48.303169 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fdbc00-64d8fb85bf-5zgk5" event={"ID":"ef8546d1-98b8-4f03-bf2c-90b013c279de","Type":"ContainerStarted","Data":"730c957185d5e82d19705cdf258606c783e38dbf23acdf216542e2f4df7e3456"} Sep 14 11:10:48.303323 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:48.303222 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fdbc00-64d8fb85bf-5zgk5" event={"ID":"ef8546d1-98b8-4f03-bf2c-90b013c279de","Type":"ContainerStarted","Data":"73333f4f5f41a8e351390923a26eaf3065975a24453aa68b0b2c749f603ae188"} Sep 14 11:10:48.303323 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:48.303253 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fdbc00-64d8fb85bf-5zgk5" Sep 14 11:10:48.326425 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:48.326366 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdbc00-67944d958-zfr26" podStartSLOduration=2.326316214 podStartE2EDuration="2.326316214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:10:48.323111794 +0000 UTC m=+1322.065513764" watchObservedRunningTime="2026-09-14 11:10:48.326316214 +0000 UTC m=+1322.068718184" Sep 14 11:10:48.348263 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:48.348201 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fdbc00-64d8fb85bf-5zgk5" podStartSLOduration=2.348181321 podStartE2EDuration="2.348181321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:10:48.346528064 +0000 UTC m=+1322.088930036" watchObservedRunningTime="2026-09-14 11:10:48.348181321 +0000 UTC m=+1322.090583294" Sep 14 11:10:57.937556 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:57.937522 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e6e0b4a3-6669d65c98-l8xz9"] Sep 14 11:10:57.942020 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:57.942002 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e6e0b4a3-6669d65c98-l8xz9" Sep 14 11:10:57.944771 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:57.944754 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e6e0b4a3\"" Sep 14 11:10:57.944928 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:57.944912 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e6e0b4a3-dockercfg-zd8pp\"" Sep 14 11:10:57.951172 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:57.951151 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e6e0b4a3-6669d65c98-l8xz9"] Sep 14 11:10:58.034991 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:58.034945 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfdzl\" (UniqueName: \"kubernetes.io/projected/4b0c80e5-881f-45d8-a7f9-77dac57c9d4c-kube-api-access-rfdzl\") pod \"payload-pre-processing-e2e-mt-e6e0b4a3-6669d65c98-l8xz9\" (UID: \"4b0c80e5-881f-45d8-a7f9-77dac57c9d4c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e6e0b4a3-6669d65c98-l8xz9" Sep 14 11:10:58.035202 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:58.035180 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b0c80e5-881f-45d8-a7f9-77dac57c9d4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e6e0b4a3-6669d65c98-l8xz9\" (UID: \"4b0c80e5-881f-45d8-a7f9-77dac57c9d4c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e6e0b4a3-6669d65c98-l8xz9" Sep 14 11:10:58.136472 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:58.136431 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b0c80e5-881f-45d8-a7f9-77dac57c9d4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e6e0b4a3-6669d65c98-l8xz9\" (UID: \"4b0c80e5-881f-45d8-a7f9-77dac57c9d4c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e6e0b4a3-6669d65c98-l8xz9" Sep 14 11:10:58.136653 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:58.136491 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfdzl\" (UniqueName: \"kubernetes.io/projected/4b0c80e5-881f-45d8-a7f9-77dac57c9d4c-kube-api-access-rfdzl\") pod \"payload-pre-processing-e2e-mt-e6e0b4a3-6669d65c98-l8xz9\" (UID: \"4b0c80e5-881f-45d8-a7f9-77dac57c9d4c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e6e0b4a3-6669d65c98-l8xz9" Sep 14 11:10:58.137189 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:58.137163 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b0c80e5-881f-45d8-a7f9-77dac57c9d4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e6e0b4a3-6669d65c98-l8xz9\" (UID: \"4b0c80e5-881f-45d8-a7f9-77dac57c9d4c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e6e0b4a3-6669d65c98-l8xz9" Sep 14 11:10:58.145984 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:58.145957 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfdzl\" (UniqueName: \"kubernetes.io/projected/4b0c80e5-881f-45d8-a7f9-77dac57c9d4c-kube-api-access-rfdzl\") pod \"payload-pre-processing-e2e-mt-e6e0b4a3-6669d65c98-l8xz9\" (UID: \"4b0c80e5-881f-45d8-a7f9-77dac57c9d4c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e6e0b4a3-6669d65c98-l8xz9" Sep 14 11:10:58.254742 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:58.254653 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e6e0b4a3-6669d65c98-l8xz9" Sep 14 11:10:58.531262 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:58.531227 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e6e0b4a3-6669d65c98-l8xz9"] Sep 14 11:10:59.310451 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:59.310422 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdbc00-67944d958-zfr26" Sep 14 11:10:59.310837 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:59.310488 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fdbc00-64d8fb85bf-5zgk5" Sep 14 11:10:59.372971 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:59.372926 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e6e0b4a3-6669d65c98-l8xz9" event={"ID":"4b0c80e5-881f-45d8-a7f9-77dac57c9d4c","Type":"ContainerStarted","Data":"923981358b5de4da1ec239e347feb196e5f5b03cc9e27ddaa43336814abfea6b"} Sep 14 11:10:59.373164 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:59.372989 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e6e0b4a3-6669d65c98-l8xz9" event={"ID":"4b0c80e5-881f-45d8-a7f9-77dac57c9d4c","Type":"ContainerStarted","Data":"394ecae0e71e0605e43c0ed8361eff5da848bd5bfd339c6481095e7ec79cfbdd"} Sep 14 11:10:59.373164 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:59.373035 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e6e0b4a3-6669d65c98-l8xz9" Sep 14 11:10:59.402613 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:10:59.402554 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-e6e0b4a3-6669d65c98-l8xz9" podStartSLOduration=2.40253645 podStartE2EDuration="2.40253645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:10:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:10:59.400505016 +0000 UTC m=+1333.142906988" watchObservedRunningTime="2026-09-14 11:10:59.40253645 +0000 UTC m=+1333.144938421" Sep 14 11:11:00.922194 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:00.922160 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7984b1/distinct-2-w0-7984b1-kserve-59576ff7c7-zh827"] Sep 14 11:11:00.925920 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:00.925904 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7984b1/distinct-2-w0-7984b1-kserve-59576ff7c7-zh827" Sep 14 11:11:00.928953 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:00.928932 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7984b1\"/\"distinct-2-w0-7984b1-kserve-self-signed-certs\"" Sep 14 11:11:00.934645 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:00.934618 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7984b1/distinct-2-w0-7984b1-kserve-59576ff7c7-zh827"] Sep 14 11:11:00.970403 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:00.970327 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv7rz\" (UniqueName: \"kubernetes.io/projected/6b3d88ab-62c4-4b21-adf7-ab2c8e529b57-kube-api-access-xv7rz\") pod \"distinct-2-w0-7984b1-kserve-59576ff7c7-zh827\" (UID: \"6b3d88ab-62c4-4b21-adf7-ab2c8e529b57\") " pod="e2e-models-e2e-worker-w0-7984b1/distinct-2-w0-7984b1-kserve-59576ff7c7-zh827" Sep 14 11:11:00.970566 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:00.970418 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b3d88ab-62c4-4b21-adf7-ab2c8e529b57-tls-certs\") pod \"distinct-2-w0-7984b1-kserve-59576ff7c7-zh827\" (UID: \"6b3d88ab-62c4-4b21-adf7-ab2c8e529b57\") " pod="e2e-models-e2e-worker-w0-7984b1/distinct-2-w0-7984b1-kserve-59576ff7c7-zh827" Sep 14 11:11:00.970566 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:00.970456 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b3d88ab-62c4-4b21-adf7-ab2c8e529b57-tmp-dir\") pod \"distinct-2-w0-7984b1-kserve-59576ff7c7-zh827\" (UID: \"6b3d88ab-62c4-4b21-adf7-ab2c8e529b57\") " pod="e2e-models-e2e-worker-w0-7984b1/distinct-2-w0-7984b1-kserve-59576ff7c7-zh827" Sep 14 11:11:00.970641 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:00.970567 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b3d88ab-62c4-4b21-adf7-ab2c8e529b57-dshm\") pod \"distinct-2-w0-7984b1-kserve-59576ff7c7-zh827\" (UID: \"6b3d88ab-62c4-4b21-adf7-ab2c8e529b57\") " pod="e2e-models-e2e-worker-w0-7984b1/distinct-2-w0-7984b1-kserve-59576ff7c7-zh827" Sep 14 11:11:00.970641 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:00.970604 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b3d88ab-62c4-4b21-adf7-ab2c8e529b57-model-cache\") pod \"distinct-2-w0-7984b1-kserve-59576ff7c7-zh827\" (UID: \"6b3d88ab-62c4-4b21-adf7-ab2c8e529b57\") " pod="e2e-models-e2e-worker-w0-7984b1/distinct-2-w0-7984b1-kserve-59576ff7c7-zh827" Sep 14 11:11:00.970710 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:00.970667 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b3d88ab-62c4-4b21-adf7-ab2c8e529b57-home\") pod \"distinct-2-w0-7984b1-kserve-59576ff7c7-zh827\" (UID: \"6b3d88ab-62c4-4b21-adf7-ab2c8e529b57\") " pod="e2e-models-e2e-worker-w0-7984b1/distinct-2-w0-7984b1-kserve-59576ff7c7-zh827" Sep 14 11:11:01.071932 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:01.071899 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b3d88ab-62c4-4b21-adf7-ab2c8e529b57-dshm\") pod \"distinct-2-w0-7984b1-kserve-59576ff7c7-zh827\" (UID: \"6b3d88ab-62c4-4b21-adf7-ab2c8e529b57\") " pod="e2e-models-e2e-worker-w0-7984b1/distinct-2-w0-7984b1-kserve-59576ff7c7-zh827" Sep 14 11:11:01.072144 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:01.071944 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b3d88ab-62c4-4b21-adf7-ab2c8e529b57-model-cache\") pod \"distinct-2-w0-7984b1-kserve-59576ff7c7-zh827\" (UID: \"6b3d88ab-62c4-4b21-adf7-ab2c8e529b57\") " pod="e2e-models-e2e-worker-w0-7984b1/distinct-2-w0-7984b1-kserve-59576ff7c7-zh827" Sep 14 11:11:01.072144 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:01.071998 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b3d88ab-62c4-4b21-adf7-ab2c8e529b57-home\") pod \"distinct-2-w0-7984b1-kserve-59576ff7c7-zh827\" (UID: \"6b3d88ab-62c4-4b21-adf7-ab2c8e529b57\") " pod="e2e-models-e2e-worker-w0-7984b1/distinct-2-w0-7984b1-kserve-59576ff7c7-zh827" Sep 14 11:11:01.072144 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:01.072024 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv7rz\" (UniqueName: \"kubernetes.io/projected/6b3d88ab-62c4-4b21-adf7-ab2c8e529b57-kube-api-access-xv7rz\") pod \"distinct-2-w0-7984b1-kserve-59576ff7c7-zh827\" (UID: \"6b3d88ab-62c4-4b21-adf7-ab2c8e529b57\") " pod="e2e-models-e2e-worker-w0-7984b1/distinct-2-w0-7984b1-kserve-59576ff7c7-zh827" Sep 14 11:11:01.072144 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:01.072060 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b3d88ab-62c4-4b21-adf7-ab2c8e529b57-tls-certs\") pod \"distinct-2-w0-7984b1-kserve-59576ff7c7-zh827\" (UID: \"6b3d88ab-62c4-4b21-adf7-ab2c8e529b57\") " pod="e2e-models-e2e-worker-w0-7984b1/distinct-2-w0-7984b1-kserve-59576ff7c7-zh827" Sep 14 11:11:01.072144 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:01.072109 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b3d88ab-62c4-4b21-adf7-ab2c8e529b57-tmp-dir\") pod \"distinct-2-w0-7984b1-kserve-59576ff7c7-zh827\" (UID: \"6b3d88ab-62c4-4b21-adf7-ab2c8e529b57\") " pod="e2e-models-e2e-worker-w0-7984b1/distinct-2-w0-7984b1-kserve-59576ff7c7-zh827" Sep 14 11:11:01.072463 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:01.072433 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b3d88ab-62c4-4b21-adf7-ab2c8e529b57-home\") pod \"distinct-2-w0-7984b1-kserve-59576ff7c7-zh827\" (UID: \"6b3d88ab-62c4-4b21-adf7-ab2c8e529b57\") " pod="e2e-models-e2e-worker-w0-7984b1/distinct-2-w0-7984b1-kserve-59576ff7c7-zh827" Sep 14 11:11:01.072463 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:01.072451 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b3d88ab-62c4-4b21-adf7-ab2c8e529b57-model-cache\") pod \"distinct-2-w0-7984b1-kserve-59576ff7c7-zh827\" (UID: \"6b3d88ab-62c4-4b21-adf7-ab2c8e529b57\") " pod="e2e-models-e2e-worker-w0-7984b1/distinct-2-w0-7984b1-kserve-59576ff7c7-zh827" Sep 14 11:11:01.072563 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:01.072501 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b3d88ab-62c4-4b21-adf7-ab2c8e529b57-tmp-dir\") pod \"distinct-2-w0-7984b1-kserve-59576ff7c7-zh827\" (UID: \"6b3d88ab-62c4-4b21-adf7-ab2c8e529b57\") " pod="e2e-models-e2e-worker-w0-7984b1/distinct-2-w0-7984b1-kserve-59576ff7c7-zh827" Sep 14 11:11:01.074315 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:01.074297 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b3d88ab-62c4-4b21-adf7-ab2c8e529b57-dshm\") pod \"distinct-2-w0-7984b1-kserve-59576ff7c7-zh827\" (UID: \"6b3d88ab-62c4-4b21-adf7-ab2c8e529b57\") " pod="e2e-models-e2e-worker-w0-7984b1/distinct-2-w0-7984b1-kserve-59576ff7c7-zh827" Sep 14 11:11:01.074689 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:01.074671 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b3d88ab-62c4-4b21-adf7-ab2c8e529b57-tls-certs\") pod \"distinct-2-w0-7984b1-kserve-59576ff7c7-zh827\" (UID: \"6b3d88ab-62c4-4b21-adf7-ab2c8e529b57\") " pod="e2e-models-e2e-worker-w0-7984b1/distinct-2-w0-7984b1-kserve-59576ff7c7-zh827" Sep 14 11:11:01.082565 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:01.082542 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv7rz\" (UniqueName: \"kubernetes.io/projected/6b3d88ab-62c4-4b21-adf7-ab2c8e529b57-kube-api-access-xv7rz\") pod \"distinct-2-w0-7984b1-kserve-59576ff7c7-zh827\" (UID: \"6b3d88ab-62c4-4b21-adf7-ab2c8e529b57\") " pod="e2e-models-e2e-worker-w0-7984b1/distinct-2-w0-7984b1-kserve-59576ff7c7-zh827" Sep 14 11:11:01.237158 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:01.237066 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7984b1/distinct-2-w0-7984b1-kserve-59576ff7c7-zh827" Sep 14 11:11:01.519894 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:01.519778 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7984b1/distinct-2-w0-7984b1-kserve-59576ff7c7-zh827"] Sep 14 11:11:01.522812 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:11:01.522782 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b3d88ab_62c4_4b21_adf7_ab2c8e529b57.slice/crio-b3627564c96778ce5afefd3b6c2edfaaa9b34715a1c558defc59717406a4fd76 WatchSource:0}: Error finding container b3627564c96778ce5afefd3b6c2edfaaa9b34715a1c558defc59717406a4fd76: Status 404 returned error can't find the container with id b3627564c96778ce5afefd3b6c2edfaaa9b34715a1c558defc59717406a4fd76 Sep 14 11:11:02.391324 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:02.391287 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7984b1/distinct-2-w0-7984b1-kserve-59576ff7c7-zh827" event={"ID":"6b3d88ab-62c4-4b21-adf7-ab2c8e529b57","Type":"ContainerStarted","Data":"b1854c670eb797d8b7165f723c85dfc690a5e028d842243006f56a1742cfe7ef"} Sep 14 11:11:02.391759 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:02.391363 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7984b1/distinct-2-w0-7984b1-kserve-59576ff7c7-zh827" event={"ID":"6b3d88ab-62c4-4b21-adf7-ab2c8e529b57","Type":"ContainerStarted","Data":"b3627564c96778ce5afefd3b6c2edfaaa9b34715a1c558defc59717406a4fd76"} Sep 14 11:11:02.411469 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:02.411421 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7984b1/distinct-2-w0-7984b1-kserve-59576ff7c7-zh827" podStartSLOduration=2.411404784 podStartE2EDuration="2.411404784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:11:02.410162936 +0000 UTC m=+1336.152564907" watchObservedRunningTime="2026-09-14 11:11:02.411404784 +0000 UTC m=+1336.153806800" Sep 14 11:11:10.380583 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:10.380510 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e6e0b4a3-6669d65c98-l8xz9" Sep 14 11:11:11.237480 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:11.237441 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7984b1/distinct-2-w0-7984b1-kserve-59576ff7c7-zh827" Sep 14 11:11:11.237703 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:11.237515 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7984b1/distinct-2-w0-7984b1-kserve-59576ff7c7-zh827" Sep 14 11:11:11.250310 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:11.250278 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7984b1/distinct-2-w0-7984b1-kserve-59576ff7c7-zh827" Sep 14 11:11:11.456351 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:11.456302 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7984b1/distinct-2-w0-7984b1-kserve-59576ff7c7-zh827" Sep 14 11:11:12.334050 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:12.334016 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0fb86d-6658f598c7-xtzs5"] Sep 14 11:11:12.338001 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:12.337982 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0fb86d-6658f598c7-xtzs5" Sep 14 11:11:12.340624 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:12.340602 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-0fb86d\"" Sep 14 11:11:12.341201 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:12.341181 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-0fb86d-dockercfg-pt98f\"" Sep 14 11:11:12.355849 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:12.355821 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0fb86d-6658f598c7-xtzs5"] Sep 14 11:11:12.393739 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:12.393704 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0fb86d-7c8c54f78-s9p7j"] Sep 14 11:11:12.398365 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:12.398299 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0fb86d-7c8c54f78-s9p7j" Sep 14 11:11:12.411982 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:12.411950 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0fb86d-7c8c54f78-s9p7j"] Sep 14 11:11:12.493635 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:12.493292 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39cb0e24-66a1-474c-9c19-767520c1cf0d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-0fb86d-6658f598c7-xtzs5\" (UID: \"39cb0e24-66a1-474c-9c19-767520c1cf0d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0fb86d-6658f598c7-xtzs5" Sep 14 11:11:12.493635 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:12.493396 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e75f7100-543f-44de-b464-ca5487d50b5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-0fb86d-7c8c54f78-s9p7j\" (UID: \"e75f7100-543f-44de-b464-ca5487d50b5a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0fb86d-7c8c54f78-s9p7j" Sep 14 11:11:12.493635 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:12.493451 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbw5t\" (UniqueName: \"kubernetes.io/projected/39cb0e24-66a1-474c-9c19-767520c1cf0d-kube-api-access-jbw5t\") pod \"payload-processing-e2e-shared-b-w3-0fb86d-6658f598c7-xtzs5\" (UID: \"39cb0e24-66a1-474c-9c19-767520c1cf0d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0fb86d-6658f598c7-xtzs5" Sep 14 11:11:12.493635 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:12.493512 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrgjp\" (UniqueName: \"kubernetes.io/projected/e75f7100-543f-44de-b464-ca5487d50b5a-kube-api-access-qrgjp\") pod \"payload-pre-processing-e2e-shared-b-w3-0fb86d-7c8c54f78-s9p7j\" (UID: \"e75f7100-543f-44de-b464-ca5487d50b5a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0fb86d-7c8c54f78-s9p7j" Sep 14 11:11:12.594876 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:12.594787 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39cb0e24-66a1-474c-9c19-767520c1cf0d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-0fb86d-6658f598c7-xtzs5\" (UID: \"39cb0e24-66a1-474c-9c19-767520c1cf0d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0fb86d-6658f598c7-xtzs5" Sep 14 11:11:12.595021 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:12.594954 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e75f7100-543f-44de-b464-ca5487d50b5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-0fb86d-7c8c54f78-s9p7j\" (UID: \"e75f7100-543f-44de-b464-ca5487d50b5a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0fb86d-7c8c54f78-s9p7j" Sep 14 11:11:12.595089 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:12.595021 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbw5t\" (UniqueName: \"kubernetes.io/projected/39cb0e24-66a1-474c-9c19-767520c1cf0d-kube-api-access-jbw5t\") pod \"payload-processing-e2e-shared-b-w3-0fb86d-6658f598c7-xtzs5\" (UID: \"39cb0e24-66a1-474c-9c19-767520c1cf0d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0fb86d-6658f598c7-xtzs5" Sep 14 11:11:12.595148 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:12.595106 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrgjp\" (UniqueName: \"kubernetes.io/projected/e75f7100-543f-44de-b464-ca5487d50b5a-kube-api-access-qrgjp\") pod \"payload-pre-processing-e2e-shared-b-w3-0fb86d-7c8c54f78-s9p7j\" (UID: \"e75f7100-543f-44de-b464-ca5487d50b5a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0fb86d-7c8c54f78-s9p7j" Sep 14 11:11:12.595630 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:12.595611 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39cb0e24-66a1-474c-9c19-767520c1cf0d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-0fb86d-6658f598c7-xtzs5\" (UID: \"39cb0e24-66a1-474c-9c19-767520c1cf0d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0fb86d-6658f598c7-xtzs5" Sep 14 11:11:12.595708 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:12.595642 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e75f7100-543f-44de-b464-ca5487d50b5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-0fb86d-7c8c54f78-s9p7j\" (UID: \"e75f7100-543f-44de-b464-ca5487d50b5a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0fb86d-7c8c54f78-s9p7j" Sep 14 11:11:12.607851 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:12.607823 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbw5t\" (UniqueName: \"kubernetes.io/projected/39cb0e24-66a1-474c-9c19-767520c1cf0d-kube-api-access-jbw5t\") pod \"payload-processing-e2e-shared-b-w3-0fb86d-6658f598c7-xtzs5\" (UID: \"39cb0e24-66a1-474c-9c19-767520c1cf0d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0fb86d-6658f598c7-xtzs5" Sep 14 11:11:12.612149 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:12.612121 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrgjp\" (UniqueName: \"kubernetes.io/projected/e75f7100-543f-44de-b464-ca5487d50b5a-kube-api-access-qrgjp\") pod \"payload-pre-processing-e2e-shared-b-w3-0fb86d-7c8c54f78-s9p7j\" (UID: \"e75f7100-543f-44de-b464-ca5487d50b5a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0fb86d-7c8c54f78-s9p7j" Sep 14 11:11:12.651518 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:12.651480 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0fb86d-6658f598c7-xtzs5" Sep 14 11:11:12.712899 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:12.712569 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0fb86d-7c8c54f78-s9p7j" Sep 14 11:11:12.928056 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:12.928031 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0fb86d-6658f598c7-xtzs5"] Sep 14 11:11:12.930582 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:11:12.930560 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39cb0e24_66a1_474c_9c19_767520c1cf0d.slice/crio-ba142a3c5afcea804d68798a4c05deedceb13e89575066086a7e79345154ffc6 WatchSource:0}: Error finding container ba142a3c5afcea804d68798a4c05deedceb13e89575066086a7e79345154ffc6: Status 404 returned error can't find the container with id ba142a3c5afcea804d68798a4c05deedceb13e89575066086a7e79345154ffc6 Sep 14 11:11:12.998489 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:12.998463 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0fb86d-7c8c54f78-s9p7j"] Sep 14 11:11:13.000076 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:11:13.000019 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode75f7100_543f_44de_b464_ca5487d50b5a.slice/crio-5e90427d491e275fa20fe100d16a5e086ecc87063aacc33cc8bb69dc6ee4e018 WatchSource:0}: Error finding container 5e90427d491e275fa20fe100d16a5e086ecc87063aacc33cc8bb69dc6ee4e018: Status 404 returned error can't find the container with id 5e90427d491e275fa20fe100d16a5e086ecc87063aacc33cc8bb69dc6ee4e018 Sep 14 11:11:13.460472 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:13.460439 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0fb86d-7c8c54f78-s9p7j" event={"ID":"e75f7100-543f-44de-b464-ca5487d50b5a","Type":"ContainerStarted","Data":"b2bc4f40fdfbd6b3b20680fec7b0447e43c92789dfc1528c95a925ce1715b3ee"} Sep 14 11:11:13.460663 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:13.460497 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0fb86d-7c8c54f78-s9p7j" event={"ID":"e75f7100-543f-44de-b464-ca5487d50b5a","Type":"ContainerStarted","Data":"5e90427d491e275fa20fe100d16a5e086ecc87063aacc33cc8bb69dc6ee4e018"} Sep 14 11:11:13.460718 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:13.460656 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0fb86d-7c8c54f78-s9p7j" Sep 14 11:11:13.462172 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:13.462154 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0fb86d-6658f598c7-xtzs5" event={"ID":"39cb0e24-66a1-474c-9c19-767520c1cf0d","Type":"ContainerStarted","Data":"a76408f7184a2b4d0a6cc92d047999260e4cb855aa9c74a52960a967051a42c5"} Sep 14 11:11:13.462260 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:13.462192 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0fb86d-6658f598c7-xtzs5" event={"ID":"39cb0e24-66a1-474c-9c19-767520c1cf0d","Type":"ContainerStarted","Data":"ba142a3c5afcea804d68798a4c05deedceb13e89575066086a7e79345154ffc6"} Sep 14 11:11:13.477760 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:13.477701 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0fb86d-7c8c54f78-s9p7j" podStartSLOduration=1.477687342 podStartE2EDuration="1.477687342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:11:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:11:13.476769432 +0000 UTC m=+1347.219171445" watchObservedRunningTime="2026-09-14 11:11:13.477687342 +0000 UTC m=+1347.220089311" Sep 14 11:11:13.495639 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:13.495587 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0fb86d-6658f598c7-xtzs5" podStartSLOduration=1.495567327 podStartE2EDuration="1.495567327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:11:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:11:13.494005011 +0000 UTC m=+1347.236406981" watchObservedRunningTime="2026-09-14 11:11:13.495567327 +0000 UTC m=+1347.237969297" Sep 14 11:11:14.467796 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:14.467767 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0fb86d-6658f598c7-xtzs5" Sep 14 11:11:24.469900 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:24.469865 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0fb86d-7c8c54f78-s9p7j" Sep 14 11:11:25.476473 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:25.476444 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0fb86d-6658f598c7-xtzs5" Sep 14 11:11:46.908994 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:11:46.908965 2581 scope.go:117] "RemoveContainer" containerID="b6a0395114b6deb8d95eaae3f60df62375f24d7e422e4b6d73ede821f2664674" Sep 14 11:12:14.344555 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:14.344518 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-fdbc00/auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w"] Sep 14 11:12:14.349798 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:14.349766 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fdbc00/auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w" Sep 14 11:12:14.352849 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:14.352826 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-fdbc00\"/\"kube-root-ca.crt\"" Sep 14 11:12:14.352985 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:14.352864 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-fdbc00\"/\"auth-test-model-fdbc00-kserve-self-signed-certs\"" Sep 14 11:12:14.353633 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:14.353616 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-fdbc00\"/\"openshift-service-ca.crt\"" Sep 14 11:12:14.353862 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:14.353847 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-fdbc00\"/\"default-dockercfg-xwbwm\"" Sep 14 11:12:14.359712 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:14.359692 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fdbc00/auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w"] Sep 14 11:12:14.449887 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:14.449852 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a22802be-62a1-4055-b7e2-46a6c25a2bd2-home\") pod \"auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w\" (UID: \"a22802be-62a1-4055-b7e2-46a6c25a2bd2\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w" Sep 14 11:12:14.450056 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:14.449926 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a22802be-62a1-4055-b7e2-46a6c25a2bd2-dshm\") pod \"auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w\" (UID: \"a22802be-62a1-4055-b7e2-46a6c25a2bd2\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w" Sep 14 11:12:14.450056 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:14.449994 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a22802be-62a1-4055-b7e2-46a6c25a2bd2-tmp-dir\") pod \"auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w\" (UID: \"a22802be-62a1-4055-b7e2-46a6c25a2bd2\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w" Sep 14 11:12:14.450056 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:14.450018 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a22802be-62a1-4055-b7e2-46a6c25a2bd2-tls-certs\") pod \"auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w\" (UID: \"a22802be-62a1-4055-b7e2-46a6c25a2bd2\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w" Sep 14 11:12:14.450160 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:14.450065 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n846x\" (UniqueName: \"kubernetes.io/projected/a22802be-62a1-4055-b7e2-46a6c25a2bd2-kube-api-access-n846x\") pod \"auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w\" (UID: \"a22802be-62a1-4055-b7e2-46a6c25a2bd2\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w" Sep 14 11:12:14.450160 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:14.450108 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a22802be-62a1-4055-b7e2-46a6c25a2bd2-model-cache\") pod \"auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w\" (UID: \"a22802be-62a1-4055-b7e2-46a6c25a2bd2\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w" Sep 14 11:12:14.551014 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:14.550981 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n846x\" (UniqueName: \"kubernetes.io/projected/a22802be-62a1-4055-b7e2-46a6c25a2bd2-kube-api-access-n846x\") pod \"auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w\" (UID: \"a22802be-62a1-4055-b7e2-46a6c25a2bd2\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w" Sep 14 11:12:14.551216 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:14.551029 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a22802be-62a1-4055-b7e2-46a6c25a2bd2-model-cache\") pod \"auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w\" (UID: \"a22802be-62a1-4055-b7e2-46a6c25a2bd2\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w" Sep 14 11:12:14.551216 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:14.551072 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a22802be-62a1-4055-b7e2-46a6c25a2bd2-home\") pod \"auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w\" (UID: \"a22802be-62a1-4055-b7e2-46a6c25a2bd2\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w" Sep 14 11:12:14.551216 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:14.551126 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a22802be-62a1-4055-b7e2-46a6c25a2bd2-dshm\") pod \"auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w\" (UID: \"a22802be-62a1-4055-b7e2-46a6c25a2bd2\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w" Sep 14 11:12:14.551216 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:14.551160 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a22802be-62a1-4055-b7e2-46a6c25a2bd2-tmp-dir\") pod \"auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w\" (UID: \"a22802be-62a1-4055-b7e2-46a6c25a2bd2\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w" Sep 14 11:12:14.551477 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:14.551276 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a22802be-62a1-4055-b7e2-46a6c25a2bd2-tls-certs\") pod \"auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w\" (UID: \"a22802be-62a1-4055-b7e2-46a6c25a2bd2\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w" Sep 14 11:12:14.551528 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:14.551487 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a22802be-62a1-4055-b7e2-46a6c25a2bd2-home\") pod \"auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w\" (UID: \"a22802be-62a1-4055-b7e2-46a6c25a2bd2\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w" Sep 14 11:12:14.551634 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:14.551614 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a22802be-62a1-4055-b7e2-46a6c25a2bd2-tmp-dir\") pod \"auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w\" (UID: \"a22802be-62a1-4055-b7e2-46a6c25a2bd2\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w" Sep 14 11:12:14.551678 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:14.551639 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a22802be-62a1-4055-b7e2-46a6c25a2bd2-model-cache\") pod \"auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w\" (UID: \"a22802be-62a1-4055-b7e2-46a6c25a2bd2\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w" Sep 14 11:12:14.553661 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:14.553642 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a22802be-62a1-4055-b7e2-46a6c25a2bd2-dshm\") pod \"auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w\" (UID: \"a22802be-62a1-4055-b7e2-46a6c25a2bd2\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w" Sep 14 11:12:14.553766 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:14.553750 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a22802be-62a1-4055-b7e2-46a6c25a2bd2-tls-certs\") pod \"auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w\" (UID: \"a22802be-62a1-4055-b7e2-46a6c25a2bd2\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w" Sep 14 11:12:14.561415 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:14.561390 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n846x\" (UniqueName: \"kubernetes.io/projected/a22802be-62a1-4055-b7e2-46a6c25a2bd2-kube-api-access-n846x\") pod \"auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w\" (UID: \"a22802be-62a1-4055-b7e2-46a6c25a2bd2\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w" Sep 14 11:12:14.660285 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:14.660194 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fdbc00/auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w" Sep 14 11:12:14.937597 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:14.937564 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fdbc00/auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w"] Sep 14 11:12:14.939193 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:12:14.939166 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda22802be_62a1_4055_b7e2_46a6c25a2bd2.slice/crio-c0c46be97e9f2ddccb16af88c8342563690ef2cc4d1fcd307c887dd2d3752e54 WatchSource:0}: Error finding container c0c46be97e9f2ddccb16af88c8342563690ef2cc4d1fcd307c887dd2d3752e54: Status 404 returned error can't find the container with id c0c46be97e9f2ddccb16af88c8342563690ef2cc4d1fcd307c887dd2d3752e54 Sep 14 11:12:15.834732 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:15.834695 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fdbc00/auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w" event={"ID":"a22802be-62a1-4055-b7e2-46a6c25a2bd2","Type":"ContainerStarted","Data":"49ef8ac2aca4567f7052e260171692a109d357c37a62c6cda6bb702d9e6123ed"} Sep 14 11:12:15.834732 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:15.834736 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fdbc00/auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w" event={"ID":"a22802be-62a1-4055-b7e2-46a6c25a2bd2","Type":"ContainerStarted","Data":"c0c46be97e9f2ddccb16af88c8342563690ef2cc4d1fcd307c887dd2d3752e54"} Sep 14 11:12:15.911072 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:15.911016 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-fdbc00/auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w" podStartSLOduration=1.910995535 podStartE2EDuration="1.910995535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:12:15.906798251 +0000 UTC m=+1409.649200221" watchObservedRunningTime="2026-09-14 11:12:15.910995535 +0000 UTC m=+1409.653397505" Sep 14 11:12:22.864334 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:22.864303 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4ebf73/distinct-2-w4-4ebf73-kserve-7584f57865-l9888"] Sep 14 11:12:22.868469 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:22.868447 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ebf73/distinct-2-w4-4ebf73-kserve-7584f57865-l9888" Sep 14 11:12:22.871481 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:22.871457 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4ebf73\"/\"distinct-2-w4-4ebf73-kserve-self-signed-certs\"" Sep 14 11:12:22.888119 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:22.888092 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4ebf73/distinct-2-w4-4ebf73-kserve-7584f57865-l9888"] Sep 14 11:12:23.041070 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:23.041031 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75d16066-49c1-4a32-b80b-ca4634eb0fb2-tls-certs\") pod \"distinct-2-w4-4ebf73-kserve-7584f57865-l9888\" (UID: \"75d16066-49c1-4a32-b80b-ca4634eb0fb2\") " pod="e2e-models-e2e-worker-w4-4ebf73/distinct-2-w4-4ebf73-kserve-7584f57865-l9888" Sep 14 11:12:23.041070 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:23.041089 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75d16066-49c1-4a32-b80b-ca4634eb0fb2-tmp-dir\") pod \"distinct-2-w4-4ebf73-kserve-7584f57865-l9888\" (UID: \"75d16066-49c1-4a32-b80b-ca4634eb0fb2\") " pod="e2e-models-e2e-worker-w4-4ebf73/distinct-2-w4-4ebf73-kserve-7584f57865-l9888" Sep 14 11:12:23.041401 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:23.041177 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75d16066-49c1-4a32-b80b-ca4634eb0fb2-model-cache\") pod \"distinct-2-w4-4ebf73-kserve-7584f57865-l9888\" (UID: \"75d16066-49c1-4a32-b80b-ca4634eb0fb2\") " pod="e2e-models-e2e-worker-w4-4ebf73/distinct-2-w4-4ebf73-kserve-7584f57865-l9888" Sep 14 11:12:23.041401 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:23.041247 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75d16066-49c1-4a32-b80b-ca4634eb0fb2-dshm\") pod \"distinct-2-w4-4ebf73-kserve-7584f57865-l9888\" (UID: \"75d16066-49c1-4a32-b80b-ca4634eb0fb2\") " pod="e2e-models-e2e-worker-w4-4ebf73/distinct-2-w4-4ebf73-kserve-7584f57865-l9888" Sep 14 11:12:23.041504 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:23.041416 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlpzw\" (UniqueName: \"kubernetes.io/projected/75d16066-49c1-4a32-b80b-ca4634eb0fb2-kube-api-access-rlpzw\") pod \"distinct-2-w4-4ebf73-kserve-7584f57865-l9888\" (UID: \"75d16066-49c1-4a32-b80b-ca4634eb0fb2\") " pod="e2e-models-e2e-worker-w4-4ebf73/distinct-2-w4-4ebf73-kserve-7584f57865-l9888" Sep 14 11:12:23.041504 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:23.041451 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75d16066-49c1-4a32-b80b-ca4634eb0fb2-home\") pod \"distinct-2-w4-4ebf73-kserve-7584f57865-l9888\" (UID: \"75d16066-49c1-4a32-b80b-ca4634eb0fb2\") " pod="e2e-models-e2e-worker-w4-4ebf73/distinct-2-w4-4ebf73-kserve-7584f57865-l9888" Sep 14 11:12:23.142507 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:23.142413 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75d16066-49c1-4a32-b80b-ca4634eb0fb2-model-cache\") pod \"distinct-2-w4-4ebf73-kserve-7584f57865-l9888\" (UID: \"75d16066-49c1-4a32-b80b-ca4634eb0fb2\") " pod="e2e-models-e2e-worker-w4-4ebf73/distinct-2-w4-4ebf73-kserve-7584f57865-l9888" Sep 14 11:12:23.142507 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:23.142473 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75d16066-49c1-4a32-b80b-ca4634eb0fb2-dshm\") pod \"distinct-2-w4-4ebf73-kserve-7584f57865-l9888\" (UID: \"75d16066-49c1-4a32-b80b-ca4634eb0fb2\") " pod="e2e-models-e2e-worker-w4-4ebf73/distinct-2-w4-4ebf73-kserve-7584f57865-l9888" Sep 14 11:12:23.142741 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:23.142544 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlpzw\" (UniqueName: \"kubernetes.io/projected/75d16066-49c1-4a32-b80b-ca4634eb0fb2-kube-api-access-rlpzw\") pod \"distinct-2-w4-4ebf73-kserve-7584f57865-l9888\" (UID: \"75d16066-49c1-4a32-b80b-ca4634eb0fb2\") " pod="e2e-models-e2e-worker-w4-4ebf73/distinct-2-w4-4ebf73-kserve-7584f57865-l9888" Sep 14 11:12:23.142741 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:23.142568 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75d16066-49c1-4a32-b80b-ca4634eb0fb2-home\") pod \"distinct-2-w4-4ebf73-kserve-7584f57865-l9888\" (UID: \"75d16066-49c1-4a32-b80b-ca4634eb0fb2\") " pod="e2e-models-e2e-worker-w4-4ebf73/distinct-2-w4-4ebf73-kserve-7584f57865-l9888" Sep 14 11:12:23.142741 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:23.142607 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75d16066-49c1-4a32-b80b-ca4634eb0fb2-tls-certs\") pod \"distinct-2-w4-4ebf73-kserve-7584f57865-l9888\" (UID: \"75d16066-49c1-4a32-b80b-ca4634eb0fb2\") " pod="e2e-models-e2e-worker-w4-4ebf73/distinct-2-w4-4ebf73-kserve-7584f57865-l9888" Sep 14 11:12:23.142741 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:23.142638 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75d16066-49c1-4a32-b80b-ca4634eb0fb2-tmp-dir\") pod \"distinct-2-w4-4ebf73-kserve-7584f57865-l9888\" (UID: \"75d16066-49c1-4a32-b80b-ca4634eb0fb2\") " pod="e2e-models-e2e-worker-w4-4ebf73/distinct-2-w4-4ebf73-kserve-7584f57865-l9888" Sep 14 11:12:23.143056 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:23.142848 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75d16066-49c1-4a32-b80b-ca4634eb0fb2-model-cache\") pod \"distinct-2-w4-4ebf73-kserve-7584f57865-l9888\" (UID: \"75d16066-49c1-4a32-b80b-ca4634eb0fb2\") " pod="e2e-models-e2e-worker-w4-4ebf73/distinct-2-w4-4ebf73-kserve-7584f57865-l9888" Sep 14 11:12:23.143056 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:23.142975 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75d16066-49c1-4a32-b80b-ca4634eb0fb2-tmp-dir\") pod \"distinct-2-w4-4ebf73-kserve-7584f57865-l9888\" (UID: \"75d16066-49c1-4a32-b80b-ca4634eb0fb2\") " pod="e2e-models-e2e-worker-w4-4ebf73/distinct-2-w4-4ebf73-kserve-7584f57865-l9888" Sep 14 11:12:23.143056 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:23.143036 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75d16066-49c1-4a32-b80b-ca4634eb0fb2-home\") pod \"distinct-2-w4-4ebf73-kserve-7584f57865-l9888\" (UID: \"75d16066-49c1-4a32-b80b-ca4634eb0fb2\") " pod="e2e-models-e2e-worker-w4-4ebf73/distinct-2-w4-4ebf73-kserve-7584f57865-l9888" Sep 14 11:12:23.144935 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:23.144913 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75d16066-49c1-4a32-b80b-ca4634eb0fb2-dshm\") pod \"distinct-2-w4-4ebf73-kserve-7584f57865-l9888\" (UID: \"75d16066-49c1-4a32-b80b-ca4634eb0fb2\") " pod="e2e-models-e2e-worker-w4-4ebf73/distinct-2-w4-4ebf73-kserve-7584f57865-l9888" Sep 14 11:12:23.145175 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:23.145156 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75d16066-49c1-4a32-b80b-ca4634eb0fb2-tls-certs\") pod \"distinct-2-w4-4ebf73-kserve-7584f57865-l9888\" (UID: \"75d16066-49c1-4a32-b80b-ca4634eb0fb2\") " pod="e2e-models-e2e-worker-w4-4ebf73/distinct-2-w4-4ebf73-kserve-7584f57865-l9888" Sep 14 11:12:23.153851 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:23.153824 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlpzw\" (UniqueName: \"kubernetes.io/projected/75d16066-49c1-4a32-b80b-ca4634eb0fb2-kube-api-access-rlpzw\") pod \"distinct-2-w4-4ebf73-kserve-7584f57865-l9888\" (UID: \"75d16066-49c1-4a32-b80b-ca4634eb0fb2\") " pod="e2e-models-e2e-worker-w4-4ebf73/distinct-2-w4-4ebf73-kserve-7584f57865-l9888" Sep 14 11:12:23.178783 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:23.178751 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ebf73/distinct-2-w4-4ebf73-kserve-7584f57865-l9888" Sep 14 11:12:23.459492 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:23.459464 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4ebf73/distinct-2-w4-4ebf73-kserve-7584f57865-l9888"] Sep 14 11:12:23.460585 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:12:23.460561 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75d16066_49c1_4a32_b80b_ca4634eb0fb2.slice/crio-e4707521ece3bd4019204362979dadc39a23ffc535f66f6f3ab0869861a4a0a4 WatchSource:0}: Error finding container e4707521ece3bd4019204362979dadc39a23ffc535f66f6f3ab0869861a4a0a4: Status 404 returned error can't find the container with id e4707521ece3bd4019204362979dadc39a23ffc535f66f6f3ab0869861a4a0a4 Sep 14 11:12:23.880494 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:23.880459 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ebf73/distinct-2-w4-4ebf73-kserve-7584f57865-l9888" event={"ID":"75d16066-49c1-4a32-b80b-ca4634eb0fb2","Type":"ContainerStarted","Data":"22454accd77a6aa481849496786062adc3bdd1903f00da80cf92b8dd67ad1f5d"} Sep 14 11:12:23.880494 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:23.880499 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ebf73/distinct-2-w4-4ebf73-kserve-7584f57865-l9888" event={"ID":"75d16066-49c1-4a32-b80b-ca4634eb0fb2","Type":"ContainerStarted","Data":"e4707521ece3bd4019204362979dadc39a23ffc535f66f6f3ab0869861a4a0a4"} Sep 14 11:12:23.903030 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:23.902974 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4ebf73/distinct-2-w4-4ebf73-kserve-7584f57865-l9888" podStartSLOduration=1.902953804 podStartE2EDuration="1.902953804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:12:23.902421069 +0000 UTC m=+1417.644823040" watchObservedRunningTime="2026-09-14 11:12:23.902953804 +0000 UTC m=+1417.645355777" Sep 14 11:12:24.663425 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:24.663380 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-fdbc00/auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w" Sep 14 11:12:24.663646 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:24.663460 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-fdbc00/auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w" Sep 14 11:12:24.676887 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:24.676859 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-fdbc00/auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w" Sep 14 11:12:24.897834 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:24.897804 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-fdbc00/auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w" Sep 14 11:12:33.178912 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:33.178870 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ebf73/distinct-2-w4-4ebf73-kserve-7584f57865-l9888" Sep 14 11:12:33.179322 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:33.178928 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4ebf73/distinct-2-w4-4ebf73-kserve-7584f57865-l9888" Sep 14 11:12:33.191456 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:33.191428 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4ebf73/distinct-2-w4-4ebf73-kserve-7584f57865-l9888" Sep 14 11:12:33.947287 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:33.947249 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ebf73/distinct-2-w4-4ebf73-kserve-7584f57865-l9888" Sep 14 11:12:46.931799 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:46.931723 2581 scope.go:117] "RemoveContainer" containerID="caeda90321ee9230cf29a52cff9a136a3a1c6dd689a2bd98f5c77ef170fc4dc5" Sep 14 11:12:46.940633 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:46.940611 2581 scope.go:117] "RemoveContainer" containerID="fba0b43155baae8e2a31d3d1ca2b214cafe853ffe3154cc5a6b92433db06689d" Sep 14 11:12:46.949130 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:46.949107 2581 scope.go:117] "RemoveContainer" containerID="e4bf9ab2502da1272f15047302a28908e59cf1cd525c55199552d705e1019f9f" Sep 14 11:12:58.852196 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:58.852163 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6e0b4a3-rfwp4"] Sep 14 11:12:58.857321 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:58.857302 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6e0b4a3-rfwp4" Sep 14 11:12:58.860983 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:58.860966 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pv9g8\"" Sep 14 11:12:58.868633 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:58.868604 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6e0b4a3-rfwp4"] Sep 14 11:12:58.917091 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:58.917059 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc965586-a194-4c90-819d-b2a9706ace78-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e6e0b4a3-rfwp4\" (UID: \"fc965586-a194-4c90-819d-b2a9706ace78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6e0b4a3-rfwp4" Sep 14 11:12:59.017761 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:59.017718 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc965586-a194-4c90-819d-b2a9706ace78-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e6e0b4a3-rfwp4\" (UID: \"fc965586-a194-4c90-819d-b2a9706ace78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6e0b4a3-rfwp4" Sep 14 11:12:59.018354 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:59.018319 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc965586-a194-4c90-819d-b2a9706ace78-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e6e0b4a3-rfwp4\" (UID: \"fc965586-a194-4c90-819d-b2a9706ace78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6e0b4a3-rfwp4" Sep 14 11:12:59.168884 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:59.168788 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6e0b4a3-rfwp4" Sep 14 11:12:59.448015 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:12:59.447937 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6e0b4a3-rfwp4"] Sep 14 11:12:59.451538 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:12:59.451510 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc965586_a194_4c90_819d_b2a9706ace78.slice/crio-084861ae8d20c0f17b56811e770a7de713f76f43ab8d2176f6f87beeb118da8e WatchSource:0}: Error finding container 084861ae8d20c0f17b56811e770a7de713f76f43ab8d2176f6f87beeb118da8e: Status 404 returned error can't find the container with id 084861ae8d20c0f17b56811e770a7de713f76f43ab8d2176f6f87beeb118da8e Sep 14 11:13:00.088324 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:13:00.088275 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6e0b4a3-rfwp4" event={"ID":"fc965586-a194-4c90-819d-b2a9706ace78","Type":"ContainerStarted","Data":"816c17b6ab3e26d79360976fcbe200fa3117a3900c66ade0849bd589e2b2ac72"} Sep 14 11:13:00.088324 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:13:00.088351 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6e0b4a3-rfwp4" event={"ID":"fc965586-a194-4c90-819d-b2a9706ace78","Type":"ContainerStarted","Data":"084861ae8d20c0f17b56811e770a7de713f76f43ab8d2176f6f87beeb118da8e"} Sep 14 11:13:00.107856 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:13:00.107798 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6e0b4a3-rfwp4" podStartSLOduration=2.107781157 podStartE2EDuration="2.107781157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:12:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:13:00.106757188 +0000 UTC m=+1453.849159158" watchObservedRunningTime="2026-09-14 11:13:00.107781157 +0000 UTC m=+1453.850183127" Sep 14 11:13:01.095469 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:13:01.095429 2581 generic.go:356] "Generic (PLEG): container finished" podID="fc965586-a194-4c90-819d-b2a9706ace78" containerID="816c17b6ab3e26d79360976fcbe200fa3117a3900c66ade0849bd589e2b2ac72" exitCode=0 Sep 14 11:13:01.095839 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:13:01.095523 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6e0b4a3-rfwp4" event={"ID":"fc965586-a194-4c90-819d-b2a9706ace78","Type":"ContainerDied","Data":"816c17b6ab3e26d79360976fcbe200fa3117a3900c66ade0849bd589e2b2ac72"} Sep 14 11:13:02.236610 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:13:02.236586 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6e0b4a3-rfwp4" Sep 14 11:13:02.245479 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:13:02.245451 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc965586-a194-4c90-819d-b2a9706ace78-maas-api-service-ca\") pod \"fc965586-a194-4c90-819d-b2a9706ace78\" (UID: \"fc965586-a194-4c90-819d-b2a9706ace78\") " Sep 14 11:13:02.245865 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:13:02.245840 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc965586-a194-4c90-819d-b2a9706ace78-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fc965586-a194-4c90-819d-b2a9706ace78" (UID: "fc965586-a194-4c90-819d-b2a9706ace78"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:13:02.347214 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:13:02.347177 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc965586-a194-4c90-819d-b2a9706ace78-maas-api-service-ca\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:13:03.108220 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:13:03.108132 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6e0b4a3-rfwp4" Sep 14 11:13:03.108220 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:13:03.108168 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6e0b4a3-rfwp4" event={"ID":"fc965586-a194-4c90-819d-b2a9706ace78","Type":"ContainerDied","Data":"084861ae8d20c0f17b56811e770a7de713f76f43ab8d2176f6f87beeb118da8e"} Sep 14 11:13:03.108220 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:13:03.108211 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="084861ae8d20c0f17b56811e770a7de713f76f43ab8d2176f6f87beeb118da8e" Sep 14 11:13:14.496772 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:13:14.496737 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e6e0b4a3-6669d65c98-l8xz9"] Sep 14 11:13:14.497224 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:13:14.496988 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-e6e0b4a3-6669d65c98-l8xz9" podUID="4b0c80e5-881f-45d8-a7f9-77dac57c9d4c" containerName="payload-pre-processing" containerID="cri-o://923981358b5de4da1ec239e347feb196e5f5b03cc9e27ddaa43336814abfea6b" gracePeriod=30 Sep 14 11:13:14.751477 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:13:14.751408 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e6e0b4a3-6669d65c98-l8xz9" Sep 14 11:13:14.767938 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:13:14.767904 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6e0b4a3-rfwp4"] Sep 14 11:13:14.769963 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:13:14.769941 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e6e0b4a3-rfwp4"] Sep 14 11:13:14.832993 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:13:14.832957 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc965586-a194-4c90-819d-b2a9706ace78" path="/var/lib/kubelet/pods/fc965586-a194-4c90-819d-b2a9706ace78/volumes" Sep 14 11:13:14.873451 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:13:14.873422 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b0c80e5-881f-45d8-a7f9-77dac57c9d4c-plugins-config-volume\") pod \"4b0c80e5-881f-45d8-a7f9-77dac57c9d4c\" (UID: \"4b0c80e5-881f-45d8-a7f9-77dac57c9d4c\") " Sep 14 11:13:14.873632 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:13:14.873494 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfdzl\" (UniqueName: \"kubernetes.io/projected/4b0c80e5-881f-45d8-a7f9-77dac57c9d4c-kube-api-access-rfdzl\") pod \"4b0c80e5-881f-45d8-a7f9-77dac57c9d4c\" (UID: \"4b0c80e5-881f-45d8-a7f9-77dac57c9d4c\") " Sep 14 11:13:14.873906 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:13:14.873877 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b0c80e5-881f-45d8-a7f9-77dac57c9d4c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4b0c80e5-881f-45d8-a7f9-77dac57c9d4c" (UID: "4b0c80e5-881f-45d8-a7f9-77dac57c9d4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:13:14.875764 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:13:14.875740 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b0c80e5-881f-45d8-a7f9-77dac57c9d4c-kube-api-access-rfdzl" (OuterVolumeSpecName: "kube-api-access-rfdzl") pod "4b0c80e5-881f-45d8-a7f9-77dac57c9d4c" (UID: "4b0c80e5-881f-45d8-a7f9-77dac57c9d4c"). InnerVolumeSpecName "kube-api-access-rfdzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:13:14.974958 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:13:14.974919 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfdzl\" (UniqueName: \"kubernetes.io/projected/4b0c80e5-881f-45d8-a7f9-77dac57c9d4c-kube-api-access-rfdzl\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:13:14.974958 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:13:14.974947 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b0c80e5-881f-45d8-a7f9-77dac57c9d4c-plugins-config-volume\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:13:15.177826 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:13:15.177789 2581 generic.go:356] "Generic (PLEG): container finished" podID="4b0c80e5-881f-45d8-a7f9-77dac57c9d4c" containerID="923981358b5de4da1ec239e347feb196e5f5b03cc9e27ddaa43336814abfea6b" exitCode=0 Sep 14 11:13:15.178000 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:13:15.177837 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e6e0b4a3-6669d65c98-l8xz9" Sep 14 11:13:15.178000 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:13:15.177860 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e6e0b4a3-6669d65c98-l8xz9" event={"ID":"4b0c80e5-881f-45d8-a7f9-77dac57c9d4c","Type":"ContainerDied","Data":"923981358b5de4da1ec239e347feb196e5f5b03cc9e27ddaa43336814abfea6b"} Sep 14 11:13:15.178000 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:13:15.177914 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e6e0b4a3-6669d65c98-l8xz9" event={"ID":"4b0c80e5-881f-45d8-a7f9-77dac57c9d4c","Type":"ContainerDied","Data":"394ecae0e71e0605e43c0ed8361eff5da848bd5bfd339c6481095e7ec79cfbdd"} Sep 14 11:13:15.178000 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:13:15.177940 2581 scope.go:117] "RemoveContainer" containerID="923981358b5de4da1ec239e347feb196e5f5b03cc9e27ddaa43336814abfea6b" Sep 14 11:13:15.187182 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:13:15.187159 2581 scope.go:117] "RemoveContainer" containerID="923981358b5de4da1ec239e347feb196e5f5b03cc9e27ddaa43336814abfea6b" Sep 14 11:13:15.187529 ip-10-0-137-53 kubenswrapper[2581]: E0914 11:13:15.187508 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"923981358b5de4da1ec239e347feb196e5f5b03cc9e27ddaa43336814abfea6b\": container with ID starting with 923981358b5de4da1ec239e347feb196e5f5b03cc9e27ddaa43336814abfea6b not found: ID does not exist" containerID="923981358b5de4da1ec239e347feb196e5f5b03cc9e27ddaa43336814abfea6b" Sep 14 11:13:15.187616 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:13:15.187536 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"923981358b5de4da1ec239e347feb196e5f5b03cc9e27ddaa43336814abfea6b"} err="failed to get container status \"923981358b5de4da1ec239e347feb196e5f5b03cc9e27ddaa43336814abfea6b\": rpc error: code = NotFound desc = could not find container \"923981358b5de4da1ec239e347feb196e5f5b03cc9e27ddaa43336814abfea6b\": container with ID starting with 923981358b5de4da1ec239e347feb196e5f5b03cc9e27ddaa43336814abfea6b not found: ID does not exist" Sep 14 11:13:15.233032 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:13:15.232996 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e6e0b4a3-6669d65c98-l8xz9"] Sep 14 11:13:15.242380 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:13:15.242333 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e6e0b4a3-6669d65c98-l8xz9"] Sep 14 11:13:16.832992 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:13:16.832960 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b0c80e5-881f-45d8-a7f9-77dac57c9d4c" path="/var/lib/kubelet/pods/4b0c80e5-881f-45d8-a7f9-77dac57c9d4c/volumes" Sep 14 11:14:46.994330 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:14:46.994298 2581 scope.go:117] "RemoveContainer" containerID="2c79ce167dc78e047da9ff4524dc1f4cffe4326c678b5acb2006235dda3db5ce" Sep 14 11:15:00.198784 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:00.198749 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fdbc00-29823075-dkww5"] Sep 14 11:15:00.199232 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:00.199218 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b0c80e5-881f-45d8-a7f9-77dac57c9d4c" containerName="payload-pre-processing" Sep 14 11:15:00.199289 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:00.199234 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b0c80e5-881f-45d8-a7f9-77dac57c9d4c" containerName="payload-pre-processing" Sep 14 11:15:00.199289 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:00.199249 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc965586-a194-4c90-819d-b2a9706ace78" containerName="revoke-keys" Sep 14 11:15:00.199289 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:00.199255 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc965586-a194-4c90-819d-b2a9706ace78" containerName="revoke-keys" Sep 14 11:15:00.199422 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:00.199322 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc965586-a194-4c90-819d-b2a9706ace78" containerName="revoke-keys" Sep 14 11:15:00.199422 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:00.199332 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b0c80e5-881f-45d8-a7f9-77dac57c9d4c" containerName="payload-pre-processing" Sep 14 11:15:00.202589 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:00.202573 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29823075-f8dpz"] Sep 14 11:15:00.202709 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:00.202689 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fdbc00-29823075-dkww5" Sep 14 11:15:00.206175 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:00.206154 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823075-f8dpz" Sep 14 11:15:00.206312 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:00.206158 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pv9g8\"" Sep 14 11:15:00.214062 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:00.214034 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ebf73-29823075-p5cnj"] Sep 14 11:15:00.214472 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:00.214453 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fdbc00-29823075-dkww5" Sep 14 11:15:00.218172 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:00.217742 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ebf73-29823075-p5cnj" Sep 14 11:15:00.218976 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:00.218950 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7984b1-29823075-pjsfd"] Sep 14 11:15:00.220545 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:00.220529 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823075-f8dpz" Sep 14 11:15:00.222775 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:00.222757 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7984b1-29823075-pjsfd" Sep 14 11:15:00.235518 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:00.235428 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fdbc00-29823075-dkww5"] Sep 14 11:15:00.242069 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:00.242042 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29823075-f8dpz"] Sep 14 11:15:00.250363 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:00.249943 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ebf73-29823075-p5cnj"] Sep 14 11:15:00.264722 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:00.264526 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ebf73-29823075-p5cnj" Sep 14 11:15:00.266902 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:00.266811 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7984b1-29823075-pjsfd"] Sep 14 11:15:00.275006 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:00.274979 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7984b1-29823075-pjsfd" Sep 14 11:15:00.531317 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:15:00.531286 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48b1176c_8bbd_4ff7_9f23_3d5bd1458afb.slice/crio-c604166f02120b92fc444b4f6d0e3d2cf7f65ab1c062af6eabe275323589c863 WatchSource:0}: Error finding container c604166f02120b92fc444b4f6d0e3d2cf7f65ab1c062af6eabe275323589c863: Status 404 returned error can't find the container with id c604166f02120b92fc444b4f6d0e3d2cf7f65ab1c062af6eabe275323589c863 Sep 14 11:15:00.532426 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:00.532380 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fdbc00-29823075-dkww5"] Sep 14 11:15:00.535097 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:00.535076 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29823075-f8dpz"] Sep 14 11:15:00.536788 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:15:00.536764 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a6c9875_9c61_4b2a_80c1_52397cc003db.slice/crio-fd6d0a48ec2a4e1dfc42fa25d230fdbfe611cb65dc61a82af68024f083ad86e8 WatchSource:0}: Error finding container fd6d0a48ec2a4e1dfc42fa25d230fdbfe611cb65dc61a82af68024f083ad86e8: Status 404 returned error can't find the container with id fd6d0a48ec2a4e1dfc42fa25d230fdbfe611cb65dc61a82af68024f083ad86e8 Sep 14 11:15:00.606847 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:00.606808 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ebf73-29823075-p5cnj"] Sep 14 11:15:00.607453 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:15:00.607424 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ff251f5_a134_4d57_804f_372254f418b1.slice/crio-43d1352e832fa914fd4a20b097dbdc0044227031bb941f94dea3f3d9f3e4b2eb WatchSource:0}: Error finding container 43d1352e832fa914fd4a20b097dbdc0044227031bb941f94dea3f3d9f3e4b2eb: Status 404 returned error can't find the container with id 43d1352e832fa914fd4a20b097dbdc0044227031bb941f94dea3f3d9f3e4b2eb Sep 14 11:15:00.608544 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:15:00.608524 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41401723_4af0_421d_ace9_a8d3353f39f6.slice/crio-929d51915f460899e0ec3bd51113920f45ae1fce6894438694bfc7eae83b2183 WatchSource:0}: Error finding container 929d51915f460899e0ec3bd51113920f45ae1fce6894438694bfc7eae83b2183: Status 404 returned error can't find the container with id 929d51915f460899e0ec3bd51113920f45ae1fce6894438694bfc7eae83b2183 Sep 14 11:15:00.609319 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:00.609295 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7984b1-29823075-pjsfd"] Sep 14 11:15:00.770627 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:00.770596 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7984b1-29823075-pjsfd" event={"ID":"5ff251f5-a134-4d57-804f-372254f418b1","Type":"ContainerStarted","Data":"43d1352e832fa914fd4a20b097dbdc0044227031bb941f94dea3f3d9f3e4b2eb"} Sep 14 11:15:00.772002 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:00.771974 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ebf73-29823075-p5cnj" event={"ID":"41401723-4af0-421d-ace9-a8d3353f39f6","Type":"ContainerStarted","Data":"929d51915f460899e0ec3bd51113920f45ae1fce6894438694bfc7eae83b2183"} Sep 14 11:15:00.773165 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:00.773143 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fdbc00-29823075-dkww5" event={"ID":"48b1176c-8bbd-4ff7-9f23-3d5bd1458afb","Type":"ContainerStarted","Data":"c604166f02120b92fc444b4f6d0e3d2cf7f65ab1c062af6eabe275323589c863"} Sep 14 11:15:00.774404 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:00.774386 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823075-f8dpz" event={"ID":"0a6c9875-9c61-4b2a-80c1-52397cc003db","Type":"ContainerStarted","Data":"fd6d0a48ec2a4e1dfc42fa25d230fdbfe611cb65dc61a82af68024f083ad86e8"} Sep 14 11:15:01.782406 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:01.782367 2581 generic.go:356] "Generic (PLEG): container finished" podID="41401723-4af0-421d-ace9-a8d3353f39f6" containerID="c121336c35a4bef2685976d6b3e9dd4f217121fb0fca4536e1eb746e48b4a0a2" exitCode=0 Sep 14 11:15:01.782847 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:01.782450 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ebf73-29823075-p5cnj" event={"ID":"41401723-4af0-421d-ace9-a8d3353f39f6","Type":"ContainerDied","Data":"c121336c35a4bef2685976d6b3e9dd4f217121fb0fca4536e1eb746e48b4a0a2"} Sep 14 11:15:01.784016 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:01.783995 2581 generic.go:356] "Generic (PLEG): container finished" podID="48b1176c-8bbd-4ff7-9f23-3d5bd1458afb" containerID="63f04906ff95af31b5c9a3cb314d796f23c294a460b5a0e84310963762fb76fe" exitCode=0 Sep 14 11:15:01.784142 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:01.784064 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fdbc00-29823075-dkww5" event={"ID":"48b1176c-8bbd-4ff7-9f23-3d5bd1458afb","Type":"ContainerDied","Data":"63f04906ff95af31b5c9a3cb314d796f23c294a460b5a0e84310963762fb76fe"} Sep 14 11:15:01.785491 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:01.785465 2581 generic.go:356] "Generic (PLEG): container finished" podID="0a6c9875-9c61-4b2a-80c1-52397cc003db" containerID="e879a1d3f0ee2448a341b065d053b00b6e07712ed949699e3d699fc59aba9217" exitCode=0 Sep 14 11:15:01.785575 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:01.785491 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823075-f8dpz" event={"ID":"0a6c9875-9c61-4b2a-80c1-52397cc003db","Type":"ContainerDied","Data":"e879a1d3f0ee2448a341b065d053b00b6e07712ed949699e3d699fc59aba9217"} Sep 14 11:15:01.787042 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:01.787023 2581 generic.go:356] "Generic (PLEG): container finished" podID="5ff251f5-a134-4d57-804f-372254f418b1" containerID="c868048b83c9da07c10e15fbdfbe789bf2c99ff6a2c3b6c276aae6a3310c46ba" exitCode=0 Sep 14 11:15:01.787134 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:01.787070 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7984b1-29823075-pjsfd" event={"ID":"5ff251f5-a134-4d57-804f-372254f418b1","Type":"ContainerDied","Data":"c868048b83c9da07c10e15fbdfbe789bf2c99ff6a2c3b6c276aae6a3310c46ba"} Sep 14 11:15:02.950210 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:02.950186 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7984b1-29823075-pjsfd" Sep 14 11:15:03.000890 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:03.000863 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823075-f8dpz" Sep 14 11:15:03.042436 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:03.042407 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fdbc00-29823075-dkww5" Sep 14 11:15:03.045865 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:03.045799 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ebf73-29823075-p5cnj" Sep 14 11:15:03.800246 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:03.800217 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ebf73-29823075-p5cnj" Sep 14 11:15:03.800482 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:03.800222 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ebf73-29823075-p5cnj" event={"ID":"41401723-4af0-421d-ace9-a8d3353f39f6","Type":"ContainerDied","Data":"929d51915f460899e0ec3bd51113920f45ae1fce6894438694bfc7eae83b2183"} Sep 14 11:15:03.800482 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:03.800370 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="929d51915f460899e0ec3bd51113920f45ae1fce6894438694bfc7eae83b2183" Sep 14 11:15:03.801606 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:03.801588 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fdbc00-29823075-dkww5" Sep 14 11:15:03.801606 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:03.801599 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fdbc00-29823075-dkww5" event={"ID":"48b1176c-8bbd-4ff7-9f23-3d5bd1458afb","Type":"ContainerDied","Data":"c604166f02120b92fc444b4f6d0e3d2cf7f65ab1c062af6eabe275323589c863"} Sep 14 11:15:03.801811 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:03.801635 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c604166f02120b92fc444b4f6d0e3d2cf7f65ab1c062af6eabe275323589c863" Sep 14 11:15:03.803106 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:03.803080 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823075-f8dpz" Sep 14 11:15:03.803219 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:03.803115 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29823075-f8dpz" event={"ID":"0a6c9875-9c61-4b2a-80c1-52397cc003db","Type":"ContainerDied","Data":"fd6d0a48ec2a4e1dfc42fa25d230fdbfe611cb65dc61a82af68024f083ad86e8"} Sep 14 11:15:03.803219 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:03.803155 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd6d0a48ec2a4e1dfc42fa25d230fdbfe611cb65dc61a82af68024f083ad86e8" Sep 14 11:15:03.804663 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:03.804641 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7984b1-29823075-pjsfd" Sep 14 11:15:03.804768 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:03.804649 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7984b1-29823075-pjsfd" event={"ID":"5ff251f5-a134-4d57-804f-372254f418b1","Type":"ContainerDied","Data":"43d1352e832fa914fd4a20b097dbdc0044227031bb941f94dea3f3d9f3e4b2eb"} Sep 14 11:15:03.804768 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:03.804712 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43d1352e832fa914fd4a20b097dbdc0044227031bb941f94dea3f3d9f3e4b2eb" Sep 14 11:15:19.109609 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:19.109578 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7984b1-jrhw9"] Sep 14 11:15:19.110050 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:19.110036 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48b1176c-8bbd-4ff7-9f23-3d5bd1458afb" containerName="cleanup" Sep 14 11:15:19.110100 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:19.110052 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="48b1176c-8bbd-4ff7-9f23-3d5bd1458afb" containerName="cleanup" Sep 14 11:15:19.110100 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:19.110064 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a6c9875-9c61-4b2a-80c1-52397cc003db" containerName="cleanup" Sep 14 11:15:19.110100 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:19.110071 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a6c9875-9c61-4b2a-80c1-52397cc003db" containerName="cleanup" Sep 14 11:15:19.110100 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:19.110077 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ff251f5-a134-4d57-804f-372254f418b1" containerName="cleanup" Sep 14 11:15:19.110100 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:19.110082 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ff251f5-a134-4d57-804f-372254f418b1" containerName="cleanup" Sep 14 11:15:19.110100 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:19.110101 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41401723-4af0-421d-ace9-a8d3353f39f6" containerName="cleanup" Sep 14 11:15:19.110311 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:19.110107 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="41401723-4af0-421d-ace9-a8d3353f39f6" containerName="cleanup" Sep 14 11:15:19.110311 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:19.110176 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a6c9875-9c61-4b2a-80c1-52397cc003db" containerName="cleanup" Sep 14 11:15:19.110311 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:19.110184 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="48b1176c-8bbd-4ff7-9f23-3d5bd1458afb" containerName="cleanup" Sep 14 11:15:19.110311 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:19.110189 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="41401723-4af0-421d-ace9-a8d3353f39f6" containerName="cleanup" Sep 14 11:15:19.110311 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:19.110198 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ff251f5-a134-4d57-804f-372254f418b1" containerName="cleanup" Sep 14 11:15:19.113416 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:19.113398 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7984b1-jrhw9" Sep 14 11:15:19.119158 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:19.119134 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pv9g8\"" Sep 14 11:15:19.131275 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:19.131245 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7984b1-jrhw9"] Sep 14 11:15:19.177406 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:19.177374 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adf17cd1-b246-4f08-8850-06b89f24652e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-7984b1-jrhw9\" (UID: \"adf17cd1-b246-4f08-8850-06b89f24652e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7984b1-jrhw9" Sep 14 11:15:19.278968 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:19.278934 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adf17cd1-b246-4f08-8850-06b89f24652e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-7984b1-jrhw9\" (UID: \"adf17cd1-b246-4f08-8850-06b89f24652e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7984b1-jrhw9" Sep 14 11:15:19.279624 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:19.279602 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adf17cd1-b246-4f08-8850-06b89f24652e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-7984b1-jrhw9\" (UID: \"adf17cd1-b246-4f08-8850-06b89f24652e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7984b1-jrhw9" Sep 14 11:15:19.424649 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:19.424562 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7984b1-jrhw9" Sep 14 11:15:19.712493 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:19.712470 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7984b1-jrhw9"] Sep 14 11:15:19.714960 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:15:19.714927 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadf17cd1_b246_4f08_8850_06b89f24652e.slice/crio-e4614e036d6eb2088d0ee5f8264b8838a37ce1c1d53422523376343f374325d7 WatchSource:0}: Error finding container e4614e036d6eb2088d0ee5f8264b8838a37ce1c1d53422523376343f374325d7: Status 404 returned error can't find the container with id e4614e036d6eb2088d0ee5f8264b8838a37ce1c1d53422523376343f374325d7 Sep 14 11:15:19.899132 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:19.899096 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7984b1-jrhw9" event={"ID":"adf17cd1-b246-4f08-8850-06b89f24652e","Type":"ContainerStarted","Data":"e4614e036d6eb2088d0ee5f8264b8838a37ce1c1d53422523376343f374325d7"} Sep 14 11:15:20.906623 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:20.906571 2581 generic.go:356] "Generic (PLEG): container finished" podID="adf17cd1-b246-4f08-8850-06b89f24652e" containerID="aabd091a12dd4bdd9d59ddaa3fa59b11f662ab3e2905d77aedb3ba16264144f7" exitCode=0 Sep 14 11:15:20.907029 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:20.906633 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7984b1-jrhw9" event={"ID":"adf17cd1-b246-4f08-8850-06b89f24652e","Type":"ContainerDied","Data":"aabd091a12dd4bdd9d59ddaa3fa59b11f662ab3e2905d77aedb3ba16264144f7"} Sep 14 11:15:22.060870 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:22.060844 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7984b1-jrhw9" Sep 14 11:15:22.108156 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:22.108110 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adf17cd1-b246-4f08-8850-06b89f24652e-maas-api-service-ca\") pod \"adf17cd1-b246-4f08-8850-06b89f24652e\" (UID: \"adf17cd1-b246-4f08-8850-06b89f24652e\") " Sep 14 11:15:22.108592 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:22.108568 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adf17cd1-b246-4f08-8850-06b89f24652e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "adf17cd1-b246-4f08-8850-06b89f24652e" (UID: "adf17cd1-b246-4f08-8850-06b89f24652e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:15:22.209688 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:22.209603 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adf17cd1-b246-4f08-8850-06b89f24652e-maas-api-service-ca\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:15:22.918303 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:22.918274 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7984b1-jrhw9" Sep 14 11:15:22.918495 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:22.918301 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7984b1-jrhw9" event={"ID":"adf17cd1-b246-4f08-8850-06b89f24652e","Type":"ContainerDied","Data":"e4614e036d6eb2088d0ee5f8264b8838a37ce1c1d53422523376343f374325d7"} Sep 14 11:15:22.918495 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:22.918362 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4614e036d6eb2088d0ee5f8264b8838a37ce1c1d53422523376343f374325d7" Sep 14 11:15:24.044960 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:24.044925 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7984b1/distinct-2-w0-7984b1-kserve-59576ff7c7-zh827"] Sep 14 11:15:24.054539 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:24.054504 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7984b1/distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk"] Sep 14 11:15:24.098389 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:24.098333 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7984b1/simulator-w0-7984b1-kserve-77b7979974-n6lvt"] Sep 14 11:15:33.392744 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:33.392707 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ebf73-vzmcr"] Sep 14 11:15:33.393154 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:33.393141 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adf17cd1-b246-4f08-8850-06b89f24652e" containerName="revoke-keys" Sep 14 11:15:33.393154 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:33.393155 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="adf17cd1-b246-4f08-8850-06b89f24652e" containerName="revoke-keys" Sep 14 11:15:33.393251 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:33.393242 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="adf17cd1-b246-4f08-8850-06b89f24652e" containerName="revoke-keys" Sep 14 11:15:33.397680 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:33.397661 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ebf73-vzmcr" Sep 14 11:15:33.400324 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:33.400301 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pv9g8\"" Sep 14 11:15:33.406118 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:33.406091 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ebf73-vzmcr"] Sep 14 11:15:33.526865 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:33.526827 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37971749-8dbe-4d9e-b348-1a9a33c7d3a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4ebf73-vzmcr\" (UID: \"37971749-8dbe-4d9e-b348-1a9a33c7d3a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ebf73-vzmcr" Sep 14 11:15:33.628409 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:33.628369 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37971749-8dbe-4d9e-b348-1a9a33c7d3a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4ebf73-vzmcr\" (UID: \"37971749-8dbe-4d9e-b348-1a9a33c7d3a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ebf73-vzmcr" Sep 14 11:15:33.628990 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:33.628968 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37971749-8dbe-4d9e-b348-1a9a33c7d3a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4ebf73-vzmcr\" (UID: \"37971749-8dbe-4d9e-b348-1a9a33c7d3a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ebf73-vzmcr" Sep 14 11:15:33.709301 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:33.709215 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ebf73-vzmcr" Sep 14 11:15:33.994163 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:33.994138 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ebf73-vzmcr"] Sep 14 11:15:33.996019 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:15:33.995992 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37971749_8dbe_4d9e_b348_1a9a33c7d3a9.slice/crio-bbec415dfdd3df1fe458526a4665144a9fdb5795b6901ff974cd8f9e2e65233b WatchSource:0}: Error finding container bbec415dfdd3df1fe458526a4665144a9fdb5795b6901ff974cd8f9e2e65233b: Status 404 returned error can't find the container with id bbec415dfdd3df1fe458526a4665144a9fdb5795b6901ff974cd8f9e2e65233b Sep 14 11:15:34.990305 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:34.990215 2581 generic.go:356] "Generic (PLEG): container finished" podID="37971749-8dbe-4d9e-b348-1a9a33c7d3a9" containerID="7073b8cf5694b654d1fa2d82e9dccc5df1ebfb2aa81387b20bbd68552e3d8e58" exitCode=0 Sep 14 11:15:34.990686 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:34.990307 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ebf73-vzmcr" event={"ID":"37971749-8dbe-4d9e-b348-1a9a33c7d3a9","Type":"ContainerDied","Data":"7073b8cf5694b654d1fa2d82e9dccc5df1ebfb2aa81387b20bbd68552e3d8e58"} Sep 14 11:15:34.990686 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:34.990381 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ebf73-vzmcr" event={"ID":"37971749-8dbe-4d9e-b348-1a9a33c7d3a9","Type":"ContainerStarted","Data":"bbec415dfdd3df1fe458526a4665144a9fdb5795b6901ff974cd8f9e2e65233b"} Sep 14 11:15:36.140126 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:36.140097 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ebf73-vzmcr" Sep 14 11:15:36.257031 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:36.256942 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37971749-8dbe-4d9e-b348-1a9a33c7d3a9-maas-api-service-ca\") pod \"37971749-8dbe-4d9e-b348-1a9a33c7d3a9\" (UID: \"37971749-8dbe-4d9e-b348-1a9a33c7d3a9\") " Sep 14 11:15:36.257396 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:36.257373 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37971749-8dbe-4d9e-b348-1a9a33c7d3a9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "37971749-8dbe-4d9e-b348-1a9a33c7d3a9" (UID: "37971749-8dbe-4d9e-b348-1a9a33c7d3a9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:15:36.358191 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:36.358155 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37971749-8dbe-4d9e-b348-1a9a33c7d3a9-maas-api-service-ca\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:15:37.002405 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:37.002369 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ebf73-vzmcr" event={"ID":"37971749-8dbe-4d9e-b348-1a9a33c7d3a9","Type":"ContainerDied","Data":"bbec415dfdd3df1fe458526a4665144a9fdb5795b6901ff974cd8f9e2e65233b"} Sep 14 11:15:37.002405 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:37.002394 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ebf73-vzmcr" Sep 14 11:15:37.002612 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:37.002428 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbec415dfdd3df1fe458526a4665144a9fdb5795b6901ff974cd8f9e2e65233b" Sep 14 11:15:38.304632 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:38.304532 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4ebf73/distinct-2-w4-4ebf73-kserve-7584f57865-l9888"] Sep 14 11:15:38.313151 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:38.313116 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4ebf73/distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks"] Sep 14 11:15:38.696159 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:38.696078 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7984b1-29823075-pjsfd"] Sep 14 11:15:38.722547 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:38.722513 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7984b1-29823075-pjsfd"] Sep 14 11:15:38.814370 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:38.814320 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7984b1-56cdcd4d45-bfgfv"] Sep 14 11:15:38.814554 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:38.814533 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-7984b1-56cdcd4d45-bfgfv" podUID="a7e74eea-f3ea-492e-9dab-fdf852dad5c0" containerName="payload-processing" containerID="cri-o://3037e1017bb01e8faf3b3713d471839746e14045eb2e8dd97237daa7f9a5e2e8" gracePeriod=30 Sep 14 11:15:38.835237 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:38.835202 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ff251f5-a134-4d57-804f-372254f418b1" path="/var/lib/kubelet/pods/5ff251f5-a134-4d57-804f-372254f418b1/volumes" Sep 14 11:15:38.877561 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:38.877527 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7984b1-6999cf6c6b-8vbcv"] Sep 14 11:15:38.877791 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:38.877768 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7984b1-6999cf6c6b-8vbcv" podUID="5c3e8eac-9ead-40d3-a619-2021c3a203de" containerName="payload-pre-processing" containerID="cri-o://d8937e07a02f4804ff9ea0b7da1661c955132a9cb6c0ae8515c5c5a873a0b213" gracePeriod=30 Sep 14 11:15:39.018919 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.018248 2581 generic.go:356] "Generic (PLEG): container finished" podID="a7e74eea-f3ea-492e-9dab-fdf852dad5c0" containerID="3037e1017bb01e8faf3b3713d471839746e14045eb2e8dd97237daa7f9a5e2e8" exitCode=0 Sep 14 11:15:39.018919 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.018410 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-7984b1-56cdcd4d45-bfgfv" event={"ID":"a7e74eea-f3ea-492e-9dab-fdf852dad5c0","Type":"ContainerDied","Data":"3037e1017bb01e8faf3b3713d471839746e14045eb2e8dd97237daa7f9a5e2e8"} Sep 14 11:15:39.021716 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.021636 2581 generic.go:356] "Generic (PLEG): container finished" podID="5c3e8eac-9ead-40d3-a619-2021c3a203de" containerID="d8937e07a02f4804ff9ea0b7da1661c955132a9cb6c0ae8515c5c5a873a0b213" exitCode=0 Sep 14 11:15:39.021716 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.021689 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7984b1-6999cf6c6b-8vbcv" event={"ID":"5c3e8eac-9ead-40d3-a619-2021c3a203de","Type":"ContainerDied","Data":"d8937e07a02f4804ff9ea0b7da1661c955132a9cb6c0ae8515c5c5a873a0b213"} Sep 14 11:15:39.066729 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.066663 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7984b1/distinct-2-w0-7984b1-kserve-59576ff7c7-zh827" podUID="6b3d88ab-62c4-4b21-adf7-ab2c8e529b57" containerName="main" containerID="cri-o://b1854c670eb797d8b7165f723c85dfc690a5e028d842243006f56a1742cfe7ef" gracePeriod=45 Sep 14 11:15:39.074812 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.074754 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7984b1/distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk" podUID="762c6088-d668-4d97-9700-d68b50d3277f" containerName="main" containerID="cri-o://d29a0f2f7e2fcc922f9c702b4e1ec8c1cd080948fc9ae4793ed4a39c801c3ba1" gracePeriod=45 Sep 14 11:15:39.118373 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.118315 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7984b1/simulator-w0-7984b1-kserve-77b7979974-n6lvt" podUID="57bc673d-5288-43bc-8712-45dc88500013" containerName="main" containerID="cri-o://8b72d79d244473e9414cdf680b76e8545707d2a05d348821b421b144c8a1ee1a" gracePeriod=45 Sep 14 11:15:39.270538 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.270495 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-7984b1-56cdcd4d45-bfgfv" Sep 14 11:15:39.276259 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.276175 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7984b1-6999cf6c6b-8vbcv" Sep 14 11:15:39.374410 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.374375 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7984b1/distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk" Sep 14 11:15:39.390665 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.390637 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7e74eea-f3ea-492e-9dab-fdf852dad5c0-plugins-config-volume\") pod \"a7e74eea-f3ea-492e-9dab-fdf852dad5c0\" (UID: \"a7e74eea-f3ea-492e-9dab-fdf852dad5c0\") " Sep 14 11:15:39.390833 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.390731 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vblw\" (UniqueName: \"kubernetes.io/projected/5c3e8eac-9ead-40d3-a619-2021c3a203de-kube-api-access-9vblw\") pod \"5c3e8eac-9ead-40d3-a619-2021c3a203de\" (UID: \"5c3e8eac-9ead-40d3-a619-2021c3a203de\") " Sep 14 11:15:39.390833 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.390818 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znw97\" (UniqueName: \"kubernetes.io/projected/a7e74eea-f3ea-492e-9dab-fdf852dad5c0-kube-api-access-znw97\") pod \"a7e74eea-f3ea-492e-9dab-fdf852dad5c0\" (UID: \"a7e74eea-f3ea-492e-9dab-fdf852dad5c0\") " Sep 14 11:15:39.390985 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.390901 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c3e8eac-9ead-40d3-a619-2021c3a203de-plugins-config-volume\") pod \"5c3e8eac-9ead-40d3-a619-2021c3a203de\" (UID: \"5c3e8eac-9ead-40d3-a619-2021c3a203de\") " Sep 14 11:15:39.391522 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.391492 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7e74eea-f3ea-492e-9dab-fdf852dad5c0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a7e74eea-f3ea-492e-9dab-fdf852dad5c0" (UID: "a7e74eea-f3ea-492e-9dab-fdf852dad5c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:15:39.391662 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.391557 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c3e8eac-9ead-40d3-a619-2021c3a203de-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5c3e8eac-9ead-40d3-a619-2021c3a203de" (UID: "5c3e8eac-9ead-40d3-a619-2021c3a203de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:15:39.395111 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.395083 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c3e8eac-9ead-40d3-a619-2021c3a203de-kube-api-access-9vblw" (OuterVolumeSpecName: "kube-api-access-9vblw") pod "5c3e8eac-9ead-40d3-a619-2021c3a203de" (UID: "5c3e8eac-9ead-40d3-a619-2021c3a203de"). InnerVolumeSpecName "kube-api-access-9vblw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:15:39.395225 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.395109 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7e74eea-f3ea-492e-9dab-fdf852dad5c0-kube-api-access-znw97" (OuterVolumeSpecName: "kube-api-access-znw97") pod "a7e74eea-f3ea-492e-9dab-fdf852dad5c0" (UID: "a7e74eea-f3ea-492e-9dab-fdf852dad5c0"). InnerVolumeSpecName "kube-api-access-znw97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:15:39.431397 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.431330 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7984b1/distinct-2-w0-7984b1-kserve-59576ff7c7-zh827" Sep 14 11:15:39.441993 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.441971 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7984b1/simulator-w0-7984b1-kserve-77b7979974-n6lvt" Sep 14 11:15:39.492699 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.492610 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/762c6088-d668-4d97-9700-d68b50d3277f-model-cache\") pod \"762c6088-d668-4d97-9700-d68b50d3277f\" (UID: \"762c6088-d668-4d97-9700-d68b50d3277f\") " Sep 14 11:15:39.492699 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.492685 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7l8qg\" (UniqueName: \"kubernetes.io/projected/762c6088-d668-4d97-9700-d68b50d3277f-kube-api-access-7l8qg\") pod \"762c6088-d668-4d97-9700-d68b50d3277f\" (UID: \"762c6088-d668-4d97-9700-d68b50d3277f\") " Sep 14 11:15:39.492910 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.492730 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/762c6088-d668-4d97-9700-d68b50d3277f-dshm\") pod \"762c6088-d668-4d97-9700-d68b50d3277f\" (UID: \"762c6088-d668-4d97-9700-d68b50d3277f\") " Sep 14 11:15:39.492910 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.492845 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/762c6088-d668-4d97-9700-d68b50d3277f-home\") pod \"762c6088-d668-4d97-9700-d68b50d3277f\" (UID: \"762c6088-d668-4d97-9700-d68b50d3277f\") " Sep 14 11:15:39.492910 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.492885 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/762c6088-d668-4d97-9700-d68b50d3277f-tmp-dir\") pod \"762c6088-d668-4d97-9700-d68b50d3277f\" (UID: \"762c6088-d668-4d97-9700-d68b50d3277f\") " Sep 14 11:15:39.493079 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.492907 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762c6088-d668-4d97-9700-d68b50d3277f-model-cache" (OuterVolumeSpecName: "model-cache") pod "762c6088-d668-4d97-9700-d68b50d3277f" (UID: "762c6088-d668-4d97-9700-d68b50d3277f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:15:39.493079 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.492940 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/762c6088-d668-4d97-9700-d68b50d3277f-tls-certs\") pod \"762c6088-d668-4d97-9700-d68b50d3277f\" (UID: \"762c6088-d668-4d97-9700-d68b50d3277f\") " Sep 14 11:15:39.493238 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.493216 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762c6088-d668-4d97-9700-d68b50d3277f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "762c6088-d668-4d97-9700-d68b50d3277f" (UID: "762c6088-d668-4d97-9700-d68b50d3277f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:15:39.493387 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.493367 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znw97\" (UniqueName: \"kubernetes.io/projected/a7e74eea-f3ea-492e-9dab-fdf852dad5c0-kube-api-access-znw97\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:15:39.493604 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.493392 2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/762c6088-d668-4d97-9700-d68b50d3277f-model-cache\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:15:39.493604 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.493403 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c3e8eac-9ead-40d3-a619-2021c3a203de-plugins-config-volume\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:15:39.493604 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.493412 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7e74eea-f3ea-492e-9dab-fdf852dad5c0-plugins-config-volume\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:15:39.493604 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.493425 2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/762c6088-d668-4d97-9700-d68b50d3277f-tmp-dir\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:15:39.493604 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.493440 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vblw\" (UniqueName: \"kubernetes.io/projected/5c3e8eac-9ead-40d3-a619-2021c3a203de-kube-api-access-9vblw\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:15:39.493604 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.493148 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762c6088-d668-4d97-9700-d68b50d3277f-home" (OuterVolumeSpecName: "home") pod "762c6088-d668-4d97-9700-d68b50d3277f" (UID: "762c6088-d668-4d97-9700-d68b50d3277f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:15:39.495264 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.495234 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/762c6088-d668-4d97-9700-d68b50d3277f-dshm" (OuterVolumeSpecName: "dshm") pod "762c6088-d668-4d97-9700-d68b50d3277f" (UID: "762c6088-d668-4d97-9700-d68b50d3277f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:15:39.495380 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.495290 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/762c6088-d668-4d97-9700-d68b50d3277f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "762c6088-d668-4d97-9700-d68b50d3277f" (UID: "762c6088-d668-4d97-9700-d68b50d3277f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:15:39.495478 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.495461 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/762c6088-d668-4d97-9700-d68b50d3277f-kube-api-access-7l8qg" (OuterVolumeSpecName: "kube-api-access-7l8qg") pod "762c6088-d668-4d97-9700-d68b50d3277f" (UID: "762c6088-d668-4d97-9700-d68b50d3277f"). InnerVolumeSpecName "kube-api-access-7l8qg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:15:39.534516 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.534482 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7984b1-jrhw9"] Sep 14 11:15:39.554019 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.553984 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7984b1-jrhw9"] Sep 14 11:15:39.594483 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.594452 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57bc673d-5288-43bc-8712-45dc88500013-tls-certs\") pod \"57bc673d-5288-43bc-8712-45dc88500013\" (UID: \"57bc673d-5288-43bc-8712-45dc88500013\") " Sep 14 11:15:39.594677 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.594508 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6trj\" (UniqueName: \"kubernetes.io/projected/57bc673d-5288-43bc-8712-45dc88500013-kube-api-access-z6trj\") pod \"57bc673d-5288-43bc-8712-45dc88500013\" (UID: \"57bc673d-5288-43bc-8712-45dc88500013\") " Sep 14 11:15:39.594677 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.594538 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57bc673d-5288-43bc-8712-45dc88500013-dshm\") pod \"57bc673d-5288-43bc-8712-45dc88500013\" (UID: \"57bc673d-5288-43bc-8712-45dc88500013\") " Sep 14 11:15:39.594677 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.594577 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57bc673d-5288-43bc-8712-45dc88500013-model-cache\") pod \"57bc673d-5288-43bc-8712-45dc88500013\" (UID: \"57bc673d-5288-43bc-8712-45dc88500013\") " Sep 14 11:15:39.594677 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.594606 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b3d88ab-62c4-4b21-adf7-ab2c8e529b57-tls-certs\") pod \"6b3d88ab-62c4-4b21-adf7-ab2c8e529b57\" (UID: \"6b3d88ab-62c4-4b21-adf7-ab2c8e529b57\") " Sep 14 11:15:39.594677 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.594641 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b3d88ab-62c4-4b21-adf7-ab2c8e529b57-model-cache\") pod \"6b3d88ab-62c4-4b21-adf7-ab2c8e529b57\" (UID: \"6b3d88ab-62c4-4b21-adf7-ab2c8e529b57\") " Sep 14 11:15:39.594677 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.594673 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b3d88ab-62c4-4b21-adf7-ab2c8e529b57-tmp-dir\") pod \"6b3d88ab-62c4-4b21-adf7-ab2c8e529b57\" (UID: \"6b3d88ab-62c4-4b21-adf7-ab2c8e529b57\") " Sep 14 11:15:39.595034 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.594739 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xv7rz\" (UniqueName: \"kubernetes.io/projected/6b3d88ab-62c4-4b21-adf7-ab2c8e529b57-kube-api-access-xv7rz\") pod \"6b3d88ab-62c4-4b21-adf7-ab2c8e529b57\" (UID: \"6b3d88ab-62c4-4b21-adf7-ab2c8e529b57\") " Sep 14 11:15:39.595034 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.594795 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b3d88ab-62c4-4b21-adf7-ab2c8e529b57-home\") pod \"6b3d88ab-62c4-4b21-adf7-ab2c8e529b57\" (UID: \"6b3d88ab-62c4-4b21-adf7-ab2c8e529b57\") " Sep 14 11:15:39.595034 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.594838 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b3d88ab-62c4-4b21-adf7-ab2c8e529b57-dshm\") pod \"6b3d88ab-62c4-4b21-adf7-ab2c8e529b57\" (UID: \"6b3d88ab-62c4-4b21-adf7-ab2c8e529b57\") " Sep 14 11:15:39.595034 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.594849 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57bc673d-5288-43bc-8712-45dc88500013-model-cache" (OuterVolumeSpecName: "model-cache") pod "57bc673d-5288-43bc-8712-45dc88500013" (UID: "57bc673d-5288-43bc-8712-45dc88500013"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:15:39.595034 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.594879 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57bc673d-5288-43bc-8712-45dc88500013-tmp-dir\") pod \"57bc673d-5288-43bc-8712-45dc88500013\" (UID: \"57bc673d-5288-43bc-8712-45dc88500013\") " Sep 14 11:15:39.595034 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.594961 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57bc673d-5288-43bc-8712-45dc88500013-home\") pod \"57bc673d-5288-43bc-8712-45dc88500013\" (UID: \"57bc673d-5288-43bc-8712-45dc88500013\") " Sep 14 11:15:39.595381 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.595054 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b3d88ab-62c4-4b21-adf7-ab2c8e529b57-home" (OuterVolumeSpecName: "home") pod "6b3d88ab-62c4-4b21-adf7-ab2c8e529b57" (UID: "6b3d88ab-62c4-4b21-adf7-ab2c8e529b57"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:15:39.595381 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.595230 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57bc673d-5288-43bc-8712-45dc88500013-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "57bc673d-5288-43bc-8712-45dc88500013" (UID: "57bc673d-5288-43bc-8712-45dc88500013"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:15:39.595497 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.595396 2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/762c6088-d668-4d97-9700-d68b50d3277f-dshm\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:15:39.595497 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.595417 2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57bc673d-5288-43bc-8712-45dc88500013-tmp-dir\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:15:39.595497 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.595430 2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57bc673d-5288-43bc-8712-45dc88500013-model-cache\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:15:39.595497 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.595436 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b3d88ab-62c4-4b21-adf7-ab2c8e529b57-model-cache" (OuterVolumeSpecName: "model-cache") pod "6b3d88ab-62c4-4b21-adf7-ab2c8e529b57" (UID: "6b3d88ab-62c4-4b21-adf7-ab2c8e529b57"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:15:39.595497 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.595444 2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/762c6088-d668-4d97-9700-d68b50d3277f-home\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:15:39.595497 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.595477 2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/762c6088-d668-4d97-9700-d68b50d3277f-tls-certs\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:15:39.595497 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.595492 2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b3d88ab-62c4-4b21-adf7-ab2c8e529b57-home\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:15:39.595854 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.595507 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l8qg\" (UniqueName: \"kubernetes.io/projected/762c6088-d668-4d97-9700-d68b50d3277f-kube-api-access-7l8qg\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:15:39.595854 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.595647 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57bc673d-5288-43bc-8712-45dc88500013-home" (OuterVolumeSpecName: "home") pod "57bc673d-5288-43bc-8712-45dc88500013" (UID: "57bc673d-5288-43bc-8712-45dc88500013"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:15:39.595941 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.595863 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b3d88ab-62c4-4b21-adf7-ab2c8e529b57-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6b3d88ab-62c4-4b21-adf7-ab2c8e529b57" (UID: "6b3d88ab-62c4-4b21-adf7-ab2c8e529b57"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:15:39.597514 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.597482 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b3d88ab-62c4-4b21-adf7-ab2c8e529b57-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6b3d88ab-62c4-4b21-adf7-ab2c8e529b57" (UID: "6b3d88ab-62c4-4b21-adf7-ab2c8e529b57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:15:39.597514 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.597500 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57bc673d-5288-43bc-8712-45dc88500013-kube-api-access-z6trj" (OuterVolumeSpecName: "kube-api-access-z6trj") pod "57bc673d-5288-43bc-8712-45dc88500013" (UID: "57bc673d-5288-43bc-8712-45dc88500013"). InnerVolumeSpecName "kube-api-access-z6trj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:15:39.597640 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.597568 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57bc673d-5288-43bc-8712-45dc88500013-dshm" (OuterVolumeSpecName: "dshm") pod "57bc673d-5288-43bc-8712-45dc88500013" (UID: "57bc673d-5288-43bc-8712-45dc88500013"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:15:39.597640 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.597568 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b3d88ab-62c4-4b21-adf7-ab2c8e529b57-dshm" (OuterVolumeSpecName: "dshm") pod "6b3d88ab-62c4-4b21-adf7-ab2c8e529b57" (UID: "6b3d88ab-62c4-4b21-adf7-ab2c8e529b57"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:15:39.597780 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.597764 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57bc673d-5288-43bc-8712-45dc88500013-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "57bc673d-5288-43bc-8712-45dc88500013" (UID: "57bc673d-5288-43bc-8712-45dc88500013"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:15:39.597819 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.597807 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b3d88ab-62c4-4b21-adf7-ab2c8e529b57-kube-api-access-xv7rz" (OuterVolumeSpecName: "kube-api-access-xv7rz") pod "6b3d88ab-62c4-4b21-adf7-ab2c8e529b57" (UID: "6b3d88ab-62c4-4b21-adf7-ab2c8e529b57"). InnerVolumeSpecName "kube-api-access-xv7rz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:15:39.696864 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.696825 2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57bc673d-5288-43bc-8712-45dc88500013-home\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:15:39.696864 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.696858 2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57bc673d-5288-43bc-8712-45dc88500013-tls-certs\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:15:39.696864 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.696872 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6trj\" (UniqueName: \"kubernetes.io/projected/57bc673d-5288-43bc-8712-45dc88500013-kube-api-access-z6trj\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:15:39.697112 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.696885 2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57bc673d-5288-43bc-8712-45dc88500013-dshm\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:15:39.697112 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.696896 2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b3d88ab-62c4-4b21-adf7-ab2c8e529b57-tls-certs\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:15:39.697112 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.696907 2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b3d88ab-62c4-4b21-adf7-ab2c8e529b57-model-cache\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:15:39.697112 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.696916 2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b3d88ab-62c4-4b21-adf7-ab2c8e529b57-tmp-dir\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:15:39.697112 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.696925 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv7rz\" (UniqueName: \"kubernetes.io/projected/6b3d88ab-62c4-4b21-adf7-ab2c8e529b57-kube-api-access-xv7rz\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:15:39.697112 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:39.696935 2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b3d88ab-62c4-4b21-adf7-ab2c8e529b57-dshm\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:15:40.028307 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:40.028267 2581 generic.go:356] "Generic (PLEG): container finished" podID="6b3d88ab-62c4-4b21-adf7-ab2c8e529b57" containerID="b1854c670eb797d8b7165f723c85dfc690a5e028d842243006f56a1742cfe7ef" exitCode=0 Sep 14 11:15:40.028487 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:40.028312 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7984b1/distinct-2-w0-7984b1-kserve-59576ff7c7-zh827" Sep 14 11:15:40.028487 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:40.028374 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7984b1/distinct-2-w0-7984b1-kserve-59576ff7c7-zh827" event={"ID":"6b3d88ab-62c4-4b21-adf7-ab2c8e529b57","Type":"ContainerDied","Data":"b1854c670eb797d8b7165f723c85dfc690a5e028d842243006f56a1742cfe7ef"} Sep 14 11:15:40.028487 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:40.028412 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7984b1/distinct-2-w0-7984b1-kserve-59576ff7c7-zh827" event={"ID":"6b3d88ab-62c4-4b21-adf7-ab2c8e529b57","Type":"ContainerDied","Data":"b3627564c96778ce5afefd3b6c2edfaaa9b34715a1c558defc59717406a4fd76"} Sep 14 11:15:40.028487 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:40.028430 2581 scope.go:117] "RemoveContainer" containerID="b1854c670eb797d8b7165f723c85dfc690a5e028d842243006f56a1742cfe7ef" Sep 14 11:15:40.029799 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:40.029734 2581 generic.go:356] "Generic (PLEG): container finished" podID="57bc673d-5288-43bc-8712-45dc88500013" containerID="8b72d79d244473e9414cdf680b76e8545707d2a05d348821b421b144c8a1ee1a" exitCode=0 Sep 14 11:15:40.029799 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:40.029758 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7984b1/simulator-w0-7984b1-kserve-77b7979974-n6lvt" event={"ID":"57bc673d-5288-43bc-8712-45dc88500013","Type":"ContainerDied","Data":"8b72d79d244473e9414cdf680b76e8545707d2a05d348821b421b144c8a1ee1a"} Sep 14 11:15:40.029799 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:40.029778 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7984b1/simulator-w0-7984b1-kserve-77b7979974-n6lvt" Sep 14 11:15:40.029799 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:40.029791 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7984b1/simulator-w0-7984b1-kserve-77b7979974-n6lvt" event={"ID":"57bc673d-5288-43bc-8712-45dc88500013","Type":"ContainerDied","Data":"f6e0357e369936b07036f3ed4c7b55ce4736545988c21535966864c58e86e3f7"} Sep 14 11:15:40.031305 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:40.031278 2581 generic.go:356] "Generic (PLEG): container finished" podID="762c6088-d668-4d97-9700-d68b50d3277f" containerID="d29a0f2f7e2fcc922f9c702b4e1ec8c1cd080948fc9ae4793ed4a39c801c3ba1" exitCode=0 Sep 14 11:15:40.031435 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:40.031311 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7984b1/distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk" Sep 14 11:15:40.031435 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:40.031305 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7984b1/distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk" event={"ID":"762c6088-d668-4d97-9700-d68b50d3277f","Type":"ContainerDied","Data":"d29a0f2f7e2fcc922f9c702b4e1ec8c1cd080948fc9ae4793ed4a39c801c3ba1"} Sep 14 11:15:40.031575 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:40.031454 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7984b1/distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk" event={"ID":"762c6088-d668-4d97-9700-d68b50d3277f","Type":"ContainerDied","Data":"5c53c95e6ecd6fe8e8be550c7a1fb6f79f895c1aed31a7e04b983c28a9beb351"} Sep 14 11:15:40.033039 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:40.033010 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-7984b1-56cdcd4d45-bfgfv" event={"ID":"a7e74eea-f3ea-492e-9dab-fdf852dad5c0","Type":"ContainerDied","Data":"493d4bbcca1d06b222a3becfcf8626c455d41857c479989ef3337cb40bc09b46"} Sep 14 11:15:40.033151 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:40.033020 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-7984b1-56cdcd4d45-bfgfv" Sep 14 11:15:40.034658 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:40.034626 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7984b1-6999cf6c6b-8vbcv" event={"ID":"5c3e8eac-9ead-40d3-a619-2021c3a203de","Type":"ContainerDied","Data":"ef44d569095ddc5d7b50729fcc7635ddcae7c5150fa4c7b04c80560c6297cc9f"} Sep 14 11:15:40.034763 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:40.034689 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7984b1-6999cf6c6b-8vbcv" Sep 14 11:15:40.041637 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:40.041387 2581 scope.go:117] "RemoveContainer" containerID="b1854c670eb797d8b7165f723c85dfc690a5e028d842243006f56a1742cfe7ef" Sep 14 11:15:40.041758 ip-10-0-137-53 kubenswrapper[2581]: E0914 11:15:40.041723 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1854c670eb797d8b7165f723c85dfc690a5e028d842243006f56a1742cfe7ef\": container with ID starting with b1854c670eb797d8b7165f723c85dfc690a5e028d842243006f56a1742cfe7ef not found: ID does not exist" containerID="b1854c670eb797d8b7165f723c85dfc690a5e028d842243006f56a1742cfe7ef" Sep 14 11:15:40.041853 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:40.041755 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1854c670eb797d8b7165f723c85dfc690a5e028d842243006f56a1742cfe7ef"} err="failed to get container status \"b1854c670eb797d8b7165f723c85dfc690a5e028d842243006f56a1742cfe7ef\": rpc error: code = NotFound desc = could not find container \"b1854c670eb797d8b7165f723c85dfc690a5e028d842243006f56a1742cfe7ef\": container with ID starting with b1854c670eb797d8b7165f723c85dfc690a5e028d842243006f56a1742cfe7ef not found: ID does not exist" Sep 14 11:15:40.041853 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:40.041778 2581 scope.go:117] "RemoveContainer" containerID="8b72d79d244473e9414cdf680b76e8545707d2a05d348821b421b144c8a1ee1a" Sep 14 11:15:40.052190 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:40.052169 2581 scope.go:117] "RemoveContainer" containerID="8b72d79d244473e9414cdf680b76e8545707d2a05d348821b421b144c8a1ee1a" Sep 14 11:15:40.052496 ip-10-0-137-53 kubenswrapper[2581]: E0914 11:15:40.052476 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b72d79d244473e9414cdf680b76e8545707d2a05d348821b421b144c8a1ee1a\": container with ID starting with 8b72d79d244473e9414cdf680b76e8545707d2a05d348821b421b144c8a1ee1a not found: ID does not exist" containerID="8b72d79d244473e9414cdf680b76e8545707d2a05d348821b421b144c8a1ee1a" Sep 14 11:15:40.052558 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:40.052502 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b72d79d244473e9414cdf680b76e8545707d2a05d348821b421b144c8a1ee1a"} err="failed to get container status \"8b72d79d244473e9414cdf680b76e8545707d2a05d348821b421b144c8a1ee1a\": rpc error: code = NotFound desc = could not find container \"8b72d79d244473e9414cdf680b76e8545707d2a05d348821b421b144c8a1ee1a\": container with ID starting with 8b72d79d244473e9414cdf680b76e8545707d2a05d348821b421b144c8a1ee1a not found: ID does not exist" Sep 14 11:15:40.052558 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:40.052519 2581 scope.go:117] "RemoveContainer" containerID="d29a0f2f7e2fcc922f9c702b4e1ec8c1cd080948fc9ae4793ed4a39c801c3ba1" Sep 14 11:15:40.060990 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:40.060970 2581 scope.go:117] "RemoveContainer" containerID="d29a0f2f7e2fcc922f9c702b4e1ec8c1cd080948fc9ae4793ed4a39c801c3ba1" Sep 14 11:15:40.061284 ip-10-0-137-53 kubenswrapper[2581]: E0914 11:15:40.061265 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d29a0f2f7e2fcc922f9c702b4e1ec8c1cd080948fc9ae4793ed4a39c801c3ba1\": container with ID starting with d29a0f2f7e2fcc922f9c702b4e1ec8c1cd080948fc9ae4793ed4a39c801c3ba1 not found: ID does not exist" containerID="d29a0f2f7e2fcc922f9c702b4e1ec8c1cd080948fc9ae4793ed4a39c801c3ba1" Sep 14 11:15:40.061384 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:40.061295 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d29a0f2f7e2fcc922f9c702b4e1ec8c1cd080948fc9ae4793ed4a39c801c3ba1"} err="failed to get container status \"d29a0f2f7e2fcc922f9c702b4e1ec8c1cd080948fc9ae4793ed4a39c801c3ba1\": rpc error: code = NotFound desc = could not find container \"d29a0f2f7e2fcc922f9c702b4e1ec8c1cd080948fc9ae4793ed4a39c801c3ba1\": container with ID starting with d29a0f2f7e2fcc922f9c702b4e1ec8c1cd080948fc9ae4793ed4a39c801c3ba1 not found: ID does not exist" Sep 14 11:15:40.061384 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:40.061316 2581 scope.go:117] "RemoveContainer" containerID="3037e1017bb01e8faf3b3713d471839746e14045eb2e8dd97237daa7f9a5e2e8" Sep 14 11:15:40.069653 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:40.069636 2581 scope.go:117] "RemoveContainer" containerID="d8937e07a02f4804ff9ea0b7da1661c955132a9cb6c0ae8515c5c5a873a0b213" Sep 14 11:15:40.094363 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:40.094281 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7984b1/simulator-w0-7984b1-kserve-77b7979974-n6lvt"] Sep 14 11:15:40.105419 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:40.105395 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7984b1/simulator-w0-7984b1-kserve-77b7979974-n6lvt"] Sep 14 11:15:40.152178 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:40.152138 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7984b1-6999cf6c6b-8vbcv"] Sep 14 11:15:40.154845 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:40.154822 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7984b1-6999cf6c6b-8vbcv"] Sep 14 11:15:40.195711 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:40.195685 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7984b1-56cdcd4d45-bfgfv"] Sep 14 11:15:40.204269 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:40.204247 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7984b1-56cdcd4d45-bfgfv"] Sep 14 11:15:40.223694 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:40.223668 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7984b1/distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk"] Sep 14 11:15:40.233151 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:40.233123 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7984b1/distinct-w0-7984b1-kserve-56dbb8c5f4-k8bpk"] Sep 14 11:15:40.316906 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:40.316872 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7984b1/distinct-2-w0-7984b1-kserve-59576ff7c7-zh827"] Sep 14 11:15:40.326780 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:40.326752 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7984b1/distinct-2-w0-7984b1-kserve-59576ff7c7-zh827"] Sep 14 11:15:40.832287 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:40.832247 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57bc673d-5288-43bc-8712-45dc88500013" path="/var/lib/kubelet/pods/57bc673d-5288-43bc-8712-45dc88500013/volumes" Sep 14 11:15:40.832697 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:40.832681 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c3e8eac-9ead-40d3-a619-2021c3a203de" path="/var/lib/kubelet/pods/5c3e8eac-9ead-40d3-a619-2021c3a203de/volumes" Sep 14 11:15:40.833041 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:40.833029 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b3d88ab-62c4-4b21-adf7-ab2c8e529b57" path="/var/lib/kubelet/pods/6b3d88ab-62c4-4b21-adf7-ab2c8e529b57/volumes" Sep 14 11:15:40.833700 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:40.833687 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="762c6088-d668-4d97-9700-d68b50d3277f" path="/var/lib/kubelet/pods/762c6088-d668-4d97-9700-d68b50d3277f/volumes" Sep 14 11:15:40.834033 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:40.834021 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7e74eea-f3ea-492e-9dab-fdf852dad5c0" path="/var/lib/kubelet/pods/a7e74eea-f3ea-492e-9dab-fdf852dad5c0/volumes" Sep 14 11:15:40.834318 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:40.834307 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adf17cd1-b246-4f08-8850-06b89f24652e" path="/var/lib/kubelet/pods/adf17cd1-b246-4f08-8850-06b89f24652e/volumes" Sep 14 11:15:43.368859 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:43.368819 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-29e74e-85664db6-xhbmm"] Sep 14 11:15:43.369277 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:43.369246 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7e74eea-f3ea-492e-9dab-fdf852dad5c0" containerName="payload-processing" Sep 14 11:15:43.369277 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:43.369257 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7e74eea-f3ea-492e-9dab-fdf852dad5c0" containerName="payload-processing" Sep 14 11:15:43.369277 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:43.369270 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b3d88ab-62c4-4b21-adf7-ab2c8e529b57" containerName="main" Sep 14 11:15:43.369277 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:43.369275 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b3d88ab-62c4-4b21-adf7-ab2c8e529b57" containerName="main" Sep 14 11:15:43.369465 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:43.369282 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37971749-8dbe-4d9e-b348-1a9a33c7d3a9" containerName="revoke-keys" Sep 14 11:15:43.369465 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:43.369288 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="37971749-8dbe-4d9e-b348-1a9a33c7d3a9" containerName="revoke-keys" Sep 14 11:15:43.369465 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:43.369297 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="762c6088-d668-4d97-9700-d68b50d3277f" containerName="main" Sep 14 11:15:43.369465 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:43.369304 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="762c6088-d668-4d97-9700-d68b50d3277f" containerName="main" Sep 14 11:15:43.369465 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:43.369320 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c3e8eac-9ead-40d3-a619-2021c3a203de" containerName="payload-pre-processing" Sep 14 11:15:43.369465 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:43.369325 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c3e8eac-9ead-40d3-a619-2021c3a203de" containerName="payload-pre-processing" Sep 14 11:15:43.369465 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:43.369330 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57bc673d-5288-43bc-8712-45dc88500013" containerName="main" Sep 14 11:15:43.369465 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:43.369349 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="57bc673d-5288-43bc-8712-45dc88500013" containerName="main" Sep 14 11:15:43.369465 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:43.369415 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7e74eea-f3ea-492e-9dab-fdf852dad5c0" containerName="payload-processing" Sep 14 11:15:43.369465 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:43.369423 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="762c6088-d668-4d97-9700-d68b50d3277f" containerName="main" Sep 14 11:15:43.369465 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:43.369430 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="37971749-8dbe-4d9e-b348-1a9a33c7d3a9" containerName="revoke-keys" Sep 14 11:15:43.369465 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:43.369437 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b3d88ab-62c4-4b21-adf7-ab2c8e529b57" containerName="main" Sep 14 11:15:43.369465 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:43.369443 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c3e8eac-9ead-40d3-a619-2021c3a203de" containerName="payload-pre-processing" Sep 14 11:15:43.369465 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:43.369447 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="57bc673d-5288-43bc-8712-45dc88500013" containerName="main" Sep 14 11:15:43.374154 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:43.374134 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29e74e-85664db6-xhbmm" Sep 14 11:15:43.378464 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:43.378442 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-29e74e-dockercfg-prst2\"" Sep 14 11:15:43.378567 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:43.378492 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-29e74e\"" Sep 14 11:15:43.392713 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:43.392682 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-29e74e-85664db6-xhbmm"] Sep 14 11:15:43.430321 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:43.430284 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd9wg\" (UniqueName: \"kubernetes.io/projected/2a84eecc-cf96-48b6-87c4-9ed9103d214b-kube-api-access-rd9wg\") pod \"payload-pre-processing-e2e-api-a-29e74e-85664db6-xhbmm\" (UID: \"2a84eecc-cf96-48b6-87c4-9ed9103d214b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-29e74e-85664db6-xhbmm" Sep 14 11:15:43.430524 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:43.430387 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a84eecc-cf96-48b6-87c4-9ed9103d214b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-29e74e-85664db6-xhbmm\" (UID: \"2a84eecc-cf96-48b6-87c4-9ed9103d214b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-29e74e-85664db6-xhbmm" Sep 14 11:15:43.446098 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:43.446065 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-29e74e-6fb8fc5c5-bk6nv"] Sep 14 11:15:43.449651 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:43.449634 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-29e74e-6fb8fc5c5-bk6nv" Sep 14 11:15:43.469727 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:43.469701 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-29e74e-6fb8fc5c5-bk6nv"] Sep 14 11:15:43.531122 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:43.531085 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/967e30fc-237b-450e-baf2-c6c4990c460f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-29e74e-6fb8fc5c5-bk6nv\" (UID: \"967e30fc-237b-450e-baf2-c6c4990c460f\") " pod="openshift-ingress/payload-processing-e2e-api-a-29e74e-6fb8fc5c5-bk6nv" Sep 14 11:15:43.531122 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:43.531122 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a84eecc-cf96-48b6-87c4-9ed9103d214b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-29e74e-85664db6-xhbmm\" (UID: \"2a84eecc-cf96-48b6-87c4-9ed9103d214b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-29e74e-85664db6-xhbmm" Sep 14 11:15:43.531330 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:43.531197 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd9wg\" (UniqueName: \"kubernetes.io/projected/2a84eecc-cf96-48b6-87c4-9ed9103d214b-kube-api-access-rd9wg\") pod \"payload-pre-processing-e2e-api-a-29e74e-85664db6-xhbmm\" (UID: \"2a84eecc-cf96-48b6-87c4-9ed9103d214b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-29e74e-85664db6-xhbmm" Sep 14 11:15:43.531330 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:43.531228 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktzfr\" (UniqueName: \"kubernetes.io/projected/967e30fc-237b-450e-baf2-c6c4990c460f-kube-api-access-ktzfr\") pod \"payload-processing-e2e-api-a-29e74e-6fb8fc5c5-bk6nv\" (UID: \"967e30fc-237b-450e-baf2-c6c4990c460f\") " pod="openshift-ingress/payload-processing-e2e-api-a-29e74e-6fb8fc5c5-bk6nv" Sep 14 11:15:43.531781 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:43.531763 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a84eecc-cf96-48b6-87c4-9ed9103d214b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-29e74e-85664db6-xhbmm\" (UID: \"2a84eecc-cf96-48b6-87c4-9ed9103d214b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-29e74e-85664db6-xhbmm" Sep 14 11:15:43.544388 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:43.544361 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd9wg\" (UniqueName: \"kubernetes.io/projected/2a84eecc-cf96-48b6-87c4-9ed9103d214b-kube-api-access-rd9wg\") pod \"payload-pre-processing-e2e-api-a-29e74e-85664db6-xhbmm\" (UID: \"2a84eecc-cf96-48b6-87c4-9ed9103d214b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-29e74e-85664db6-xhbmm" Sep 14 11:15:43.631838 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:43.631748 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktzfr\" (UniqueName: \"kubernetes.io/projected/967e30fc-237b-450e-baf2-c6c4990c460f-kube-api-access-ktzfr\") pod \"payload-processing-e2e-api-a-29e74e-6fb8fc5c5-bk6nv\" (UID: \"967e30fc-237b-450e-baf2-c6c4990c460f\") " pod="openshift-ingress/payload-processing-e2e-api-a-29e74e-6fb8fc5c5-bk6nv" Sep 14 11:15:43.631838 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:43.631801 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/967e30fc-237b-450e-baf2-c6c4990c460f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-29e74e-6fb8fc5c5-bk6nv\" (UID: \"967e30fc-237b-450e-baf2-c6c4990c460f\") " pod="openshift-ingress/payload-processing-e2e-api-a-29e74e-6fb8fc5c5-bk6nv" Sep 14 11:15:43.632450 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:43.632434 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/967e30fc-237b-450e-baf2-c6c4990c460f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-29e74e-6fb8fc5c5-bk6nv\" (UID: \"967e30fc-237b-450e-baf2-c6c4990c460f\") " pod="openshift-ingress/payload-processing-e2e-api-a-29e74e-6fb8fc5c5-bk6nv" Sep 14 11:15:43.651317 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:43.651287 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktzfr\" (UniqueName: \"kubernetes.io/projected/967e30fc-237b-450e-baf2-c6c4990c460f-kube-api-access-ktzfr\") pod \"payload-processing-e2e-api-a-29e74e-6fb8fc5c5-bk6nv\" (UID: \"967e30fc-237b-450e-baf2-c6c4990c460f\") " pod="openshift-ingress/payload-processing-e2e-api-a-29e74e-6fb8fc5c5-bk6nv" Sep 14 11:15:43.684781 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:43.684747 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29e74e-85664db6-xhbmm" Sep 14 11:15:43.759657 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:43.759626 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-29e74e-6fb8fc5c5-bk6nv" Sep 14 11:15:43.969443 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:43.969353 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-29e74e-85664db6-xhbmm"] Sep 14 11:15:43.975540 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:15:43.975517 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a84eecc_cf96_48b6_87c4_9ed9103d214b.slice/crio-9021542f5aa75b2e0925f14bc87bbab60782b35590c9b54956faf6bb6eb25418 WatchSource:0}: Error finding container 9021542f5aa75b2e0925f14bc87bbab60782b35590c9b54956faf6bb6eb25418: Status 404 returned error can't find the container with id 9021542f5aa75b2e0925f14bc87bbab60782b35590c9b54956faf6bb6eb25418 Sep 14 11:15:44.060734 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:44.060704 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-29e74e-6fb8fc5c5-bk6nv"] Sep 14 11:15:44.063023 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:15:44.062996 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod967e30fc_237b_450e_baf2_c6c4990c460f.slice/crio-bd1730f459817ccde5c4916dd428705d49dacc0f6e3c13cd7cdcf174f42da1b5 WatchSource:0}: Error finding container bd1730f459817ccde5c4916dd428705d49dacc0f6e3c13cd7cdcf174f42da1b5: Status 404 returned error can't find the container with id bd1730f459817ccde5c4916dd428705d49dacc0f6e3c13cd7cdcf174f42da1b5 Sep 14 11:15:44.066251 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:44.066224 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29e74e-85664db6-xhbmm" event={"ID":"2a84eecc-cf96-48b6-87c4-9ed9103d214b","Type":"ContainerStarted","Data":"9a8ac1e85f268cf706f0a990fd4ae89aeaba557b05a9a305f324981cdc840727"} Sep 14 11:15:44.066385 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:44.066275 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29e74e-85664db6-xhbmm" event={"ID":"2a84eecc-cf96-48b6-87c4-9ed9103d214b","Type":"ContainerStarted","Data":"9021542f5aa75b2e0925f14bc87bbab60782b35590c9b54956faf6bb6eb25418"} Sep 14 11:15:44.066457 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:44.066397 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29e74e-85664db6-xhbmm" Sep 14 11:15:44.094471 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:44.094415 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29e74e-85664db6-xhbmm" podStartSLOduration=1.094395304 podStartE2EDuration="1.094395304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:15:44.092673481 +0000 UTC m=+1617.835075452" watchObservedRunningTime="2026-09-14 11:15:44.094395304 +0000 UTC m=+1617.836797275" Sep 14 11:15:45.073109 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:45.073067 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-29e74e-6fb8fc5c5-bk6nv" event={"ID":"967e30fc-237b-450e-baf2-c6c4990c460f","Type":"ContainerStarted","Data":"b50405f452c91930f2fda75a59bf2def69448ccaa9a65ddc922bd1c26f2a03c9"} Sep 14 11:15:45.073639 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:45.073134 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-29e74e-6fb8fc5c5-bk6nv" event={"ID":"967e30fc-237b-450e-baf2-c6c4990c460f","Type":"ContainerStarted","Data":"bd1730f459817ccde5c4916dd428705d49dacc0f6e3c13cd7cdcf174f42da1b5"} Sep 14 11:15:45.073639 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:45.073315 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-29e74e-6fb8fc5c5-bk6nv" Sep 14 11:15:45.097921 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:45.097869 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-29e74e-6fb8fc5c5-bk6nv" podStartSLOduration=2.097854079 podStartE2EDuration="2.097854079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:15:45.095490355 +0000 UTC m=+1618.837892320" watchObservedRunningTime="2026-09-14 11:15:45.097854079 +0000 UTC m=+1618.840256049" Sep 14 11:15:47.016235 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:47.016197 2581 scope.go:117] "RemoveContainer" containerID="ed78c94bf81180914f73798d534ad5f6e2ca7e0131f5afd525c64a5f5973fca6" Sep 14 11:15:53.324867 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:53.324807 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4ebf73/distinct-2-w4-4ebf73-kserve-7584f57865-l9888" podUID="75d16066-49c1-4a32-b80b-ca4634eb0fb2" containerName="main" containerID="cri-o://22454accd77a6aa481849496786062adc3bdd1903f00da80cf92b8dd67ad1f5d" gracePeriod=45 Sep 14 11:15:53.332444 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:53.332410 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4ebf73/distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks" podUID="55d476ad-c696-4d47-b3c1-d3cd01d5a658" containerName="main" containerID="cri-o://23cc0d1518d40278a85e7f162e3f2ddb3607c2574d477bf8763a983a2cb8d8d2" gracePeriod=45 Sep 14 11:15:53.597021 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:53.596990 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ebf73/distinct-2-w4-4ebf73-kserve-7584f57865-l9888" Sep 14 11:15:53.621671 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:53.621646 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ebf73/distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks" Sep 14 11:15:53.627617 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:53.627587 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75d16066-49c1-4a32-b80b-ca4634eb0fb2-tls-certs\") pod \"75d16066-49c1-4a32-b80b-ca4634eb0fb2\" (UID: \"75d16066-49c1-4a32-b80b-ca4634eb0fb2\") " Sep 14 11:15:53.627814 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:53.627641 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75d16066-49c1-4a32-b80b-ca4634eb0fb2-model-cache\") pod \"75d16066-49c1-4a32-b80b-ca4634eb0fb2\" (UID: \"75d16066-49c1-4a32-b80b-ca4634eb0fb2\") " Sep 14 11:15:53.627865 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:53.627808 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75d16066-49c1-4a32-b80b-ca4634eb0fb2-tmp-dir\") pod \"75d16066-49c1-4a32-b80b-ca4634eb0fb2\" (UID: \"75d16066-49c1-4a32-b80b-ca4634eb0fb2\") " Sep 14 11:15:53.627903 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:53.627865 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75d16066-49c1-4a32-b80b-ca4634eb0fb2-model-cache" (OuterVolumeSpecName: "model-cache") pod "75d16066-49c1-4a32-b80b-ca4634eb0fb2" (UID: "75d16066-49c1-4a32-b80b-ca4634eb0fb2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:15:53.627903 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:53.627873 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rlpzw\" (UniqueName: \"kubernetes.io/projected/75d16066-49c1-4a32-b80b-ca4634eb0fb2-kube-api-access-rlpzw\") pod \"75d16066-49c1-4a32-b80b-ca4634eb0fb2\" (UID: \"75d16066-49c1-4a32-b80b-ca4634eb0fb2\") " Sep 14 11:15:53.627982 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:53.627903 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75d16066-49c1-4a32-b80b-ca4634eb0fb2-dshm\") pod \"75d16066-49c1-4a32-b80b-ca4634eb0fb2\" (UID: \"75d16066-49c1-4a32-b80b-ca4634eb0fb2\") " Sep 14 11:15:53.628041 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:53.627999 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75d16066-49c1-4a32-b80b-ca4634eb0fb2-home\") pod \"75d16066-49c1-4a32-b80b-ca4634eb0fb2\" (UID: \"75d16066-49c1-4a32-b80b-ca4634eb0fb2\") " Sep 14 11:15:53.628095 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:53.628040 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75d16066-49c1-4a32-b80b-ca4634eb0fb2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "75d16066-49c1-4a32-b80b-ca4634eb0fb2" (UID: "75d16066-49c1-4a32-b80b-ca4634eb0fb2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:15:53.628470 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:53.628410 2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75d16066-49c1-4a32-b80b-ca4634eb0fb2-model-cache\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:15:53.628470 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:53.628431 2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75d16066-49c1-4a32-b80b-ca4634eb0fb2-tmp-dir\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:15:53.628705 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:53.628677 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75d16066-49c1-4a32-b80b-ca4634eb0fb2-home" (OuterVolumeSpecName: "home") pod "75d16066-49c1-4a32-b80b-ca4634eb0fb2" (UID: "75d16066-49c1-4a32-b80b-ca4634eb0fb2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:15:53.630370 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:53.630323 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75d16066-49c1-4a32-b80b-ca4634eb0fb2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "75d16066-49c1-4a32-b80b-ca4634eb0fb2" (UID: "75d16066-49c1-4a32-b80b-ca4634eb0fb2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:15:53.630633 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:53.630613 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75d16066-49c1-4a32-b80b-ca4634eb0fb2-dshm" (OuterVolumeSpecName: "dshm") pod "75d16066-49c1-4a32-b80b-ca4634eb0fb2" (UID: "75d16066-49c1-4a32-b80b-ca4634eb0fb2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:15:53.630723 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:53.630642 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75d16066-49c1-4a32-b80b-ca4634eb0fb2-kube-api-access-rlpzw" (OuterVolumeSpecName: "kube-api-access-rlpzw") pod "75d16066-49c1-4a32-b80b-ca4634eb0fb2" (UID: "75d16066-49c1-4a32-b80b-ca4634eb0fb2"). InnerVolumeSpecName "kube-api-access-rlpzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:15:53.729417 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:53.729386 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55d476ad-c696-4d47-b3c1-d3cd01d5a658-model-cache\") pod \"55d476ad-c696-4d47-b3c1-d3cd01d5a658\" (UID: \"55d476ad-c696-4d47-b3c1-d3cd01d5a658\") " Sep 14 11:15:53.729417 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:53.729423 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55d476ad-c696-4d47-b3c1-d3cd01d5a658-dshm\") pod \"55d476ad-c696-4d47-b3c1-d3cd01d5a658\" (UID: \"55d476ad-c696-4d47-b3c1-d3cd01d5a658\") " Sep 14 11:15:53.729654 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:53.729538 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55d476ad-c696-4d47-b3c1-d3cd01d5a658-tls-certs\") pod \"55d476ad-c696-4d47-b3c1-d3cd01d5a658\" (UID: \"55d476ad-c696-4d47-b3c1-d3cd01d5a658\") " Sep 14 11:15:53.729654 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:53.729593 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55d476ad-c696-4d47-b3c1-d3cd01d5a658-home\") pod \"55d476ad-c696-4d47-b3c1-d3cd01d5a658\" (UID: \"55d476ad-c696-4d47-b3c1-d3cd01d5a658\") " Sep 14 11:15:53.729654 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:53.729635 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55d476ad-c696-4d47-b3c1-d3cd01d5a658-tmp-dir\") pod \"55d476ad-c696-4d47-b3c1-d3cd01d5a658\" (UID: \"55d476ad-c696-4d47-b3c1-d3cd01d5a658\") " Sep 14 11:15:53.729810 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:53.729665 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgsl2\" (UniqueName: \"kubernetes.io/projected/55d476ad-c696-4d47-b3c1-d3cd01d5a658-kube-api-access-pgsl2\") pod \"55d476ad-c696-4d47-b3c1-d3cd01d5a658\" (UID: \"55d476ad-c696-4d47-b3c1-d3cd01d5a658\") " Sep 14 11:15:53.729931 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:53.729658 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55d476ad-c696-4d47-b3c1-d3cd01d5a658-model-cache" (OuterVolumeSpecName: "model-cache") pod "55d476ad-c696-4d47-b3c1-d3cd01d5a658" (UID: "55d476ad-c696-4d47-b3c1-d3cd01d5a658"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:15:53.729931 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:53.729863 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55d476ad-c696-4d47-b3c1-d3cd01d5a658-home" (OuterVolumeSpecName: "home") pod "55d476ad-c696-4d47-b3c1-d3cd01d5a658" (UID: "55d476ad-c696-4d47-b3c1-d3cd01d5a658"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:15:53.729931 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:53.729873 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55d476ad-c696-4d47-b3c1-d3cd01d5a658-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "55d476ad-c696-4d47-b3c1-d3cd01d5a658" (UID: "55d476ad-c696-4d47-b3c1-d3cd01d5a658"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:15:53.730197 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:53.730066 2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75d16066-49c1-4a32-b80b-ca4634eb0fb2-home\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:15:53.730197 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:53.730085 2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55d476ad-c696-4d47-b3c1-d3cd01d5a658-home\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:15:53.730197 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:53.730097 2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55d476ad-c696-4d47-b3c1-d3cd01d5a658-tmp-dir\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:15:53.730197 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:53.730109 2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75d16066-49c1-4a32-b80b-ca4634eb0fb2-tls-certs\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:15:53.730197 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:53.730125 2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55d476ad-c696-4d47-b3c1-d3cd01d5a658-model-cache\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:15:53.730197 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:53.730138 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlpzw\" (UniqueName: \"kubernetes.io/projected/75d16066-49c1-4a32-b80b-ca4634eb0fb2-kube-api-access-rlpzw\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:15:53.730197 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:53.730151 2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75d16066-49c1-4a32-b80b-ca4634eb0fb2-dshm\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:15:53.731792 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:53.731770 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55d476ad-c696-4d47-b3c1-d3cd01d5a658-dshm" (OuterVolumeSpecName: "dshm") pod "55d476ad-c696-4d47-b3c1-d3cd01d5a658" (UID: "55d476ad-c696-4d47-b3c1-d3cd01d5a658"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:15:53.731871 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:53.731785 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55d476ad-c696-4d47-b3c1-d3cd01d5a658-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "55d476ad-c696-4d47-b3c1-d3cd01d5a658" (UID: "55d476ad-c696-4d47-b3c1-d3cd01d5a658"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:15:53.731871 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:53.731864 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55d476ad-c696-4d47-b3c1-d3cd01d5a658-kube-api-access-pgsl2" (OuterVolumeSpecName: "kube-api-access-pgsl2") pod "55d476ad-c696-4d47-b3c1-d3cd01d5a658" (UID: "55d476ad-c696-4d47-b3c1-d3cd01d5a658"). InnerVolumeSpecName "kube-api-access-pgsl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:15:53.831189 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:53.831109 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgsl2\" (UniqueName: \"kubernetes.io/projected/55d476ad-c696-4d47-b3c1-d3cd01d5a658-kube-api-access-pgsl2\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:15:53.831189 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:53.831137 2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55d476ad-c696-4d47-b3c1-d3cd01d5a658-dshm\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:15:53.831189 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:53.831148 2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55d476ad-c696-4d47-b3c1-d3cd01d5a658-tls-certs\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:15:54.125954 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:54.125853 2581 generic.go:356] "Generic (PLEG): container finished" podID="55d476ad-c696-4d47-b3c1-d3cd01d5a658" containerID="23cc0d1518d40278a85e7f162e3f2ddb3607c2574d477bf8763a983a2cb8d8d2" exitCode=0 Sep 14 11:15:54.126119 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:54.125969 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ebf73/distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks" Sep 14 11:15:54.126119 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:54.125975 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ebf73/distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks" event={"ID":"55d476ad-c696-4d47-b3c1-d3cd01d5a658","Type":"ContainerDied","Data":"23cc0d1518d40278a85e7f162e3f2ddb3607c2574d477bf8763a983a2cb8d8d2"} Sep 14 11:15:54.126119 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:54.126021 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ebf73/distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks" event={"ID":"55d476ad-c696-4d47-b3c1-d3cd01d5a658","Type":"ContainerDied","Data":"3f3b0a80c8768f736ac609e24be20b274ff8141fc3904d4c76f5e0d04047c568"} Sep 14 11:15:54.126119 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:54.126048 2581 scope.go:117] "RemoveContainer" containerID="23cc0d1518d40278a85e7f162e3f2ddb3607c2574d477bf8763a983a2cb8d8d2" Sep 14 11:15:54.127732 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:54.127711 2581 generic.go:356] "Generic (PLEG): container finished" podID="75d16066-49c1-4a32-b80b-ca4634eb0fb2" containerID="22454accd77a6aa481849496786062adc3bdd1903f00da80cf92b8dd67ad1f5d" exitCode=0 Sep 14 11:15:54.127854 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:54.127812 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ebf73/distinct-2-w4-4ebf73-kserve-7584f57865-l9888" Sep 14 11:15:54.127918 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:54.127863 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ebf73/distinct-2-w4-4ebf73-kserve-7584f57865-l9888" event={"ID":"75d16066-49c1-4a32-b80b-ca4634eb0fb2","Type":"ContainerDied","Data":"22454accd77a6aa481849496786062adc3bdd1903f00da80cf92b8dd67ad1f5d"} Sep 14 11:15:54.127918 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:54.127894 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ebf73/distinct-2-w4-4ebf73-kserve-7584f57865-l9888" event={"ID":"75d16066-49c1-4a32-b80b-ca4634eb0fb2","Type":"ContainerDied","Data":"e4707521ece3bd4019204362979dadc39a23ffc535f66f6f3ab0869861a4a0a4"} Sep 14 11:15:54.136632 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:54.136606 2581 scope.go:117] "RemoveContainer" containerID="23cc0d1518d40278a85e7f162e3f2ddb3607c2574d477bf8763a983a2cb8d8d2" Sep 14 11:15:54.136965 ip-10-0-137-53 kubenswrapper[2581]: E0914 11:15:54.136946 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23cc0d1518d40278a85e7f162e3f2ddb3607c2574d477bf8763a983a2cb8d8d2\": container with ID starting with 23cc0d1518d40278a85e7f162e3f2ddb3607c2574d477bf8763a983a2cb8d8d2 not found: ID does not exist" containerID="23cc0d1518d40278a85e7f162e3f2ddb3607c2574d477bf8763a983a2cb8d8d2" Sep 14 11:15:54.137037 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:54.136979 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23cc0d1518d40278a85e7f162e3f2ddb3607c2574d477bf8763a983a2cb8d8d2"} err="failed to get container status \"23cc0d1518d40278a85e7f162e3f2ddb3607c2574d477bf8763a983a2cb8d8d2\": rpc error: code = NotFound desc = could not find container \"23cc0d1518d40278a85e7f162e3f2ddb3607c2574d477bf8763a983a2cb8d8d2\": container with ID starting with 23cc0d1518d40278a85e7f162e3f2ddb3607c2574d477bf8763a983a2cb8d8d2 not found: ID does not exist" Sep 14 11:15:54.137037 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:54.137005 2581 scope.go:117] "RemoveContainer" containerID="22454accd77a6aa481849496786062adc3bdd1903f00da80cf92b8dd67ad1f5d" Sep 14 11:15:54.145919 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:54.145900 2581 scope.go:117] "RemoveContainer" containerID="22454accd77a6aa481849496786062adc3bdd1903f00da80cf92b8dd67ad1f5d" Sep 14 11:15:54.146254 ip-10-0-137-53 kubenswrapper[2581]: E0914 11:15:54.146237 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22454accd77a6aa481849496786062adc3bdd1903f00da80cf92b8dd67ad1f5d\": container with ID starting with 22454accd77a6aa481849496786062adc3bdd1903f00da80cf92b8dd67ad1f5d not found: ID does not exist" containerID="22454accd77a6aa481849496786062adc3bdd1903f00da80cf92b8dd67ad1f5d" Sep 14 11:15:54.146296 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:54.146264 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22454accd77a6aa481849496786062adc3bdd1903f00da80cf92b8dd67ad1f5d"} err="failed to get container status \"22454accd77a6aa481849496786062adc3bdd1903f00da80cf92b8dd67ad1f5d\": rpc error: code = NotFound desc = could not find container \"22454accd77a6aa481849496786062adc3bdd1903f00da80cf92b8dd67ad1f5d\": container with ID starting with 22454accd77a6aa481849496786062adc3bdd1903f00da80cf92b8dd67ad1f5d not found: ID does not exist" Sep 14 11:15:54.174010 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:54.173985 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4ebf73/distinct-2-w4-4ebf73-kserve-7584f57865-l9888"] Sep 14 11:15:54.177351 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:54.177308 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4ebf73/distinct-2-w4-4ebf73-kserve-7584f57865-l9888"] Sep 14 11:15:54.189804 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:54.189781 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4ebf73/distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks"] Sep 14 11:15:54.196601 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:54.196580 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4ebf73/distinct-w4-4ebf73-kserve-7bcffb6b58-xt2ks"] Sep 14 11:15:54.832570 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:54.832538 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55d476ad-c696-4d47-b3c1-d3cd01d5a658" path="/var/lib/kubelet/pods/55d476ad-c696-4d47-b3c1-d3cd01d5a658/volumes" Sep 14 11:15:54.832986 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:54.832930 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75d16066-49c1-4a32-b80b-ca4634eb0fb2" path="/var/lib/kubelet/pods/75d16066-49c1-4a32-b80b-ca4634eb0fb2/volumes" Sep 14 11:15:55.075170 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:55.075138 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29e74e-85664db6-xhbmm" Sep 14 11:15:56.081539 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:56.081510 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-29e74e-6fb8fc5c5-bk6nv" Sep 14 11:15:57.771673 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:57.771639 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ebf73-29823075-p5cnj"] Sep 14 11:15:57.785497 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:57.785473 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ebf73-29823075-p5cnj"] Sep 14 11:15:57.852284 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:57.852241 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4ebf73-9687f9869-zpnm6"] Sep 14 11:15:57.852657 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:57.852622 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-4ebf73-9687f9869-zpnm6" podUID="c9017a38-f02d-4997-98ef-078dd6c49977" containerName="payload-processing" containerID="cri-o://78c11751d331f022030a16bce3e7d4a42096acc784207baaa24bf211803ac241" gracePeriod=30 Sep 14 11:15:57.855671 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:57.855638 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4ebf73-5567b59786-mzx5m"] Sep 14 11:15:57.855923 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:57.855890 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ebf73-5567b59786-mzx5m" podUID="3e4b7137-fa3a-43d9-aeed-fffd3fda99a3" containerName="payload-pre-processing" containerID="cri-o://24eac3aa3b20b2c94401a26fbe18f3d5bf5c0dbd2722cdcac67aad16a0d1b3f3" gracePeriod=30 Sep 14 11:15:58.111444 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:58.111410 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-4ebf73-9687f9869-zpnm6" Sep 14 11:15:58.140477 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:58.140449 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ebf73-5567b59786-mzx5m" Sep 14 11:15:58.154628 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:58.154595 2581 generic.go:356] "Generic (PLEG): container finished" podID="c9017a38-f02d-4997-98ef-078dd6c49977" containerID="78c11751d331f022030a16bce3e7d4a42096acc784207baaa24bf211803ac241" exitCode=0 Sep 14 11:15:58.154804 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:58.154657 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-4ebf73-9687f9869-zpnm6" Sep 14 11:15:58.154804 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:58.154714 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-4ebf73-9687f9869-zpnm6" event={"ID":"c9017a38-f02d-4997-98ef-078dd6c49977","Type":"ContainerDied","Data":"78c11751d331f022030a16bce3e7d4a42096acc784207baaa24bf211803ac241"} Sep 14 11:15:58.154804 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:58.154772 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-4ebf73-9687f9869-zpnm6" event={"ID":"c9017a38-f02d-4997-98ef-078dd6c49977","Type":"ContainerDied","Data":"e3feacdb50851f744bd07525e1ca574d71700d32f2204a596bc9340a8dedd58c"} Sep 14 11:15:58.154804 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:58.154800 2581 scope.go:117] "RemoveContainer" containerID="78c11751d331f022030a16bce3e7d4a42096acc784207baaa24bf211803ac241" Sep 14 11:15:58.156854 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:58.156827 2581 generic.go:356] "Generic (PLEG): container finished" podID="3e4b7137-fa3a-43d9-aeed-fffd3fda99a3" containerID="24eac3aa3b20b2c94401a26fbe18f3d5bf5c0dbd2722cdcac67aad16a0d1b3f3" exitCode=0 Sep 14 11:15:58.156990 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:58.156869 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ebf73-5567b59786-mzx5m" event={"ID":"3e4b7137-fa3a-43d9-aeed-fffd3fda99a3","Type":"ContainerDied","Data":"24eac3aa3b20b2c94401a26fbe18f3d5bf5c0dbd2722cdcac67aad16a0d1b3f3"} Sep 14 11:15:58.156990 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:58.156880 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ebf73-5567b59786-mzx5m" Sep 14 11:15:58.156990 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:58.156911 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4ebf73-5567b59786-mzx5m" event={"ID":"3e4b7137-fa3a-43d9-aeed-fffd3fda99a3","Type":"ContainerDied","Data":"20e8fba5688c081e4d6874635c8595de6fb66e2799289d9cc99d7a921081a196"} Sep 14 11:15:58.157692 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:58.157670 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ebf73-vzmcr"] Sep 14 11:15:58.165240 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:58.165223 2581 scope.go:117] "RemoveContainer" containerID="78c11751d331f022030a16bce3e7d4a42096acc784207baaa24bf211803ac241" Sep 14 11:15:58.165728 ip-10-0-137-53 kubenswrapper[2581]: E0914 11:15:58.165703 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78c11751d331f022030a16bce3e7d4a42096acc784207baaa24bf211803ac241\": container with ID starting with 78c11751d331f022030a16bce3e7d4a42096acc784207baaa24bf211803ac241 not found: ID does not exist" containerID="78c11751d331f022030a16bce3e7d4a42096acc784207baaa24bf211803ac241" Sep 14 11:15:58.165822 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:58.165739 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78c11751d331f022030a16bce3e7d4a42096acc784207baaa24bf211803ac241"} err="failed to get container status \"78c11751d331f022030a16bce3e7d4a42096acc784207baaa24bf211803ac241\": rpc error: code = NotFound desc = could not find container \"78c11751d331f022030a16bce3e7d4a42096acc784207baaa24bf211803ac241\": container with ID starting with 78c11751d331f022030a16bce3e7d4a42096acc784207baaa24bf211803ac241 not found: ID does not exist" Sep 14 11:15:58.165822 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:58.165763 2581 scope.go:117] "RemoveContainer" containerID="24eac3aa3b20b2c94401a26fbe18f3d5bf5c0dbd2722cdcac67aad16a0d1b3f3" Sep 14 11:15:58.167409 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:58.167388 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ebf73-vzmcr"] Sep 14 11:15:58.171708 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:58.171684 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9017a38-f02d-4997-98ef-078dd6c49977-plugins-config-volume\") pod \"c9017a38-f02d-4997-98ef-078dd6c49977\" (UID: \"c9017a38-f02d-4997-98ef-078dd6c49977\") " Sep 14 11:15:58.171859 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:58.171795 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8b42x\" (UniqueName: \"kubernetes.io/projected/c9017a38-f02d-4997-98ef-078dd6c49977-kube-api-access-8b42x\") pod \"c9017a38-f02d-4997-98ef-078dd6c49977\" (UID: \"c9017a38-f02d-4997-98ef-078dd6c49977\") " Sep 14 11:15:58.174524 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:58.172422 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9017a38-f02d-4997-98ef-078dd6c49977-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c9017a38-f02d-4997-98ef-078dd6c49977" (UID: "c9017a38-f02d-4997-98ef-078dd6c49977"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:15:58.174524 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:58.174447 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9017a38-f02d-4997-98ef-078dd6c49977-kube-api-access-8b42x" (OuterVolumeSpecName: "kube-api-access-8b42x") pod "c9017a38-f02d-4997-98ef-078dd6c49977" (UID: "c9017a38-f02d-4997-98ef-078dd6c49977"). InnerVolumeSpecName "kube-api-access-8b42x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:15:58.178409 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:58.178391 2581 scope.go:117] "RemoveContainer" containerID="24eac3aa3b20b2c94401a26fbe18f3d5bf5c0dbd2722cdcac67aad16a0d1b3f3" Sep 14 11:15:58.178759 ip-10-0-137-53 kubenswrapper[2581]: E0914 11:15:58.178733 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24eac3aa3b20b2c94401a26fbe18f3d5bf5c0dbd2722cdcac67aad16a0d1b3f3\": container with ID starting with 24eac3aa3b20b2c94401a26fbe18f3d5bf5c0dbd2722cdcac67aad16a0d1b3f3 not found: ID does not exist" containerID="24eac3aa3b20b2c94401a26fbe18f3d5bf5c0dbd2722cdcac67aad16a0d1b3f3" Sep 14 11:15:58.178833 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:58.178767 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24eac3aa3b20b2c94401a26fbe18f3d5bf5c0dbd2722cdcac67aad16a0d1b3f3"} err="failed to get container status \"24eac3aa3b20b2c94401a26fbe18f3d5bf5c0dbd2722cdcac67aad16a0d1b3f3\": rpc error: code = NotFound desc = could not find container \"24eac3aa3b20b2c94401a26fbe18f3d5bf5c0dbd2722cdcac67aad16a0d1b3f3\": container with ID starting with 24eac3aa3b20b2c94401a26fbe18f3d5bf5c0dbd2722cdcac67aad16a0d1b3f3 not found: ID does not exist" Sep 14 11:15:58.273229 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:58.273132 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e4b7137-fa3a-43d9-aeed-fffd3fda99a3-plugins-config-volume\") pod \"3e4b7137-fa3a-43d9-aeed-fffd3fda99a3\" (UID: \"3e4b7137-fa3a-43d9-aeed-fffd3fda99a3\") " Sep 14 11:15:58.273438 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:58.273312 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzvd7\" (UniqueName: \"kubernetes.io/projected/3e4b7137-fa3a-43d9-aeed-fffd3fda99a3-kube-api-access-zzvd7\") pod \"3e4b7137-fa3a-43d9-aeed-fffd3fda99a3\" (UID: \"3e4b7137-fa3a-43d9-aeed-fffd3fda99a3\") " Sep 14 11:15:58.273625 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:58.273600 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e4b7137-fa3a-43d9-aeed-fffd3fda99a3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3e4b7137-fa3a-43d9-aeed-fffd3fda99a3" (UID: "3e4b7137-fa3a-43d9-aeed-fffd3fda99a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:15:58.273686 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:58.273634 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9017a38-f02d-4997-98ef-078dd6c49977-plugins-config-volume\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:15:58.273686 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:58.273654 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b42x\" (UniqueName: \"kubernetes.io/projected/c9017a38-f02d-4997-98ef-078dd6c49977-kube-api-access-8b42x\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:15:58.275565 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:58.275543 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e4b7137-fa3a-43d9-aeed-fffd3fda99a3-kube-api-access-zzvd7" (OuterVolumeSpecName: "kube-api-access-zzvd7") pod "3e4b7137-fa3a-43d9-aeed-fffd3fda99a3" (UID: "3e4b7137-fa3a-43d9-aeed-fffd3fda99a3"). InnerVolumeSpecName "kube-api-access-zzvd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:15:58.374208 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:58.374172 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzvd7\" (UniqueName: \"kubernetes.io/projected/3e4b7137-fa3a-43d9-aeed-fffd3fda99a3-kube-api-access-zzvd7\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:15:58.374208 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:58.374204 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e4b7137-fa3a-43d9-aeed-fffd3fda99a3-plugins-config-volume\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:15:58.495920 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:58.495883 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4ebf73-9687f9869-zpnm6"] Sep 14 11:15:58.498641 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:58.498612 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4ebf73-9687f9869-zpnm6"] Sep 14 11:15:58.519976 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:58.519948 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4ebf73-5567b59786-mzx5m"] Sep 14 11:15:58.527198 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:58.527132 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4ebf73-5567b59786-mzx5m"] Sep 14 11:15:58.832062 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:58.831975 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37971749-8dbe-4d9e-b348-1a9a33c7d3a9" path="/var/lib/kubelet/pods/37971749-8dbe-4d9e-b348-1a9a33c7d3a9/volumes" Sep 14 11:15:58.832442 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:58.832332 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e4b7137-fa3a-43d9-aeed-fffd3fda99a3" path="/var/lib/kubelet/pods/3e4b7137-fa3a-43d9-aeed-fffd3fda99a3/volumes" Sep 14 11:15:58.832600 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:58.832587 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41401723-4af0-421d-ace9-a8d3353f39f6" path="/var/lib/kubelet/pods/41401723-4af0-421d-ace9-a8d3353f39f6/volumes" Sep 14 11:15:58.832887 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:15:58.832875 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9017a38-f02d-4997-98ef-078dd6c49977" path="/var/lib/kubelet/pods/c9017a38-f02d-4997-98ef-078dd6c49977/volumes" Sep 14 11:16:08.087647 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:08.087617 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f8e582-58b77c449c-thb87"] Sep 14 11:16:08.088189 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:08.088029 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e4b7137-fa3a-43d9-aeed-fffd3fda99a3" containerName="payload-pre-processing" Sep 14 11:16:08.088189 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:08.088040 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e4b7137-fa3a-43d9-aeed-fffd3fda99a3" containerName="payload-pre-processing" Sep 14 11:16:08.088189 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:08.088050 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9017a38-f02d-4997-98ef-078dd6c49977" containerName="payload-processing" Sep 14 11:16:08.088189 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:08.088055 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9017a38-f02d-4997-98ef-078dd6c49977" containerName="payload-processing" Sep 14 11:16:08.088189 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:08.088067 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55d476ad-c696-4d47-b3c1-d3cd01d5a658" containerName="main" Sep 14 11:16:08.088189 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:08.088073 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="55d476ad-c696-4d47-b3c1-d3cd01d5a658" containerName="main" Sep 14 11:16:08.088189 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:08.088079 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75d16066-49c1-4a32-b80b-ca4634eb0fb2" containerName="main" Sep 14 11:16:08.088189 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:08.088083 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="75d16066-49c1-4a32-b80b-ca4634eb0fb2" containerName="main" Sep 14 11:16:08.088189 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:08.088163 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9017a38-f02d-4997-98ef-078dd6c49977" containerName="payload-processing" Sep 14 11:16:08.088189 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:08.088173 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="55d476ad-c696-4d47-b3c1-d3cd01d5a658" containerName="main" Sep 14 11:16:08.088189 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:08.088178 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e4b7137-fa3a-43d9-aeed-fffd3fda99a3" containerName="payload-pre-processing" Sep 14 11:16:08.088189 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:08.088185 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="75d16066-49c1-4a32-b80b-ca4634eb0fb2" containerName="main" Sep 14 11:16:08.092959 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:08.092939 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f8e582-58b77c449c-thb87" Sep 14 11:16:08.097926 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:08.097901 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-f8e582\"" Sep 14 11:16:08.098059 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:08.098004 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-f8e582-dockercfg-lftq5\"" Sep 14 11:16:08.124985 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:08.124957 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f8e582-58b77c449c-thb87"] Sep 14 11:16:08.176034 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:08.176004 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml6jq\" (UniqueName: \"kubernetes.io/projected/f35d29eb-0414-44eb-bd0f-ada347619cbf-kube-api-access-ml6jq\") pod \"payload-processing-e2e-api-b-f8e582-58b77c449c-thb87\" (UID: \"f35d29eb-0414-44eb-bd0f-ada347619cbf\") " pod="openshift-ingress/payload-processing-e2e-api-b-f8e582-58b77c449c-thb87" Sep 14 11:16:08.176216 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:08.176087 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f35d29eb-0414-44eb-bd0f-ada347619cbf-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f8e582-58b77c449c-thb87\" (UID: \"f35d29eb-0414-44eb-bd0f-ada347619cbf\") " pod="openshift-ingress/payload-processing-e2e-api-b-f8e582-58b77c449c-thb87" Sep 14 11:16:08.276781 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:08.276749 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml6jq\" (UniqueName: \"kubernetes.io/projected/f35d29eb-0414-44eb-bd0f-ada347619cbf-kube-api-access-ml6jq\") pod \"payload-processing-e2e-api-b-f8e582-58b77c449c-thb87\" (UID: \"f35d29eb-0414-44eb-bd0f-ada347619cbf\") " pod="openshift-ingress/payload-processing-e2e-api-b-f8e582-58b77c449c-thb87" Sep 14 11:16:08.276938 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:08.276836 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f35d29eb-0414-44eb-bd0f-ada347619cbf-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f8e582-58b77c449c-thb87\" (UID: \"f35d29eb-0414-44eb-bd0f-ada347619cbf\") " pod="openshift-ingress/payload-processing-e2e-api-b-f8e582-58b77c449c-thb87" Sep 14 11:16:08.277545 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:08.277525 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f35d29eb-0414-44eb-bd0f-ada347619cbf-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f8e582-58b77c449c-thb87\" (UID: \"f35d29eb-0414-44eb-bd0f-ada347619cbf\") " pod="openshift-ingress/payload-processing-e2e-api-b-f8e582-58b77c449c-thb87" Sep 14 11:16:08.294385 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:08.294332 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml6jq\" (UniqueName: \"kubernetes.io/projected/f35d29eb-0414-44eb-bd0f-ada347619cbf-kube-api-access-ml6jq\") pod \"payload-processing-e2e-api-b-f8e582-58b77c449c-thb87\" (UID: \"f35d29eb-0414-44eb-bd0f-ada347619cbf\") " pod="openshift-ingress/payload-processing-e2e-api-b-f8e582-58b77c449c-thb87" Sep 14 11:16:08.403575 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:08.403487 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f8e582-58b77c449c-thb87" Sep 14 11:16:08.749921 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:08.749891 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f8e582-58b77c449c-thb87"] Sep 14 11:16:08.752649 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:16:08.752625 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf35d29eb_0414_44eb_bd0f_ada347619cbf.slice/crio-ba3449d111c77b45e2264a2be4468518fb950ee0738542ef9e407ad63196fb73 WatchSource:0}: Error finding container ba3449d111c77b45e2264a2be4468518fb950ee0738542ef9e407ad63196fb73: Status 404 returned error can't find the container with id ba3449d111c77b45e2264a2be4468518fb950ee0738542ef9e407ad63196fb73 Sep 14 11:16:09.221679 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:09.221638 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f8e582-58b77c449c-thb87" event={"ID":"f35d29eb-0414-44eb-bd0f-ada347619cbf","Type":"ContainerStarted","Data":"0b1c07d0fea1d9fc123602f7fd85825779e52eb7a387ba29359458db4b2f9b05"} Sep 14 11:16:09.222091 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:09.221703 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f8e582-58b77c449c-thb87" event={"ID":"f35d29eb-0414-44eb-bd0f-ada347619cbf","Type":"ContainerStarted","Data":"ba3449d111c77b45e2264a2be4468518fb950ee0738542ef9e407ad63196fb73"} Sep 14 11:16:09.222091 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:09.221838 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-f8e582-58b77c449c-thb87" Sep 14 11:16:09.270811 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:09.270753 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-f8e582-58b77c449c-thb87" podStartSLOduration=1.270731697 podStartE2EDuration="1.270731697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:16:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:16:09.268758683 +0000 UTC m=+1643.011160655" watchObservedRunningTime="2026-09-14 11:16:09.270731697 +0000 UTC m=+1643.013133669" Sep 14 11:16:20.229700 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:20.229667 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-f8e582-58b77c449c-thb87" Sep 14 11:16:37.474107 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:37.474076 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29e74e-m5h72"] Sep 14 11:16:37.477984 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:37.477967 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29e74e-m5h72" Sep 14 11:16:37.480560 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:37.480530 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pv9g8\"" Sep 14 11:16:37.490578 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:37.490554 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29e74e-m5h72"] Sep 14 11:16:37.561962 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:37.561906 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a5b7686-38d8-444f-8579-c458987fd13b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-29e74e-m5h72\" (UID: \"9a5b7686-38d8-444f-8579-c458987fd13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29e74e-m5h72" Sep 14 11:16:37.663234 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:37.663200 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a5b7686-38d8-444f-8579-c458987fd13b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-29e74e-m5h72\" (UID: \"9a5b7686-38d8-444f-8579-c458987fd13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29e74e-m5h72" Sep 14 11:16:37.663846 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:37.663826 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a5b7686-38d8-444f-8579-c458987fd13b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-29e74e-m5h72\" (UID: \"9a5b7686-38d8-444f-8579-c458987fd13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29e74e-m5h72" Sep 14 11:16:37.788566 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:37.788530 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29e74e-m5h72" Sep 14 11:16:38.073703 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:38.073676 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29e74e-m5h72"] Sep 14 11:16:38.075993 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:16:38.075963 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a5b7686_38d8_444f_8579_c458987fd13b.slice/crio-9a89dc84d17f83963332cfd0365c4fb5c1c7a733d6c679297d4950816f1576a4 WatchSource:0}: Error finding container 9a89dc84d17f83963332cfd0365c4fb5c1c7a733d6c679297d4950816f1576a4: Status 404 returned error can't find the container with id 9a89dc84d17f83963332cfd0365c4fb5c1c7a733d6c679297d4950816f1576a4 Sep 14 11:16:38.386036 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:38.385999 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29e74e-m5h72" event={"ID":"9a5b7686-38d8-444f-8579-c458987fd13b","Type":"ContainerStarted","Data":"9a89dc84d17f83963332cfd0365c4fb5c1c7a733d6c679297d4950816f1576a4"} Sep 14 11:16:39.393821 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:39.393785 2581 generic.go:356] "Generic (PLEG): container finished" podID="9a5b7686-38d8-444f-8579-c458987fd13b" containerID="daa5b28dee3ef0e17fc1e74a6217a7965cc7fb60bd194438c853bd13335a9070" exitCode=0 Sep 14 11:16:39.394275 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:39.393849 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29e74e-m5h72" event={"ID":"9a5b7686-38d8-444f-8579-c458987fd13b","Type":"ContainerDied","Data":"daa5b28dee3ef0e17fc1e74a6217a7965cc7fb60bd194438c853bd13335a9070"} Sep 14 11:16:40.536749 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:40.536727 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29e74e-m5h72" Sep 14 11:16:40.592671 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:40.592642 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a5b7686-38d8-444f-8579-c458987fd13b-maas-api-service-ca\") pod \"9a5b7686-38d8-444f-8579-c458987fd13b\" (UID: \"9a5b7686-38d8-444f-8579-c458987fd13b\") " Sep 14 11:16:40.593009 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:40.592982 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a5b7686-38d8-444f-8579-c458987fd13b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9a5b7686-38d8-444f-8579-c458987fd13b" (UID: "9a5b7686-38d8-444f-8579-c458987fd13b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:16:40.693792 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:40.693697 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a5b7686-38d8-444f-8579-c458987fd13b-maas-api-service-ca\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:16:41.406121 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:41.406084 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29e74e-m5h72" Sep 14 11:16:41.406121 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:41.406115 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29e74e-m5h72" event={"ID":"9a5b7686-38d8-444f-8579-c458987fd13b","Type":"ContainerDied","Data":"9a89dc84d17f83963332cfd0365c4fb5c1c7a733d6c679297d4950816f1576a4"} Sep 14 11:16:41.406365 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:41.406158 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a89dc84d17f83963332cfd0365c4fb5c1c7a733d6c679297d4950816f1576a4" Sep 14 11:16:47.058678 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:47.058640 2581 scope.go:117] "RemoveContainer" containerID="e964e4d87052e191a142d7052e8fe3c0093d76a5e9405e608aae1eb4098de7e8" Sep 14 11:16:47.067737 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:47.067715 2581 scope.go:117] "RemoveContainer" containerID="645c753bcb7d31fe353316c7fc845433d08c779c4a8e57201bf3c22d0fb0058f" Sep 14 11:16:53.301204 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:53.301167 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-29e74e-6fb8fc5c5-bk6nv"] Sep 14 11:16:53.301659 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:53.301410 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-29e74e-6fb8fc5c5-bk6nv" podUID="967e30fc-237b-450e-baf2-c6c4990c460f" containerName="payload-processing" containerID="cri-o://b50405f452c91930f2fda75a59bf2def69448ccaa9a65ddc922bd1c26f2a03c9" gracePeriod=30 Sep 14 11:16:53.313577 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:53.313548 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-29e74e-85664db6-xhbmm"] Sep 14 11:16:53.313774 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:53.313739 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29e74e-85664db6-xhbmm" podUID="2a84eecc-cf96-48b6-87c4-9ed9103d214b" containerName="payload-pre-processing" containerID="cri-o://9a8ac1e85f268cf706f0a990fd4ae89aeaba557b05a9a305f324981cdc840727" gracePeriod=30 Sep 14 11:16:53.477484 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:53.477305 2581 generic.go:356] "Generic (PLEG): container finished" podID="967e30fc-237b-450e-baf2-c6c4990c460f" containerID="b50405f452c91930f2fda75a59bf2def69448ccaa9a65ddc922bd1c26f2a03c9" exitCode=0 Sep 14 11:16:53.477484 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:53.477377 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-29e74e-6fb8fc5c5-bk6nv" event={"ID":"967e30fc-237b-450e-baf2-c6c4990c460f","Type":"ContainerDied","Data":"b50405f452c91930f2fda75a59bf2def69448ccaa9a65ddc922bd1c26f2a03c9"} Sep 14 11:16:53.479571 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:53.479424 2581 generic.go:356] "Generic (PLEG): container finished" podID="2a84eecc-cf96-48b6-87c4-9ed9103d214b" containerID="9a8ac1e85f268cf706f0a990fd4ae89aeaba557b05a9a305f324981cdc840727" exitCode=0 Sep 14 11:16:53.479912 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:53.479831 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29e74e-85664db6-xhbmm" event={"ID":"2a84eecc-cf96-48b6-87c4-9ed9103d214b","Type":"ContainerDied","Data":"9a8ac1e85f268cf706f0a990fd4ae89aeaba557b05a9a305f324981cdc840727"} Sep 14 11:16:53.564261 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:53.564199 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-29e74e-6fb8fc5c5-bk6nv" Sep 14 11:16:53.598675 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:53.598645 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29e74e-85664db6-xhbmm" Sep 14 11:16:53.606228 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:53.606200 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29e74e-m5h72"] Sep 14 11:16:53.608238 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:53.608215 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29e74e-m5h72"] Sep 14 11:16:53.728788 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:53.728752 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rd9wg\" (UniqueName: \"kubernetes.io/projected/2a84eecc-cf96-48b6-87c4-9ed9103d214b-kube-api-access-rd9wg\") pod \"2a84eecc-cf96-48b6-87c4-9ed9103d214b\" (UID: \"2a84eecc-cf96-48b6-87c4-9ed9103d214b\") " Sep 14 11:16:53.728966 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:53.728797 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a84eecc-cf96-48b6-87c4-9ed9103d214b-plugins-config-volume\") pod \"2a84eecc-cf96-48b6-87c4-9ed9103d214b\" (UID: \"2a84eecc-cf96-48b6-87c4-9ed9103d214b\") " Sep 14 11:16:53.728966 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:53.728840 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/967e30fc-237b-450e-baf2-c6c4990c460f-plugins-config-volume\") pod \"967e30fc-237b-450e-baf2-c6c4990c460f\" (UID: \"967e30fc-237b-450e-baf2-c6c4990c460f\") " Sep 14 11:16:53.728966 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:53.728893 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktzfr\" (UniqueName: \"kubernetes.io/projected/967e30fc-237b-450e-baf2-c6c4990c460f-kube-api-access-ktzfr\") pod \"967e30fc-237b-450e-baf2-c6c4990c460f\" (UID: \"967e30fc-237b-450e-baf2-c6c4990c460f\") " Sep 14 11:16:53.729240 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:53.729211 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/967e30fc-237b-450e-baf2-c6c4990c460f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "967e30fc-237b-450e-baf2-c6c4990c460f" (UID: "967e30fc-237b-450e-baf2-c6c4990c460f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:16:53.729240 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:53.729215 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a84eecc-cf96-48b6-87c4-9ed9103d214b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2a84eecc-cf96-48b6-87c4-9ed9103d214b" (UID: "2a84eecc-cf96-48b6-87c4-9ed9103d214b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:16:53.731158 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:53.731132 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a84eecc-cf96-48b6-87c4-9ed9103d214b-kube-api-access-rd9wg" (OuterVolumeSpecName: "kube-api-access-rd9wg") pod "2a84eecc-cf96-48b6-87c4-9ed9103d214b" (UID: "2a84eecc-cf96-48b6-87c4-9ed9103d214b"). InnerVolumeSpecName "kube-api-access-rd9wg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:16:53.731220 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:53.731169 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/967e30fc-237b-450e-baf2-c6c4990c460f-kube-api-access-ktzfr" (OuterVolumeSpecName: "kube-api-access-ktzfr") pod "967e30fc-237b-450e-baf2-c6c4990c460f" (UID: "967e30fc-237b-450e-baf2-c6c4990c460f"). InnerVolumeSpecName "kube-api-access-ktzfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:16:53.830216 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:53.830127 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/967e30fc-237b-450e-baf2-c6c4990c460f-plugins-config-volume\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:16:53.830216 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:53.830156 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktzfr\" (UniqueName: \"kubernetes.io/projected/967e30fc-237b-450e-baf2-c6c4990c460f-kube-api-access-ktzfr\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:16:53.830216 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:53.830168 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd9wg\" (UniqueName: \"kubernetes.io/projected/2a84eecc-cf96-48b6-87c4-9ed9103d214b-kube-api-access-rd9wg\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:16:53.830216 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:53.830179 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a84eecc-cf96-48b6-87c4-9ed9103d214b-plugins-config-volume\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:16:54.487568 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:54.487536 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29e74e-85664db6-xhbmm" event={"ID":"2a84eecc-cf96-48b6-87c4-9ed9103d214b","Type":"ContainerDied","Data":"9021542f5aa75b2e0925f14bc87bbab60782b35590c9b54956faf6bb6eb25418"} Sep 14 11:16:54.488038 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:54.487603 2581 scope.go:117] "RemoveContainer" containerID="9a8ac1e85f268cf706f0a990fd4ae89aeaba557b05a9a305f324981cdc840727" Sep 14 11:16:54.488038 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:54.487546 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29e74e-85664db6-xhbmm" Sep 14 11:16:54.489066 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:54.489047 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-29e74e-6fb8fc5c5-bk6nv" event={"ID":"967e30fc-237b-450e-baf2-c6c4990c460f","Type":"ContainerDied","Data":"bd1730f459817ccde5c4916dd428705d49dacc0f6e3c13cd7cdcf174f42da1b5"} Sep 14 11:16:54.489275 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:54.489089 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-29e74e-6fb8fc5c5-bk6nv" Sep 14 11:16:54.497199 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:54.497172 2581 scope.go:117] "RemoveContainer" containerID="b50405f452c91930f2fda75a59bf2def69448ccaa9a65ddc922bd1c26f2a03c9" Sep 14 11:16:54.518217 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:54.518190 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-29e74e-85664db6-xhbmm"] Sep 14 11:16:54.523771 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:54.523748 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-29e74e-85664db6-xhbmm"] Sep 14 11:16:54.540838 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:54.540817 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-29e74e-6fb8fc5c5-bk6nv"] Sep 14 11:16:54.544235 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:54.544215 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-29e74e-6fb8fc5c5-bk6nv"] Sep 14 11:16:54.832866 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:54.832827 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a84eecc-cf96-48b6-87c4-9ed9103d214b" path="/var/lib/kubelet/pods/2a84eecc-cf96-48b6-87c4-9ed9103d214b/volumes" Sep 14 11:16:54.833291 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:54.833270 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="967e30fc-237b-450e-baf2-c6c4990c460f" path="/var/lib/kubelet/pods/967e30fc-237b-450e-baf2-c6c4990c460f/volumes" Sep 14 11:16:54.833670 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:16:54.833652 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a5b7686-38d8-444f-8579-c458987fd13b" path="/var/lib/kubelet/pods/9a5b7686-38d8-444f-8579-c458987fd13b/volumes" Sep 14 11:17:01.299421 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:01.299387 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f8e582-g5545"] Sep 14 11:17:01.299812 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:01.299788 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a5b7686-38d8-444f-8579-c458987fd13b" containerName="revoke-keys" Sep 14 11:17:01.299812 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:01.299800 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a5b7686-38d8-444f-8579-c458987fd13b" containerName="revoke-keys" Sep 14 11:17:01.299812 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:01.299808 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a84eecc-cf96-48b6-87c4-9ed9103d214b" containerName="payload-pre-processing" Sep 14 11:17:01.299913 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:01.299814 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a84eecc-cf96-48b6-87c4-9ed9103d214b" containerName="payload-pre-processing" Sep 14 11:17:01.299913 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:01.299837 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="967e30fc-237b-450e-baf2-c6c4990c460f" containerName="payload-processing" Sep 14 11:17:01.299913 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:01.299842 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="967e30fc-237b-450e-baf2-c6c4990c460f" containerName="payload-processing" Sep 14 11:17:01.299913 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:01.299903 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="967e30fc-237b-450e-baf2-c6c4990c460f" containerName="payload-processing" Sep 14 11:17:01.299913 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:01.299910 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a5b7686-38d8-444f-8579-c458987fd13b" containerName="revoke-keys" Sep 14 11:17:01.300062 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:01.299919 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a84eecc-cf96-48b6-87c4-9ed9103d214b" containerName="payload-pre-processing" Sep 14 11:17:01.304457 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:01.304437 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f8e582-g5545" Sep 14 11:17:01.307718 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:01.307701 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pv9g8\"" Sep 14 11:17:01.318909 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:01.318884 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f8e582-g5545"] Sep 14 11:17:01.392541 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:01.392502 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf2fcd8d-74b0-4018-96bc-326effc87feb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f8e582-g5545\" (UID: \"bf2fcd8d-74b0-4018-96bc-326effc87feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f8e582-g5545" Sep 14 11:17:01.493835 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:01.493801 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf2fcd8d-74b0-4018-96bc-326effc87feb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f8e582-g5545\" (UID: \"bf2fcd8d-74b0-4018-96bc-326effc87feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f8e582-g5545" Sep 14 11:17:01.494453 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:01.494433 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf2fcd8d-74b0-4018-96bc-326effc87feb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f8e582-g5545\" (UID: \"bf2fcd8d-74b0-4018-96bc-326effc87feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f8e582-g5545" Sep 14 11:17:01.614976 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:01.614890 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f8e582-g5545" Sep 14 11:17:01.894271 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:01.894247 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f8e582-g5545"] Sep 14 11:17:01.896890 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:17:01.896858 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf2fcd8d_74b0_4018_96bc_326effc87feb.slice/crio-782c850ab2daad75bc6af3dbe711c60e94d404bcfbd0282e498b37cd8ba86a07 WatchSource:0}: Error finding container 782c850ab2daad75bc6af3dbe711c60e94d404bcfbd0282e498b37cd8ba86a07: Status 404 returned error can't find the container with id 782c850ab2daad75bc6af3dbe711c60e94d404bcfbd0282e498b37cd8ba86a07 Sep 14 11:17:02.535971 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:02.535930 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f8e582-g5545" event={"ID":"bf2fcd8d-74b0-4018-96bc-326effc87feb","Type":"ContainerStarted","Data":"627eb5cef607f297d0176b77e331b925c0adfff1604c7326476f864043239470"} Sep 14 11:17:02.536485 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:02.536026 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f8e582-g5545" event={"ID":"bf2fcd8d-74b0-4018-96bc-326effc87feb","Type":"ContainerStarted","Data":"782c850ab2daad75bc6af3dbe711c60e94d404bcfbd0282e498b37cd8ba86a07"} Sep 14 11:17:02.559147 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:02.559091 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f8e582-g5545" podStartSLOduration=1.5590752060000002 podStartE2EDuration="1.559075206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:17:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:17:02.557324787 +0000 UTC m=+1696.299726756" watchObservedRunningTime="2026-09-14 11:17:02.559075206 +0000 UTC m=+1696.301477176" Sep 14 11:17:03.543300 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:03.543262 2581 generic.go:356] "Generic (PLEG): container finished" podID="bf2fcd8d-74b0-4018-96bc-326effc87feb" containerID="627eb5cef607f297d0176b77e331b925c0adfff1604c7326476f864043239470" exitCode=0 Sep 14 11:17:03.543726 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:03.543328 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f8e582-g5545" event={"ID":"bf2fcd8d-74b0-4018-96bc-326effc87feb","Type":"ContainerDied","Data":"627eb5cef607f297d0176b77e331b925c0adfff1604c7326476f864043239470"} Sep 14 11:17:04.682609 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:04.682586 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f8e582-g5545" Sep 14 11:17:04.729038 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:04.729010 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf2fcd8d-74b0-4018-96bc-326effc87feb-maas-api-service-ca\") pod \"bf2fcd8d-74b0-4018-96bc-326effc87feb\" (UID: \"bf2fcd8d-74b0-4018-96bc-326effc87feb\") " Sep 14 11:17:04.729462 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:04.729436 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf2fcd8d-74b0-4018-96bc-326effc87feb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bf2fcd8d-74b0-4018-96bc-326effc87feb" (UID: "bf2fcd8d-74b0-4018-96bc-326effc87feb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:17:04.830040 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:04.829956 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf2fcd8d-74b0-4018-96bc-326effc87feb-maas-api-service-ca\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:17:05.555644 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:05.555602 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f8e582-g5545" event={"ID":"bf2fcd8d-74b0-4018-96bc-326effc87feb","Type":"ContainerDied","Data":"782c850ab2daad75bc6af3dbe711c60e94d404bcfbd0282e498b37cd8ba86a07"} Sep 14 11:17:05.555855 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:05.555663 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="782c850ab2daad75bc6af3dbe711c60e94d404bcfbd0282e498b37cd8ba86a07" Sep 14 11:17:05.555855 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:05.555618 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f8e582-g5545" Sep 14 11:17:17.196944 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:17.196855 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f8e582-58b77c449c-thb87"] Sep 14 11:17:17.197442 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:17.197138 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-f8e582-58b77c449c-thb87" podUID="f35d29eb-0414-44eb-bd0f-ada347619cbf" containerName="payload-processing" containerID="cri-o://0b1c07d0fea1d9fc123602f7fd85825779e52eb7a387ba29359458db4b2f9b05" gracePeriod=30 Sep 14 11:17:17.463113 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:17.463078 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f8e582-g5545"] Sep 14 11:17:17.465258 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:17.465219 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f8e582-58b77c449c-thb87" Sep 14 11:17:17.467479 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:17.467456 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f8e582-g5545"] Sep 14 11:17:17.559284 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:17.559245 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f35d29eb-0414-44eb-bd0f-ada347619cbf-plugins-config-volume\") pod \"f35d29eb-0414-44eb-bd0f-ada347619cbf\" (UID: \"f35d29eb-0414-44eb-bd0f-ada347619cbf\") " Sep 14 11:17:17.559578 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:17.559358 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ml6jq\" (UniqueName: \"kubernetes.io/projected/f35d29eb-0414-44eb-bd0f-ada347619cbf-kube-api-access-ml6jq\") pod \"f35d29eb-0414-44eb-bd0f-ada347619cbf\" (UID: \"f35d29eb-0414-44eb-bd0f-ada347619cbf\") " Sep 14 11:17:17.559735 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:17.559712 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f35d29eb-0414-44eb-bd0f-ada347619cbf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f35d29eb-0414-44eb-bd0f-ada347619cbf" (UID: "f35d29eb-0414-44eb-bd0f-ada347619cbf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:17:17.561644 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:17.561618 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f35d29eb-0414-44eb-bd0f-ada347619cbf-kube-api-access-ml6jq" (OuterVolumeSpecName: "kube-api-access-ml6jq") pod "f35d29eb-0414-44eb-bd0f-ada347619cbf" (UID: "f35d29eb-0414-44eb-bd0f-ada347619cbf"). InnerVolumeSpecName "kube-api-access-ml6jq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:17:17.622223 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:17.622191 2581 generic.go:356] "Generic (PLEG): container finished" podID="f35d29eb-0414-44eb-bd0f-ada347619cbf" containerID="0b1c07d0fea1d9fc123602f7fd85825779e52eb7a387ba29359458db4b2f9b05" exitCode=0 Sep 14 11:17:17.622423 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:17.622253 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f8e582-58b77c449c-thb87" Sep 14 11:17:17.622423 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:17.622265 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f8e582-58b77c449c-thb87" event={"ID":"f35d29eb-0414-44eb-bd0f-ada347619cbf","Type":"ContainerDied","Data":"0b1c07d0fea1d9fc123602f7fd85825779e52eb7a387ba29359458db4b2f9b05"} Sep 14 11:17:17.622423 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:17.622309 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f8e582-58b77c449c-thb87" event={"ID":"f35d29eb-0414-44eb-bd0f-ada347619cbf","Type":"ContainerDied","Data":"ba3449d111c77b45e2264a2be4468518fb950ee0738542ef9e407ad63196fb73"} Sep 14 11:17:17.622423 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:17.622330 2581 scope.go:117] "RemoveContainer" containerID="0b1c07d0fea1d9fc123602f7fd85825779e52eb7a387ba29359458db4b2f9b05" Sep 14 11:17:17.631686 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:17.631661 2581 scope.go:117] "RemoveContainer" containerID="0b1c07d0fea1d9fc123602f7fd85825779e52eb7a387ba29359458db4b2f9b05" Sep 14 11:17:17.632001 ip-10-0-137-53 kubenswrapper[2581]: E0914 11:17:17.631982 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b1c07d0fea1d9fc123602f7fd85825779e52eb7a387ba29359458db4b2f9b05\": container with ID starting with 0b1c07d0fea1d9fc123602f7fd85825779e52eb7a387ba29359458db4b2f9b05 not found: ID does not exist" containerID="0b1c07d0fea1d9fc123602f7fd85825779e52eb7a387ba29359458db4b2f9b05" Sep 14 11:17:17.632061 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:17.632012 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b1c07d0fea1d9fc123602f7fd85825779e52eb7a387ba29359458db4b2f9b05"} err="failed to get container status \"0b1c07d0fea1d9fc123602f7fd85825779e52eb7a387ba29359458db4b2f9b05\": rpc error: code = NotFound desc = could not find container \"0b1c07d0fea1d9fc123602f7fd85825779e52eb7a387ba29359458db4b2f9b05\": container with ID starting with 0b1c07d0fea1d9fc123602f7fd85825779e52eb7a387ba29359458db4b2f9b05 not found: ID does not exist" Sep 14 11:17:17.643940 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:17.643912 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f8e582-58b77c449c-thb87"] Sep 14 11:17:17.652408 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:17.652380 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f8e582-58b77c449c-thb87"] Sep 14 11:17:17.660885 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:17.660856 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f35d29eb-0414-44eb-bd0f-ada347619cbf-plugins-config-volume\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:17:17.661003 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:17.660888 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml6jq\" (UniqueName: \"kubernetes.io/projected/f35d29eb-0414-44eb-bd0f-ada347619cbf-kube-api-access-ml6jq\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:17:18.832159 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:18.832127 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf2fcd8d-74b0-4018-96bc-326effc87feb" path="/var/lib/kubelet/pods/bf2fcd8d-74b0-4018-96bc-326effc87feb/volumes" Sep 14 11:17:18.832592 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:18.832511 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f35d29eb-0414-44eb-bd0f-ada347619cbf" path="/var/lib/kubelet/pods/f35d29eb-0414-44eb-bd0f-ada347619cbf/volumes" Sep 14 11:17:26.442963 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:26.442926 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fdbc00/auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w"] Sep 14 11:17:29.408322 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:29.408289 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ae9d47-cx2j8"] Sep 14 11:17:29.408728 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:29.408714 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf2fcd8d-74b0-4018-96bc-326effc87feb" containerName="revoke-keys" Sep 14 11:17:29.408773 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:29.408731 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf2fcd8d-74b0-4018-96bc-326effc87feb" containerName="revoke-keys" Sep 14 11:17:29.408773 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:29.408740 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f35d29eb-0414-44eb-bd0f-ada347619cbf" containerName="payload-processing" Sep 14 11:17:29.408773 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:29.408746 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f35d29eb-0414-44eb-bd0f-ada347619cbf" containerName="payload-processing" Sep 14 11:17:29.408876 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:29.408816 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf2fcd8d-74b0-4018-96bc-326effc87feb" containerName="revoke-keys" Sep 14 11:17:29.408876 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:29.408826 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f35d29eb-0414-44eb-bd0f-ada347619cbf" containerName="payload-processing" Sep 14 11:17:29.413392 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:29.413376 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ae9d47-cx2j8" Sep 14 11:17:29.416204 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:29.416182 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pv9g8\"" Sep 14 11:17:29.423416 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:29.423383 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ae9d47-cx2j8"] Sep 14 11:17:29.578295 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:29.578253 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0736e784-3b83-4cf9-8a0c-3f832a410334-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-ae9d47-cx2j8\" (UID: \"0736e784-3b83-4cf9-8a0c-3f832a410334\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ae9d47-cx2j8" Sep 14 11:17:29.679789 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:29.679699 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0736e784-3b83-4cf9-8a0c-3f832a410334-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-ae9d47-cx2j8\" (UID: \"0736e784-3b83-4cf9-8a0c-3f832a410334\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ae9d47-cx2j8" Sep 14 11:17:29.680328 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:29.680307 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0736e784-3b83-4cf9-8a0c-3f832a410334-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-ae9d47-cx2j8\" (UID: \"0736e784-3b83-4cf9-8a0c-3f832a410334\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ae9d47-cx2j8" Sep 14 11:17:29.724323 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:29.724280 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ae9d47-cx2j8" Sep 14 11:17:30.000787 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:30.000761 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ae9d47-cx2j8"] Sep 14 11:17:30.003160 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:17:30.003138 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0736e784_3b83_4cf9_8a0c_3f832a410334.slice/crio-c8b85ed98183a2e82ea105867f91b99805e7d63ae2da2eeec2405a221421a195 WatchSource:0}: Error finding container c8b85ed98183a2e82ea105867f91b99805e7d63ae2da2eeec2405a221421a195: Status 404 returned error can't find the container with id c8b85ed98183a2e82ea105867f91b99805e7d63ae2da2eeec2405a221421a195 Sep 14 11:17:30.695930 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:30.695890 2581 generic.go:356] "Generic (PLEG): container finished" podID="0736e784-3b83-4cf9-8a0c-3f832a410334" containerID="978792c3528e385886414a8ef90c1a497acdde9f27d4288d1053716d61dee22a" exitCode=0 Sep 14 11:17:30.696334 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:30.695958 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ae9d47-cx2j8" event={"ID":"0736e784-3b83-4cf9-8a0c-3f832a410334","Type":"ContainerDied","Data":"978792c3528e385886414a8ef90c1a497acdde9f27d4288d1053716d61dee22a"} Sep 14 11:17:30.696334 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:30.696004 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ae9d47-cx2j8" event={"ID":"0736e784-3b83-4cf9-8a0c-3f832a410334","Type":"ContainerStarted","Data":"c8b85ed98183a2e82ea105867f91b99805e7d63ae2da2eeec2405a221421a195"} Sep 14 11:17:31.841943 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:31.841909 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ae9d47-cx2j8" Sep 14 11:17:32.003455 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:32.003363 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0736e784-3b83-4cf9-8a0c-3f832a410334-maas-api-service-ca\") pod \"0736e784-3b83-4cf9-8a0c-3f832a410334\" (UID: \"0736e784-3b83-4cf9-8a0c-3f832a410334\") " Sep 14 11:17:32.003675 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:32.003651 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0736e784-3b83-4cf9-8a0c-3f832a410334-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0736e784-3b83-4cf9-8a0c-3f832a410334" (UID: "0736e784-3b83-4cf9-8a0c-3f832a410334"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:17:32.003739 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:32.003717 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0736e784-3b83-4cf9-8a0c-3f832a410334-maas-api-service-ca\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:17:32.716685 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:32.716650 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ae9d47-cx2j8" Sep 14 11:17:32.716856 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:32.716650 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ae9d47-cx2j8" event={"ID":"0736e784-3b83-4cf9-8a0c-3f832a410334","Type":"ContainerDied","Data":"c8b85ed98183a2e82ea105867f91b99805e7d63ae2da2eeec2405a221421a195"} Sep 14 11:17:32.716856 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:32.716774 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8b85ed98183a2e82ea105867f91b99805e7d63ae2da2eeec2405a221421a195" Sep 14 11:17:34.525514 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:34.525480 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-fdbc00/sub-test-model-fdbc00-kserve-57d7757869-6h7q4"] Sep 14 11:17:34.525991 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:34.525886 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0736e784-3b83-4cf9-8a0c-3f832a410334" containerName="revoke-keys" Sep 14 11:17:34.525991 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:34.525897 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0736e784-3b83-4cf9-8a0c-3f832a410334" containerName="revoke-keys" Sep 14 11:17:34.525991 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:34.525973 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0736e784-3b83-4cf9-8a0c-3f832a410334" containerName="revoke-keys" Sep 14 11:17:34.528816 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:34.528801 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fdbc00/sub-test-model-fdbc00-kserve-57d7757869-6h7q4" Sep 14 11:17:34.532112 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:34.532092 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-fdbc00\"/\"sub-test-model-fdbc00-kserve-self-signed-certs\"" Sep 14 11:17:34.549462 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:34.549431 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fdbc00/sub-test-model-fdbc00-kserve-57d7757869-6h7q4"] Sep 14 11:17:34.628207 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:34.628161 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a711a5b-6a9c-4303-af43-2d9892a90c47-model-cache\") pod \"sub-test-model-fdbc00-kserve-57d7757869-6h7q4\" (UID: \"1a711a5b-6a9c-4303-af43-2d9892a90c47\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/sub-test-model-fdbc00-kserve-57d7757869-6h7q4" Sep 14 11:17:34.628207 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:34.628207 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5mc5\" (UniqueName: \"kubernetes.io/projected/1a711a5b-6a9c-4303-af43-2d9892a90c47-kube-api-access-w5mc5\") pod \"sub-test-model-fdbc00-kserve-57d7757869-6h7q4\" (UID: \"1a711a5b-6a9c-4303-af43-2d9892a90c47\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/sub-test-model-fdbc00-kserve-57d7757869-6h7q4" Sep 14 11:17:34.628505 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:34.628241 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a711a5b-6a9c-4303-af43-2d9892a90c47-tls-certs\") pod \"sub-test-model-fdbc00-kserve-57d7757869-6h7q4\" (UID: \"1a711a5b-6a9c-4303-af43-2d9892a90c47\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/sub-test-model-fdbc00-kserve-57d7757869-6h7q4" Sep 14 11:17:34.628505 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:34.628291 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a711a5b-6a9c-4303-af43-2d9892a90c47-dshm\") pod \"sub-test-model-fdbc00-kserve-57d7757869-6h7q4\" (UID: \"1a711a5b-6a9c-4303-af43-2d9892a90c47\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/sub-test-model-fdbc00-kserve-57d7757869-6h7q4" Sep 14 11:17:34.628505 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:34.628315 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a711a5b-6a9c-4303-af43-2d9892a90c47-tmp-dir\") pod \"sub-test-model-fdbc00-kserve-57d7757869-6h7q4\" (UID: \"1a711a5b-6a9c-4303-af43-2d9892a90c47\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/sub-test-model-fdbc00-kserve-57d7757869-6h7q4" Sep 14 11:17:34.628505 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:34.628395 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a711a5b-6a9c-4303-af43-2d9892a90c47-home\") pod \"sub-test-model-fdbc00-kserve-57d7757869-6h7q4\" (UID: \"1a711a5b-6a9c-4303-af43-2d9892a90c47\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/sub-test-model-fdbc00-kserve-57d7757869-6h7q4" Sep 14 11:17:34.728980 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:34.728949 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a711a5b-6a9c-4303-af43-2d9892a90c47-home\") pod \"sub-test-model-fdbc00-kserve-57d7757869-6h7q4\" (UID: \"1a711a5b-6a9c-4303-af43-2d9892a90c47\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/sub-test-model-fdbc00-kserve-57d7757869-6h7q4" Sep 14 11:17:34.729170 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:34.729016 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a711a5b-6a9c-4303-af43-2d9892a90c47-model-cache\") pod \"sub-test-model-fdbc00-kserve-57d7757869-6h7q4\" (UID: \"1a711a5b-6a9c-4303-af43-2d9892a90c47\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/sub-test-model-fdbc00-kserve-57d7757869-6h7q4" Sep 14 11:17:34.729170 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:34.729038 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5mc5\" (UniqueName: \"kubernetes.io/projected/1a711a5b-6a9c-4303-af43-2d9892a90c47-kube-api-access-w5mc5\") pod \"sub-test-model-fdbc00-kserve-57d7757869-6h7q4\" (UID: \"1a711a5b-6a9c-4303-af43-2d9892a90c47\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/sub-test-model-fdbc00-kserve-57d7757869-6h7q4" Sep 14 11:17:34.729170 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:34.729062 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a711a5b-6a9c-4303-af43-2d9892a90c47-tls-certs\") pod \"sub-test-model-fdbc00-kserve-57d7757869-6h7q4\" (UID: \"1a711a5b-6a9c-4303-af43-2d9892a90c47\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/sub-test-model-fdbc00-kserve-57d7757869-6h7q4" Sep 14 11:17:34.729170 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:34.729100 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a711a5b-6a9c-4303-af43-2d9892a90c47-dshm\") pod \"sub-test-model-fdbc00-kserve-57d7757869-6h7q4\" (UID: \"1a711a5b-6a9c-4303-af43-2d9892a90c47\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/sub-test-model-fdbc00-kserve-57d7757869-6h7q4" Sep 14 11:17:34.729170 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:34.729116 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a711a5b-6a9c-4303-af43-2d9892a90c47-tmp-dir\") pod \"sub-test-model-fdbc00-kserve-57d7757869-6h7q4\" (UID: \"1a711a5b-6a9c-4303-af43-2d9892a90c47\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/sub-test-model-fdbc00-kserve-57d7757869-6h7q4" Sep 14 11:17:34.729455 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:34.729432 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a711a5b-6a9c-4303-af43-2d9892a90c47-home\") pod \"sub-test-model-fdbc00-kserve-57d7757869-6h7q4\" (UID: \"1a711a5b-6a9c-4303-af43-2d9892a90c47\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/sub-test-model-fdbc00-kserve-57d7757869-6h7q4" Sep 14 11:17:34.729501 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:34.729463 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a711a5b-6a9c-4303-af43-2d9892a90c47-model-cache\") pod \"sub-test-model-fdbc00-kserve-57d7757869-6h7q4\" (UID: \"1a711a5b-6a9c-4303-af43-2d9892a90c47\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/sub-test-model-fdbc00-kserve-57d7757869-6h7q4" Sep 14 11:17:34.729547 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:34.729510 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a711a5b-6a9c-4303-af43-2d9892a90c47-tmp-dir\") pod \"sub-test-model-fdbc00-kserve-57d7757869-6h7q4\" (UID: \"1a711a5b-6a9c-4303-af43-2d9892a90c47\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/sub-test-model-fdbc00-kserve-57d7757869-6h7q4" Sep 14 11:17:34.731581 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:34.731560 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a711a5b-6a9c-4303-af43-2d9892a90c47-dshm\") pod \"sub-test-model-fdbc00-kserve-57d7757869-6h7q4\" (UID: \"1a711a5b-6a9c-4303-af43-2d9892a90c47\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/sub-test-model-fdbc00-kserve-57d7757869-6h7q4" Sep 14 11:17:34.731728 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:34.731713 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a711a5b-6a9c-4303-af43-2d9892a90c47-tls-certs\") pod \"sub-test-model-fdbc00-kserve-57d7757869-6h7q4\" (UID: \"1a711a5b-6a9c-4303-af43-2d9892a90c47\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/sub-test-model-fdbc00-kserve-57d7757869-6h7q4" Sep 14 11:17:34.746798 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:34.746775 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5mc5\" (UniqueName: \"kubernetes.io/projected/1a711a5b-6a9c-4303-af43-2d9892a90c47-kube-api-access-w5mc5\") pod \"sub-test-model-fdbc00-kserve-57d7757869-6h7q4\" (UID: \"1a711a5b-6a9c-4303-af43-2d9892a90c47\") " pod="ai-tenant-e2e-shared-a-w3-fdbc00/sub-test-model-fdbc00-kserve-57d7757869-6h7q4" Sep 14 11:17:34.838219 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:34.838138 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fdbc00/sub-test-model-fdbc00-kserve-57d7757869-6h7q4" Sep 14 11:17:35.151001 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:35.150591 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fdbc00/sub-test-model-fdbc00-kserve-57d7757869-6h7q4"] Sep 14 11:17:35.153205 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:17:35.153177 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a711a5b_6a9c_4303_af43_2d9892a90c47.slice/crio-44c642cd88d679fea9cd86ea9d87e4bce7f032b1211adb977973d69e781f5f02 WatchSource:0}: Error finding container 44c642cd88d679fea9cd86ea9d87e4bce7f032b1211adb977973d69e781f5f02: Status 404 returned error can't find the container with id 44c642cd88d679fea9cd86ea9d87e4bce7f032b1211adb977973d69e781f5f02 Sep 14 11:17:35.735174 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:35.735137 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fdbc00/sub-test-model-fdbc00-kserve-57d7757869-6h7q4" event={"ID":"1a711a5b-6a9c-4303-af43-2d9892a90c47","Type":"ContainerStarted","Data":"9152d32b2e904fe6a5a121ba00b133e45f75a59b77280bc4d7503b870be01be8"} Sep 14 11:17:35.735174 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:35.735181 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fdbc00/sub-test-model-fdbc00-kserve-57d7757869-6h7q4" event={"ID":"1a711a5b-6a9c-4303-af43-2d9892a90c47","Type":"ContainerStarted","Data":"44c642cd88d679fea9cd86ea9d87e4bce7f032b1211adb977973d69e781f5f02"} Sep 14 11:17:35.766756 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:35.766702 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-fdbc00/sub-test-model-fdbc00-kserve-57d7757869-6h7q4" podStartSLOduration=1.766686795 podStartE2EDuration="1.766686795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:17:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:17:35.765862899 +0000 UTC m=+1729.508264868" watchObservedRunningTime="2026-09-14 11:17:35.766686795 +0000 UTC m=+1729.509088764" Sep 14 11:17:40.580479 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:40.580442 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ae9d47-cx2j8"] Sep 14 11:17:40.586683 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:40.586657 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ae9d47-cx2j8"] Sep 14 11:17:40.832105 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:40.832024 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0736e784-3b83-4cf9-8a0c-3f832a410334" path="/var/lib/kubelet/pods/0736e784-3b83-4cf9-8a0c-3f832a410334/volumes" Sep 14 11:17:41.463134 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:41.463087 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-fdbc00/auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w" podUID="a22802be-62a1-4055-b7e2-46a6c25a2bd2" containerName="main" containerID="cri-o://49ef8ac2aca4567f7052e260171692a109d357c37a62c6cda6bb702d9e6123ed" gracePeriod=45 Sep 14 11:17:41.720424 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:41.720357 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fdbc00/auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w" Sep 14 11:17:41.770384 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:41.770330 2581 generic.go:356] "Generic (PLEG): container finished" podID="a22802be-62a1-4055-b7e2-46a6c25a2bd2" containerID="49ef8ac2aca4567f7052e260171692a109d357c37a62c6cda6bb702d9e6123ed" exitCode=0 Sep 14 11:17:41.770384 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:41.770369 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fdbc00/auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w" event={"ID":"a22802be-62a1-4055-b7e2-46a6c25a2bd2","Type":"ContainerDied","Data":"49ef8ac2aca4567f7052e260171692a109d357c37a62c6cda6bb702d9e6123ed"} Sep 14 11:17:41.770654 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:41.770403 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fdbc00/auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w" Sep 14 11:17:41.770654 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:41.770418 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fdbc00/auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w" event={"ID":"a22802be-62a1-4055-b7e2-46a6c25a2bd2","Type":"ContainerDied","Data":"c0c46be97e9f2ddccb16af88c8342563690ef2cc4d1fcd307c887dd2d3752e54"} Sep 14 11:17:41.770654 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:41.770438 2581 scope.go:117] "RemoveContainer" containerID="49ef8ac2aca4567f7052e260171692a109d357c37a62c6cda6bb702d9e6123ed" Sep 14 11:17:41.780976 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:41.780958 2581 scope.go:117] "RemoveContainer" containerID="49ef8ac2aca4567f7052e260171692a109d357c37a62c6cda6bb702d9e6123ed" Sep 14 11:17:41.781291 ip-10-0-137-53 kubenswrapper[2581]: E0914 11:17:41.781273 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49ef8ac2aca4567f7052e260171692a109d357c37a62c6cda6bb702d9e6123ed\": container with ID starting with 49ef8ac2aca4567f7052e260171692a109d357c37a62c6cda6bb702d9e6123ed not found: ID does not exist" containerID="49ef8ac2aca4567f7052e260171692a109d357c37a62c6cda6bb702d9e6123ed" Sep 14 11:17:41.781381 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:41.781299 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49ef8ac2aca4567f7052e260171692a109d357c37a62c6cda6bb702d9e6123ed"} err="failed to get container status \"49ef8ac2aca4567f7052e260171692a109d357c37a62c6cda6bb702d9e6123ed\": rpc error: code = NotFound desc = could not find container \"49ef8ac2aca4567f7052e260171692a109d357c37a62c6cda6bb702d9e6123ed\": container with ID starting with 49ef8ac2aca4567f7052e260171692a109d357c37a62c6cda6bb702d9e6123ed not found: ID does not exist" Sep 14 11:17:41.797180 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:41.797148 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a22802be-62a1-4055-b7e2-46a6c25a2bd2-tmp-dir\") pod \"a22802be-62a1-4055-b7e2-46a6c25a2bd2\" (UID: \"a22802be-62a1-4055-b7e2-46a6c25a2bd2\") " Sep 14 11:17:41.797306 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:41.797240 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a22802be-62a1-4055-b7e2-46a6c25a2bd2-tls-certs\") pod \"a22802be-62a1-4055-b7e2-46a6c25a2bd2\" (UID: \"a22802be-62a1-4055-b7e2-46a6c25a2bd2\") " Sep 14 11:17:41.797306 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:41.797270 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a22802be-62a1-4055-b7e2-46a6c25a2bd2-home\") pod \"a22802be-62a1-4055-b7e2-46a6c25a2bd2\" (UID: \"a22802be-62a1-4055-b7e2-46a6c25a2bd2\") " Sep 14 11:17:41.797306 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:41.797290 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a22802be-62a1-4055-b7e2-46a6c25a2bd2-dshm\") pod \"a22802be-62a1-4055-b7e2-46a6c25a2bd2\" (UID: \"a22802be-62a1-4055-b7e2-46a6c25a2bd2\") " Sep 14 11:17:41.797506 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:41.797366 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a22802be-62a1-4055-b7e2-46a6c25a2bd2-model-cache\") pod \"a22802be-62a1-4055-b7e2-46a6c25a2bd2\" (UID: \"a22802be-62a1-4055-b7e2-46a6c25a2bd2\") " Sep 14 11:17:41.797506 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:41.797410 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n846x\" (UniqueName: \"kubernetes.io/projected/a22802be-62a1-4055-b7e2-46a6c25a2bd2-kube-api-access-n846x\") pod \"a22802be-62a1-4055-b7e2-46a6c25a2bd2\" (UID: \"a22802be-62a1-4055-b7e2-46a6c25a2bd2\") " Sep 14 11:17:41.797506 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:41.797433 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22802be-62a1-4055-b7e2-46a6c25a2bd2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a22802be-62a1-4055-b7e2-46a6c25a2bd2" (UID: "a22802be-62a1-4055-b7e2-46a6c25a2bd2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:17:41.797666 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:41.797505 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22802be-62a1-4055-b7e2-46a6c25a2bd2-home" (OuterVolumeSpecName: "home") pod "a22802be-62a1-4055-b7e2-46a6c25a2bd2" (UID: "a22802be-62a1-4055-b7e2-46a6c25a2bd2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:17:41.797726 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:41.797668 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22802be-62a1-4055-b7e2-46a6c25a2bd2-model-cache" (OuterVolumeSpecName: "model-cache") pod "a22802be-62a1-4055-b7e2-46a6c25a2bd2" (UID: "a22802be-62a1-4055-b7e2-46a6c25a2bd2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:17:41.797784 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:41.797764 2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a22802be-62a1-4055-b7e2-46a6c25a2bd2-tmp-dir\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:17:41.797784 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:41.797780 2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a22802be-62a1-4055-b7e2-46a6c25a2bd2-home\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:17:41.797857 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:41.797790 2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a22802be-62a1-4055-b7e2-46a6c25a2bd2-model-cache\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:17:41.799644 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:41.799622 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a22802be-62a1-4055-b7e2-46a6c25a2bd2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a22802be-62a1-4055-b7e2-46a6c25a2bd2" (UID: "a22802be-62a1-4055-b7e2-46a6c25a2bd2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:17:41.799748 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:41.799666 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22802be-62a1-4055-b7e2-46a6c25a2bd2-dshm" (OuterVolumeSpecName: "dshm") pod "a22802be-62a1-4055-b7e2-46a6c25a2bd2" (UID: "a22802be-62a1-4055-b7e2-46a6c25a2bd2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:17:41.799748 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:41.799670 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a22802be-62a1-4055-b7e2-46a6c25a2bd2-kube-api-access-n846x" (OuterVolumeSpecName: "kube-api-access-n846x") pod "a22802be-62a1-4055-b7e2-46a6c25a2bd2" (UID: "a22802be-62a1-4055-b7e2-46a6c25a2bd2"). InnerVolumeSpecName "kube-api-access-n846x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:17:41.899135 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:41.899096 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n846x\" (UniqueName: \"kubernetes.io/projected/a22802be-62a1-4055-b7e2-46a6c25a2bd2-kube-api-access-n846x\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:17:41.899135 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:41.899129 2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a22802be-62a1-4055-b7e2-46a6c25a2bd2-tls-certs\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:17:41.899135 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:41.899138 2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a22802be-62a1-4055-b7e2-46a6c25a2bd2-dshm\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:17:42.106538 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:42.106505 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fdbc00/auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w"] Sep 14 11:17:42.117275 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:42.117244 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fdbc00/auth-test-model-fdbc00-kserve-76c5c6b67f-6jv4w"] Sep 14 11:17:42.831671 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:42.831635 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a22802be-62a1-4055-b7e2-46a6c25a2bd2" path="/var/lib/kubelet/pods/a22802be-62a1-4055-b7e2-46a6c25a2bd2/volumes" Sep 14 11:17:44.839025 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:44.838996 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-fdbc00/sub-test-model-fdbc00-kserve-57d7757869-6h7q4" Sep 14 11:17:44.839502 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:44.839038 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-fdbc00/sub-test-model-fdbc00-kserve-57d7757869-6h7q4" Sep 14 11:17:44.851236 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:44.851215 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-fdbc00/sub-test-model-fdbc00-kserve-57d7757869-6h7q4" Sep 14 11:17:45.804198 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:45.804167 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-fdbc00/sub-test-model-fdbc00-kserve-57d7757869-6h7q4" Sep 14 11:17:49.105207 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:49.105170 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-22b014-mtr2b"] Sep 14 11:17:49.105684 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:49.105598 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a22802be-62a1-4055-b7e2-46a6c25a2bd2" containerName="main" Sep 14 11:17:49.105684 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:49.105609 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a22802be-62a1-4055-b7e2-46a6c25a2bd2" containerName="main" Sep 14 11:17:49.105684 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:49.105676 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a22802be-62a1-4055-b7e2-46a6c25a2bd2" containerName="main" Sep 14 11:17:49.108757 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:49.108742 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-22b014-mtr2b" Sep 14 11:17:49.113412 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:49.113388 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pv9g8\"" Sep 14 11:17:49.126076 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:49.126046 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-22b014-mtr2b"] Sep 14 11:17:49.172199 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:49.172166 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bdb67aa-79d7-4f96-8db5-6fe25819cd6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-22b014-mtr2b\" (UID: \"1bdb67aa-79d7-4f96-8db5-6fe25819cd6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-22b014-mtr2b" Sep 14 11:17:49.273609 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:49.273576 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bdb67aa-79d7-4f96-8db5-6fe25819cd6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-22b014-mtr2b\" (UID: \"1bdb67aa-79d7-4f96-8db5-6fe25819cd6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-22b014-mtr2b" Sep 14 11:17:49.274221 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:49.274202 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bdb67aa-79d7-4f96-8db5-6fe25819cd6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-22b014-mtr2b\" (UID: \"1bdb67aa-79d7-4f96-8db5-6fe25819cd6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-22b014-mtr2b" Sep 14 11:17:49.418917 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:49.418831 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-22b014-mtr2b" Sep 14 11:17:49.737821 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:17:49.737765 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bdb67aa_79d7_4f96_8db5_6fe25819cd6d.slice/crio-f81ef348ac4929d83721415f35affd7b47c7f2a6bcc853ce1f69c00a9833396f WatchSource:0}: Error finding container f81ef348ac4929d83721415f35affd7b47c7f2a6bcc853ce1f69c00a9833396f: Status 404 returned error can't find the container with id f81ef348ac4929d83721415f35affd7b47c7f2a6bcc853ce1f69c00a9833396f Sep 14 11:17:49.740902 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:49.740875 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-22b014-mtr2b"] Sep 14 11:17:49.816262 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:49.816224 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-22b014-mtr2b" event={"ID":"1bdb67aa-79d7-4f96-8db5-6fe25819cd6d","Type":"ContainerStarted","Data":"f81ef348ac4929d83721415f35affd7b47c7f2a6bcc853ce1f69c00a9833396f"} Sep 14 11:17:50.823174 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:50.823140 2581 generic.go:356] "Generic (PLEG): container finished" podID="1bdb67aa-79d7-4f96-8db5-6fe25819cd6d" containerID="f7027588c3311257f6bb64c9ed414e0b17922d699d5c53f436438271b6c2eb6d" exitCode=0 Sep 14 11:17:50.823588 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:50.823222 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-22b014-mtr2b" event={"ID":"1bdb67aa-79d7-4f96-8db5-6fe25819cd6d","Type":"ContainerDied","Data":"f7027588c3311257f6bb64c9ed414e0b17922d699d5c53f436438271b6c2eb6d"} Sep 14 11:17:51.964434 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:51.964409 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-22b014-mtr2b" Sep 14 11:17:52.101216 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:52.101119 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bdb67aa-79d7-4f96-8db5-6fe25819cd6d-maas-api-service-ca\") pod \"1bdb67aa-79d7-4f96-8db5-6fe25819cd6d\" (UID: \"1bdb67aa-79d7-4f96-8db5-6fe25819cd6d\") " Sep 14 11:17:52.101579 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:52.101550 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bdb67aa-79d7-4f96-8db5-6fe25819cd6d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1bdb67aa-79d7-4f96-8db5-6fe25819cd6d" (UID: "1bdb67aa-79d7-4f96-8db5-6fe25819cd6d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:17:52.202045 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:52.202011 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bdb67aa-79d7-4f96-8db5-6fe25819cd6d-maas-api-service-ca\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:17:52.842942 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:52.842909 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-22b014-mtr2b" Sep 14 11:17:52.843160 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:52.842951 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-22b014-mtr2b" event={"ID":"1bdb67aa-79d7-4f96-8db5-6fe25819cd6d","Type":"ContainerDied","Data":"f81ef348ac4929d83721415f35affd7b47c7f2a6bcc853ce1f69c00a9833396f"} Sep 14 11:17:52.843160 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:52.843037 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f81ef348ac4929d83721415f35affd7b47c7f2a6bcc853ce1f69c00a9833396f" Sep 14 11:17:59.902325 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:59.902288 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22b014-55ddd8cc54-m4lfm"] Sep 14 11:17:59.902796 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:17:59.902607 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22b014-55ddd8cc54-m4lfm" podUID="00f56b03-0457-4ffd-87b9-110d2eba6e6d" containerName="payload-pre-processing" containerID="cri-o://b83e87c6281ef3a6437aef42799143b5c8f4556ef23b63df747cc396d71c2269" gracePeriod=30 Sep 14 11:18:00.155815 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:00.155704 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-22b014-mtr2b"] Sep 14 11:18:00.156642 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:00.156624 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22b014-55ddd8cc54-m4lfm" Sep 14 11:18:00.157770 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:00.157751 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-22b014-mtr2b"] Sep 14 11:18:00.284271 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:00.284237 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00f56b03-0457-4ffd-87b9-110d2eba6e6d-plugins-config-volume\") pod \"00f56b03-0457-4ffd-87b9-110d2eba6e6d\" (UID: \"00f56b03-0457-4ffd-87b9-110d2eba6e6d\") " Sep 14 11:18:00.284271 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:00.284269 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7zxw\" (UniqueName: \"kubernetes.io/projected/00f56b03-0457-4ffd-87b9-110d2eba6e6d-kube-api-access-x7zxw\") pod \"00f56b03-0457-4ffd-87b9-110d2eba6e6d\" (UID: \"00f56b03-0457-4ffd-87b9-110d2eba6e6d\") " Sep 14 11:18:00.284683 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:00.284661 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00f56b03-0457-4ffd-87b9-110d2eba6e6d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "00f56b03-0457-4ffd-87b9-110d2eba6e6d" (UID: "00f56b03-0457-4ffd-87b9-110d2eba6e6d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:18:00.286529 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:00.286508 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00f56b03-0457-4ffd-87b9-110d2eba6e6d-kube-api-access-x7zxw" (OuterVolumeSpecName: "kube-api-access-x7zxw") pod "00f56b03-0457-4ffd-87b9-110d2eba6e6d" (UID: "00f56b03-0457-4ffd-87b9-110d2eba6e6d"). InnerVolumeSpecName "kube-api-access-x7zxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:18:00.385210 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:00.385181 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00f56b03-0457-4ffd-87b9-110d2eba6e6d-plugins-config-volume\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:18:00.385210 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:00.385207 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7zxw\" (UniqueName: \"kubernetes.io/projected/00f56b03-0457-4ffd-87b9-110d2eba6e6d-kube-api-access-x7zxw\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:18:00.832907 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:00.832872 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bdb67aa-79d7-4f96-8db5-6fe25819cd6d" path="/var/lib/kubelet/pods/1bdb67aa-79d7-4f96-8db5-6fe25819cd6d/volumes" Sep 14 11:18:00.889706 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:00.889669 2581 generic.go:356] "Generic (PLEG): container finished" podID="00f56b03-0457-4ffd-87b9-110d2eba6e6d" containerID="b83e87c6281ef3a6437aef42799143b5c8f4556ef23b63df747cc396d71c2269" exitCode=0 Sep 14 11:18:00.889879 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:00.889714 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22b014-55ddd8cc54-m4lfm" Sep 14 11:18:00.889879 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:00.889755 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22b014-55ddd8cc54-m4lfm" event={"ID":"00f56b03-0457-4ffd-87b9-110d2eba6e6d","Type":"ContainerDied","Data":"b83e87c6281ef3a6437aef42799143b5c8f4556ef23b63df747cc396d71c2269"} Sep 14 11:18:00.889879 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:00.889816 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22b014-55ddd8cc54-m4lfm" event={"ID":"00f56b03-0457-4ffd-87b9-110d2eba6e6d","Type":"ContainerDied","Data":"75411b97c3fa5b69157786d26e159c9bc18e1139246fd72e3a17255c41c4463f"} Sep 14 11:18:00.889879 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:00.889846 2581 scope.go:117] "RemoveContainer" containerID="b83e87c6281ef3a6437aef42799143b5c8f4556ef23b63df747cc396d71c2269" Sep 14 11:18:00.899125 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:00.899106 2581 scope.go:117] "RemoveContainer" containerID="b83e87c6281ef3a6437aef42799143b5c8f4556ef23b63df747cc396d71c2269" Sep 14 11:18:00.899389 ip-10-0-137-53 kubenswrapper[2581]: E0914 11:18:00.899374 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b83e87c6281ef3a6437aef42799143b5c8f4556ef23b63df747cc396d71c2269\": container with ID starting with b83e87c6281ef3a6437aef42799143b5c8f4556ef23b63df747cc396d71c2269 not found: ID does not exist" containerID="b83e87c6281ef3a6437aef42799143b5c8f4556ef23b63df747cc396d71c2269" Sep 14 11:18:00.899444 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:00.899396 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b83e87c6281ef3a6437aef42799143b5c8f4556ef23b63df747cc396d71c2269"} err="failed to get container status \"b83e87c6281ef3a6437aef42799143b5c8f4556ef23b63df747cc396d71c2269\": rpc error: code = NotFound desc = could not find container \"b83e87c6281ef3a6437aef42799143b5c8f4556ef23b63df747cc396d71c2269\": container with ID starting with b83e87c6281ef3a6437aef42799143b5c8f4556ef23b63df747cc396d71c2269 not found: ID does not exist" Sep 14 11:18:00.920733 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:00.920695 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22b014-55ddd8cc54-m4lfm"] Sep 14 11:18:00.923475 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:00.923441 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-22b014-55ddd8cc54-m4lfm"] Sep 14 11:18:02.831620 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:02.831589 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00f56b03-0457-4ffd-87b9-110d2eba6e6d" path="/var/lib/kubelet/pods/00f56b03-0457-4ffd-87b9-110d2eba6e6d/volumes" Sep 14 11:18:43.126018 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:43.125934 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fdbc00/sub-test-model-fdbc00-kserve-57d7757869-6h7q4"] Sep 14 11:18:58.145135 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:58.145085 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-fdbc00/sub-test-model-fdbc00-kserve-57d7757869-6h7q4" podUID="1a711a5b-6a9c-4303-af43-2d9892a90c47" containerName="main" containerID="cri-o://9152d32b2e904fe6a5a121ba00b133e45f75a59b77280bc4d7503b870be01be8" gracePeriod=45 Sep 14 11:18:58.390052 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:58.390028 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fdbc00/sub-test-model-fdbc00-kserve-57d7757869-6h7q4" Sep 14 11:18:58.445896 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:58.445807 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a711a5b-6a9c-4303-af43-2d9892a90c47-home\") pod \"1a711a5b-6a9c-4303-af43-2d9892a90c47\" (UID: \"1a711a5b-6a9c-4303-af43-2d9892a90c47\") " Sep 14 11:18:58.445896 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:58.445887 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a711a5b-6a9c-4303-af43-2d9892a90c47-model-cache\") pod \"1a711a5b-6a9c-4303-af43-2d9892a90c47\" (UID: \"1a711a5b-6a9c-4303-af43-2d9892a90c47\") " Sep 14 11:18:58.446095 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:58.445919 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5mc5\" (UniqueName: \"kubernetes.io/projected/1a711a5b-6a9c-4303-af43-2d9892a90c47-kube-api-access-w5mc5\") pod \"1a711a5b-6a9c-4303-af43-2d9892a90c47\" (UID: \"1a711a5b-6a9c-4303-af43-2d9892a90c47\") " Sep 14 11:18:58.446095 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:58.445961 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a711a5b-6a9c-4303-af43-2d9892a90c47-dshm\") pod \"1a711a5b-6a9c-4303-af43-2d9892a90c47\" (UID: \"1a711a5b-6a9c-4303-af43-2d9892a90c47\") " Sep 14 11:18:58.446095 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:58.445989 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a711a5b-6a9c-4303-af43-2d9892a90c47-tmp-dir\") pod \"1a711a5b-6a9c-4303-af43-2d9892a90c47\" (UID: \"1a711a5b-6a9c-4303-af43-2d9892a90c47\") " Sep 14 11:18:58.446095 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:58.446043 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a711a5b-6a9c-4303-af43-2d9892a90c47-tls-certs\") pod \"1a711a5b-6a9c-4303-af43-2d9892a90c47\" (UID: \"1a711a5b-6a9c-4303-af43-2d9892a90c47\") " Sep 14 11:18:58.446310 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:58.446218 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a711a5b-6a9c-4303-af43-2d9892a90c47-home" (OuterVolumeSpecName: "home") pod "1a711a5b-6a9c-4303-af43-2d9892a90c47" (UID: "1a711a5b-6a9c-4303-af43-2d9892a90c47"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:18:58.446310 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:58.446254 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a711a5b-6a9c-4303-af43-2d9892a90c47-model-cache" (OuterVolumeSpecName: "model-cache") pod "1a711a5b-6a9c-4303-af43-2d9892a90c47" (UID: "1a711a5b-6a9c-4303-af43-2d9892a90c47"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:18:58.446451 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:58.446301 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a711a5b-6a9c-4303-af43-2d9892a90c47-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1a711a5b-6a9c-4303-af43-2d9892a90c47" (UID: "1a711a5b-6a9c-4303-af43-2d9892a90c47"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:18:58.446540 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:58.446514 2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a711a5b-6a9c-4303-af43-2d9892a90c47-tmp-dir\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:18:58.446540 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:58.446540 2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a711a5b-6a9c-4303-af43-2d9892a90c47-home\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:18:58.446679 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:58.446550 2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a711a5b-6a9c-4303-af43-2d9892a90c47-model-cache\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:18:58.448417 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:58.448392 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a711a5b-6a9c-4303-af43-2d9892a90c47-dshm" (OuterVolumeSpecName: "dshm") pod "1a711a5b-6a9c-4303-af43-2d9892a90c47" (UID: "1a711a5b-6a9c-4303-af43-2d9892a90c47"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:18:58.448816 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:58.448788 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a711a5b-6a9c-4303-af43-2d9892a90c47-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1a711a5b-6a9c-4303-af43-2d9892a90c47" (UID: "1a711a5b-6a9c-4303-af43-2d9892a90c47"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:18:58.448816 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:58.448802 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a711a5b-6a9c-4303-af43-2d9892a90c47-kube-api-access-w5mc5" (OuterVolumeSpecName: "kube-api-access-w5mc5") pod "1a711a5b-6a9c-4303-af43-2d9892a90c47" (UID: "1a711a5b-6a9c-4303-af43-2d9892a90c47"). InnerVolumeSpecName "kube-api-access-w5mc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:18:58.547656 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:58.547614 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5mc5\" (UniqueName: \"kubernetes.io/projected/1a711a5b-6a9c-4303-af43-2d9892a90c47-kube-api-access-w5mc5\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:18:58.547656 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:58.547648 2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a711a5b-6a9c-4303-af43-2d9892a90c47-dshm\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:18:58.547656 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:58.547661 2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a711a5b-6a9c-4303-af43-2d9892a90c47-tls-certs\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:18:59.207359 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:59.207243 2581 generic.go:356] "Generic (PLEG): container finished" podID="1a711a5b-6a9c-4303-af43-2d9892a90c47" containerID="9152d32b2e904fe6a5a121ba00b133e45f75a59b77280bc4d7503b870be01be8" exitCode=0 Sep 14 11:18:59.207359 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:59.207286 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-fdbc00/sub-test-model-fdbc00-kserve-57d7757869-6h7q4" Sep 14 11:18:59.207359 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:59.207303 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fdbc00/sub-test-model-fdbc00-kserve-57d7757869-6h7q4" event={"ID":"1a711a5b-6a9c-4303-af43-2d9892a90c47","Type":"ContainerDied","Data":"9152d32b2e904fe6a5a121ba00b133e45f75a59b77280bc4d7503b870be01be8"} Sep 14 11:18:59.207359 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:59.207352 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-fdbc00/sub-test-model-fdbc00-kserve-57d7757869-6h7q4" event={"ID":"1a711a5b-6a9c-4303-af43-2d9892a90c47","Type":"ContainerDied","Data":"44c642cd88d679fea9cd86ea9d87e4bce7f032b1211adb977973d69e781f5f02"} Sep 14 11:18:59.207949 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:59.207365 2581 scope.go:117] "RemoveContainer" containerID="9152d32b2e904fe6a5a121ba00b133e45f75a59b77280bc4d7503b870be01be8" Sep 14 11:18:59.216894 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:59.216875 2581 scope.go:117] "RemoveContainer" containerID="9152d32b2e904fe6a5a121ba00b133e45f75a59b77280bc4d7503b870be01be8" Sep 14 11:18:59.217177 ip-10-0-137-53 kubenswrapper[2581]: E0914 11:18:59.217153 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9152d32b2e904fe6a5a121ba00b133e45f75a59b77280bc4d7503b870be01be8\": container with ID starting with 9152d32b2e904fe6a5a121ba00b133e45f75a59b77280bc4d7503b870be01be8 not found: ID does not exist" containerID="9152d32b2e904fe6a5a121ba00b133e45f75a59b77280bc4d7503b870be01be8" Sep 14 11:18:59.217240 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:59.217186 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9152d32b2e904fe6a5a121ba00b133e45f75a59b77280bc4d7503b870be01be8"} err="failed to get container status \"9152d32b2e904fe6a5a121ba00b133e45f75a59b77280bc4d7503b870be01be8\": rpc error: code = NotFound desc = could not find container \"9152d32b2e904fe6a5a121ba00b133e45f75a59b77280bc4d7503b870be01be8\": container with ID starting with 9152d32b2e904fe6a5a121ba00b133e45f75a59b77280bc4d7503b870be01be8 not found: ID does not exist" Sep 14 11:18:59.227332 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:59.227305 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fdbc00/sub-test-model-fdbc00-kserve-57d7757869-6h7q4"] Sep 14 11:18:59.231573 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:18:59.231547 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-fdbc00/sub-test-model-fdbc00-kserve-57d7757869-6h7q4"] Sep 14 11:19:00.831918 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:19:00.831882 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a711a5b-6a9c-4303-af43-2d9892a90c47" path="/var/lib/kubelet/pods/1a711a5b-6a9c-4303-af43-2d9892a90c47/volumes" Sep 14 11:19:47.157052 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:19:47.157019 2581 scope.go:117] "RemoveContainer" containerID="816c17b6ab3e26d79360976fcbe200fa3117a3900c66ade0849bd589e2b2ac72" Sep 14 11:20:00.990044 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:00.990012 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf45e0-8f49c4bbb-jk5nj"] Sep 14 11:20:00.990457 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:00.990444 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bdb67aa-79d7-4f96-8db5-6fe25819cd6d" containerName="revoke-keys" Sep 14 11:20:00.990457 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:00.990457 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bdb67aa-79d7-4f96-8db5-6fe25819cd6d" containerName="revoke-keys" Sep 14 11:20:00.990533 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:00.990466 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00f56b03-0457-4ffd-87b9-110d2eba6e6d" containerName="payload-pre-processing" Sep 14 11:20:00.990533 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:00.990471 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="00f56b03-0457-4ffd-87b9-110d2eba6e6d" containerName="payload-pre-processing" Sep 14 11:20:00.990533 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:00.990483 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a711a5b-6a9c-4303-af43-2d9892a90c47" containerName="main" Sep 14 11:20:00.990533 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:00.990488 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a711a5b-6a9c-4303-af43-2d9892a90c47" containerName="main" Sep 14 11:20:00.990661 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:00.990547 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bdb67aa-79d7-4f96-8db5-6fe25819cd6d" containerName="revoke-keys" Sep 14 11:20:00.990661 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:00.990557 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="00f56b03-0457-4ffd-87b9-110d2eba6e6d" containerName="payload-pre-processing" Sep 14 11:20:00.990661 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:00.990565 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a711a5b-6a9c-4303-af43-2d9892a90c47" containerName="main" Sep 14 11:20:00.993678 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:00.993657 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf45e0-8f49c4bbb-jk5nj" Sep 14 11:20:01.003242 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:01.003224 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-bf45e0-dockercfg-c7xhd\"" Sep 14 11:20:01.012015 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:01.011991 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-bf45e0\"" Sep 14 11:20:01.029263 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:01.029237 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf45e0-8f49c4bbb-jk5nj"] Sep 14 11:20:01.064189 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:01.064159 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de247584-a356-4585-8b49-99e8c2985941-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bf45e0-8f49c4bbb-jk5nj\" (UID: \"de247584-a356-4585-8b49-99e8c2985941\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf45e0-8f49c4bbb-jk5nj" Sep 14 11:20:01.064377 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:01.064224 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff2wl\" (UniqueName: \"kubernetes.io/projected/de247584-a356-4585-8b49-99e8c2985941-kube-api-access-ff2wl\") pod \"payload-pre-processing-e2e-ipp-a-bf45e0-8f49c4bbb-jk5nj\" (UID: \"de247584-a356-4585-8b49-99e8c2985941\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf45e0-8f49c4bbb-jk5nj" Sep 14 11:20:01.165002 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:01.164949 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de247584-a356-4585-8b49-99e8c2985941-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bf45e0-8f49c4bbb-jk5nj\" (UID: \"de247584-a356-4585-8b49-99e8c2985941\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf45e0-8f49c4bbb-jk5nj" Sep 14 11:20:01.165192 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:01.165045 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff2wl\" (UniqueName: \"kubernetes.io/projected/de247584-a356-4585-8b49-99e8c2985941-kube-api-access-ff2wl\") pod \"payload-pre-processing-e2e-ipp-a-bf45e0-8f49c4bbb-jk5nj\" (UID: \"de247584-a356-4585-8b49-99e8c2985941\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf45e0-8f49c4bbb-jk5nj" Sep 14 11:20:01.165695 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:01.165677 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de247584-a356-4585-8b49-99e8c2985941-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bf45e0-8f49c4bbb-jk5nj\" (UID: \"de247584-a356-4585-8b49-99e8c2985941\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf45e0-8f49c4bbb-jk5nj" Sep 14 11:20:01.179834 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:01.179805 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff2wl\" (UniqueName: \"kubernetes.io/projected/de247584-a356-4585-8b49-99e8c2985941-kube-api-access-ff2wl\") pod \"payload-pre-processing-e2e-ipp-a-bf45e0-8f49c4bbb-jk5nj\" (UID: \"de247584-a356-4585-8b49-99e8c2985941\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf45e0-8f49c4bbb-jk5nj" Sep 14 11:20:01.303411 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:01.303375 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf45e0-8f49c4bbb-jk5nj" Sep 14 11:20:01.581668 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:01.581641 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf45e0-8f49c4bbb-jk5nj"] Sep 14 11:20:01.583550 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:20:01.583518 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde247584_a356_4585_8b49_99e8c2985941.slice/crio-28f94aee1a39922f2e496d64dafd43435cdf623897a92baa7e14807a9fb67619 WatchSource:0}: Error finding container 28f94aee1a39922f2e496d64dafd43435cdf623897a92baa7e14807a9fb67619: Status 404 returned error can't find the container with id 28f94aee1a39922f2e496d64dafd43435cdf623897a92baa7e14807a9fb67619 Sep 14 11:20:02.546484 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:02.546448 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf45e0-8f49c4bbb-jk5nj" event={"ID":"de247584-a356-4585-8b49-99e8c2985941","Type":"ContainerStarted","Data":"f1b4985a5d5ceaa51eb64a685352bb6a91e23e608bbd886991409e2410c56bce"} Sep 14 11:20:02.546993 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:02.546509 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf45e0-8f49c4bbb-jk5nj" event={"ID":"de247584-a356-4585-8b49-99e8c2985941","Type":"ContainerStarted","Data":"28f94aee1a39922f2e496d64dafd43435cdf623897a92baa7e14807a9fb67619"} Sep 14 11:20:02.546993 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:02.546568 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf45e0-8f49c4bbb-jk5nj" Sep 14 11:20:02.585469 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:02.585415 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf45e0-8f49c4bbb-jk5nj" podStartSLOduration=2.585398059 podStartE2EDuration="2.585398059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:20:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:20:02.583172651 +0000 UTC m=+1876.325574620" watchObservedRunningTime="2026-09-14 11:20:02.585398059 +0000 UTC m=+1876.327800029" Sep 14 11:20:13.555236 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:13.555153 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf45e0-8f49c4bbb-jk5nj" Sep 14 11:20:30.751554 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:30.751519 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a521b5-9dcdbb9d7-x2f6n"] Sep 14 11:20:30.755135 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:30.755113 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a521b5-9dcdbb9d7-x2f6n" Sep 14 11:20:30.758798 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:30.758776 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a521b5\"" Sep 14 11:20:30.759254 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:30.759236 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a521b5-dockercfg-7cpqg\"" Sep 14 11:20:30.768765 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:30.768744 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a521b5-9dcdbb9d7-x2f6n"] Sep 14 11:20:30.856760 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:30.856717 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82ba6ab8-0d94-4a59-a0bf-8999b2fe6c81-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a521b5-9dcdbb9d7-x2f6n\" (UID: \"82ba6ab8-0d94-4a59-a0bf-8999b2fe6c81\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a521b5-9dcdbb9d7-x2f6n" Sep 14 11:20:30.856942 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:30.856770 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p8c8\" (UniqueName: \"kubernetes.io/projected/82ba6ab8-0d94-4a59-a0bf-8999b2fe6c81-kube-api-access-6p8c8\") pod \"payload-processing-e2e-ipp-b-a521b5-9dcdbb9d7-x2f6n\" (UID: \"82ba6ab8-0d94-4a59-a0bf-8999b2fe6c81\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a521b5-9dcdbb9d7-x2f6n" Sep 14 11:20:30.958267 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:30.958227 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82ba6ab8-0d94-4a59-a0bf-8999b2fe6c81-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a521b5-9dcdbb9d7-x2f6n\" (UID: \"82ba6ab8-0d94-4a59-a0bf-8999b2fe6c81\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a521b5-9dcdbb9d7-x2f6n" Sep 14 11:20:30.958267 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:30.958271 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p8c8\" (UniqueName: \"kubernetes.io/projected/82ba6ab8-0d94-4a59-a0bf-8999b2fe6c81-kube-api-access-6p8c8\") pod \"payload-processing-e2e-ipp-b-a521b5-9dcdbb9d7-x2f6n\" (UID: \"82ba6ab8-0d94-4a59-a0bf-8999b2fe6c81\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a521b5-9dcdbb9d7-x2f6n" Sep 14 11:20:30.959035 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:30.959008 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82ba6ab8-0d94-4a59-a0bf-8999b2fe6c81-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a521b5-9dcdbb9d7-x2f6n\" (UID: \"82ba6ab8-0d94-4a59-a0bf-8999b2fe6c81\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a521b5-9dcdbb9d7-x2f6n" Sep 14 11:20:30.967923 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:30.967899 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p8c8\" (UniqueName: \"kubernetes.io/projected/82ba6ab8-0d94-4a59-a0bf-8999b2fe6c81-kube-api-access-6p8c8\") pod \"payload-processing-e2e-ipp-b-a521b5-9dcdbb9d7-x2f6n\" (UID: \"82ba6ab8-0d94-4a59-a0bf-8999b2fe6c81\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a521b5-9dcdbb9d7-x2f6n" Sep 14 11:20:31.064850 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:31.064818 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a521b5-9dcdbb9d7-x2f6n" Sep 14 11:20:31.341119 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:31.341091 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a521b5-9dcdbb9d7-x2f6n"] Sep 14 11:20:31.343053 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:20:31.343025 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ba6ab8_0d94_4a59_a0bf_8999b2fe6c81.slice/crio-02b3123c62fe55d78a03bf6d176742829be52ca12731fc527324554d1cff36ea WatchSource:0}: Error finding container 02b3123c62fe55d78a03bf6d176742829be52ca12731fc527324554d1cff36ea: Status 404 returned error can't find the container with id 02b3123c62fe55d78a03bf6d176742829be52ca12731fc527324554d1cff36ea Sep 14 11:20:31.709895 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:31.709797 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a521b5-9dcdbb9d7-x2f6n" event={"ID":"82ba6ab8-0d94-4a59-a0bf-8999b2fe6c81","Type":"ContainerStarted","Data":"61b85f754d16a4d925f62e74b3dace9e752e79bfb6c0cfcabe519c487e603e7f"} Sep 14 11:20:31.709895 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:31.709849 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a521b5-9dcdbb9d7-x2f6n" event={"ID":"82ba6ab8-0d94-4a59-a0bf-8999b2fe6c81","Type":"ContainerStarted","Data":"02b3123c62fe55d78a03bf6d176742829be52ca12731fc527324554d1cff36ea"} Sep 14 11:20:31.710092 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:31.709925 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a521b5-9dcdbb9d7-x2f6n" Sep 14 11:20:31.727289 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:31.727238 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a521b5-9dcdbb9d7-x2f6n" podStartSLOduration=1.7272214510000001 podStartE2EDuration="1.727221451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:20:31.726419011 +0000 UTC m=+1905.468820979" watchObservedRunningTime="2026-09-14 11:20:31.727221451 +0000 UTC m=+1905.469623420" Sep 14 11:20:42.718023 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:20:42.717992 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a521b5-9dcdbb9d7-x2f6n" Sep 14 11:21:02.199169 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:02.199139 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-a521b5-9dcdbb9d7-x2f6n_82ba6ab8-0d94-4a59-a0bf-8999b2fe6c81/payload-processing/0.log" Sep 14 11:21:11.342668 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:11.342626 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-bf45e0/ipp-route-bf45e0-kserve-8659d7675-mklh9"] Sep 14 11:21:11.346545 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:11.346525 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-bf45e0/ipp-route-bf45e0-kserve-8659d7675-mklh9" Sep 14 11:21:11.349292 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:11.349268 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-bf45e0\"/\"openshift-service-ca.crt\"" Sep 14 11:21:11.350349 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:11.350322 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-bf45e0\"/\"default-dockercfg-4jvsg\"" Sep 14 11:21:11.350491 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:11.350409 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-bf45e0\"/\"kube-root-ca.crt\"" Sep 14 11:21:11.350491 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:11.350377 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-bf45e0\"/\"ipp-route-bf45e0-kserve-self-signed-certs\"" Sep 14 11:21:11.357263 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:11.357239 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-bf45e0/ipp-route-bf45e0-kserve-8659d7675-mklh9"] Sep 14 11:21:11.359952 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:11.359930 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/976aeaed-5bf5-4d48-b72d-a71e7fda745e-home\") pod \"ipp-route-bf45e0-kserve-8659d7675-mklh9\" (UID: \"976aeaed-5bf5-4d48-b72d-a71e7fda745e\") " pod="ai-tenant-e2e-ipp-a-bf45e0/ipp-route-bf45e0-kserve-8659d7675-mklh9" Sep 14 11:21:11.360072 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:11.359965 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/976aeaed-5bf5-4d48-b72d-a71e7fda745e-dshm\") pod \"ipp-route-bf45e0-kserve-8659d7675-mklh9\" (UID: \"976aeaed-5bf5-4d48-b72d-a71e7fda745e\") " pod="ai-tenant-e2e-ipp-a-bf45e0/ipp-route-bf45e0-kserve-8659d7675-mklh9" Sep 14 11:21:11.360072 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:11.359992 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/976aeaed-5bf5-4d48-b72d-a71e7fda745e-tmp-dir\") pod \"ipp-route-bf45e0-kserve-8659d7675-mklh9\" (UID: \"976aeaed-5bf5-4d48-b72d-a71e7fda745e\") " pod="ai-tenant-e2e-ipp-a-bf45e0/ipp-route-bf45e0-kserve-8659d7675-mklh9" Sep 14 11:21:11.360192 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:11.360094 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdcfm\" (UniqueName: \"kubernetes.io/projected/976aeaed-5bf5-4d48-b72d-a71e7fda745e-kube-api-access-cdcfm\") pod \"ipp-route-bf45e0-kserve-8659d7675-mklh9\" (UID: \"976aeaed-5bf5-4d48-b72d-a71e7fda745e\") " pod="ai-tenant-e2e-ipp-a-bf45e0/ipp-route-bf45e0-kserve-8659d7675-mklh9" Sep 14 11:21:11.360192 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:11.360131 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/976aeaed-5bf5-4d48-b72d-a71e7fda745e-model-cache\") pod \"ipp-route-bf45e0-kserve-8659d7675-mklh9\" (UID: \"976aeaed-5bf5-4d48-b72d-a71e7fda745e\") " pod="ai-tenant-e2e-ipp-a-bf45e0/ipp-route-bf45e0-kserve-8659d7675-mklh9" Sep 14 11:21:11.360192 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:11.360177 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/976aeaed-5bf5-4d48-b72d-a71e7fda745e-tls-certs\") pod \"ipp-route-bf45e0-kserve-8659d7675-mklh9\" (UID: \"976aeaed-5bf5-4d48-b72d-a71e7fda745e\") " pod="ai-tenant-e2e-ipp-a-bf45e0/ipp-route-bf45e0-kserve-8659d7675-mklh9" Sep 14 11:21:11.461079 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:11.461040 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/976aeaed-5bf5-4d48-b72d-a71e7fda745e-dshm\") pod \"ipp-route-bf45e0-kserve-8659d7675-mklh9\" (UID: \"976aeaed-5bf5-4d48-b72d-a71e7fda745e\") " pod="ai-tenant-e2e-ipp-a-bf45e0/ipp-route-bf45e0-kserve-8659d7675-mklh9" Sep 14 11:21:11.461079 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:11.461081 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/976aeaed-5bf5-4d48-b72d-a71e7fda745e-tmp-dir\") pod \"ipp-route-bf45e0-kserve-8659d7675-mklh9\" (UID: \"976aeaed-5bf5-4d48-b72d-a71e7fda745e\") " pod="ai-tenant-e2e-ipp-a-bf45e0/ipp-route-bf45e0-kserve-8659d7675-mklh9" Sep 14 11:21:11.461329 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:11.461110 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdcfm\" (UniqueName: \"kubernetes.io/projected/976aeaed-5bf5-4d48-b72d-a71e7fda745e-kube-api-access-cdcfm\") pod \"ipp-route-bf45e0-kserve-8659d7675-mklh9\" (UID: \"976aeaed-5bf5-4d48-b72d-a71e7fda745e\") " pod="ai-tenant-e2e-ipp-a-bf45e0/ipp-route-bf45e0-kserve-8659d7675-mklh9" Sep 14 11:21:11.461329 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:11.461135 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/976aeaed-5bf5-4d48-b72d-a71e7fda745e-model-cache\") pod \"ipp-route-bf45e0-kserve-8659d7675-mklh9\" (UID: \"976aeaed-5bf5-4d48-b72d-a71e7fda745e\") " pod="ai-tenant-e2e-ipp-a-bf45e0/ipp-route-bf45e0-kserve-8659d7675-mklh9" Sep 14 11:21:11.461329 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:11.461164 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/976aeaed-5bf5-4d48-b72d-a71e7fda745e-tls-certs\") pod \"ipp-route-bf45e0-kserve-8659d7675-mklh9\" (UID: \"976aeaed-5bf5-4d48-b72d-a71e7fda745e\") " pod="ai-tenant-e2e-ipp-a-bf45e0/ipp-route-bf45e0-kserve-8659d7675-mklh9" Sep 14 11:21:11.461329 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:11.461271 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/976aeaed-5bf5-4d48-b72d-a71e7fda745e-home\") pod \"ipp-route-bf45e0-kserve-8659d7675-mklh9\" (UID: \"976aeaed-5bf5-4d48-b72d-a71e7fda745e\") " pod="ai-tenant-e2e-ipp-a-bf45e0/ipp-route-bf45e0-kserve-8659d7675-mklh9" Sep 14 11:21:11.461594 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:11.461579 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/976aeaed-5bf5-4d48-b72d-a71e7fda745e-tmp-dir\") pod \"ipp-route-bf45e0-kserve-8659d7675-mklh9\" (UID: \"976aeaed-5bf5-4d48-b72d-a71e7fda745e\") " pod="ai-tenant-e2e-ipp-a-bf45e0/ipp-route-bf45e0-kserve-8659d7675-mklh9" Sep 14 11:21:11.461659 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:11.461644 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/976aeaed-5bf5-4d48-b72d-a71e7fda745e-model-cache\") pod \"ipp-route-bf45e0-kserve-8659d7675-mklh9\" (UID: \"976aeaed-5bf5-4d48-b72d-a71e7fda745e\") " pod="ai-tenant-e2e-ipp-a-bf45e0/ipp-route-bf45e0-kserve-8659d7675-mklh9" Sep 14 11:21:11.461706 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:11.461677 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/976aeaed-5bf5-4d48-b72d-a71e7fda745e-home\") pod \"ipp-route-bf45e0-kserve-8659d7675-mklh9\" (UID: \"976aeaed-5bf5-4d48-b72d-a71e7fda745e\") " pod="ai-tenant-e2e-ipp-a-bf45e0/ipp-route-bf45e0-kserve-8659d7675-mklh9" Sep 14 11:21:11.463550 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:11.463531 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/976aeaed-5bf5-4d48-b72d-a71e7fda745e-dshm\") pod \"ipp-route-bf45e0-kserve-8659d7675-mklh9\" (UID: \"976aeaed-5bf5-4d48-b72d-a71e7fda745e\") " pod="ai-tenant-e2e-ipp-a-bf45e0/ipp-route-bf45e0-kserve-8659d7675-mklh9" Sep 14 11:21:11.463818 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:11.463803 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/976aeaed-5bf5-4d48-b72d-a71e7fda745e-tls-certs\") pod \"ipp-route-bf45e0-kserve-8659d7675-mklh9\" (UID: \"976aeaed-5bf5-4d48-b72d-a71e7fda745e\") " pod="ai-tenant-e2e-ipp-a-bf45e0/ipp-route-bf45e0-kserve-8659d7675-mklh9" Sep 14 11:21:11.469633 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:11.469610 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdcfm\" (UniqueName: \"kubernetes.io/projected/976aeaed-5bf5-4d48-b72d-a71e7fda745e-kube-api-access-cdcfm\") pod \"ipp-route-bf45e0-kserve-8659d7675-mklh9\" (UID: \"976aeaed-5bf5-4d48-b72d-a71e7fda745e\") " pod="ai-tenant-e2e-ipp-a-bf45e0/ipp-route-bf45e0-kserve-8659d7675-mklh9" Sep 14 11:21:11.656375 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:11.656242 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-bf45e0/ipp-route-bf45e0-kserve-8659d7675-mklh9" Sep 14 11:21:11.929724 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:11.929700 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-bf45e0/ipp-route-bf45e0-kserve-8659d7675-mklh9"] Sep 14 11:21:11.932031 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:21:11.932003 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod976aeaed_5bf5_4d48_b72d_a71e7fda745e.slice/crio-ccad00994ea81c109a6f72bb6362e8ca583f2c907cbd0b3d36aa0a07bfed62f1 WatchSource:0}: Error finding container ccad00994ea81c109a6f72bb6362e8ca583f2c907cbd0b3d36aa0a07bfed62f1: Status 404 returned error can't find the container with id ccad00994ea81c109a6f72bb6362e8ca583f2c907cbd0b3d36aa0a07bfed62f1 Sep 14 11:21:12.927185 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:12.927146 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-bf45e0/ipp-route-bf45e0-kserve-8659d7675-mklh9" event={"ID":"976aeaed-5bf5-4d48-b72d-a71e7fda745e","Type":"ContainerStarted","Data":"bc016ae137b9dd6df3ca3b3a4db6972c9d58d8aacfbdf971e502ac6442c99c4b"} Sep 14 11:21:12.927781 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:12.927194 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-bf45e0/ipp-route-bf45e0-kserve-8659d7675-mklh9" event={"ID":"976aeaed-5bf5-4d48-b72d-a71e7fda745e","Type":"ContainerStarted","Data":"ccad00994ea81c109a6f72bb6362e8ca583f2c907cbd0b3d36aa0a07bfed62f1"} Sep 14 11:21:12.948327 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:12.948275 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-bf45e0/ipp-route-bf45e0-kserve-8659d7675-mklh9" podStartSLOduration=1.9482607939999999 podStartE2EDuration="1.948260794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:21:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:21:12.94649711 +0000 UTC m=+1946.688899077" watchObservedRunningTime="2026-09-14 11:21:12.948260794 +0000 UTC m=+1946.690662797" Sep 14 11:21:21.656678 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:21.656639 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-bf45e0/ipp-route-bf45e0-kserve-8659d7675-mklh9" Sep 14 11:21:21.657187 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:21.656695 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-bf45e0/ipp-route-bf45e0-kserve-8659d7675-mklh9" Sep 14 11:21:21.669242 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:21.669217 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-bf45e0/ipp-route-bf45e0-kserve-8659d7675-mklh9" Sep 14 11:21:21.987908 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:21.987826 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-bf45e0/ipp-route-bf45e0-kserve-8659d7675-mklh9" Sep 14 11:21:43.275326 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:43.275252 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-a521b5-9dcdbb9d7-x2f6n_82ba6ab8-0d94-4a59-a0bf-8999b2fe6c81/payload-processing/0.log" Sep 14 11:21:47.190548 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:47.190514 2581 scope.go:117] "RemoveContainer" containerID="aabd091a12dd4bdd9d59ddaa3fa59b11f662ab3e2905d77aedb3ba16264144f7" Sep 14 11:21:47.200074 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:47.200056 2581 scope.go:117] "RemoveContainer" containerID="c868048b83c9da07c10e15fbdfbe789bf2c99ff6a2c3b6c276aae6a3310c46ba" Sep 14 11:21:47.208322 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:47.208303 2581 scope.go:117] "RemoveContainer" containerID="c121336c35a4bef2685976d6b3e9dd4f217121fb0fca4536e1eb746e48b4a0a2" Sep 14 11:21:47.217006 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:47.216976 2581 scope.go:117] "RemoveContainer" containerID="7073b8cf5694b654d1fa2d82e9dccc5df1ebfb2aa81387b20bbd68552e3d8e58" Sep 14 11:21:52.746375 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:52.746324 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d5e418-85cfbd5f8-48mvs"] Sep 14 11:21:52.749847 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:52.749828 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d5e418-85cfbd5f8-48mvs" Sep 14 11:21:52.752673 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:52.752641 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-d5e418\"" Sep 14 11:21:52.752792 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:52.752717 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-d5e418-dockercfg-s7szn\"" Sep 14 11:21:52.760628 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:52.760598 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d5e418-85cfbd5f8-48mvs"] Sep 14 11:21:52.784877 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:52.784847 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsd9r\" (UniqueName: \"kubernetes.io/projected/fe259678-0e3d-4c72-99ab-59e0fb86e962-kube-api-access-zsd9r\") pod \"payload-pre-processing-e2e-ipp-cleanup-d5e418-85cfbd5f8-48mvs\" (UID: \"fe259678-0e3d-4c72-99ab-59e0fb86e962\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d5e418-85cfbd5f8-48mvs" Sep 14 11:21:52.785041 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:52.784911 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe259678-0e3d-4c72-99ab-59e0fb86e962-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-d5e418-85cfbd5f8-48mvs\" (UID: \"fe259678-0e3d-4c72-99ab-59e0fb86e962\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d5e418-85cfbd5f8-48mvs" Sep 14 11:21:52.886023 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:52.885987 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe259678-0e3d-4c72-99ab-59e0fb86e962-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-d5e418-85cfbd5f8-48mvs\" (UID: \"fe259678-0e3d-4c72-99ab-59e0fb86e962\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d5e418-85cfbd5f8-48mvs" Sep 14 11:21:52.886206 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:52.886067 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsd9r\" (UniqueName: \"kubernetes.io/projected/fe259678-0e3d-4c72-99ab-59e0fb86e962-kube-api-access-zsd9r\") pod \"payload-pre-processing-e2e-ipp-cleanup-d5e418-85cfbd5f8-48mvs\" (UID: \"fe259678-0e3d-4c72-99ab-59e0fb86e962\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d5e418-85cfbd5f8-48mvs" Sep 14 11:21:52.886693 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:52.886673 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe259678-0e3d-4c72-99ab-59e0fb86e962-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-d5e418-85cfbd5f8-48mvs\" (UID: \"fe259678-0e3d-4c72-99ab-59e0fb86e962\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d5e418-85cfbd5f8-48mvs" Sep 14 11:21:52.894684 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:52.894663 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsd9r\" (UniqueName: \"kubernetes.io/projected/fe259678-0e3d-4c72-99ab-59e0fb86e962-kube-api-access-zsd9r\") pod \"payload-pre-processing-e2e-ipp-cleanup-d5e418-85cfbd5f8-48mvs\" (UID: \"fe259678-0e3d-4c72-99ab-59e0fb86e962\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d5e418-85cfbd5f8-48mvs" Sep 14 11:21:53.060786 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:53.060732 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d5e418-85cfbd5f8-48mvs" Sep 14 11:21:53.338528 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:53.338502 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d5e418-85cfbd5f8-48mvs"] Sep 14 11:21:53.339619 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:21:53.339598 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe259678_0e3d_4c72_99ab_59e0fb86e962.slice/crio-f3819c712af531aae9cc51ab2f694601b79c0731d5a9ccc1bb4bb59e1ae33646 WatchSource:0}: Error finding container f3819c712af531aae9cc51ab2f694601b79c0731d5a9ccc1bb4bb59e1ae33646: Status 404 returned error can't find the container with id f3819c712af531aae9cc51ab2f694601b79c0731d5a9ccc1bb4bb59e1ae33646 Sep 14 11:21:54.149984 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:54.149946 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d5e418-85cfbd5f8-48mvs" event={"ID":"fe259678-0e3d-4c72-99ab-59e0fb86e962","Type":"ContainerStarted","Data":"ee37a20a2c7063b0a7ed522cdeb987ed97e87b5e24a48aba660d5cbda784ca4b"} Sep 14 11:21:54.150519 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:54.150006 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d5e418-85cfbd5f8-48mvs" event={"ID":"fe259678-0e3d-4c72-99ab-59e0fb86e962","Type":"ContainerStarted","Data":"f3819c712af531aae9cc51ab2f694601b79c0731d5a9ccc1bb4bb59e1ae33646"} Sep 14 11:21:54.150519 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:54.150086 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d5e418-85cfbd5f8-48mvs" Sep 14 11:21:54.167649 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:21:54.167601 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d5e418-85cfbd5f8-48mvs" podStartSLOduration=2.167585175 podStartE2EDuration="2.167585175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:21:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:21:54.166629795 +0000 UTC m=+1987.909031760" watchObservedRunningTime="2026-09-14 11:21:54.167585175 +0000 UTC m=+1987.909987145" Sep 14 11:22:05.158106 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:05.158075 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d5e418-85cfbd5f8-48mvs" Sep 14 11:22:18.046286 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:18.046251 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d5e418-p4bwg"] Sep 14 11:22:18.050082 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:18.050066 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d5e418-p4bwg" Sep 14 11:22:18.053548 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:18.053528 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pv9g8\"" Sep 14 11:22:18.091712 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:18.091679 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d5e418-p4bwg"] Sep 14 11:22:18.240924 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:18.240890 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83992a3f-dc0b-40e5-9e1f-d4b1fc0cd4bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-d5e418-p4bwg\" (UID: \"83992a3f-dc0b-40e5-9e1f-d4b1fc0cd4bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d5e418-p4bwg" Sep 14 11:22:18.341722 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:18.341629 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83992a3f-dc0b-40e5-9e1f-d4b1fc0cd4bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-d5e418-p4bwg\" (UID: \"83992a3f-dc0b-40e5-9e1f-d4b1fc0cd4bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d5e418-p4bwg" Sep 14 11:22:18.342300 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:18.342275 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83992a3f-dc0b-40e5-9e1f-d4b1fc0cd4bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-d5e418-p4bwg\" (UID: \"83992a3f-dc0b-40e5-9e1f-d4b1fc0cd4bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d5e418-p4bwg" Sep 14 11:22:18.360380 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:18.360330 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d5e418-p4bwg" Sep 14 11:22:18.647095 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:18.647069 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d5e418-p4bwg"] Sep 14 11:22:18.649571 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:22:18.649542 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83992a3f_dc0b_40e5_9e1f_d4b1fc0cd4bf.slice/crio-9a8fd0a95b8a5777571ea7a7c68378a4161f2109b2dc6bb3e3e7843a308c78bb WatchSource:0}: Error finding container 9a8fd0a95b8a5777571ea7a7c68378a4161f2109b2dc6bb3e3e7843a308c78bb: Status 404 returned error can't find the container with id 9a8fd0a95b8a5777571ea7a7c68378a4161f2109b2dc6bb3e3e7843a308c78bb Sep 14 11:22:19.286678 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:19.286642 2581 generic.go:356] "Generic (PLEG): container finished" podID="83992a3f-dc0b-40e5-9e1f-d4b1fc0cd4bf" containerID="073894664bb1a052abcc71443bac92f1d935f9fa81cf94574d3e3934a7bc0d42" exitCode=0 Sep 14 11:22:19.287049 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:19.286727 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d5e418-p4bwg" event={"ID":"83992a3f-dc0b-40e5-9e1f-d4b1fc0cd4bf","Type":"ContainerDied","Data":"073894664bb1a052abcc71443bac92f1d935f9fa81cf94574d3e3934a7bc0d42"} Sep 14 11:22:19.287049 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:19.286786 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d5e418-p4bwg" event={"ID":"83992a3f-dc0b-40e5-9e1f-d4b1fc0cd4bf","Type":"ContainerStarted","Data":"9a8fd0a95b8a5777571ea7a7c68378a4161f2109b2dc6bb3e3e7843a308c78bb"} Sep 14 11:22:20.429518 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:20.429489 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d5e418-p4bwg" Sep 14 11:22:20.461201 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:20.461165 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83992a3f-dc0b-40e5-9e1f-d4b1fc0cd4bf-maas-api-service-ca\") pod \"83992a3f-dc0b-40e5-9e1f-d4b1fc0cd4bf\" (UID: \"83992a3f-dc0b-40e5-9e1f-d4b1fc0cd4bf\") " Sep 14 11:22:20.461579 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:20.461559 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83992a3f-dc0b-40e5-9e1f-d4b1fc0cd4bf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "83992a3f-dc0b-40e5-9e1f-d4b1fc0cd4bf" (UID: "83992a3f-dc0b-40e5-9e1f-d4b1fc0cd4bf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:22:20.562363 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:20.562265 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83992a3f-dc0b-40e5-9e1f-d4b1fc0cd4bf-maas-api-service-ca\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:22:21.299988 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:21.299958 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d5e418-p4bwg" Sep 14 11:22:21.300191 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:21.299957 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d5e418-p4bwg" event={"ID":"83992a3f-dc0b-40e5-9e1f-d4b1fc0cd4bf","Type":"ContainerDied","Data":"9a8fd0a95b8a5777571ea7a7c68378a4161f2109b2dc6bb3e3e7843a308c78bb"} Sep 14 11:22:21.300191 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:21.300084 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a8fd0a95b8a5777571ea7a7c68378a4161f2109b2dc6bb3e3e7843a308c78bb" Sep 14 11:22:29.703247 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:29.703205 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d5e418-85cfbd5f8-48mvs"] Sep 14 11:22:29.703753 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:29.703523 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d5e418-85cfbd5f8-48mvs" podUID="fe259678-0e3d-4c72-99ab-59e0fb86e962" containerName="payload-pre-processing" containerID="cri-o://ee37a20a2c7063b0a7ed522cdeb987ed97e87b5e24a48aba660d5cbda784ca4b" gracePeriod=30 Sep 14 11:22:29.969818 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:29.969782 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d5e418-85cfbd5f8-48mvs" Sep 14 11:22:30.044902 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:30.044866 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d5e418-p4bwg"] Sep 14 11:22:30.049167 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:30.049137 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d5e418-p4bwg"] Sep 14 11:22:30.049446 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:30.049426 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsd9r\" (UniqueName: \"kubernetes.io/projected/fe259678-0e3d-4c72-99ab-59e0fb86e962-kube-api-access-zsd9r\") pod \"fe259678-0e3d-4c72-99ab-59e0fb86e962\" (UID: \"fe259678-0e3d-4c72-99ab-59e0fb86e962\") " Sep 14 11:22:30.049500 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:30.049460 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe259678-0e3d-4c72-99ab-59e0fb86e962-plugins-config-volume\") pod \"fe259678-0e3d-4c72-99ab-59e0fb86e962\" (UID: \"fe259678-0e3d-4c72-99ab-59e0fb86e962\") " Sep 14 11:22:30.049864 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:30.049842 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe259678-0e3d-4c72-99ab-59e0fb86e962-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fe259678-0e3d-4c72-99ab-59e0fb86e962" (UID: "fe259678-0e3d-4c72-99ab-59e0fb86e962"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:22:30.051772 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:30.051746 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe259678-0e3d-4c72-99ab-59e0fb86e962-kube-api-access-zsd9r" (OuterVolumeSpecName: "kube-api-access-zsd9r") pod "fe259678-0e3d-4c72-99ab-59e0fb86e962" (UID: "fe259678-0e3d-4c72-99ab-59e0fb86e962"). InnerVolumeSpecName "kube-api-access-zsd9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:22:30.151126 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:30.151092 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsd9r\" (UniqueName: \"kubernetes.io/projected/fe259678-0e3d-4c72-99ab-59e0fb86e962-kube-api-access-zsd9r\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:22:30.151126 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:30.151122 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe259678-0e3d-4c72-99ab-59e0fb86e962-plugins-config-volume\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:22:30.348556 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:30.348514 2581 generic.go:356] "Generic (PLEG): container finished" podID="fe259678-0e3d-4c72-99ab-59e0fb86e962" containerID="ee37a20a2c7063b0a7ed522cdeb987ed97e87b5e24a48aba660d5cbda784ca4b" exitCode=0 Sep 14 11:22:30.348731 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:30.348565 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d5e418-85cfbd5f8-48mvs" Sep 14 11:22:30.348731 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:30.348601 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d5e418-85cfbd5f8-48mvs" event={"ID":"fe259678-0e3d-4c72-99ab-59e0fb86e962","Type":"ContainerDied","Data":"ee37a20a2c7063b0a7ed522cdeb987ed97e87b5e24a48aba660d5cbda784ca4b"} Sep 14 11:22:30.348731 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:30.348663 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d5e418-85cfbd5f8-48mvs" event={"ID":"fe259678-0e3d-4c72-99ab-59e0fb86e962","Type":"ContainerDied","Data":"f3819c712af531aae9cc51ab2f694601b79c0731d5a9ccc1bb4bb59e1ae33646"} Sep 14 11:22:30.348731 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:30.348689 2581 scope.go:117] "RemoveContainer" containerID="ee37a20a2c7063b0a7ed522cdeb987ed97e87b5e24a48aba660d5cbda784ca4b" Sep 14 11:22:30.358076 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:30.358049 2581 scope.go:117] "RemoveContainer" containerID="ee37a20a2c7063b0a7ed522cdeb987ed97e87b5e24a48aba660d5cbda784ca4b" Sep 14 11:22:30.358408 ip-10-0-137-53 kubenswrapper[2581]: E0914 11:22:30.358373 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee37a20a2c7063b0a7ed522cdeb987ed97e87b5e24a48aba660d5cbda784ca4b\": container with ID starting with ee37a20a2c7063b0a7ed522cdeb987ed97e87b5e24a48aba660d5cbda784ca4b not found: ID does not exist" containerID="ee37a20a2c7063b0a7ed522cdeb987ed97e87b5e24a48aba660d5cbda784ca4b" Sep 14 11:22:30.358499 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:30.358413 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee37a20a2c7063b0a7ed522cdeb987ed97e87b5e24a48aba660d5cbda784ca4b"} err="failed to get container status \"ee37a20a2c7063b0a7ed522cdeb987ed97e87b5e24a48aba660d5cbda784ca4b\": rpc error: code = NotFound desc = could not find container \"ee37a20a2c7063b0a7ed522cdeb987ed97e87b5e24a48aba660d5cbda784ca4b\": container with ID starting with ee37a20a2c7063b0a7ed522cdeb987ed97e87b5e24a48aba660d5cbda784ca4b not found: ID does not exist" Sep 14 11:22:30.400251 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:30.400220 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d5e418-85cfbd5f8-48mvs"] Sep 14 11:22:30.412884 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:30.412854 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d5e418-85cfbd5f8-48mvs"] Sep 14 11:22:30.832242 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:30.832208 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83992a3f-dc0b-40e5-9e1f-d4b1fc0cd4bf" path="/var/lib/kubelet/pods/83992a3f-dc0b-40e5-9e1f-d4b1fc0cd4bf/volumes" Sep 14 11:22:30.832651 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:30.832617 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe259678-0e3d-4c72-99ab-59e0fb86e962" path="/var/lib/kubelet/pods/fe259678-0e3d-4c72-99ab-59e0fb86e962/volumes" Sep 14 11:22:41.323842 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:41.323806 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf45e0-tfvvf"] Sep 14 11:22:41.324231 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:41.324211 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe259678-0e3d-4c72-99ab-59e0fb86e962" containerName="payload-pre-processing" Sep 14 11:22:41.324231 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:41.324224 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe259678-0e3d-4c72-99ab-59e0fb86e962" containerName="payload-pre-processing" Sep 14 11:22:41.324305 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:41.324243 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83992a3f-dc0b-40e5-9e1f-d4b1fc0cd4bf" containerName="revoke-keys" Sep 14 11:22:41.324305 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:41.324248 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="83992a3f-dc0b-40e5-9e1f-d4b1fc0cd4bf" containerName="revoke-keys" Sep 14 11:22:41.324305 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:41.324301 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="83992a3f-dc0b-40e5-9e1f-d4b1fc0cd4bf" containerName="revoke-keys" Sep 14 11:22:41.324479 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:41.324313 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe259678-0e3d-4c72-99ab-59e0fb86e962" containerName="payload-pre-processing" Sep 14 11:22:41.328973 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:41.328957 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf45e0-tfvvf" Sep 14 11:22:41.332433 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:41.332411 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pv9g8\"" Sep 14 11:22:41.338741 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:41.338712 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf45e0-tfvvf"] Sep 14 11:22:41.463440 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:41.463406 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87930bfb-3b38-4b66-bdd1-ecb13fa034b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bf45e0-tfvvf\" (UID: \"87930bfb-3b38-4b66-bdd1-ecb13fa034b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf45e0-tfvvf" Sep 14 11:22:41.564463 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:41.564423 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87930bfb-3b38-4b66-bdd1-ecb13fa034b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bf45e0-tfvvf\" (UID: \"87930bfb-3b38-4b66-bdd1-ecb13fa034b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf45e0-tfvvf" Sep 14 11:22:41.565063 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:41.565037 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87930bfb-3b38-4b66-bdd1-ecb13fa034b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bf45e0-tfvvf\" (UID: \"87930bfb-3b38-4b66-bdd1-ecb13fa034b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf45e0-tfvvf" Sep 14 11:22:41.640694 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:41.640609 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf45e0-tfvvf" Sep 14 11:22:41.947127 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:41.947049 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf45e0-tfvvf"] Sep 14 11:22:41.951160 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:22:41.951132 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87930bfb_3b38_4b66_bdd1_ecb13fa034b8.slice/crio-d850cc96649bcd334de6f5a64b66dd1fe4eff3ef3dce78d58a6b57a7acba91b4 WatchSource:0}: Error finding container d850cc96649bcd334de6f5a64b66dd1fe4eff3ef3dce78d58a6b57a7acba91b4: Status 404 returned error can't find the container with id d850cc96649bcd334de6f5a64b66dd1fe4eff3ef3dce78d58a6b57a7acba91b4 Sep 14 11:22:42.412113 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:42.412078 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf45e0-tfvvf" event={"ID":"87930bfb-3b38-4b66-bdd1-ecb13fa034b8","Type":"ContainerStarted","Data":"a732983009a7c0528d3169390ffefaa85b7ee774266a8918e3b1ab82d8e75954"} Sep 14 11:22:42.412519 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:42.412130 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf45e0-tfvvf" event={"ID":"87930bfb-3b38-4b66-bdd1-ecb13fa034b8","Type":"ContainerStarted","Data":"d850cc96649bcd334de6f5a64b66dd1fe4eff3ef3dce78d58a6b57a7acba91b4"} Sep 14 11:22:42.433119 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:42.433047 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf45e0-tfvvf" podStartSLOduration=1.433031404 podStartE2EDuration="1.433031404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:22:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:22:42.432251157 +0000 UTC m=+2036.174653128" watchObservedRunningTime="2026-09-14 11:22:42.433031404 +0000 UTC m=+2036.175433374" Sep 14 11:22:43.418627 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:43.418591 2581 generic.go:356] "Generic (PLEG): container finished" podID="87930bfb-3b38-4b66-bdd1-ecb13fa034b8" containerID="a732983009a7c0528d3169390ffefaa85b7ee774266a8918e3b1ab82d8e75954" exitCode=0 Sep 14 11:22:43.419027 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:43.418684 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf45e0-tfvvf" event={"ID":"87930bfb-3b38-4b66-bdd1-ecb13fa034b8","Type":"ContainerDied","Data":"a732983009a7c0528d3169390ffefaa85b7ee774266a8918e3b1ab82d8e75954"} Sep 14 11:22:44.563135 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:44.563106 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf45e0-tfvvf" Sep 14 11:22:44.696504 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:44.696408 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87930bfb-3b38-4b66-bdd1-ecb13fa034b8-maas-api-service-ca\") pod \"87930bfb-3b38-4b66-bdd1-ecb13fa034b8\" (UID: \"87930bfb-3b38-4b66-bdd1-ecb13fa034b8\") " Sep 14 11:22:44.696857 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:44.696832 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87930bfb-3b38-4b66-bdd1-ecb13fa034b8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "87930bfb-3b38-4b66-bdd1-ecb13fa034b8" (UID: "87930bfb-3b38-4b66-bdd1-ecb13fa034b8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:22:44.797548 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:44.797509 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87930bfb-3b38-4b66-bdd1-ecb13fa034b8-maas-api-service-ca\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:22:45.431469 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:45.431436 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf45e0-tfvvf" Sep 14 11:22:45.431469 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:45.431453 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf45e0-tfvvf" event={"ID":"87930bfb-3b38-4b66-bdd1-ecb13fa034b8","Type":"ContainerDied","Data":"d850cc96649bcd334de6f5a64b66dd1fe4eff3ef3dce78d58a6b57a7acba91b4"} Sep 14 11:22:45.431656 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:45.431500 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d850cc96649bcd334de6f5a64b66dd1fe4eff3ef3dce78d58a6b57a7acba91b4" Sep 14 11:22:47.248880 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:47.248848 2581 scope.go:117] "RemoveContainer" containerID="daa5b28dee3ef0e17fc1e74a6217a7965cc7fb60bd194438c853bd13335a9070" Sep 14 11:22:57.088442 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:57.088403 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf45e0-8f49c4bbb-jk5nj"] Sep 14 11:22:57.088960 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:57.088663 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf45e0-8f49c4bbb-jk5nj" podUID="de247584-a356-4585-8b49-99e8c2985941" containerName="payload-pre-processing" containerID="cri-o://f1b4985a5d5ceaa51eb64a685352bb6a91e23e608bbd886991409e2410c56bce" gracePeriod=30 Sep 14 11:22:57.323883 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:57.323848 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf45e0-tfvvf"] Sep 14 11:22:57.326456 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:57.326428 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf45e0-tfvvf"] Sep 14 11:22:57.352842 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:57.352775 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf45e0-8f49c4bbb-jk5nj" Sep 14 11:22:57.494644 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:57.494610 2581 generic.go:356] "Generic (PLEG): container finished" podID="de247584-a356-4585-8b49-99e8c2985941" containerID="f1b4985a5d5ceaa51eb64a685352bb6a91e23e608bbd886991409e2410c56bce" exitCode=0 Sep 14 11:22:57.494850 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:57.494656 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf45e0-8f49c4bbb-jk5nj" Sep 14 11:22:57.494850 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:57.494702 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf45e0-8f49c4bbb-jk5nj" event={"ID":"de247584-a356-4585-8b49-99e8c2985941","Type":"ContainerDied","Data":"f1b4985a5d5ceaa51eb64a685352bb6a91e23e608bbd886991409e2410c56bce"} Sep 14 11:22:57.494850 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:57.494759 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf45e0-8f49c4bbb-jk5nj" event={"ID":"de247584-a356-4585-8b49-99e8c2985941","Type":"ContainerDied","Data":"28f94aee1a39922f2e496d64dafd43435cdf623897a92baa7e14807a9fb67619"} Sep 14 11:22:57.494850 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:57.494790 2581 scope.go:117] "RemoveContainer" containerID="f1b4985a5d5ceaa51eb64a685352bb6a91e23e608bbd886991409e2410c56bce" Sep 14 11:22:57.503402 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:57.503382 2581 scope.go:117] "RemoveContainer" containerID="f1b4985a5d5ceaa51eb64a685352bb6a91e23e608bbd886991409e2410c56bce" Sep 14 11:22:57.503738 ip-10-0-137-53 kubenswrapper[2581]: E0914 11:22:57.503719 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1b4985a5d5ceaa51eb64a685352bb6a91e23e608bbd886991409e2410c56bce\": container with ID starting with f1b4985a5d5ceaa51eb64a685352bb6a91e23e608bbd886991409e2410c56bce not found: ID does not exist" containerID="f1b4985a5d5ceaa51eb64a685352bb6a91e23e608bbd886991409e2410c56bce" Sep 14 11:22:57.503790 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:57.503749 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1b4985a5d5ceaa51eb64a685352bb6a91e23e608bbd886991409e2410c56bce"} err="failed to get container status \"f1b4985a5d5ceaa51eb64a685352bb6a91e23e608bbd886991409e2410c56bce\": rpc error: code = NotFound desc = could not find container \"f1b4985a5d5ceaa51eb64a685352bb6a91e23e608bbd886991409e2410c56bce\": container with ID starting with f1b4985a5d5ceaa51eb64a685352bb6a91e23e608bbd886991409e2410c56bce not found: ID does not exist" Sep 14 11:22:57.519856 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:57.519828 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de247584-a356-4585-8b49-99e8c2985941-plugins-config-volume\") pod \"de247584-a356-4585-8b49-99e8c2985941\" (UID: \"de247584-a356-4585-8b49-99e8c2985941\") " Sep 14 11:22:57.520029 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:57.519869 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ff2wl\" (UniqueName: \"kubernetes.io/projected/de247584-a356-4585-8b49-99e8c2985941-kube-api-access-ff2wl\") pod \"de247584-a356-4585-8b49-99e8c2985941\" (UID: \"de247584-a356-4585-8b49-99e8c2985941\") " Sep 14 11:22:57.520201 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:57.520180 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de247584-a356-4585-8b49-99e8c2985941-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "de247584-a356-4585-8b49-99e8c2985941" (UID: "de247584-a356-4585-8b49-99e8c2985941"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:22:57.522036 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:57.522016 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de247584-a356-4585-8b49-99e8c2985941-kube-api-access-ff2wl" (OuterVolumeSpecName: "kube-api-access-ff2wl") pod "de247584-a356-4585-8b49-99e8c2985941" (UID: "de247584-a356-4585-8b49-99e8c2985941"). InnerVolumeSpecName "kube-api-access-ff2wl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:22:57.621071 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:57.620978 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de247584-a356-4585-8b49-99e8c2985941-plugins-config-volume\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:22:57.621071 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:57.621011 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff2wl\" (UniqueName: \"kubernetes.io/projected/de247584-a356-4585-8b49-99e8c2985941-kube-api-access-ff2wl\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:22:57.838593 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:57.838558 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf45e0-8f49c4bbb-jk5nj"] Sep 14 11:22:57.846395 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:57.846359 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf45e0-8f49c4bbb-jk5nj"] Sep 14 11:22:58.832026 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:58.831982 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87930bfb-3b38-4b66-bdd1-ecb13fa034b8" path="/var/lib/kubelet/pods/87930bfb-3b38-4b66-bdd1-ecb13fa034b8/volumes" Sep 14 11:22:58.832561 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:22:58.832539 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de247584-a356-4585-8b49-99e8c2985941" path="/var/lib/kubelet/pods/de247584-a356-4585-8b49-99e8c2985941/volumes" Sep 14 11:23:02.732291 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:23:02.732237 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-bf45e0/ipp-route-bf45e0-kserve-8659d7675-mklh9"] Sep 14 11:23:17.752437 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:23:17.752294 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-bf45e0/ipp-route-bf45e0-kserve-8659d7675-mklh9" podUID="976aeaed-5bf5-4d48-b72d-a71e7fda745e" containerName="main" containerID="cri-o://bc016ae137b9dd6df3ca3b3a4db6972c9d58d8aacfbdf971e502ac6442c99c4b" gracePeriod=45 Sep 14 11:23:18.004038 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:23:18.003976 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-bf45e0/ipp-route-bf45e0-kserve-8659d7675-mklh9" Sep 14 11:23:18.112599 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:23:18.112569 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/976aeaed-5bf5-4d48-b72d-a71e7fda745e-home\") pod \"976aeaed-5bf5-4d48-b72d-a71e7fda745e\" (UID: \"976aeaed-5bf5-4d48-b72d-a71e7fda745e\") " Sep 14 11:23:18.112798 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:23:18.112620 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/976aeaed-5bf5-4d48-b72d-a71e7fda745e-tmp-dir\") pod \"976aeaed-5bf5-4d48-b72d-a71e7fda745e\" (UID: \"976aeaed-5bf5-4d48-b72d-a71e7fda745e\") " Sep 14 11:23:18.112798 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:23:18.112667 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/976aeaed-5bf5-4d48-b72d-a71e7fda745e-model-cache\") pod \"976aeaed-5bf5-4d48-b72d-a71e7fda745e\" (UID: \"976aeaed-5bf5-4d48-b72d-a71e7fda745e\") " Sep 14 11:23:18.112798 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:23:18.112724 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdcfm\" (UniqueName: \"kubernetes.io/projected/976aeaed-5bf5-4d48-b72d-a71e7fda745e-kube-api-access-cdcfm\") pod \"976aeaed-5bf5-4d48-b72d-a71e7fda745e\" (UID: \"976aeaed-5bf5-4d48-b72d-a71e7fda745e\") " Sep 14 11:23:18.112798 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:23:18.112755 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/976aeaed-5bf5-4d48-b72d-a71e7fda745e-dshm\") pod \"976aeaed-5bf5-4d48-b72d-a71e7fda745e\" (UID: \"976aeaed-5bf5-4d48-b72d-a71e7fda745e\") " Sep 14 11:23:18.113013 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:23:18.112806 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/976aeaed-5bf5-4d48-b72d-a71e7fda745e-tls-certs\") pod \"976aeaed-5bf5-4d48-b72d-a71e7fda745e\" (UID: \"976aeaed-5bf5-4d48-b72d-a71e7fda745e\") " Sep 14 11:23:18.113013 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:23:18.112922 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/976aeaed-5bf5-4d48-b72d-a71e7fda745e-home" (OuterVolumeSpecName: "home") pod "976aeaed-5bf5-4d48-b72d-a71e7fda745e" (UID: "976aeaed-5bf5-4d48-b72d-a71e7fda745e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:23:18.113013 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:23:18.112960 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/976aeaed-5bf5-4d48-b72d-a71e7fda745e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "976aeaed-5bf5-4d48-b72d-a71e7fda745e" (UID: "976aeaed-5bf5-4d48-b72d-a71e7fda745e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:23:18.113458 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:23:18.113007 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/976aeaed-5bf5-4d48-b72d-a71e7fda745e-model-cache" (OuterVolumeSpecName: "model-cache") pod "976aeaed-5bf5-4d48-b72d-a71e7fda745e" (UID: "976aeaed-5bf5-4d48-b72d-a71e7fda745e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:23:18.113458 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:23:18.113160 2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/976aeaed-5bf5-4d48-b72d-a71e7fda745e-home\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:23:18.113458 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:23:18.113177 2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/976aeaed-5bf5-4d48-b72d-a71e7fda745e-tmp-dir\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:23:18.113458 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:23:18.113191 2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/976aeaed-5bf5-4d48-b72d-a71e7fda745e-model-cache\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:23:18.115127 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:23:18.115105 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/976aeaed-5bf5-4d48-b72d-a71e7fda745e-dshm" (OuterVolumeSpecName: "dshm") pod "976aeaed-5bf5-4d48-b72d-a71e7fda745e" (UID: "976aeaed-5bf5-4d48-b72d-a71e7fda745e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:23:18.115127 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:23:18.115106 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/976aeaed-5bf5-4d48-b72d-a71e7fda745e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "976aeaed-5bf5-4d48-b72d-a71e7fda745e" (UID: "976aeaed-5bf5-4d48-b72d-a71e7fda745e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:23:18.115277 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:23:18.115134 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/976aeaed-5bf5-4d48-b72d-a71e7fda745e-kube-api-access-cdcfm" (OuterVolumeSpecName: "kube-api-access-cdcfm") pod "976aeaed-5bf5-4d48-b72d-a71e7fda745e" (UID: "976aeaed-5bf5-4d48-b72d-a71e7fda745e"). InnerVolumeSpecName "kube-api-access-cdcfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:23:18.220500 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:23:18.220452 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdcfm\" (UniqueName: \"kubernetes.io/projected/976aeaed-5bf5-4d48-b72d-a71e7fda745e-kube-api-access-cdcfm\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:23:18.220500 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:23:18.220485 2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/976aeaed-5bf5-4d48-b72d-a71e7fda745e-dshm\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:23:18.220500 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:23:18.220500 2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/976aeaed-5bf5-4d48-b72d-a71e7fda745e-tls-certs\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:23:18.610921 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:23:18.610880 2581 generic.go:356] "Generic (PLEG): container finished" podID="976aeaed-5bf5-4d48-b72d-a71e7fda745e" containerID="bc016ae137b9dd6df3ca3b3a4db6972c9d58d8aacfbdf971e502ac6442c99c4b" exitCode=0 Sep 14 11:23:18.611115 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:23:18.610955 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-bf45e0/ipp-route-bf45e0-kserve-8659d7675-mklh9" Sep 14 11:23:18.611115 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:23:18.610958 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-bf45e0/ipp-route-bf45e0-kserve-8659d7675-mklh9" event={"ID":"976aeaed-5bf5-4d48-b72d-a71e7fda745e","Type":"ContainerDied","Data":"bc016ae137b9dd6df3ca3b3a4db6972c9d58d8aacfbdf971e502ac6442c99c4b"} Sep 14 11:23:18.611115 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:23:18.611005 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-bf45e0/ipp-route-bf45e0-kserve-8659d7675-mklh9" event={"ID":"976aeaed-5bf5-4d48-b72d-a71e7fda745e","Type":"ContainerDied","Data":"ccad00994ea81c109a6f72bb6362e8ca583f2c907cbd0b3d36aa0a07bfed62f1"} Sep 14 11:23:18.611115 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:23:18.611022 2581 scope.go:117] "RemoveContainer" containerID="bc016ae137b9dd6df3ca3b3a4db6972c9d58d8aacfbdf971e502ac6442c99c4b" Sep 14 11:23:18.620734 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:23:18.620716 2581 scope.go:117] "RemoveContainer" containerID="bc016ae137b9dd6df3ca3b3a4db6972c9d58d8aacfbdf971e502ac6442c99c4b" Sep 14 11:23:18.620999 ip-10-0-137-53 kubenswrapper[2581]: E0914 11:23:18.620979 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc016ae137b9dd6df3ca3b3a4db6972c9d58d8aacfbdf971e502ac6442c99c4b\": container with ID starting with bc016ae137b9dd6df3ca3b3a4db6972c9d58d8aacfbdf971e502ac6442c99c4b not found: ID does not exist" containerID="bc016ae137b9dd6df3ca3b3a4db6972c9d58d8aacfbdf971e502ac6442c99c4b" Sep 14 11:23:18.621074 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:23:18.621006 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc016ae137b9dd6df3ca3b3a4db6972c9d58d8aacfbdf971e502ac6442c99c4b"} err="failed to get container status \"bc016ae137b9dd6df3ca3b3a4db6972c9d58d8aacfbdf971e502ac6442c99c4b\": rpc error: code = NotFound desc = could not find container \"bc016ae137b9dd6df3ca3b3a4db6972c9d58d8aacfbdf971e502ac6442c99c4b\": container with ID starting with bc016ae137b9dd6df3ca3b3a4db6972c9d58d8aacfbdf971e502ac6442c99c4b not found: ID does not exist" Sep 14 11:23:18.642739 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:23:18.642705 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-bf45e0/ipp-route-bf45e0-kserve-8659d7675-mklh9"] Sep 14 11:23:18.655610 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:23:18.655583 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-bf45e0/ipp-route-bf45e0-kserve-8659d7675-mklh9"] Sep 14 11:23:18.832390 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:23:18.832357 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="976aeaed-5bf5-4d48-b72d-a71e7fda745e" path="/var/lib/kubelet/pods/976aeaed-5bf5-4d48-b72d-a71e7fda745e/volumes" Sep 14 11:23:47.275420 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:23:47.275389 2581 scope.go:117] "RemoveContainer" containerID="978792c3528e385886414a8ef90c1a497acdde9f27d4288d1053716d61dee22a" Sep 14 11:23:47.284370 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:23:47.284324 2581 scope.go:117] "RemoveContainer" containerID="627eb5cef607f297d0176b77e331b925c0adfff1604c7326476f864043239470" Sep 14 11:24:12.616972 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:12.616931 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a521b5-xp7vx"] Sep 14 11:24:12.617650 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:12.617630 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="976aeaed-5bf5-4d48-b72d-a71e7fda745e" containerName="main" Sep 14 11:24:12.617734 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:12.617655 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="976aeaed-5bf5-4d48-b72d-a71e7fda745e" containerName="main" Sep 14 11:24:12.617734 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:12.617686 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87930bfb-3b38-4b66-bdd1-ecb13fa034b8" containerName="revoke-keys" Sep 14 11:24:12.617734 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:12.617695 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="87930bfb-3b38-4b66-bdd1-ecb13fa034b8" containerName="revoke-keys" Sep 14 11:24:12.617734 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:12.617717 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de247584-a356-4585-8b49-99e8c2985941" containerName="payload-pre-processing" Sep 14 11:24:12.617734 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:12.617725 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="de247584-a356-4585-8b49-99e8c2985941" containerName="payload-pre-processing" Sep 14 11:24:12.618007 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:12.617817 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="976aeaed-5bf5-4d48-b72d-a71e7fda745e" containerName="main" Sep 14 11:24:12.618007 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:12.617831 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="de247584-a356-4585-8b49-99e8c2985941" containerName="payload-pre-processing" Sep 14 11:24:12.618007 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:12.617841 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="87930bfb-3b38-4b66-bdd1-ecb13fa034b8" containerName="revoke-keys" Sep 14 11:24:12.621226 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:12.621206 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a521b5-xp7vx" Sep 14 11:24:12.625133 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:12.625111 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pv9g8\"" Sep 14 11:24:12.635116 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:12.635086 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a521b5-xp7vx"] Sep 14 11:24:12.755042 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:12.754983 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42d4dbf7-2b43-40e2-b701-100cd9b36bfe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a521b5-xp7vx\" (UID: \"42d4dbf7-2b43-40e2-b701-100cd9b36bfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a521b5-xp7vx" Sep 14 11:24:12.856181 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:12.856146 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42d4dbf7-2b43-40e2-b701-100cd9b36bfe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a521b5-xp7vx\" (UID: \"42d4dbf7-2b43-40e2-b701-100cd9b36bfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a521b5-xp7vx" Sep 14 11:24:12.856813 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:12.856791 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42d4dbf7-2b43-40e2-b701-100cd9b36bfe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a521b5-xp7vx\" (UID: \"42d4dbf7-2b43-40e2-b701-100cd9b36bfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a521b5-xp7vx" Sep 14 11:24:12.932041 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:12.931939 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a521b5-xp7vx" Sep 14 11:24:13.226253 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:13.226228 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a521b5-xp7vx"] Sep 14 11:24:13.228624 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:24:13.228602 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d4dbf7_2b43_40e2_b701_100cd9b36bfe.slice/crio-5a6199de10efb5cba14f7e3046f358dce6ec7afbec3912ef6777e2819fad94e1 WatchSource:0}: Error finding container 5a6199de10efb5cba14f7e3046f358dce6ec7afbec3912ef6777e2819fad94e1: Status 404 returned error can't find the container with id 5a6199de10efb5cba14f7e3046f358dce6ec7afbec3912ef6777e2819fad94e1 Sep 14 11:24:13.896866 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:13.896824 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a521b5-xp7vx" event={"ID":"42d4dbf7-2b43-40e2-b701-100cd9b36bfe","Type":"ContainerStarted","Data":"57b6c4c9bd46b237fab8f41e15c902759462ca57ef25beb94e7ee1e36e14ac53"} Sep 14 11:24:13.897298 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:13.896889 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a521b5-xp7vx" event={"ID":"42d4dbf7-2b43-40e2-b701-100cd9b36bfe","Type":"ContainerStarted","Data":"5a6199de10efb5cba14f7e3046f358dce6ec7afbec3912ef6777e2819fad94e1"} Sep 14 11:24:13.935039 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:13.934991 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a521b5-xp7vx" podStartSLOduration=1.934975428 podStartE2EDuration="1.934975428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:24:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:24:13.934713974 +0000 UTC m=+2127.677115945" watchObservedRunningTime="2026-09-14 11:24:13.934975428 +0000 UTC m=+2127.677377397" Sep 14 11:24:14.903563 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:14.903527 2581 generic.go:356] "Generic (PLEG): container finished" podID="42d4dbf7-2b43-40e2-b701-100cd9b36bfe" containerID="57b6c4c9bd46b237fab8f41e15c902759462ca57ef25beb94e7ee1e36e14ac53" exitCode=0 Sep 14 11:24:14.903993 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:14.903624 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a521b5-xp7vx" event={"ID":"42d4dbf7-2b43-40e2-b701-100cd9b36bfe","Type":"ContainerDied","Data":"57b6c4c9bd46b237fab8f41e15c902759462ca57ef25beb94e7ee1e36e14ac53"} Sep 14 11:24:16.045663 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:16.045634 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a521b5-xp7vx" Sep 14 11:24:16.188578 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:16.188477 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42d4dbf7-2b43-40e2-b701-100cd9b36bfe-maas-api-service-ca\") pod \"42d4dbf7-2b43-40e2-b701-100cd9b36bfe\" (UID: \"42d4dbf7-2b43-40e2-b701-100cd9b36bfe\") " Sep 14 11:24:16.188891 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:16.188864 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42d4dbf7-2b43-40e2-b701-100cd9b36bfe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "42d4dbf7-2b43-40e2-b701-100cd9b36bfe" (UID: "42d4dbf7-2b43-40e2-b701-100cd9b36bfe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:24:16.289612 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:16.289571 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42d4dbf7-2b43-40e2-b701-100cd9b36bfe-maas-api-service-ca\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:24:16.915232 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:16.915196 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a521b5-xp7vx" event={"ID":"42d4dbf7-2b43-40e2-b701-100cd9b36bfe","Type":"ContainerDied","Data":"5a6199de10efb5cba14f7e3046f358dce6ec7afbec3912ef6777e2819fad94e1"} Sep 14 11:24:16.915232 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:16.915226 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a521b5-xp7vx" Sep 14 11:24:16.915463 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:16.915257 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a6199de10efb5cba14f7e3046f358dce6ec7afbec3912ef6777e2819fad94e1" Sep 14 11:24:23.330931 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:23.330895 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a521b5-9dcdbb9d7-x2f6n"] Sep 14 11:24:23.331432 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:23.331326 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a521b5-9dcdbb9d7-x2f6n" podUID="82ba6ab8-0d94-4a59-a0bf-8999b2fe6c81" containerName="payload-processing" containerID="cri-o://61b85f754d16a4d925f62e74b3dace9e752e79bfb6c0cfcabe519c487e603e7f" gracePeriod=30 Sep 14 11:24:23.587186 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:23.587122 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a521b5-9dcdbb9d7-x2f6n" Sep 14 11:24:23.659097 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:23.659063 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82ba6ab8-0d94-4a59-a0bf-8999b2fe6c81-plugins-config-volume\") pod \"82ba6ab8-0d94-4a59-a0bf-8999b2fe6c81\" (UID: \"82ba6ab8-0d94-4a59-a0bf-8999b2fe6c81\") " Sep 14 11:24:23.659267 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:23.659202 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6p8c8\" (UniqueName: \"kubernetes.io/projected/82ba6ab8-0d94-4a59-a0bf-8999b2fe6c81-kube-api-access-6p8c8\") pod \"82ba6ab8-0d94-4a59-a0bf-8999b2fe6c81\" (UID: \"82ba6ab8-0d94-4a59-a0bf-8999b2fe6c81\") " Sep 14 11:24:23.659480 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:23.659457 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82ba6ab8-0d94-4a59-a0bf-8999b2fe6c81-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "82ba6ab8-0d94-4a59-a0bf-8999b2fe6c81" (UID: "82ba6ab8-0d94-4a59-a0bf-8999b2fe6c81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:24:23.661462 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:23.661439 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82ba6ab8-0d94-4a59-a0bf-8999b2fe6c81-kube-api-access-6p8c8" (OuterVolumeSpecName: "kube-api-access-6p8c8") pod "82ba6ab8-0d94-4a59-a0bf-8999b2fe6c81" (UID: "82ba6ab8-0d94-4a59-a0bf-8999b2fe6c81"). InnerVolumeSpecName "kube-api-access-6p8c8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:24:23.684205 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:23.684170 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a521b5-xp7vx"] Sep 14 11:24:23.691305 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:23.691280 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a521b5-xp7vx"] Sep 14 11:24:23.760298 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:23.760261 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p8c8\" (UniqueName: \"kubernetes.io/projected/82ba6ab8-0d94-4a59-a0bf-8999b2fe6c81-kube-api-access-6p8c8\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:24:23.760298 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:23.760290 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82ba6ab8-0d94-4a59-a0bf-8999b2fe6c81-plugins-config-volume\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:24:23.954568 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:23.954473 2581 generic.go:356] "Generic (PLEG): container finished" podID="82ba6ab8-0d94-4a59-a0bf-8999b2fe6c81" containerID="61b85f754d16a4d925f62e74b3dace9e752e79bfb6c0cfcabe519c487e603e7f" exitCode=0 Sep 14 11:24:23.954568 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:23.954529 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a521b5-9dcdbb9d7-x2f6n" Sep 14 11:24:23.954568 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:23.954558 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a521b5-9dcdbb9d7-x2f6n" event={"ID":"82ba6ab8-0d94-4a59-a0bf-8999b2fe6c81","Type":"ContainerDied","Data":"61b85f754d16a4d925f62e74b3dace9e752e79bfb6c0cfcabe519c487e603e7f"} Sep 14 11:24:23.954814 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:23.954641 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a521b5-9dcdbb9d7-x2f6n" event={"ID":"82ba6ab8-0d94-4a59-a0bf-8999b2fe6c81","Type":"ContainerDied","Data":"02b3123c62fe55d78a03bf6d176742829be52ca12731fc527324554d1cff36ea"} Sep 14 11:24:23.954814 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:23.954663 2581 scope.go:117] "RemoveContainer" containerID="61b85f754d16a4d925f62e74b3dace9e752e79bfb6c0cfcabe519c487e603e7f" Sep 14 11:24:23.964149 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:23.964129 2581 scope.go:117] "RemoveContainer" containerID="61b85f754d16a4d925f62e74b3dace9e752e79bfb6c0cfcabe519c487e603e7f" Sep 14 11:24:23.964488 ip-10-0-137-53 kubenswrapper[2581]: E0914 11:24:23.964469 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61b85f754d16a4d925f62e74b3dace9e752e79bfb6c0cfcabe519c487e603e7f\": container with ID starting with 61b85f754d16a4d925f62e74b3dace9e752e79bfb6c0cfcabe519c487e603e7f not found: ID does not exist" containerID="61b85f754d16a4d925f62e74b3dace9e752e79bfb6c0cfcabe519c487e603e7f" Sep 14 11:24:23.964556 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:23.964496 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61b85f754d16a4d925f62e74b3dace9e752e79bfb6c0cfcabe519c487e603e7f"} err="failed to get container status \"61b85f754d16a4d925f62e74b3dace9e752e79bfb6c0cfcabe519c487e603e7f\": rpc error: code = NotFound desc = could not find container \"61b85f754d16a4d925f62e74b3dace9e752e79bfb6c0cfcabe519c487e603e7f\": container with ID starting with 61b85f754d16a4d925f62e74b3dace9e752e79bfb6c0cfcabe519c487e603e7f not found: ID does not exist" Sep 14 11:24:23.991105 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:23.991071 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a521b5-9dcdbb9d7-x2f6n"] Sep 14 11:24:23.998652 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:23.998623 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a521b5-9dcdbb9d7-x2f6n"] Sep 14 11:24:24.831837 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:24.831801 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42d4dbf7-2b43-40e2-b701-100cd9b36bfe" path="/var/lib/kubelet/pods/42d4dbf7-2b43-40e2-b701-100cd9b36bfe/volumes" Sep 14 11:24:24.832227 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:24.832163 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82ba6ab8-0d94-4a59-a0bf-8999b2fe6c81" path="/var/lib/kubelet/pods/82ba6ab8-0d94-4a59-a0bf-8999b2fe6c81/volumes" Sep 14 11:24:31.921054 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:31.921014 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fdbc00-bm6ds"] Sep 14 11:24:31.921462 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:31.921408 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42d4dbf7-2b43-40e2-b701-100cd9b36bfe" containerName="revoke-keys" Sep 14 11:24:31.921462 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:31.921419 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="42d4dbf7-2b43-40e2-b701-100cd9b36bfe" containerName="revoke-keys" Sep 14 11:24:31.921462 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:31.921430 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82ba6ab8-0d94-4a59-a0bf-8999b2fe6c81" containerName="payload-processing" Sep 14 11:24:31.921462 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:31.921435 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="82ba6ab8-0d94-4a59-a0bf-8999b2fe6c81" containerName="payload-processing" Sep 14 11:24:31.921597 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:31.921514 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="82ba6ab8-0d94-4a59-a0bf-8999b2fe6c81" containerName="payload-processing" Sep 14 11:24:31.921597 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:31.921522 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="42d4dbf7-2b43-40e2-b701-100cd9b36bfe" containerName="revoke-keys" Sep 14 11:24:31.924584 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:31.924568 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fdbc00-bm6ds" Sep 14 11:24:31.928480 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:31.928460 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pv9g8\"" Sep 14 11:24:31.950325 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:31.950293 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fdbc00-bm6ds"] Sep 14 11:24:32.040675 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:32.040636 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43702cb6-6807-40b1-ad8b-c7ba10bcf65d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-fdbc00-bm6ds\" (UID: \"43702cb6-6807-40b1-ad8b-c7ba10bcf65d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fdbc00-bm6ds" Sep 14 11:24:32.142322 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:32.142286 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43702cb6-6807-40b1-ad8b-c7ba10bcf65d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-fdbc00-bm6ds\" (UID: \"43702cb6-6807-40b1-ad8b-c7ba10bcf65d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fdbc00-bm6ds" Sep 14 11:24:32.142991 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:32.142969 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43702cb6-6807-40b1-ad8b-c7ba10bcf65d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-fdbc00-bm6ds\" (UID: \"43702cb6-6807-40b1-ad8b-c7ba10bcf65d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fdbc00-bm6ds" Sep 14 11:24:32.234259 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:32.234172 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fdbc00-bm6ds" Sep 14 11:24:32.553968 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:32.553945 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fdbc00-bm6ds"] Sep 14 11:24:32.556564 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:24:32.556540 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43702cb6_6807_40b1_ad8b_c7ba10bcf65d.slice/crio-4e6b0340a654af473f21ed348952a0da1e00da2c330570d811487294c43e3217 WatchSource:0}: Error finding container 4e6b0340a654af473f21ed348952a0da1e00da2c330570d811487294c43e3217: Status 404 returned error can't find the container with id 4e6b0340a654af473f21ed348952a0da1e00da2c330570d811487294c43e3217 Sep 14 11:24:33.002529 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:33.002492 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fdbc00-bm6ds" event={"ID":"43702cb6-6807-40b1-ad8b-c7ba10bcf65d","Type":"ContainerStarted","Data":"2160fde79123edaea75d5b8eb5db333eabb96ed38471f6a091c1a05acdcd5870"} Sep 14 11:24:33.002916 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:33.002548 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fdbc00-bm6ds" event={"ID":"43702cb6-6807-40b1-ad8b-c7ba10bcf65d","Type":"ContainerStarted","Data":"4e6b0340a654af473f21ed348952a0da1e00da2c330570d811487294c43e3217"} Sep 14 11:24:33.042069 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:33.042013 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fdbc00-bm6ds" podStartSLOduration=2.041994795 podStartE2EDuration="2.041994795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:24:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:24:33.03660515 +0000 UTC m=+2146.779007119" watchObservedRunningTime="2026-09-14 11:24:33.041994795 +0000 UTC m=+2146.784396767" Sep 14 11:24:34.009565 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:34.009523 2581 generic.go:356] "Generic (PLEG): container finished" podID="43702cb6-6807-40b1-ad8b-c7ba10bcf65d" containerID="2160fde79123edaea75d5b8eb5db333eabb96ed38471f6a091c1a05acdcd5870" exitCode=0 Sep 14 11:24:34.009987 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:34.009620 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fdbc00-bm6ds" event={"ID":"43702cb6-6807-40b1-ad8b-c7ba10bcf65d","Type":"ContainerDied","Data":"2160fde79123edaea75d5b8eb5db333eabb96ed38471f6a091c1a05acdcd5870"} Sep 14 11:24:35.150713 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:35.150690 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fdbc00-bm6ds" Sep 14 11:24:35.274988 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:35.274960 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43702cb6-6807-40b1-ad8b-c7ba10bcf65d-maas-api-service-ca\") pod \"43702cb6-6807-40b1-ad8b-c7ba10bcf65d\" (UID: \"43702cb6-6807-40b1-ad8b-c7ba10bcf65d\") " Sep 14 11:24:35.275450 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:35.275423 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43702cb6-6807-40b1-ad8b-c7ba10bcf65d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "43702cb6-6807-40b1-ad8b-c7ba10bcf65d" (UID: "43702cb6-6807-40b1-ad8b-c7ba10bcf65d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:24:35.376523 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:35.376486 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43702cb6-6807-40b1-ad8b-c7ba10bcf65d-maas-api-service-ca\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:24:36.021070 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:36.021040 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fdbc00-bm6ds" Sep 14 11:24:36.021246 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:36.021039 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fdbc00-bm6ds" event={"ID":"43702cb6-6807-40b1-ad8b-c7ba10bcf65d","Type":"ContainerDied","Data":"4e6b0340a654af473f21ed348952a0da1e00da2c330570d811487294c43e3217"} Sep 14 11:24:36.021246 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:36.021166 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e6b0340a654af473f21ed348952a0da1e00da2c330570d811487294c43e3217" Sep 14 11:24:42.642825 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:42.642789 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fdbc00-29823075-dkww5"] Sep 14 11:24:42.647062 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:42.647036 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-fdbc00-29823075-dkww5"] Sep 14 11:24:42.685789 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:42.685742 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-fdbc00-64d8fb85bf-5zgk5"] Sep 14 11:24:42.686072 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:42.686041 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fdbc00-64d8fb85bf-5zgk5" podUID="ef8546d1-98b8-4f03-bf2c-90b013c279de" containerName="payload-processing" containerID="cri-o://730c957185d5e82d19705cdf258606c783e38dbf23acdf216542e2f4df7e3456" gracePeriod=30 Sep 14 11:24:42.688808 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:42.688742 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdbc00-67944d958-zfr26"] Sep 14 11:24:42.689013 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:42.688983 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdbc00-67944d958-zfr26" podUID="d4d3967c-eeb2-4edd-af84-18df43e18c09" containerName="payload-pre-processing" containerID="cri-o://a6ab1f990a938445a70dbb698b1e4a2165919f3f9b056caa4e15eac40e19b5cf" gracePeriod=30 Sep 14 11:24:42.833582 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:42.833524 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48b1176c-8bbd-4ff7-9f23-3d5bd1458afb" path="/var/lib/kubelet/pods/48b1176c-8bbd-4ff7-9f23-3d5bd1458afb/volumes" Sep 14 11:24:42.954748 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:42.954721 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fdbc00-bm6ds"] Sep 14 11:24:42.963816 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:42.963792 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-fdbc00-bm6ds"] Sep 14 11:24:42.972132 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:42.972107 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fdbc00-64d8fb85bf-5zgk5" Sep 14 11:24:42.975068 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:42.975050 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdbc00-67944d958-zfr26" Sep 14 11:24:43.047385 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:43.047327 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbzvq\" (UniqueName: \"kubernetes.io/projected/ef8546d1-98b8-4f03-bf2c-90b013c279de-kube-api-access-qbzvq\") pod \"ef8546d1-98b8-4f03-bf2c-90b013c279de\" (UID: \"ef8546d1-98b8-4f03-bf2c-90b013c279de\") " Sep 14 11:24:43.047548 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:43.047455 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bz7xd\" (UniqueName: \"kubernetes.io/projected/d4d3967c-eeb2-4edd-af84-18df43e18c09-kube-api-access-bz7xd\") pod \"d4d3967c-eeb2-4edd-af84-18df43e18c09\" (UID: \"d4d3967c-eeb2-4edd-af84-18df43e18c09\") " Sep 14 11:24:43.047548 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:43.047508 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4d3967c-eeb2-4edd-af84-18df43e18c09-plugins-config-volume\") pod \"d4d3967c-eeb2-4edd-af84-18df43e18c09\" (UID: \"d4d3967c-eeb2-4edd-af84-18df43e18c09\") " Sep 14 11:24:43.047548 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:43.047538 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef8546d1-98b8-4f03-bf2c-90b013c279de-plugins-config-volume\") pod \"ef8546d1-98b8-4f03-bf2c-90b013c279de\" (UID: \"ef8546d1-98b8-4f03-bf2c-90b013c279de\") " Sep 14 11:24:43.047971 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:43.047942 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef8546d1-98b8-4f03-bf2c-90b013c279de-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ef8546d1-98b8-4f03-bf2c-90b013c279de" (UID: "ef8546d1-98b8-4f03-bf2c-90b013c279de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:24:43.048109 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:43.047969 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4d3967c-eeb2-4edd-af84-18df43e18c09-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d4d3967c-eeb2-4edd-af84-18df43e18c09" (UID: "d4d3967c-eeb2-4edd-af84-18df43e18c09"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:24:43.049809 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:43.049783 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4d3967c-eeb2-4edd-af84-18df43e18c09-kube-api-access-bz7xd" (OuterVolumeSpecName: "kube-api-access-bz7xd") pod "d4d3967c-eeb2-4edd-af84-18df43e18c09" (UID: "d4d3967c-eeb2-4edd-af84-18df43e18c09"). InnerVolumeSpecName "kube-api-access-bz7xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:24:43.050123 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:43.050103 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef8546d1-98b8-4f03-bf2c-90b013c279de-kube-api-access-qbzvq" (OuterVolumeSpecName: "kube-api-access-qbzvq") pod "ef8546d1-98b8-4f03-bf2c-90b013c279de" (UID: "ef8546d1-98b8-4f03-bf2c-90b013c279de"). InnerVolumeSpecName "kube-api-access-qbzvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:24:43.060646 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:43.060612 2581 generic.go:356] "Generic (PLEG): container finished" podID="ef8546d1-98b8-4f03-bf2c-90b013c279de" containerID="730c957185d5e82d19705cdf258606c783e38dbf23acdf216542e2f4df7e3456" exitCode=0 Sep 14 11:24:43.060773 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:43.060659 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fdbc00-64d8fb85bf-5zgk5" Sep 14 11:24:43.060773 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:43.060697 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fdbc00-64d8fb85bf-5zgk5" event={"ID":"ef8546d1-98b8-4f03-bf2c-90b013c279de","Type":"ContainerDied","Data":"730c957185d5e82d19705cdf258606c783e38dbf23acdf216542e2f4df7e3456"} Sep 14 11:24:43.060773 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:43.060752 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-fdbc00-64d8fb85bf-5zgk5" event={"ID":"ef8546d1-98b8-4f03-bf2c-90b013c279de","Type":"ContainerDied","Data":"73333f4f5f41a8e351390923a26eaf3065975a24453aa68b0b2c749f603ae188"} Sep 14 11:24:43.060915 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:43.060783 2581 scope.go:117] "RemoveContainer" containerID="730c957185d5e82d19705cdf258606c783e38dbf23acdf216542e2f4df7e3456" Sep 14 11:24:43.062822 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:43.062797 2581 generic.go:356] "Generic (PLEG): container finished" podID="d4d3967c-eeb2-4edd-af84-18df43e18c09" containerID="a6ab1f990a938445a70dbb698b1e4a2165919f3f9b056caa4e15eac40e19b5cf" exitCode=0 Sep 14 11:24:43.062931 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:43.062835 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdbc00-67944d958-zfr26" event={"ID":"d4d3967c-eeb2-4edd-af84-18df43e18c09","Type":"ContainerDied","Data":"a6ab1f990a938445a70dbb698b1e4a2165919f3f9b056caa4e15eac40e19b5cf"} Sep 14 11:24:43.062931 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:43.062848 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdbc00-67944d958-zfr26" Sep 14 11:24:43.062931 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:43.062881 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdbc00-67944d958-zfr26" event={"ID":"d4d3967c-eeb2-4edd-af84-18df43e18c09","Type":"ContainerDied","Data":"a7aa19a5768edb8e9c07f523e9c56c0f3fb5b98ec1eb49cb2582c961136ea7c8"} Sep 14 11:24:43.071886 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:43.071869 2581 scope.go:117] "RemoveContainer" containerID="730c957185d5e82d19705cdf258606c783e38dbf23acdf216542e2f4df7e3456" Sep 14 11:24:43.072209 ip-10-0-137-53 kubenswrapper[2581]: E0914 11:24:43.072191 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"730c957185d5e82d19705cdf258606c783e38dbf23acdf216542e2f4df7e3456\": container with ID starting with 730c957185d5e82d19705cdf258606c783e38dbf23acdf216542e2f4df7e3456 not found: ID does not exist" containerID="730c957185d5e82d19705cdf258606c783e38dbf23acdf216542e2f4df7e3456" Sep 14 11:24:43.072258 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:43.072219 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"730c957185d5e82d19705cdf258606c783e38dbf23acdf216542e2f4df7e3456"} err="failed to get container status \"730c957185d5e82d19705cdf258606c783e38dbf23acdf216542e2f4df7e3456\": rpc error: code = NotFound desc = could not find container \"730c957185d5e82d19705cdf258606c783e38dbf23acdf216542e2f4df7e3456\": container with ID starting with 730c957185d5e82d19705cdf258606c783e38dbf23acdf216542e2f4df7e3456 not found: ID does not exist" Sep 14 11:24:43.072258 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:43.072237 2581 scope.go:117] "RemoveContainer" containerID="a6ab1f990a938445a70dbb698b1e4a2165919f3f9b056caa4e15eac40e19b5cf" Sep 14 11:24:43.081835 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:43.081813 2581 scope.go:117] "RemoveContainer" containerID="a6ab1f990a938445a70dbb698b1e4a2165919f3f9b056caa4e15eac40e19b5cf" Sep 14 11:24:43.082202 ip-10-0-137-53 kubenswrapper[2581]: E0914 11:24:43.082180 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6ab1f990a938445a70dbb698b1e4a2165919f3f9b056caa4e15eac40e19b5cf\": container with ID starting with a6ab1f990a938445a70dbb698b1e4a2165919f3f9b056caa4e15eac40e19b5cf not found: ID does not exist" containerID="a6ab1f990a938445a70dbb698b1e4a2165919f3f9b056caa4e15eac40e19b5cf" Sep 14 11:24:43.082276 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:43.082214 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6ab1f990a938445a70dbb698b1e4a2165919f3f9b056caa4e15eac40e19b5cf"} err="failed to get container status \"a6ab1f990a938445a70dbb698b1e4a2165919f3f9b056caa4e15eac40e19b5cf\": rpc error: code = NotFound desc = could not find container \"a6ab1f990a938445a70dbb698b1e4a2165919f3f9b056caa4e15eac40e19b5cf\": container with ID starting with a6ab1f990a938445a70dbb698b1e4a2165919f3f9b056caa4e15eac40e19b5cf not found: ID does not exist" Sep 14 11:24:43.102560 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:43.102526 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-fdbc00-64d8fb85bf-5zgk5"] Sep 14 11:24:43.120419 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:43.120386 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-fdbc00-64d8fb85bf-5zgk5"] Sep 14 11:24:43.147643 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:43.147606 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdbc00-67944d958-zfr26"] Sep 14 11:24:43.148906 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:43.148853 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz7xd\" (UniqueName: \"kubernetes.io/projected/d4d3967c-eeb2-4edd-af84-18df43e18c09-kube-api-access-bz7xd\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:24:43.148906 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:43.148876 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4d3967c-eeb2-4edd-af84-18df43e18c09-plugins-config-volume\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:24:43.148906 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:43.148887 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef8546d1-98b8-4f03-bf2c-90b013c279de-plugins-config-volume\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:24:43.148906 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:43.148897 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbzvq\" (UniqueName: \"kubernetes.io/projected/ef8546d1-98b8-4f03-bf2c-90b013c279de-kube-api-access-qbzvq\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:24:43.155192 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:43.155166 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-fdbc00-67944d958-zfr26"] Sep 14 11:24:44.831768 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:44.831736 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43702cb6-6807-40b1-ad8b-c7ba10bcf65d" path="/var/lib/kubelet/pods/43702cb6-6807-40b1-ad8b-c7ba10bcf65d/volumes" Sep 14 11:24:44.832232 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:44.832085 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4d3967c-eeb2-4edd-af84-18df43e18c09" path="/var/lib/kubelet/pods/d4d3967c-eeb2-4edd-af84-18df43e18c09/volumes" Sep 14 11:24:44.832415 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:44.832402 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef8546d1-98b8-4f03-bf2c-90b013c279de" path="/var/lib/kubelet/pods/ef8546d1-98b8-4f03-bf2c-90b013c279de/volumes" Sep 14 11:24:47.322419 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:47.322384 2581 scope.go:117] "RemoveContainer" containerID="f7027588c3311257f6bb64c9ed414e0b17922d699d5c53f436438271b6c2eb6d" Sep 14 11:24:47.331290 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:47.331269 2581 scope.go:117] "RemoveContainer" containerID="63f04906ff95af31b5c9a3cb314d796f23c294a460b5a0e84310963762fb76fe" Sep 14 11:24:50.466021 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:50.465990 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0fb86d-ljk9v"] Sep 14 11:24:50.466458 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:50.466389 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4d3967c-eeb2-4edd-af84-18df43e18c09" containerName="payload-pre-processing" Sep 14 11:24:50.466458 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:50.466401 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4d3967c-eeb2-4edd-af84-18df43e18c09" containerName="payload-pre-processing" Sep 14 11:24:50.466458 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:50.466419 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef8546d1-98b8-4f03-bf2c-90b013c279de" containerName="payload-processing" Sep 14 11:24:50.466458 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:50.466426 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef8546d1-98b8-4f03-bf2c-90b013c279de" containerName="payload-processing" Sep 14 11:24:50.466458 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:50.466440 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43702cb6-6807-40b1-ad8b-c7ba10bcf65d" containerName="revoke-keys" Sep 14 11:24:50.466458 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:50.466445 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="43702cb6-6807-40b1-ad8b-c7ba10bcf65d" containerName="revoke-keys" Sep 14 11:24:50.466654 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:50.466500 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="43702cb6-6807-40b1-ad8b-c7ba10bcf65d" containerName="revoke-keys" Sep 14 11:24:50.466654 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:50.466508 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef8546d1-98b8-4f03-bf2c-90b013c279de" containerName="payload-processing" Sep 14 11:24:50.466654 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:50.466517 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4d3967c-eeb2-4edd-af84-18df43e18c09" containerName="payload-pre-processing" Sep 14 11:24:50.469551 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:50.469536 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0fb86d-ljk9v" Sep 14 11:24:50.472212 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:50.472190 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pv9g8\"" Sep 14 11:24:50.479813 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:50.479787 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0fb86d-ljk9v"] Sep 14 11:24:50.617801 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:50.617758 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/483c76b1-81c7-4458-beac-569e1960e982-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-0fb86d-ljk9v\" (UID: \"483c76b1-81c7-4458-beac-569e1960e982\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0fb86d-ljk9v" Sep 14 11:24:50.719356 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:50.719243 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/483c76b1-81c7-4458-beac-569e1960e982-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-0fb86d-ljk9v\" (UID: \"483c76b1-81c7-4458-beac-569e1960e982\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0fb86d-ljk9v" Sep 14 11:24:50.719914 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:50.719896 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/483c76b1-81c7-4458-beac-569e1960e982-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-0fb86d-ljk9v\" (UID: \"483c76b1-81c7-4458-beac-569e1960e982\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0fb86d-ljk9v" Sep 14 11:24:50.780196 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:50.780163 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0fb86d-ljk9v" Sep 14 11:24:51.060384 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:51.060355 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0fb86d-ljk9v"] Sep 14 11:24:51.062598 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:24:51.062574 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod483c76b1_81c7_4458_beac_569e1960e982.slice/crio-3461fc7644e8762c76c7388d357dc866c5a7fba4297b8c031b85a50eabf1e2dd WatchSource:0}: Error finding container 3461fc7644e8762c76c7388d357dc866c5a7fba4297b8c031b85a50eabf1e2dd: Status 404 returned error can't find the container with id 3461fc7644e8762c76c7388d357dc866c5a7fba4297b8c031b85a50eabf1e2dd Sep 14 11:24:51.116637 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:51.116602 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0fb86d-ljk9v" event={"ID":"483c76b1-81c7-4458-beac-569e1960e982","Type":"ContainerStarted","Data":"3461fc7644e8762c76c7388d357dc866c5a7fba4297b8c031b85a50eabf1e2dd"} Sep 14 11:24:52.123799 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:52.123694 2581 generic.go:356] "Generic (PLEG): container finished" podID="483c76b1-81c7-4458-beac-569e1960e982" containerID="0bded94d2066d6bcd063f69d66ba5d7fe1464a6bd5d2748844444d6c71af9329" exitCode=0 Sep 14 11:24:52.123799 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:52.123738 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0fb86d-ljk9v" event={"ID":"483c76b1-81c7-4458-beac-569e1960e982","Type":"ContainerDied","Data":"0bded94d2066d6bcd063f69d66ba5d7fe1464a6bd5d2748844444d6c71af9329"} Sep 14 11:24:53.256518 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:53.256493 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0fb86d-ljk9v" Sep 14 11:24:53.342407 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:53.342366 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/483c76b1-81c7-4458-beac-569e1960e982-maas-api-service-ca\") pod \"483c76b1-81c7-4458-beac-569e1960e982\" (UID: \"483c76b1-81c7-4458-beac-569e1960e982\") " Sep 14 11:24:53.342731 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:53.342707 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/483c76b1-81c7-4458-beac-569e1960e982-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "483c76b1-81c7-4458-beac-569e1960e982" (UID: "483c76b1-81c7-4458-beac-569e1960e982"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:24:53.443543 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:53.443451 2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/483c76b1-81c7-4458-beac-569e1960e982-maas-api-service-ca\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:24:54.135240 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:54.135215 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0fb86d-ljk9v" Sep 14 11:24:54.135425 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:54.135212 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0fb86d-ljk9v" event={"ID":"483c76b1-81c7-4458-beac-569e1960e982","Type":"ContainerDied","Data":"3461fc7644e8762c76c7388d357dc866c5a7fba4297b8c031b85a50eabf1e2dd"} Sep 14 11:24:54.135425 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:24:54.135365 2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3461fc7644e8762c76c7388d357dc866c5a7fba4297b8c031b85a50eabf1e2dd" Sep 14 11:25:01.396265 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:25:01.396230 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0fb86d-6658f598c7-xtzs5"] Sep 14 11:25:01.396785 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:25:01.396546 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0fb86d-6658f598c7-xtzs5" podUID="39cb0e24-66a1-474c-9c19-767520c1cf0d" containerName="payload-processing" containerID="cri-o://a76408f7184a2b4d0a6cc92d047999260e4cb855aa9c74a52960a967051a42c5" gracePeriod=30 Sep 14 11:25:01.422969 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:25:01.422934 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0fb86d-7c8c54f78-s9p7j"] Sep 14 11:25:01.423194 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:25:01.423168 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0fb86d-7c8c54f78-s9p7j" podUID="e75f7100-543f-44de-b464-ca5487d50b5a" containerName="payload-pre-processing" containerID="cri-o://b2bc4f40fdfbd6b3b20680fec7b0447e43c92789dfc1528c95a925ce1715b3ee" gracePeriod=30 Sep 14 11:25:01.625958 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:25:01.625919 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0fb86d-ljk9v"] Sep 14 11:25:01.629385 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:25:01.629358 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-0fb86d-ljk9v"] Sep 14 11:25:01.704165 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:25:01.704136 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0fb86d-6658f598c7-xtzs5" Sep 14 11:25:01.706925 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:25:01.706902 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0fb86d-7c8c54f78-s9p7j" Sep 14 11:25:01.829671 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:25:01.829622 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbw5t\" (UniqueName: \"kubernetes.io/projected/39cb0e24-66a1-474c-9c19-767520c1cf0d-kube-api-access-jbw5t\") pod \"39cb0e24-66a1-474c-9c19-767520c1cf0d\" (UID: \"39cb0e24-66a1-474c-9c19-767520c1cf0d\") " Sep 14 11:25:01.829883 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:25:01.829685 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrgjp\" (UniqueName: \"kubernetes.io/projected/e75f7100-543f-44de-b464-ca5487d50b5a-kube-api-access-qrgjp\") pod \"e75f7100-543f-44de-b464-ca5487d50b5a\" (UID: \"e75f7100-543f-44de-b464-ca5487d50b5a\") " Sep 14 11:25:01.829883 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:25:01.829725 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39cb0e24-66a1-474c-9c19-767520c1cf0d-plugins-config-volume\") pod \"39cb0e24-66a1-474c-9c19-767520c1cf0d\" (UID: \"39cb0e24-66a1-474c-9c19-767520c1cf0d\") " Sep 14 11:25:01.829883 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:25:01.829794 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e75f7100-543f-44de-b464-ca5487d50b5a-plugins-config-volume\") pod \"e75f7100-543f-44de-b464-ca5487d50b5a\" (UID: \"e75f7100-543f-44de-b464-ca5487d50b5a\") " Sep 14 11:25:01.830148 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:25:01.830124 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e75f7100-543f-44de-b464-ca5487d50b5a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e75f7100-543f-44de-b464-ca5487d50b5a" (UID: "e75f7100-543f-44de-b464-ca5487d50b5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:25:01.830209 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:25:01.830123 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39cb0e24-66a1-474c-9c19-767520c1cf0d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "39cb0e24-66a1-474c-9c19-767520c1cf0d" (UID: "39cb0e24-66a1-474c-9c19-767520c1cf0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 11:25:01.831925 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:25:01.831903 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39cb0e24-66a1-474c-9c19-767520c1cf0d-kube-api-access-jbw5t" (OuterVolumeSpecName: "kube-api-access-jbw5t") pod "39cb0e24-66a1-474c-9c19-767520c1cf0d" (UID: "39cb0e24-66a1-474c-9c19-767520c1cf0d"). InnerVolumeSpecName "kube-api-access-jbw5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:25:01.831983 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:25:01.831963 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e75f7100-543f-44de-b464-ca5487d50b5a-kube-api-access-qrgjp" (OuterVolumeSpecName: "kube-api-access-qrgjp") pod "e75f7100-543f-44de-b464-ca5487d50b5a" (UID: "e75f7100-543f-44de-b464-ca5487d50b5a"). InnerVolumeSpecName "kube-api-access-qrgjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:25:01.930710 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:25:01.930619 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrgjp\" (UniqueName: \"kubernetes.io/projected/e75f7100-543f-44de-b464-ca5487d50b5a-kube-api-access-qrgjp\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:25:01.930710 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:25:01.930651 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39cb0e24-66a1-474c-9c19-767520c1cf0d-plugins-config-volume\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:25:01.930710 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:25:01.930661 2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e75f7100-543f-44de-b464-ca5487d50b5a-plugins-config-volume\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:25:01.930710 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:25:01.930669 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbw5t\" (UniqueName: \"kubernetes.io/projected/39cb0e24-66a1-474c-9c19-767520c1cf0d-kube-api-access-jbw5t\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:25:02.176475 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:25:02.176433 2581 generic.go:356] "Generic (PLEG): container finished" podID="39cb0e24-66a1-474c-9c19-767520c1cf0d" containerID="a76408f7184a2b4d0a6cc92d047999260e4cb855aa9c74a52960a967051a42c5" exitCode=0 Sep 14 11:25:02.176654 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:25:02.176483 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0fb86d-6658f598c7-xtzs5" Sep 14 11:25:02.176654 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:25:02.176526 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0fb86d-6658f598c7-xtzs5" event={"ID":"39cb0e24-66a1-474c-9c19-767520c1cf0d","Type":"ContainerDied","Data":"a76408f7184a2b4d0a6cc92d047999260e4cb855aa9c74a52960a967051a42c5"} Sep 14 11:25:02.176654 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:25:02.176588 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0fb86d-6658f598c7-xtzs5" event={"ID":"39cb0e24-66a1-474c-9c19-767520c1cf0d","Type":"ContainerDied","Data":"ba142a3c5afcea804d68798a4c05deedceb13e89575066086a7e79345154ffc6"} Sep 14 11:25:02.176654 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:25:02.176618 2581 scope.go:117] "RemoveContainer" containerID="a76408f7184a2b4d0a6cc92d047999260e4cb855aa9c74a52960a967051a42c5" Sep 14 11:25:02.177915 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:25:02.177886 2581 generic.go:356] "Generic (PLEG): container finished" podID="e75f7100-543f-44de-b464-ca5487d50b5a" containerID="b2bc4f40fdfbd6b3b20680fec7b0447e43c92789dfc1528c95a925ce1715b3ee" exitCode=0 Sep 14 11:25:02.178009 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:25:02.177946 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0fb86d-7c8c54f78-s9p7j" Sep 14 11:25:02.178071 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:25:02.177947 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0fb86d-7c8c54f78-s9p7j" event={"ID":"e75f7100-543f-44de-b464-ca5487d50b5a","Type":"ContainerDied","Data":"b2bc4f40fdfbd6b3b20680fec7b0447e43c92789dfc1528c95a925ce1715b3ee"} Sep 14 11:25:02.178123 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:25:02.178068 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0fb86d-7c8c54f78-s9p7j" event={"ID":"e75f7100-543f-44de-b464-ca5487d50b5a","Type":"ContainerDied","Data":"5e90427d491e275fa20fe100d16a5e086ecc87063aacc33cc8bb69dc6ee4e018"} Sep 14 11:25:02.185724 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:25:02.185702 2581 scope.go:117] "RemoveContainer" containerID="a76408f7184a2b4d0a6cc92d047999260e4cb855aa9c74a52960a967051a42c5" Sep 14 11:25:02.186096 ip-10-0-137-53 kubenswrapper[2581]: E0914 11:25:02.186076 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a76408f7184a2b4d0a6cc92d047999260e4cb855aa9c74a52960a967051a42c5\": container with ID starting with a76408f7184a2b4d0a6cc92d047999260e4cb855aa9c74a52960a967051a42c5 not found: ID does not exist" containerID="a76408f7184a2b4d0a6cc92d047999260e4cb855aa9c74a52960a967051a42c5" Sep 14 11:25:02.186172 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:25:02.186104 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a76408f7184a2b4d0a6cc92d047999260e4cb855aa9c74a52960a967051a42c5"} err="failed to get container status \"a76408f7184a2b4d0a6cc92d047999260e4cb855aa9c74a52960a967051a42c5\": rpc error: code = NotFound desc = could not find container \"a76408f7184a2b4d0a6cc92d047999260e4cb855aa9c74a52960a967051a42c5\": container with ID starting with a76408f7184a2b4d0a6cc92d047999260e4cb855aa9c74a52960a967051a42c5 not found: ID does not exist" Sep 14 11:25:02.186172 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:25:02.186119 2581 scope.go:117] "RemoveContainer" containerID="b2bc4f40fdfbd6b3b20680fec7b0447e43c92789dfc1528c95a925ce1715b3ee" Sep 14 11:25:02.194643 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:25:02.194623 2581 scope.go:117] "RemoveContainer" containerID="b2bc4f40fdfbd6b3b20680fec7b0447e43c92789dfc1528c95a925ce1715b3ee" Sep 14 11:25:02.194939 ip-10-0-137-53 kubenswrapper[2581]: E0914 11:25:02.194920 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2bc4f40fdfbd6b3b20680fec7b0447e43c92789dfc1528c95a925ce1715b3ee\": container with ID starting with b2bc4f40fdfbd6b3b20680fec7b0447e43c92789dfc1528c95a925ce1715b3ee not found: ID does not exist" containerID="b2bc4f40fdfbd6b3b20680fec7b0447e43c92789dfc1528c95a925ce1715b3ee" Sep 14 11:25:02.194999 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:25:02.194951 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2bc4f40fdfbd6b3b20680fec7b0447e43c92789dfc1528c95a925ce1715b3ee"} err="failed to get container status \"b2bc4f40fdfbd6b3b20680fec7b0447e43c92789dfc1528c95a925ce1715b3ee\": rpc error: code = NotFound desc = could not find container \"b2bc4f40fdfbd6b3b20680fec7b0447e43c92789dfc1528c95a925ce1715b3ee\": container with ID starting with b2bc4f40fdfbd6b3b20680fec7b0447e43c92789dfc1528c95a925ce1715b3ee not found: ID does not exist" Sep 14 11:25:02.219002 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:25:02.218968 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0fb86d-6658f598c7-xtzs5"] Sep 14 11:25:02.224988 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:25:02.224963 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0fb86d-6658f598c7-xtzs5"] Sep 14 11:25:02.241090 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:25:02.241057 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0fb86d-7c8c54f78-s9p7j"] Sep 14 11:25:02.251835 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:25:02.251811 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0fb86d-7c8c54f78-s9p7j"] Sep 14 11:25:02.833173 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:25:02.833126 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39cb0e24-66a1-474c-9c19-767520c1cf0d" path="/var/lib/kubelet/pods/39cb0e24-66a1-474c-9c19-767520c1cf0d/volumes" Sep 14 11:25:02.833694 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:25:02.833635 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="483c76b1-81c7-4458-beac-569e1960e982" path="/var/lib/kubelet/pods/483c76b1-81c7-4458-beac-569e1960e982/volumes" Sep 14 11:25:02.834029 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:25:02.834008 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e75f7100-543f-44de-b464-ca5487d50b5a" path="/var/lib/kubelet/pods/e75f7100-543f-44de-b464-ca5487d50b5a/volumes" Sep 14 11:28:47.401499 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:28:47.401468 2581 scope.go:117] "RemoveContainer" containerID="073894664bb1a052abcc71443bac92f1d935f9fa81cf94574d3e3934a7bc0d42" Sep 14 11:28:47.410471 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:28:47.410447 2581 scope.go:117] "RemoveContainer" containerID="a732983009a7c0528d3169390ffefaa85b7ee774266a8918e3b1ab82d8e75954" Sep 14 11:29:33.740970 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:33.740934 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk"] Sep 14 11:29:33.741559 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:33.741311 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39cb0e24-66a1-474c-9c19-767520c1cf0d" containerName="payload-processing" Sep 14 11:29:33.741559 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:33.741324 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="39cb0e24-66a1-474c-9c19-767520c1cf0d" containerName="payload-processing" Sep 14 11:29:33.741559 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:33.741356 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e75f7100-543f-44de-b464-ca5487d50b5a" containerName="payload-pre-processing" Sep 14 11:29:33.741559 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:33.741369 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e75f7100-543f-44de-b464-ca5487d50b5a" containerName="payload-pre-processing" Sep 14 11:29:33.741559 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:33.741388 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="483c76b1-81c7-4458-beac-569e1960e982" containerName="revoke-keys" Sep 14 11:29:33.741559 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:33.741394 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="483c76b1-81c7-4458-beac-569e1960e982" containerName="revoke-keys" Sep 14 11:29:33.741559 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:33.741466 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e75f7100-543f-44de-b464-ca5487d50b5a" containerName="payload-pre-processing" Sep 14 11:29:33.741559 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:33.741475 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="39cb0e24-66a1-474c-9c19-767520c1cf0d" containerName="payload-processing" Sep 14 11:29:33.741559 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:33.741486 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="483c76b1-81c7-4458-beac-569e1960e982" containerName="revoke-keys" Sep 14 11:29:33.744494 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:33.744474 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk" Sep 14 11:29:33.747163 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:33.747134 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-2ee2f4f3-kserve-self-signed-certs\"" Sep 14 11:29:33.754998 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:33.754971 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk"] Sep 14 11:29:33.850714 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:33.850671 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73f96125-cd38-4508-9106-ce46565328c6-home\") pod \"e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk\" (UID: \"73f96125-cd38-4508-9106-ce46565328c6\") " pod="llm/e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk" Sep 14 11:29:33.850714 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:33.850716 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73f96125-cd38-4508-9106-ce46565328c6-model-cache\") pod \"e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk\" (UID: \"73f96125-cd38-4508-9106-ce46565328c6\") " pod="llm/e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk" Sep 14 11:29:33.850966 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:33.850787 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73f96125-cd38-4508-9106-ce46565328c6-tls-certs\") pod \"e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk\" (UID: \"73f96125-cd38-4508-9106-ce46565328c6\") " pod="llm/e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk" Sep 14 11:29:33.850966 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:33.850868 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73f96125-cd38-4508-9106-ce46565328c6-dshm\") pod \"e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk\" (UID: \"73f96125-cd38-4508-9106-ce46565328c6\") " pod="llm/e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk" Sep 14 11:29:33.850966 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:33.850923 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwbrt\" (UniqueName: \"kubernetes.io/projected/73f96125-cd38-4508-9106-ce46565328c6-kube-api-access-bwbrt\") pod \"e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk\" (UID: \"73f96125-cd38-4508-9106-ce46565328c6\") " pod="llm/e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk" Sep 14 11:29:33.851090 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:33.850972 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73f96125-cd38-4508-9106-ce46565328c6-tmp-dir\") pod \"e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk\" (UID: \"73f96125-cd38-4508-9106-ce46565328c6\") " pod="llm/e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk" Sep 14 11:29:33.952196 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:33.952157 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73f96125-cd38-4508-9106-ce46565328c6-home\") pod \"e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk\" (UID: \"73f96125-cd38-4508-9106-ce46565328c6\") " pod="llm/e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk" Sep 14 11:29:33.952196 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:33.952199 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73f96125-cd38-4508-9106-ce46565328c6-model-cache\") pod \"e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk\" (UID: \"73f96125-cd38-4508-9106-ce46565328c6\") " pod="llm/e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk" Sep 14 11:29:33.952507 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:33.952245 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73f96125-cd38-4508-9106-ce46565328c6-tls-certs\") pod \"e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk\" (UID: \"73f96125-cd38-4508-9106-ce46565328c6\") " pod="llm/e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk" Sep 14 11:29:33.952507 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:33.952283 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73f96125-cd38-4508-9106-ce46565328c6-dshm\") pod \"e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk\" (UID: \"73f96125-cd38-4508-9106-ce46565328c6\") " pod="llm/e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk" Sep 14 11:29:33.952507 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:33.952368 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwbrt\" (UniqueName: \"kubernetes.io/projected/73f96125-cd38-4508-9106-ce46565328c6-kube-api-access-bwbrt\") pod \"e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk\" (UID: \"73f96125-cd38-4508-9106-ce46565328c6\") " pod="llm/e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk" Sep 14 11:29:33.952507 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:33.952423 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73f96125-cd38-4508-9106-ce46565328c6-tmp-dir\") pod \"e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk\" (UID: \"73f96125-cd38-4508-9106-ce46565328c6\") " pod="llm/e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk" Sep 14 11:29:33.952693 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:33.952656 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73f96125-cd38-4508-9106-ce46565328c6-home\") pod \"e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk\" (UID: \"73f96125-cd38-4508-9106-ce46565328c6\") " pod="llm/e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk" Sep 14 11:29:33.952693 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:33.952669 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73f96125-cd38-4508-9106-ce46565328c6-model-cache\") pod \"e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk\" (UID: \"73f96125-cd38-4508-9106-ce46565328c6\") " pod="llm/e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk" Sep 14 11:29:33.952972 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:33.952951 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73f96125-cd38-4508-9106-ce46565328c6-tmp-dir\") pod \"e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk\" (UID: \"73f96125-cd38-4508-9106-ce46565328c6\") " pod="llm/e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk" Sep 14 11:29:33.954806 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:33.954786 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73f96125-cd38-4508-9106-ce46565328c6-dshm\") pod \"e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk\" (UID: \"73f96125-cd38-4508-9106-ce46565328c6\") " pod="llm/e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk" Sep 14 11:29:33.954969 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:33.954952 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73f96125-cd38-4508-9106-ce46565328c6-tls-certs\") pod \"e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk\" (UID: \"73f96125-cd38-4508-9106-ce46565328c6\") " pod="llm/e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk" Sep 14 11:29:33.962568 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:33.962549 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwbrt\" (UniqueName: \"kubernetes.io/projected/73f96125-cd38-4508-9106-ce46565328c6-kube-api-access-bwbrt\") pod \"e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk\" (UID: \"73f96125-cd38-4508-9106-ce46565328c6\") " pod="llm/e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk" Sep 14 11:29:34.053523 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:34.053488 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk" Sep 14 11:29:34.326513 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:34.326486 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk"] Sep 14 11:29:34.328715 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:29:34.328687 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73f96125_cd38_4508_9106_ce46565328c6.slice/crio-246a6a66ce2c25331d5f9fb6a418f72cd62a6e7c3f1511722bdda74b4dc84773 WatchSource:0}: Error finding container 246a6a66ce2c25331d5f9fb6a418f72cd62a6e7c3f1511722bdda74b4dc84773: Status 404 returned error can't find the container with id 246a6a66ce2c25331d5f9fb6a418f72cd62a6e7c3f1511722bdda74b4dc84773 Sep 14 11:29:34.532822 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:34.532782 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk" event={"ID":"73f96125-cd38-4508-9106-ce46565328c6","Type":"ContainerStarted","Data":"23d211691af2f396a643753b0e145fbe70090c42fd4a40b4fd32c3072103352e"} Sep 14 11:29:34.532822 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:34.532827 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk" event={"ID":"73f96125-cd38-4508-9106-ce46565328c6","Type":"ContainerStarted","Data":"246a6a66ce2c25331d5f9fb6a418f72cd62a6e7c3f1511722bdda74b4dc84773"} Sep 14 11:29:34.556248 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:34.556193 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk" podStartSLOduration=1.5561780710000002 podStartE2EDuration="1.556178071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:29:34.554100742 +0000 UTC m=+2448.296502713" watchObservedRunningTime="2026-09-14 11:29:34.556178071 +0000 UTC m=+2448.298580041" Sep 14 11:29:43.033106 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:43.033070 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m"] Sep 14 11:29:43.036896 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:43.036874 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m" Sep 14 11:29:43.039587 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:43.039562 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-2ee2f4f3-kserve-self-signed-certs\"" Sep 14 11:29:43.043296 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:43.043266 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef34636c-a3e5-45fa-b93e-56758063a046-model-cache\") pod \"e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m\" (UID: \"ef34636c-a3e5-45fa-b93e-56758063a046\") " pod="llm/e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m" Sep 14 11:29:43.043483 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:43.043354 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef34636c-a3e5-45fa-b93e-56758063a046-tmp-dir\") pod \"e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m\" (UID: \"ef34636c-a3e5-45fa-b93e-56758063a046\") " pod="llm/e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m" Sep 14 11:29:43.043483 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:43.043397 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef34636c-a3e5-45fa-b93e-56758063a046-home\") pod \"e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m\" (UID: \"ef34636c-a3e5-45fa-b93e-56758063a046\") " pod="llm/e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m" Sep 14 11:29:43.043483 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:43.043434 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef34636c-a3e5-45fa-b93e-56758063a046-tls-certs\") pod \"e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m\" (UID: \"ef34636c-a3e5-45fa-b93e-56758063a046\") " pod="llm/e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m" Sep 14 11:29:43.043708 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:43.043590 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef34636c-a3e5-45fa-b93e-56758063a046-dshm\") pod \"e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m\" (UID: \"ef34636c-a3e5-45fa-b93e-56758063a046\") " pod="llm/e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m" Sep 14 11:29:43.043708 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:43.043657 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvwh6\" (UniqueName: \"kubernetes.io/projected/ef34636c-a3e5-45fa-b93e-56758063a046-kube-api-access-vvwh6\") pod \"e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m\" (UID: \"ef34636c-a3e5-45fa-b93e-56758063a046\") " pod="llm/e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m" Sep 14 11:29:43.046840 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:43.046810 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m"] Sep 14 11:29:43.144213 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:43.144172 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef34636c-a3e5-45fa-b93e-56758063a046-model-cache\") pod \"e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m\" (UID: \"ef34636c-a3e5-45fa-b93e-56758063a046\") " pod="llm/e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m" Sep 14 11:29:43.144213 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:43.144224 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef34636c-a3e5-45fa-b93e-56758063a046-tmp-dir\") pod \"e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m\" (UID: \"ef34636c-a3e5-45fa-b93e-56758063a046\") " pod="llm/e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m" Sep 14 11:29:43.144512 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:43.144244 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef34636c-a3e5-45fa-b93e-56758063a046-home\") pod \"e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m\" (UID: \"ef34636c-a3e5-45fa-b93e-56758063a046\") " pod="llm/e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m" Sep 14 11:29:43.144512 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:43.144276 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef34636c-a3e5-45fa-b93e-56758063a046-tls-certs\") pod \"e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m\" (UID: \"ef34636c-a3e5-45fa-b93e-56758063a046\") " pod="llm/e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m" Sep 14 11:29:43.144512 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:43.144392 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef34636c-a3e5-45fa-b93e-56758063a046-dshm\") pod \"e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m\" (UID: \"ef34636c-a3e5-45fa-b93e-56758063a046\") " pod="llm/e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m" Sep 14 11:29:43.144512 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:43.144429 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvwh6\" (UniqueName: \"kubernetes.io/projected/ef34636c-a3e5-45fa-b93e-56758063a046-kube-api-access-vvwh6\") pod \"e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m\" (UID: \"ef34636c-a3e5-45fa-b93e-56758063a046\") " pod="llm/e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m" Sep 14 11:29:43.144731 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:43.144665 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef34636c-a3e5-45fa-b93e-56758063a046-home\") pod \"e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m\" (UID: \"ef34636c-a3e5-45fa-b93e-56758063a046\") " pod="llm/e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m" Sep 14 11:29:43.144791 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:43.144747 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef34636c-a3e5-45fa-b93e-56758063a046-model-cache\") pod \"e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m\" (UID: \"ef34636c-a3e5-45fa-b93e-56758063a046\") " pod="llm/e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m" Sep 14 11:29:43.144791 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:43.144753 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef34636c-a3e5-45fa-b93e-56758063a046-tmp-dir\") pod \"e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m\" (UID: \"ef34636c-a3e5-45fa-b93e-56758063a046\") " pod="llm/e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m" Sep 14 11:29:43.146745 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:43.146722 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef34636c-a3e5-45fa-b93e-56758063a046-dshm\") pod \"e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m\" (UID: \"ef34636c-a3e5-45fa-b93e-56758063a046\") " pod="llm/e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m" Sep 14 11:29:43.147015 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:43.146996 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef34636c-a3e5-45fa-b93e-56758063a046-tls-certs\") pod \"e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m\" (UID: \"ef34636c-a3e5-45fa-b93e-56758063a046\") " pod="llm/e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m" Sep 14 11:29:43.153732 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:43.153704 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvwh6\" (UniqueName: \"kubernetes.io/projected/ef34636c-a3e5-45fa-b93e-56758063a046-kube-api-access-vvwh6\") pod \"e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m\" (UID: \"ef34636c-a3e5-45fa-b93e-56758063a046\") " pod="llm/e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m" Sep 14 11:29:43.347651 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:43.347556 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m" Sep 14 11:29:43.621771 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:43.621685 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m"] Sep 14 11:29:43.624958 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:29:43.624924 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef34636c_a3e5_45fa_b93e_56758063a046.slice/crio-36140e2bbe8ada2c85c65ec70175ae888fc9910e4906d90c893fd8f8a3b9c4a6 WatchSource:0}: Error finding container 36140e2bbe8ada2c85c65ec70175ae888fc9910e4906d90c893fd8f8a3b9c4a6: Status 404 returned error can't find the container with id 36140e2bbe8ada2c85c65ec70175ae888fc9910e4906d90c893fd8f8a3b9c4a6 Sep 14 11:29:44.053835 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:44.053793 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk" Sep 14 11:29:44.054297 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:44.053850 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk" Sep 14 11:29:44.066535 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:44.066503 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk" Sep 14 11:29:44.581978 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:44.581938 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m" event={"ID":"ef34636c-a3e5-45fa-b93e-56758063a046","Type":"ContainerStarted","Data":"890fd8527feb4563bcd3f4d9d482f28d82e3f23a9a212e4e239b3e8b27e16fb3"} Sep 14 11:29:44.582184 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:44.581988 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m" event={"ID":"ef34636c-a3e5-45fa-b93e-56758063a046","Type":"ContainerStarted","Data":"36140e2bbe8ada2c85c65ec70175ae888fc9910e4906d90c893fd8f8a3b9c4a6"} Sep 14 11:29:44.593470 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:44.593439 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk" Sep 14 11:29:44.601522 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:44.601445 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m" podStartSLOduration=1.601431569 podStartE2EDuration="1.601431569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 11:29:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:29:44.600178737 +0000 UTC m=+2458.342580709" watchObservedRunningTime="2026-09-14 11:29:44.601431569 +0000 UTC m=+2458.343833542" Sep 14 11:29:53.347913 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:53.347874 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m" Sep 14 11:29:53.349970 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:53.347931 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m" Sep 14 11:29:53.360802 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:53.360771 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m" Sep 14 11:29:53.635944 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:53.635857 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m" Sep 14 11:29:57.807429 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:57.807388 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk"] Sep 14 11:29:59.737967 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:29:59.737933 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m"] Sep 14 11:30:12.827888 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:12.827828 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk" podUID="73f96125-cd38-4508-9106-ce46565328c6" containerName="main" containerID="cri-o://23d211691af2f396a643753b0e145fbe70090c42fd4a40b4fd32c3072103352e" gracePeriod=45 Sep 14 11:30:13.078003 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:13.077930 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk" Sep 14 11:30:13.242518 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:13.242482 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73f96125-cd38-4508-9106-ce46565328c6-home\") pod \"73f96125-cd38-4508-9106-ce46565328c6\" (UID: \"73f96125-cd38-4508-9106-ce46565328c6\") " Sep 14 11:30:13.242518 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:13.242520 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwbrt\" (UniqueName: \"kubernetes.io/projected/73f96125-cd38-4508-9106-ce46565328c6-kube-api-access-bwbrt\") pod \"73f96125-cd38-4508-9106-ce46565328c6\" (UID: \"73f96125-cd38-4508-9106-ce46565328c6\") " Sep 14 11:30:13.242775 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:13.242553 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73f96125-cd38-4508-9106-ce46565328c6-model-cache\") pod \"73f96125-cd38-4508-9106-ce46565328c6\" (UID: \"73f96125-cd38-4508-9106-ce46565328c6\") " Sep 14 11:30:13.242775 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:13.242602 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73f96125-cd38-4508-9106-ce46565328c6-tls-certs\") pod \"73f96125-cd38-4508-9106-ce46565328c6\" (UID: \"73f96125-cd38-4508-9106-ce46565328c6\") " Sep 14 11:30:13.242775 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:13.242655 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73f96125-cd38-4508-9106-ce46565328c6-dshm\") pod \"73f96125-cd38-4508-9106-ce46565328c6\" (UID: \"73f96125-cd38-4508-9106-ce46565328c6\") " Sep 14 11:30:13.242939 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:13.242787 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73f96125-cd38-4508-9106-ce46565328c6-home" (OuterVolumeSpecName: "home") pod "73f96125-cd38-4508-9106-ce46565328c6" (UID: "73f96125-cd38-4508-9106-ce46565328c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:30:13.242939 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:13.242850 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73f96125-cd38-4508-9106-ce46565328c6-tmp-dir\") pod \"73f96125-cd38-4508-9106-ce46565328c6\" (UID: \"73f96125-cd38-4508-9106-ce46565328c6\") " Sep 14 11:30:13.242939 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:13.242849 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73f96125-cd38-4508-9106-ce46565328c6-model-cache" (OuterVolumeSpecName: "model-cache") pod "73f96125-cd38-4508-9106-ce46565328c6" (UID: "73f96125-cd38-4508-9106-ce46565328c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:30:13.243090 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:13.243029 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73f96125-cd38-4508-9106-ce46565328c6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "73f96125-cd38-4508-9106-ce46565328c6" (UID: "73f96125-cd38-4508-9106-ce46565328c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:30:13.243239 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:13.243224 2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73f96125-cd38-4508-9106-ce46565328c6-home\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:30:13.243294 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:13.243242 2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73f96125-cd38-4508-9106-ce46565328c6-model-cache\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:30:13.243294 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:13.243253 2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73f96125-cd38-4508-9106-ce46565328c6-tmp-dir\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:30:13.245032 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:13.245009 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73f96125-cd38-4508-9106-ce46565328c6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "73f96125-cd38-4508-9106-ce46565328c6" (UID: "73f96125-cd38-4508-9106-ce46565328c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:30:13.245452 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:13.245433 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73f96125-cd38-4508-9106-ce46565328c6-dshm" (OuterVolumeSpecName: "dshm") pod "73f96125-cd38-4508-9106-ce46565328c6" (UID: "73f96125-cd38-4508-9106-ce46565328c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:30:13.245533 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:13.245448 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73f96125-cd38-4508-9106-ce46565328c6-kube-api-access-bwbrt" (OuterVolumeSpecName: "kube-api-access-bwbrt") pod "73f96125-cd38-4508-9106-ce46565328c6" (UID: "73f96125-cd38-4508-9106-ce46565328c6"). InnerVolumeSpecName "kube-api-access-bwbrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:30:13.344042 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:13.343956 2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73f96125-cd38-4508-9106-ce46565328c6-dshm\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:30:13.344042 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:13.343986 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwbrt\" (UniqueName: \"kubernetes.io/projected/73f96125-cd38-4508-9106-ce46565328c6-kube-api-access-bwbrt\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:30:13.344042 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:13.343998 2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73f96125-cd38-4508-9106-ce46565328c6-tls-certs\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:30:13.727593 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:13.727507 2581 generic.go:356] "Generic (PLEG): container finished" podID="73f96125-cd38-4508-9106-ce46565328c6" containerID="23d211691af2f396a643753b0e145fbe70090c42fd4a40b4fd32c3072103352e" exitCode=0 Sep 14 11:30:13.727593 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:13.727552 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk" Sep 14 11:30:13.727776 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:13.727584 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk" event={"ID":"73f96125-cd38-4508-9106-ce46565328c6","Type":"ContainerDied","Data":"23d211691af2f396a643753b0e145fbe70090c42fd4a40b4fd32c3072103352e"} Sep 14 11:30:13.727776 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:13.727626 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk" event={"ID":"73f96125-cd38-4508-9106-ce46565328c6","Type":"ContainerDied","Data":"246a6a66ce2c25331d5f9fb6a418f72cd62a6e7c3f1511722bdda74b4dc84773"} Sep 14 11:30:13.727776 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:13.727644 2581 scope.go:117] "RemoveContainer" containerID="23d211691af2f396a643753b0e145fbe70090c42fd4a40b4fd32c3072103352e" Sep 14 11:30:13.737152 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:13.737132 2581 scope.go:117] "RemoveContainer" containerID="23d211691af2f396a643753b0e145fbe70090c42fd4a40b4fd32c3072103352e" Sep 14 11:30:13.737493 ip-10-0-137-53 kubenswrapper[2581]: E0914 11:30:13.737474 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23d211691af2f396a643753b0e145fbe70090c42fd4a40b4fd32c3072103352e\": container with ID starting with 23d211691af2f396a643753b0e145fbe70090c42fd4a40b4fd32c3072103352e not found: ID does not exist" containerID="23d211691af2f396a643753b0e145fbe70090c42fd4a40b4fd32c3072103352e" Sep 14 11:30:13.737569 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:13.737505 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23d211691af2f396a643753b0e145fbe70090c42fd4a40b4fd32c3072103352e"} err="failed to get container status \"23d211691af2f396a643753b0e145fbe70090c42fd4a40b4fd32c3072103352e\": rpc error: code = NotFound desc = could not find container \"23d211691af2f396a643753b0e145fbe70090c42fd4a40b4fd32c3072103352e\": container with ID starting with 23d211691af2f396a643753b0e145fbe70090c42fd4a40b4fd32c3072103352e not found: ID does not exist" Sep 14 11:30:13.749501 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:13.749471 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk"] Sep 14 11:30:13.751390 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:13.751367 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-2ee2f4f3-kserve-c7fb59b9f-r2cxk"] Sep 14 11:30:14.757594 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:14.757547 2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m" podUID="ef34636c-a3e5-45fa-b93e-56758063a046" containerName="main" containerID="cri-o://890fd8527feb4563bcd3f4d9d482f28d82e3f23a9a212e4e239b3e8b27e16fb3" gracePeriod=45 Sep 14 11:30:14.831313 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:14.831278 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73f96125-cd38-4508-9106-ce46565328c6" path="/var/lib/kubelet/pods/73f96125-cd38-4508-9106-ce46565328c6/volumes" Sep 14 11:30:15.003650 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:15.003620 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m" Sep 14 11:30:15.162168 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:15.162143 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef34636c-a3e5-45fa-b93e-56758063a046-model-cache\") pod \"ef34636c-a3e5-45fa-b93e-56758063a046\" (UID: \"ef34636c-a3e5-45fa-b93e-56758063a046\") " Sep 14 11:30:15.162379 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:15.162193 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvwh6\" (UniqueName: \"kubernetes.io/projected/ef34636c-a3e5-45fa-b93e-56758063a046-kube-api-access-vvwh6\") pod \"ef34636c-a3e5-45fa-b93e-56758063a046\" (UID: \"ef34636c-a3e5-45fa-b93e-56758063a046\") " Sep 14 11:30:15.162379 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:15.162218 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef34636c-a3e5-45fa-b93e-56758063a046-home\") pod \"ef34636c-a3e5-45fa-b93e-56758063a046\" (UID: \"ef34636c-a3e5-45fa-b93e-56758063a046\") " Sep 14 11:30:15.162379 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:15.162239 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef34636c-a3e5-45fa-b93e-56758063a046-dshm\") pod \"ef34636c-a3e5-45fa-b93e-56758063a046\" (UID: \"ef34636c-a3e5-45fa-b93e-56758063a046\") " Sep 14 11:30:15.162379 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:15.162267 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef34636c-a3e5-45fa-b93e-56758063a046-tmp-dir\") pod \"ef34636c-a3e5-45fa-b93e-56758063a046\" (UID: \"ef34636c-a3e5-45fa-b93e-56758063a046\") " Sep 14 11:30:15.162612 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:15.162422 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef34636c-a3e5-45fa-b93e-56758063a046-tls-certs\") pod \"ef34636c-a3e5-45fa-b93e-56758063a046\" (UID: \"ef34636c-a3e5-45fa-b93e-56758063a046\") " Sep 14 11:30:15.162612 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:15.162512 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef34636c-a3e5-45fa-b93e-56758063a046-home" (OuterVolumeSpecName: "home") pod "ef34636c-a3e5-45fa-b93e-56758063a046" (UID: "ef34636c-a3e5-45fa-b93e-56758063a046"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:30:15.162612 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:15.162526 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef34636c-a3e5-45fa-b93e-56758063a046-model-cache" (OuterVolumeSpecName: "model-cache") pod "ef34636c-a3e5-45fa-b93e-56758063a046" (UID: "ef34636c-a3e5-45fa-b93e-56758063a046"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:30:15.162730 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:15.162703 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef34636c-a3e5-45fa-b93e-56758063a046-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ef34636c-a3e5-45fa-b93e-56758063a046" (UID: "ef34636c-a3e5-45fa-b93e-56758063a046"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:30:15.162829 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:15.162806 2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef34636c-a3e5-45fa-b93e-56758063a046-model-cache\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:30:15.162829 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:15.162830 2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef34636c-a3e5-45fa-b93e-56758063a046-home\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:30:15.163016 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:15.162842 2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef34636c-a3e5-45fa-b93e-56758063a046-tmp-dir\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:30:15.164589 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:15.164555 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef34636c-a3e5-45fa-b93e-56758063a046-dshm" (OuterVolumeSpecName: "dshm") pod "ef34636c-a3e5-45fa-b93e-56758063a046" (UID: "ef34636c-a3e5-45fa-b93e-56758063a046"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 11:30:15.164705 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:15.164614 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef34636c-a3e5-45fa-b93e-56758063a046-kube-api-access-vvwh6" (OuterVolumeSpecName: "kube-api-access-vvwh6") pod "ef34636c-a3e5-45fa-b93e-56758063a046" (UID: "ef34636c-a3e5-45fa-b93e-56758063a046"). InnerVolumeSpecName "kube-api-access-vvwh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 11:30:15.164705 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:15.164662 2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef34636c-a3e5-45fa-b93e-56758063a046-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ef34636c-a3e5-45fa-b93e-56758063a046" (UID: "ef34636c-a3e5-45fa-b93e-56758063a046"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 11:30:15.263621 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:15.263582 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvwh6\" (UniqueName: \"kubernetes.io/projected/ef34636c-a3e5-45fa-b93e-56758063a046-kube-api-access-vvwh6\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:30:15.263621 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:15.263615 2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef34636c-a3e5-45fa-b93e-56758063a046-dshm\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:30:15.263621 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:15.263625 2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef34636c-a3e5-45fa-b93e-56758063a046-tls-certs\") on node \"ip-10-0-137-53.ec2.internal\" DevicePath \"\"" Sep 14 11:30:15.740405 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:15.740365 2581 generic.go:356] "Generic (PLEG): container finished" podID="ef34636c-a3e5-45fa-b93e-56758063a046" containerID="890fd8527feb4563bcd3f4d9d482f28d82e3f23a9a212e4e239b3e8b27e16fb3" exitCode=0 Sep 14 11:30:15.740589 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:15.740425 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m" Sep 14 11:30:15.740589 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:15.740424 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m" event={"ID":"ef34636c-a3e5-45fa-b93e-56758063a046","Type":"ContainerDied","Data":"890fd8527feb4563bcd3f4d9d482f28d82e3f23a9a212e4e239b3e8b27e16fb3"} Sep 14 11:30:15.740589 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:15.740537 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m" event={"ID":"ef34636c-a3e5-45fa-b93e-56758063a046","Type":"ContainerDied","Data":"36140e2bbe8ada2c85c65ec70175ae888fc9910e4906d90c893fd8f8a3b9c4a6"} Sep 14 11:30:15.740589 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:15.740554 2581 scope.go:117] "RemoveContainer" containerID="890fd8527feb4563bcd3f4d9d482f28d82e3f23a9a212e4e239b3e8b27e16fb3" Sep 14 11:30:15.750779 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:15.750759 2581 scope.go:117] "RemoveContainer" containerID="890fd8527feb4563bcd3f4d9d482f28d82e3f23a9a212e4e239b3e8b27e16fb3" Sep 14 11:30:15.751098 ip-10-0-137-53 kubenswrapper[2581]: E0914 11:30:15.751075 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"890fd8527feb4563bcd3f4d9d482f28d82e3f23a9a212e4e239b3e8b27e16fb3\": container with ID starting with 890fd8527feb4563bcd3f4d9d482f28d82e3f23a9a212e4e239b3e8b27e16fb3 not found: ID does not exist" containerID="890fd8527feb4563bcd3f4d9d482f28d82e3f23a9a212e4e239b3e8b27e16fb3" Sep 14 11:30:15.751159 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:15.751110 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"890fd8527feb4563bcd3f4d9d482f28d82e3f23a9a212e4e239b3e8b27e16fb3"} err="failed to get container status \"890fd8527feb4563bcd3f4d9d482f28d82e3f23a9a212e4e239b3e8b27e16fb3\": rpc error: code = NotFound desc = could not find container \"890fd8527feb4563bcd3f4d9d482f28d82e3f23a9a212e4e239b3e8b27e16fb3\": container with ID starting with 890fd8527feb4563bcd3f4d9d482f28d82e3f23a9a212e4e239b3e8b27e16fb3 not found: ID does not exist" Sep 14 11:30:15.763236 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:15.763203 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m"] Sep 14 11:30:15.769483 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:15.769454 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-2ee2f4f3-kserve-f7f4dd49f-bgx2m"] Sep 14 11:30:16.831757 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:16.831724 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef34636c-a3e5-45fa-b93e-56758063a046" path="/var/lib/kubelet/pods/ef34636c-a3e5-45fa-b93e-56758063a046/volumes" Sep 14 11:30:47.447525 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:47.447436 2581 scope.go:117] "RemoveContainer" containerID="2160fde79123edaea75d5b8eb5db333eabb96ed38471f6a091c1a05acdcd5870" Sep 14 11:30:47.458262 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:47.458234 2581 scope.go:117] "RemoveContainer" containerID="57b6c4c9bd46b237fab8f41e15c902759462ca57ef25beb94e7ee1e36e14ac53" Sep 14 11:30:58.418828 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:58.418797 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f6d976668-jh7ms_1c88dbf8-5536-496c-ac71-5b0de205f57d/copy-manifests/0.log" Sep 14 11:30:58.425257 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:58.425233 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f6d976668-jh7ms_1c88dbf8-5536-496c-ac71-5b0de205f57d/manager/0.log" Sep 14 11:30:59.068982 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:59.068952 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29823075-f8dpz_0a6c9875-9c61-4b2a-80c1-52397cc003db/cleanup/0.log" Sep 14 11:30:59.280197 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:30:59.280166 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-s29t2_5a2ce312-ab2f-4f61-86d1-730e597f3b63/postgres/0.log" Sep 14 11:31:00.353677 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:00.353651 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-4ws2n_81dca805-7c69-4e16-8cf8-bafb46cdc3ab/kuadrant-console-plugin/0.log" Sep 14 11:31:01.262070 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:01.262036 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-848c99f856-g64c6_b8c2cd16-075b-41b4-8c6c-332a716d1012/kube-auth-proxy/0.log" Sep 14 11:31:01.598723 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:01.598690 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-xkx49_b03e91e7-a889-45d6-81d9-6937b92a85aa/payload-pre-processing/0.log" Sep 14 11:31:02.251521 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:02.251483 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-wvxgt_7c6eb772-a931-41ca-9ef1-c3e3feaaedd3/main/0.log" Sep 14 11:31:02.467602 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:02.467567 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-szb5g_3312c8da-f19a-4279-9e7f-d1799c31c8d5/main/0.log" Sep 14 11:31:02.577726 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:02.577698 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgjhhf_5f79f3b5-6765-4f84-9325-d766b9baf87c/main/0.log" Sep 14 11:31:02.807657 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:02.807626 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-5vrfh_501ae912-66e1-478f-987c-218f6151677f/main/0.log" Sep 14 11:31:11.012438 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:11.012410 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-b62cb_6f7b7b9c-eb99-4eb6-b619-8db12670cef0/global-pull-secret-syncer/0.log" Sep 14 11:31:11.182839 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:11.182806 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-js9xt_5cc7f97c-133f-4c01-a394-1e06cb81d372/konnectivity-agent/0.log" Sep 14 11:31:11.264991 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:11.264917 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-53.ec2.internal_3ed51660b0e0cea0ccab4eff146b4e99/haproxy/0.log" Sep 14 11:31:15.870888 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:15.870855 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-4ws2n_81dca805-7c69-4e16-8cf8-bafb46cdc3ab/kuadrant-console-plugin/0.log" Sep 14 11:31:17.694217 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:17.694117 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-bmlrv_923b7386-9d29-4313-9a63-d4f43e4c2932/cluster-monitoring-operator/0.log" Sep 14 11:31:17.792868 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:17.792841 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7d4578988b-8clwn_191cacad-284e-4312-9ef5-ede80f847126/metrics-server/0.log" Sep 14 11:31:17.922255 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:17.922225 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bm7pz_1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46/node-exporter/0.log" Sep 14 11:31:17.949847 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:17.949757 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bm7pz_1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46/kube-rbac-proxy/0.log" Sep 14 11:31:17.968452 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:17.968423 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bm7pz_1f4d9097-9d34-44ff-a9d1-cff6c8cb6e46/init-textfile/0.log" Sep 14 11:31:18.196052 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:18.196023 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8dedac73-f046-46df-88df-536b7db9ed6b/prometheus/0.log" Sep 14 11:31:18.228765 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:18.228685 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8dedac73-f046-46df-88df-536b7db9ed6b/config-reloader/0.log" Sep 14 11:31:18.265305 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:18.265276 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8dedac73-f046-46df-88df-536b7db9ed6b/thanos-sidecar/0.log" Sep 14 11:31:18.299171 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:18.299146 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8dedac73-f046-46df-88df-536b7db9ed6b/kube-rbac-proxy-web/0.log" Sep 14 11:31:18.344718 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:18.344691 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8dedac73-f046-46df-88df-536b7db9ed6b/kube-rbac-proxy/0.log" Sep 14 11:31:18.398477 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:18.398448 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8dedac73-f046-46df-88df-536b7db9ed6b/kube-rbac-proxy-thanos/0.log" Sep 14 11:31:18.474279 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:18.474259 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8dedac73-f046-46df-88df-536b7db9ed6b/init-config-reloader/0.log" Sep 14 11:31:18.555791 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:18.555759 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-6z6p6_57632e3a-064f-4b7a-9063-e254aa47b7f9/prometheus-operator/0.log" Sep 14 11:31:18.593195 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:18.593169 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-6z6p6_57632e3a-064f-4b7a-9063-e254aa47b7f9/kube-rbac-proxy/0.log" Sep 14 11:31:18.771503 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:18.771468 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84b587f84f-bbcq9_0ae46dae-1601-4782-b36e-3f824d74c904/thanos-query/0.log" Sep 14 11:31:18.808297 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:18.808223 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84b587f84f-bbcq9_0ae46dae-1601-4782-b36e-3f824d74c904/kube-rbac-proxy-web/0.log" Sep 14 11:31:18.836238 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:18.836205 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84b587f84f-bbcq9_0ae46dae-1601-4782-b36e-3f824d74c904/kube-rbac-proxy/0.log" Sep 14 11:31:18.866180 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:18.866146 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84b587f84f-bbcq9_0ae46dae-1601-4782-b36e-3f824d74c904/prom-label-proxy/0.log" Sep 14 11:31:18.902134 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:18.902107 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84b587f84f-bbcq9_0ae46dae-1601-4782-b36e-3f824d74c904/kube-rbac-proxy-rules/0.log" Sep 14 11:31:18.933765 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:18.933730 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84b587f84f-bbcq9_0ae46dae-1601-4782-b36e-3f824d74c904/kube-rbac-proxy-metrics/0.log" Sep 14 11:31:20.400702 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:20.400674 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-rfn74_6ebfcbad-f943-4bfa-8ace-27cca1543c26/networking-console-plugin/0.log" Sep 14 11:31:21.062391 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:21.062362 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lp9t4/perf-node-gather-daemonset-z2h5g"] Sep 14 11:31:21.062774 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:21.062762 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73f96125-cd38-4508-9106-ce46565328c6" containerName="main" Sep 14 11:31:21.062819 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:21.062776 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="73f96125-cd38-4508-9106-ce46565328c6" containerName="main" Sep 14 11:31:21.062819 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:21.062785 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef34636c-a3e5-45fa-b93e-56758063a046" containerName="main" Sep 14 11:31:21.062819 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:21.062792 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef34636c-a3e5-45fa-b93e-56758063a046" containerName="main" Sep 14 11:31:21.062918 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:21.062854 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="73f96125-cd38-4508-9106-ce46565328c6" containerName="main" Sep 14 11:31:21.062918 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:21.062865 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef34636c-a3e5-45fa-b93e-56758063a046" containerName="main" Sep 14 11:31:21.066104 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:21.066084 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lp9t4/perf-node-gather-daemonset-z2h5g" Sep 14 11:31:21.068492 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:21.068470 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lp9t4\"/\"kube-root-ca.crt\"" Sep 14 11:31:21.069577 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:21.069562 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lp9t4\"/\"default-dockercfg-4h7ts\"" Sep 14 11:31:21.069657 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:21.069573 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lp9t4\"/\"openshift-service-ca.crt\"" Sep 14 11:31:21.083119 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:21.077724 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lp9t4/perf-node-gather-daemonset-z2h5g"] Sep 14 11:31:21.179299 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:21.179257 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r4fc\" (UniqueName: \"kubernetes.io/projected/16372e4e-75fd-420a-9bea-314cc3c3f762-kube-api-access-5r4fc\") pod \"perf-node-gather-daemonset-z2h5g\" (UID: \"16372e4e-75fd-420a-9bea-314cc3c3f762\") " pod="openshift-must-gather-lp9t4/perf-node-gather-daemonset-z2h5g" Sep 14 11:31:21.179520 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:21.179394 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16372e4e-75fd-420a-9bea-314cc3c3f762-sys\") pod \"perf-node-gather-daemonset-z2h5g\" (UID: \"16372e4e-75fd-420a-9bea-314cc3c3f762\") " pod="openshift-must-gather-lp9t4/perf-node-gather-daemonset-z2h5g" Sep 14 11:31:21.179520 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:21.179416 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/16372e4e-75fd-420a-9bea-314cc3c3f762-podres\") pod \"perf-node-gather-daemonset-z2h5g\" (UID: \"16372e4e-75fd-420a-9bea-314cc3c3f762\") " pod="openshift-must-gather-lp9t4/perf-node-gather-daemonset-z2h5g" Sep 14 11:31:21.179520 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:21.179437 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/16372e4e-75fd-420a-9bea-314cc3c3f762-proc\") pod \"perf-node-gather-daemonset-z2h5g\" (UID: \"16372e4e-75fd-420a-9bea-314cc3c3f762\") " pod="openshift-must-gather-lp9t4/perf-node-gather-daemonset-z2h5g" Sep 14 11:31:21.179659 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:21.179536 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16372e4e-75fd-420a-9bea-314cc3c3f762-lib-modules\") pod \"perf-node-gather-daemonset-z2h5g\" (UID: \"16372e4e-75fd-420a-9bea-314cc3c3f762\") " pod="openshift-must-gather-lp9t4/perf-node-gather-daemonset-z2h5g" Sep 14 11:31:21.280526 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:21.280481 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16372e4e-75fd-420a-9bea-314cc3c3f762-sys\") pod \"perf-node-gather-daemonset-z2h5g\" (UID: \"16372e4e-75fd-420a-9bea-314cc3c3f762\") " pod="openshift-must-gather-lp9t4/perf-node-gather-daemonset-z2h5g" Sep 14 11:31:21.280526 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:21.280522 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/16372e4e-75fd-420a-9bea-314cc3c3f762-podres\") pod \"perf-node-gather-daemonset-z2h5g\" (UID: \"16372e4e-75fd-420a-9bea-314cc3c3f762\") " pod="openshift-must-gather-lp9t4/perf-node-gather-daemonset-z2h5g" Sep 14 11:31:21.280784 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:21.280552 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/16372e4e-75fd-420a-9bea-314cc3c3f762-proc\") pod \"perf-node-gather-daemonset-z2h5g\" (UID: \"16372e4e-75fd-420a-9bea-314cc3c3f762\") " pod="openshift-must-gather-lp9t4/perf-node-gather-daemonset-z2h5g" Sep 14 11:31:21.280784 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:21.280594 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16372e4e-75fd-420a-9bea-314cc3c3f762-lib-modules\") pod \"perf-node-gather-daemonset-z2h5g\" (UID: \"16372e4e-75fd-420a-9bea-314cc3c3f762\") " pod="openshift-must-gather-lp9t4/perf-node-gather-daemonset-z2h5g" Sep 14 11:31:21.280784 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:21.280615 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/16372e4e-75fd-420a-9bea-314cc3c3f762-sys\") pod \"perf-node-gather-daemonset-z2h5g\" (UID: \"16372e4e-75fd-420a-9bea-314cc3c3f762\") " pod="openshift-must-gather-lp9t4/perf-node-gather-daemonset-z2h5g" Sep 14 11:31:21.280784 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:21.280637 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r4fc\" (UniqueName: \"kubernetes.io/projected/16372e4e-75fd-420a-9bea-314cc3c3f762-kube-api-access-5r4fc\") pod \"perf-node-gather-daemonset-z2h5g\" (UID: \"16372e4e-75fd-420a-9bea-314cc3c3f762\") " pod="openshift-must-gather-lp9t4/perf-node-gather-daemonset-z2h5g" Sep 14 11:31:21.280784 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:21.280672 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/16372e4e-75fd-420a-9bea-314cc3c3f762-podres\") pod \"perf-node-gather-daemonset-z2h5g\" (UID: \"16372e4e-75fd-420a-9bea-314cc3c3f762\") " pod="openshift-must-gather-lp9t4/perf-node-gather-daemonset-z2h5g" Sep 14 11:31:21.280784 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:21.280676 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/16372e4e-75fd-420a-9bea-314cc3c3f762-proc\") pod \"perf-node-gather-daemonset-z2h5g\" (UID: \"16372e4e-75fd-420a-9bea-314cc3c3f762\") " pod="openshift-must-gather-lp9t4/perf-node-gather-daemonset-z2h5g" Sep 14 11:31:21.280784 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:21.280731 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/16372e4e-75fd-420a-9bea-314cc3c3f762-lib-modules\") pod \"perf-node-gather-daemonset-z2h5g\" (UID: \"16372e4e-75fd-420a-9bea-314cc3c3f762\") " pod="openshift-must-gather-lp9t4/perf-node-gather-daemonset-z2h5g" Sep 14 11:31:21.288898 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:21.288865 2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r4fc\" (UniqueName: \"kubernetes.io/projected/16372e4e-75fd-420a-9bea-314cc3c3f762-kube-api-access-5r4fc\") pod \"perf-node-gather-daemonset-z2h5g\" (UID: \"16372e4e-75fd-420a-9bea-314cc3c3f762\") " pod="openshift-must-gather-lp9t4/perf-node-gather-daemonset-z2h5g" Sep 14 11:31:21.384248 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:21.384156 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lp9t4/perf-node-gather-daemonset-z2h5g" Sep 14 11:31:21.657379 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:21.657328 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lp9t4/perf-node-gather-daemonset-z2h5g"] Sep 14 11:31:21.659709 ip-10-0-137-53 kubenswrapper[2581]: W0914 11:31:21.659679 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod16372e4e_75fd_420a_9bea_314cc3c3f762.slice/crio-828eeda9db1f37105ce93c403f157d6e608952b7a1f60ca8a460ae70a8562e73 WatchSource:0}: Error finding container 828eeda9db1f37105ce93c403f157d6e608952b7a1f60ca8a460ae70a8562e73: Status 404 returned error can't find the container with id 828eeda9db1f37105ce93c403f157d6e608952b7a1f60ca8a460ae70a8562e73 Sep 14 11:31:22.055938 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:22.055896 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lp9t4/perf-node-gather-daemonset-z2h5g" event={"ID":"16372e4e-75fd-420a-9bea-314cc3c3f762","Type":"ContainerStarted","Data":"ac13bf3427cd16a0cf994fa2a90a54db6f6bb8a5be4d381efc1e1d03ab7c1c9c"} Sep 14 11:31:22.056093 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:22.055958 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lp9t4/perf-node-gather-daemonset-z2h5g" event={"ID":"16372e4e-75fd-420a-9bea-314cc3c3f762","Type":"ContainerStarted","Data":"828eeda9db1f37105ce93c403f157d6e608952b7a1f60ca8a460ae70a8562e73"} Sep 14 11:31:22.056093 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:22.055993 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lp9t4/perf-node-gather-daemonset-z2h5g" Sep 14 11:31:22.072932 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:22.072876 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lp9t4/perf-node-gather-daemonset-z2h5g" podStartSLOduration=1.0728613280000001 podStartE2EDuration="1.072861328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 11:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 11:31:22.07106455 +0000 UTC m=+2555.813466519" watchObservedRunningTime="2026-09-14 11:31:22.072861328 +0000 UTC m=+2555.815263330" Sep 14 11:31:22.235315 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:22.235270 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-r5rfv_854f87cd-8f75-45dc-9518-3dd0242fb0dc/volume-data-source-validator/0.log" Sep 14 11:31:23.386219 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:23.386193 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zp5bd_305e2ad1-1b29-4c6b-92e9-b21fa53847bc/dns/0.log" Sep 14 11:31:23.406254 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:23.406226 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zp5bd_305e2ad1-1b29-4c6b-92e9-b21fa53847bc/kube-rbac-proxy/0.log" Sep 14 11:31:23.430941 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:23.430909 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5nq5v_ceba844b-2781-40ea-96e0-9eacd1e13ef6/dns-node-resolver/0.log" Sep 14 11:31:23.995478 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:23.995448 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-78c748dd77-fzgm4_227e5dcb-0c45-4294-b6db-c6fc6edac409/registry/0.log" Sep 14 11:31:24.042978 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:24.042938 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rfppn_5b6399c5-592d-463e-a9a4-337834d10604/node-ca/0.log" Sep 14 11:31:25.392327 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:25.392299 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-848c99f856-g64c6_b8c2cd16-075b-41b4-8c6c-332a716d1012/kube-auth-proxy/0.log" Sep 14 11:31:25.486036 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:25.485996 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-xkx49_b03e91e7-a889-45d6-81d9-6937b92a85aa/payload-pre-processing/0.log" Sep 14 11:31:26.443809 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:26.443778 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wrtgl_3112f5c3-15ae-4606-8da4-c066b3c92af8/serve-healthcheck-canary/0.log" Sep 14 11:31:26.881768 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:26.881719 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-9tvnj_e2424929-e528-4e66-bba3-04fe380324af/insights-operator/0.log" Sep 14 11:31:26.883097 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:26.883077 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-9tvnj_e2424929-e528-4e66-bba3-04fe380324af/insights-operator/1.log" Sep 14 11:31:26.906934 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:26.906899 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4mmlv_8073aabc-9fac-40a5-bfaa-b150a5623e1c/kube-rbac-proxy/0.log" Sep 14 11:31:26.927854 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:26.927820 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4mmlv_8073aabc-9fac-40a5-bfaa-b150a5623e1c/exporter/0.log" Sep 14 11:31:26.947737 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:26.947704 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4mmlv_8073aabc-9fac-40a5-bfaa-b150a5623e1c/extractor/0.log" Sep 14 11:31:28.072620 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:28.072591 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lp9t4/perf-node-gather-daemonset-z2h5g" Sep 14 11:31:29.658536 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:29.658506 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f6d976668-jh7ms_1c88dbf8-5536-496c-ac71-5b0de205f57d/manager/0.log" Sep 14 11:31:29.681912 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:29.681868 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f6d976668-jh7ms_1c88dbf8-5536-496c-ac71-5b0de205f57d/copy-manifests/0.log" Sep 14 11:31:31.209480 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:31.209400 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-587bf9cb9-nss2w_fbd52720-5820-4d71-89b1-dbce23230c7e/manager/0.log" Sep 14 11:31:32.231308 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:32.231276 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-848c99f856-g64c6_b8c2cd16-075b-41b4-8c6c-332a716d1012/kube-auth-proxy/0.log" Sep 14 11:31:32.291050 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:32.291017 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-xkx49_b03e91e7-a889-45d6-81d9-6937b92a85aa/payload-pre-processing/0.log" Sep 14 11:31:34.263056 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:34.263021 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-848c99f856-g64c6_b8c2cd16-075b-41b4-8c6c-332a716d1012/kube-auth-proxy/0.log" Sep 14 11:31:34.328586 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:34.328558 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-xkx49_b03e91e7-a889-45d6-81d9-6937b92a85aa/payload-pre-processing/0.log" Sep 14 11:31:36.801422 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:36.801383 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-g76j6_0cedfa94-2214-4710-86cb-448821993687/kube-storage-version-migrator-operator/1.log" Sep 14 11:31:36.802092 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:36.802075 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-g76j6_0cedfa94-2214-4710-86cb-448821993687/kube-storage-version-migrator-operator/0.log" Sep 14 11:31:37.736231 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:37.736204 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nfqwx_f44ee762-76b2-4d0a-aabd-8b68c6aaffdb/kube-multus-additional-cni-plugins/0.log" Sep 14 11:31:37.768778 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:37.768751 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nfqwx_f44ee762-76b2-4d0a-aabd-8b68c6aaffdb/egress-router-binary-copy/0.log" Sep 14 11:31:37.788100 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:37.788078 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nfqwx_f44ee762-76b2-4d0a-aabd-8b68c6aaffdb/cni-plugins/0.log" Sep 14 11:31:37.807035 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:37.806998 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nfqwx_f44ee762-76b2-4d0a-aabd-8b68c6aaffdb/bond-cni-plugin/0.log" Sep 14 11:31:37.826113 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:37.826088 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nfqwx_f44ee762-76b2-4d0a-aabd-8b68c6aaffdb/routeoverride-cni/0.log" Sep 14 11:31:37.847378 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:37.847349 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nfqwx_f44ee762-76b2-4d0a-aabd-8b68c6aaffdb/whereabouts-cni-bincopy/0.log" Sep 14 11:31:37.866065 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:37.866036 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nfqwx_f44ee762-76b2-4d0a-aabd-8b68c6aaffdb/whereabouts-cni/0.log" Sep 14 11:31:38.238580 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:38.238545 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-zblrl_e95ca08f-2a1d-45b7-b0de-959c0b546a18/kube-multus/0.log" Sep 14 11:31:38.260258 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:38.260224 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-29jrs_04ff0710-e09c-48e9-9769-7ffce3645b66/network-metrics-daemon/0.log" Sep 14 11:31:38.277170 ip-10-0-137-53 kubenswrapper[2581]: I0914 11:31:38.277144 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-29jrs_04ff0710-e09c-48e9-9769-7ffce3645b66/kube-rbac-proxy/0.log"